Oct 30 16:46:26.022610 ip-10-0-128-218 systemd[1]: Starting Kubernetes Kubelet... Oct 30 16:46:26.384382 ip-10-0-128-218 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 16:46:26.384382 ip-10-0-128-218 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 16:46:26.384382 ip-10-0-128-218 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 16:46:26.384382 ip-10-0-128-218 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 16:46:26.387355 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387261 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:46:26.387355 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387355 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:46:26.387355 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387359 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:46:26.387451 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387362 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:46:26.387451 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387366 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:46:26.387451 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387368 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:46:26.387451 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387372 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:46:26.387451 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387375 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:46:26.387451 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387378 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:46:26.387451 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387380 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:46:26.387451 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387383 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:46:26.387451 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387386 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:46:26.387451 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387388 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:46:26.387451 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387391 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:46:26.387451 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387394 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:46:26.387451 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387397 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:46:26.387451 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387399 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:46:26.387451 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387402 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:46:26.387451 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387405 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:46:26.387451 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387407 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:46:26.387451 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387410 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:46:26.387451 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387413 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:46:26.387451 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387415 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:46:26.387944 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387418 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:46:26.387944 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387420 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:46:26.387944 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387423 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:46:26.387944 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387425 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:46:26.387944 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387428 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:46:26.387944 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387430 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:46:26.387944 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387437 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:46:26.387944 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387440 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:46:26.387944 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387443 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:46:26.387944 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387445 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:46:26.387944 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387448 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:46:26.387944 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387451 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:46:26.387944 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387453 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:46:26.387944 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387457 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:46:26.387944 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387460 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:46:26.387944 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387462 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:46:26.387944 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387465 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:46:26.387944 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387468 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:46:26.387944 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387470 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:46:26.387944 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387473 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:46:26.388502 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387475 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:46:26.388502 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387478 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:46:26.388502 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387480 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:46:26.388502 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387483 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:46:26.388502 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387485 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:46:26.388502 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387488 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:46:26.388502 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387491 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:46:26.388502 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387493 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:46:26.388502 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387496 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:46:26.388502 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387499 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:46:26.388502 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387501 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:46:26.388502 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387504 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:46:26.388502 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387508 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:46:26.388502 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387511 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:46:26.388502 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387514 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:46:26.388502 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387517 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:46:26.388502 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387519 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:46:26.388502 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387522 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:46:26.388502 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387524 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:46:26.388502 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387527 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:46:26.389003 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387529 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:46:26.389003 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387532 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:46:26.389003 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387534 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:46:26.389003 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387537 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:46:26.389003 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387539 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:46:26.389003 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387542 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:46:26.389003 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387544 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:46:26.389003 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387547 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:46:26.389003 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387550 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:46:26.389003 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387552 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:46:26.389003 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387555 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:46:26.389003 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387557 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:46:26.389003 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387561 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:46:26.389003 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387565 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:46:26.389003 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387568 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:46:26.389003 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387571 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:46:26.389003 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387574 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:46:26.389003 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387576 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:46:26.389003 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387580 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:46:26.389471 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387585 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:46:26.389471 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387588 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:46:26.389471 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387591 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:46:26.389471 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387593 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:46:26.389471 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387596 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:46:26.389471 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387598 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:46:26.389471 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387603 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:46:26.389471 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387605 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:46:26.389471 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387608 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:46:26.389471 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387611 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:46:26.389471 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387613 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:46:26.389471 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387616 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:46:26.389471 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387618 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:46:26.389471 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387621 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:46:26.389471 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387624 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:46:26.389471 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387626 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:46:26.389471 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387629 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:46:26.389471 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387631 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:46:26.389471 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387633 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:46:26.389908 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387636 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:46:26.389908 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387640 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:46:26.389908 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387642 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:46:26.389908 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387645 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:46:26.389908 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.387647 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:46:26.389908 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388441 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:46:26.389908 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388448 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:46:26.389908 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388451 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:46:26.389908 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388453 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:46:26.389908 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388456 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:46:26.389908 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388458 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:46:26.389908 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388461 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:46:26.389908 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388464 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:46:26.389908 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388467 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:46:26.389908 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388470 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:46:26.389908 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388472 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:46:26.389908 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388476 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:46:26.389908 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388478 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:46:26.389908 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388481 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:46:26.390387 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388486 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:46:26.390387 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388490 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:46:26.390387 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388493 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:46:26.390387 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388496 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:46:26.390387 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388500 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:46:26.390387 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388502 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:46:26.390387 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388505 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:46:26.390387 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388508 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:46:26.390387 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388511 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:46:26.390387 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388514 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:46:26.390387 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388516 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:46:26.390387 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388519 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:46:26.390387 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388522 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:46:26.390387 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388524 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:46:26.390387 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388527 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:46:26.390387 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388530 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:46:26.390387 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388532 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:46:26.390387 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388535 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:46:26.390387 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388538 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:46:26.390847 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388540 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:46:26.390847 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388543 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:46:26.390847 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388545 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:46:26.390847 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388547 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:46:26.390847 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388550 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:46:26.390847 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388553 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:46:26.390847 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388556 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:46:26.390847 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388559 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:46:26.390847 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388562 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:46:26.390847 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388564 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:46:26.390847 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388567 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:46:26.390847 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388569 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:46:26.390847 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388572 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:46:26.390847 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388575 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:46:26.390847 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388578 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:46:26.390847 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388581 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:46:26.390847 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388584 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:46:26.390847 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388586 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:46:26.390847 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388589 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:46:26.391291 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388591 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:46:26.391291 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388594 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:46:26.391291 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388596 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:46:26.391291 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388599 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:46:26.391291 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388601 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:46:26.391291 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388604 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:46:26.391291 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388606 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:46:26.391291 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388608 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:46:26.391291 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388611 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:46:26.391291 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388613 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:46:26.391291 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388616 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:46:26.391291 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388622 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:46:26.391291 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388624 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:46:26.391291 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388627 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:46:26.391291 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388629 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:46:26.391291 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388632 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:46:26.391291 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388634 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:46:26.391291 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388637 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:46:26.391291 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388640 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:46:26.391291 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388643 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:46:26.391822 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388645 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:46:26.391822 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388648 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:46:26.391822 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388651 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:46:26.391822 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388654 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:46:26.391822 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388657 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:46:26.391822 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388659 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:46:26.391822 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388662 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:46:26.391822 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388664 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:46:26.391822 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388667 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:46:26.391822 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388670 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:46:26.391822 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388672 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:46:26.391822 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388675 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:46:26.391822 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388677 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:46:26.391822 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388679 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:46:26.391822 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388682 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:46:26.391822 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388684 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:46:26.391822 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388687 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:46:26.391822 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388689 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:46:26.391822 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388693 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:46:26.391822 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388696 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:46:26.392299 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388698 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:46:26.392299 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388702 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:46:26.392299 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388707 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:46:26.392299 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388712 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:46:26.392299 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388715 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:46:26.392299 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388718 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:46:26.392299 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388720 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:46:26.392299 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388723 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:46:26.392299 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388725 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:46:26.392299 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388729 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:46:26.392299 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388732 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:46:26.392299 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388734 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:46:26.392299 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388737 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:46:26.392299 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.388741 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:46:26.392299 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389230 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:46:26.392299 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389235 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:46:26.392299 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389238 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:46:26.392299 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389241 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:46:26.392299 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389244 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:46:26.392768 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389246 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:46:26.392768 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389249 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:46:26.392768 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389252 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:46:26.392768 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389254 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:46:26.392768 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389258 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:46:26.392768 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389260 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:46:26.392768 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389263 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:46:26.392768 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389267 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:46:26.392768 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389270 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:46:26.392768 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389273 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:46:26.392768 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389276 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:46:26.392768 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389279 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:46:26.392768 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389281 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:46:26.392768 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389284 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:46:26.392768 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389286 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:46:26.392768 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389290 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:46:26.392768 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389295 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:46:26.392768 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389299 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:46:26.392768 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389301 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:46:26.393220 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389317 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:46:26.393220 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389320 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:46:26.393220 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389323 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:46:26.393220 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389326 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:46:26.393220 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389328 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:46:26.393220 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389331 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:46:26.393220 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389334 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:46:26.393220 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389336 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:46:26.393220 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389340 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:46:26.393220 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389342 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:46:26.393220 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389345 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:46:26.393220 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389348 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:46:26.393220 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389350 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:46:26.393220 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389353 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:46:26.393220 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389355 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:46:26.393220 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389358 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:46:26.393220 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389362 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:46:26.393220 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389365 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:46:26.393220 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389368 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:46:26.393664 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389372 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:46:26.393664 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389377 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:46:26.393664 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389380 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:46:26.393664 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389383 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:46:26.393664 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389386 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:46:26.393664 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389388 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:46:26.393664 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389391 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:46:26.393664 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389393 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:46:26.393664 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389396 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:46:26.393664 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389399 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:46:26.393664 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389403 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:46:26.393664 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389405 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:46:26.393664 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389408 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:46:26.393664 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389410 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:46:26.393664 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389413 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:46:26.393664 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389415 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:46:26.393664 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389418 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:46:26.393664 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389420 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:46:26.393664 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389423 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:46:26.394106 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389425 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:46:26.394106 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389428 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:46:26.394106 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389430 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:46:26.394106 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389433 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:46:26.394106 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389436 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:46:26.394106 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389439 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:46:26.394106 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389441 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:46:26.394106 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389444 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:46:26.394106 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389446 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:46:26.394106 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389449 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:46:26.394106 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389451 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:46:26.394106 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389454 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:46:26.394106 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389457 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:46:26.394106 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389459 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:46:26.394106 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389463 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:46:26.394106 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389466 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:46:26.394106 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389468 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:46:26.394106 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389471 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:46:26.394106 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389474 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:46:26.394579 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389477 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:46:26.394579 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389480 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:46:26.394579 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389483 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:46:26.394579 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389485 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:46:26.394579 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389489 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:46:26.394579 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389492 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:46:26.394579 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389494 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:46:26.394579 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389497 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:46:26.394579 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389499 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:46:26.394579 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389503 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:46:26.394579 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389505 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:46:26.394579 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389508 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:46:26.394579 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389510 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:46:26.394579 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389513 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:46:26.394579 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389516 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:46:26.394579 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389518 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:46:26.394579 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389521 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:46:26.394579 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389524 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:46:26.394579 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389526 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:46:26.394579 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389529 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:46:26.395046 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389531 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:46:26.395046 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389534 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:46:26.395046 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389536 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:46:26.395046 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389539 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:46:26.395046 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.389541 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:46:26.395046 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389627 2580 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 16:46:26.395046 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389635 2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 16:46:26.395046 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389643 2580 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 16:46:26.395046 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389647 2580 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 16:46:26.395046 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389652 2580 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 16:46:26.395046 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389655 2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 16:46:26.395046 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389660 2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 16:46:26.395046 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389664 2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 16:46:26.395046 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389686 2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 16:46:26.395046 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389690 2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 16:46:26.395046 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389695 2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 16:46:26.395046 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389700 2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 16:46:26.395046 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389703 2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 16:46:26.395046 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389706 2580 flags.go:64] FLAG: --cgroup-root="" Oct 30 16:46:26.395046 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389709 2580 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 16:46:26.395046 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389712 2580 flags.go:64] FLAG: --client-ca-file="" Oct 30 16:46:26.395046 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389715 2580 flags.go:64] FLAG: --cloud-provider="external" Oct 30 16:46:26.395046 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389718 2580 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 16:46:26.395593 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389722 2580 flags.go:64] FLAG: --cluster-domain="" Oct 30 16:46:26.395593 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389725 2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 16:46:26.395593 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389728 2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 16:46:26.395593 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389731 2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 16:46:26.395593 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389735 2580 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 16:46:26.395593 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389739 2580 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 16:46:26.395593 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389742 2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 16:46:26.395593 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389746 2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 16:46:26.395593 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389749 2580 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 16:46:26.395593 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389752 2580 flags.go:64] FLAG: --contention-profiling="false" Oct 30 16:46:26.395593 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389755 2580 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 16:46:26.395593 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389758 2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 16:46:26.395593 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389762 2580 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 16:46:26.395593 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389765 2580 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 16:46:26.395593 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389769 2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 16:46:26.395593 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389772 2580 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 16:46:26.395593 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389776 2580 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 16:46:26.395593 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389780 2580 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 16:46:26.395593 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389783 2580 flags.go:64] FLAG: --enable-server="true" Oct 30 16:46:26.395593 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389786 2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 16:46:26.395593 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389790 2580 flags.go:64] FLAG: --event-burst="100" Oct 30 16:46:26.395593 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389793 2580 flags.go:64] FLAG: --event-qps="50" Oct 30 16:46:26.395593 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389797 2580 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 16:46:26.395593 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389800 2580 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 16:46:26.396229 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389803 2580 flags.go:64] FLAG: --eviction-hard="" Oct 30 16:46:26.396229 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389809 2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 16:46:26.396229 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389813 2580 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 16:46:26.396229 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389816 2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 16:46:26.396229 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389819 2580 flags.go:64] FLAG: --eviction-soft="" Oct 30 16:46:26.396229 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389822 2580 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 16:46:26.396229 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389825 2580 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 16:46:26.396229 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389828 2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 16:46:26.396229 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389831 2580 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 16:46:26.396229 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389834 2580 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 16:46:26.396229 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389837 2580 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 16:46:26.396229 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389840 2580 flags.go:64] FLAG: --feature-gates="" Oct 30 16:46:26.396229 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389844 2580 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 16:46:26.396229 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389847 2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 16:46:26.396229 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389850 2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 16:46:26.396229 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389853 2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 16:46:26.396229 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389858 2580 flags.go:64] FLAG: --healthz-port="10248" Oct 30 16:46:26.396229 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389861 2580 flags.go:64] FLAG: --help="false" Oct 30 16:46:26.396229 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389864 2580 flags.go:64] FLAG: --hostname-override="ip-10-0-128-218.ec2.internal" Oct 30 16:46:26.396229 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389867 2580 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 16:46:26.396229 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389871 2580 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 16:46:26.396229 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389874 2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 16:46:26.396229 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389877 2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 16:46:26.396229 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389880 2580 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 16:46:26.396815 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389885 2580 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 16:46:26.396815 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389888 2580 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 16:46:26.396815 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389891 2580 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 16:46:26.396815 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389894 2580 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 16:46:26.396815 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389898 2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 16:46:26.396815 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389901 2580 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 16:46:26.396815 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389904 2580 flags.go:64] FLAG: --kube-reserved="" Oct 30 16:46:26.396815 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389907 2580 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 16:46:26.396815 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389910 2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 16:46:26.396815 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389914 2580 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 16:46:26.396815 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389918 2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 16:46:26.396815 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389921 2580 flags.go:64] FLAG: --lock-file="" Oct 30 16:46:26.396815 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389924 2580 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 16:46:26.396815 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389927 2580 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 16:46:26.396815 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389931 2580 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 16:46:26.396815 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389937 2580 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 16:46:26.396815 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389940 2580 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 16:46:26.396815 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389943 2580 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 16:46:26.396815 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389946 2580 flags.go:64] FLAG: --logging-format="text" Oct 30 16:46:26.396815 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389949 2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 16:46:26.396815 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389952 2580 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 16:46:26.396815 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389955 2580 flags.go:64] FLAG: --manifest-url="" Oct 30 16:46:26.396815 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389958 2580 flags.go:64] FLAG: --manifest-url-header="" Oct 30 16:46:26.396815 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389965 2580 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 16:46:26.396815 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389968 2580 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 16:46:26.397415 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389973 2580 flags.go:64] FLAG: --max-pods="110" Oct 30 16:46:26.397415 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389976 2580 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 16:46:26.397415 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389980 2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 16:46:26.397415 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389983 2580 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 16:46:26.397415 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389986 2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 16:46:26.397415 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389989 2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 16:46:26.397415 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389992 2580 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 16:46:26.397415 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.389996 2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 16:46:26.397415 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.390003 2580 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 16:46:26.397415 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.390006 2580 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 16:46:26.397415 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.390009 2580 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 16:46:26.397415 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.390012 2580 flags.go:64] FLAG: --pod-cidr="" Oct 30 16:46:26.397415 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.390015 2580 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 16:46:26.397415 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.390018 2580 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 16:46:26.397415 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.390021 2580 flags.go:64] FLAG: --pods-per-core="0" Oct 30 16:46:26.397415 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.390024 2580 flags.go:64] FLAG: --port="10250" Oct 30 16:46:26.397415 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.390028 2580 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 16:46:26.397415 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.390031 2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ce764e851bffdc03" Oct 30 16:46:26.397415 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.390034 2580 flags.go:64] FLAG: --qos-reserved="" Oct 30 16:46:26.397415 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.390037 2580 flags.go:64] FLAG: --read-only-port="10255" Oct 30 16:46:26.397415 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.390040 2580 flags.go:64] FLAG: --register-node="true" Oct 30 16:46:26.397415 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.390043 2580 flags.go:64] FLAG: --register-with-taints="" Oct 30 16:46:26.397415 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.390047 2580 flags.go:64] FLAG: --registry-burst="10" Oct 30 16:46:26.397415 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.390050 2580 flags.go:64] FLAG: --registry-qps="5" Oct 30 16:46:26.397415 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.390053 2580 flags.go:64] FLAG: --reserved-cpus="" Oct 30 16:46:26.397415 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.390055 2580 flags.go:64] FLAG: --reserved-memory="" Oct 30 16:46:26.398028 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.390059 2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 16:46:26.398028 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.390062 2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 16:46:26.398028 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.390065 2580 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 16:46:26.398028 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.390068 2580 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 16:46:26.398028 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.390071 2580 flags.go:64] FLAG: --runonce="false" Oct 30 16:46:26.398028 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.390074 2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 16:46:26.398028 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.390077 2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 16:46:26.398028 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.390080 2580 flags.go:64] FLAG: --seccomp-default="false" Oct 30 16:46:26.398028 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.390083 2580 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 16:46:26.398028 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.390086 2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 16:46:26.398028 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.390089 2580 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 16:46:26.398028 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.390093 2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 16:46:26.398028 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.390096 2580 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 16:46:26.398028 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.390099 2580 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 16:46:26.398028 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.390102 2580 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 16:46:26.398028 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.390105 2580 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 16:46:26.398028 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.390108 2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 16:46:26.398028 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.390111 2580 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 16:46:26.398028 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.390115 2580 flags.go:64] FLAG: --system-cgroups="" Oct 30 16:46:26.398028 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.390118 2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 16:46:26.398028 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.390123 2580 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 16:46:26.398028 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.390126 2580 flags.go:64] FLAG: --tls-cert-file="" Oct 30 16:46:26.398028 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.390129 2580 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 16:46:26.398028 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.390134 2580 flags.go:64] FLAG: --tls-min-version="" Oct 30 16:46:26.398028 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.390136 2580 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 16:46:26.398655 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.390139 2580 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 16:46:26.398655 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.390142 2580 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 16:46:26.398655 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.390146 2580 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 16:46:26.398655 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.390149 2580 flags.go:64] FLAG: --v="2" Oct 30 16:46:26.398655 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.390153 2580 flags.go:64] FLAG: --version="false" Oct 30 16:46:26.398655 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.390157 2580 flags.go:64] FLAG: --vmodule="" Oct 30 16:46:26.398655 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.390161 2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 16:46:26.398655 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.390164 2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 16:46:26.398655 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390297 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:46:26.398655 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390301 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:46:26.398655 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390319 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:46:26.398655 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390322 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:46:26.398655 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390325 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:46:26.398655 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390328 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:46:26.398655 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390331 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:46:26.398655 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390334 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:46:26.398655 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390337 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:46:26.398655 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390340 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:46:26.398655 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390342 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:46:26.398655 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390345 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:46:26.398655 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390348 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:46:26.399154 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390351 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:46:26.399154 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390354 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:46:26.399154 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390356 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:46:26.399154 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390359 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:46:26.399154 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390362 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:46:26.399154 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390364 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:46:26.399154 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390367 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:46:26.399154 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390369 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:46:26.399154 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390372 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:46:26.399154 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390374 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:46:26.399154 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390377 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:46:26.399154 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390379 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:46:26.399154 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390382 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:46:26.399154 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390384 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:46:26.399154 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390387 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:46:26.399154 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390390 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:46:26.399154 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390392 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:46:26.399154 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390395 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:46:26.399154 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390397 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:46:26.399154 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390400 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:46:26.399645 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390403 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:46:26.399645 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390405 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:46:26.399645 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390408 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:46:26.399645 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390410 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:46:26.399645 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390413 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:46:26.399645 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390415 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:46:26.399645 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390418 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:46:26.399645 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390421 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:46:26.399645 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390423 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:46:26.399645 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390435 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:46:26.399645 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390439 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:46:26.399645 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390442 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:46:26.399645 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390444 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:46:26.399645 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390447 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:46:26.399645 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390450 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:46:26.399645 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390452 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:46:26.399645 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390455 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:46:26.399645 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390457 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:46:26.399645 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390460 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:46:26.399645 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390462 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:46:26.400115 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390465 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:46:26.400115 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390467 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:46:26.400115 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390470 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:46:26.400115 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390472 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:46:26.400115 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390475 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:46:26.400115 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390477 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:46:26.400115 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390480 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:46:26.400115 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390482 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:46:26.400115 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390485 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:46:26.400115 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390487 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:46:26.400115 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390490 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:46:26.400115 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390492 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:46:26.400115 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390495 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:46:26.400115 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390497 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:46:26.400115 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390500 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:46:26.400115 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390502 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:46:26.400115 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390505 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:46:26.400115 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390507 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:46:26.400115 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390510 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:46:26.400115 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390512 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:46:26.400681 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390515 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:46:26.400681 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390518 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:46:26.400681 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390521 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:46:26.400681 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390525 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:46:26.400681 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390527 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:46:26.400681 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390531 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:46:26.400681 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390534 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:46:26.400681 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390537 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:46:26.400681 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390539 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:46:26.400681 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390541 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:46:26.400681 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390544 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:46:26.400681 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390546 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:46:26.400681 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390549 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:46:26.400681 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390552 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:46:26.400681 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390554 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:46:26.400681 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390557 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:46:26.400681 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390559 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:46:26.400681 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390562 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:46:26.400681 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390565 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:46:26.400681 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390568 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:46:26.401171 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390572 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:46:26.401171 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390576 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:46:26.401171 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390579 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:46:26.401171 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390582 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:46:26.401171 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390585 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:46:26.401171 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390588 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:46:26.401171 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390590 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:46:26.401171 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390594 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:46:26.401171 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390598 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:46:26.401171 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390601 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:46:26.401171 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390604 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:46:26.401171 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390607 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:46:26.401171 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390609 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:46:26.401171 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.390636 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 16:46:26.401171 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390678 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:46:26.401551 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390682 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:46:26.401551 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390686 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:46:26.401551 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390691 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:46:26.401551 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390694 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:46:26.401551 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390697 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:46:26.401551 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390699 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:46:26.401551 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390702 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:46:26.401551 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390704 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:46:26.401551 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390707 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:46:26.401551 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390710 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:46:26.401551 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390712 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:46:26.401551 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390715 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:46:26.401551 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390718 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:46:26.401551 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390721 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:46:26.401551 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390723 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:46:26.401551 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390725 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:46:26.401551 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390728 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:46:26.401551 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390731 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:46:26.401551 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390733 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:46:26.401551 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390736 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:46:26.402031 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390738 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:46:26.402031 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390740 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:46:26.402031 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390743 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:46:26.402031 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390745 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:46:26.402031 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390748 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:46:26.402031 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390751 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:46:26.402031 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390753 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:46:26.402031 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390756 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:46:26.402031 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390758 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:46:26.402031 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390761 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:46:26.402031 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390763 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:46:26.402031 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390766 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:46:26.402031 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390769 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:46:26.402031 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390773 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:46:26.402031 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390778 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:46:26.402031 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390780 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:46:26.402031 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390784 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:46:26.402031 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390786 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:46:26.402031 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390789 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:46:26.402489 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390791 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:46:26.402489 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390794 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:46:26.402489 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390797 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:46:26.402489 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390799 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:46:26.402489 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390802 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:46:26.402489 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390804 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:46:26.402489 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390807 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:46:26.402489 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390810 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:46:26.402489 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390812 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:46:26.402489 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390815 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:46:26.402489 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390817 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:46:26.402489 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390820 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:46:26.402489 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390823 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:46:26.402489 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390825 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:46:26.402489 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390828 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:46:26.402489 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390831 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:46:26.402489 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390833 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:46:26.402489 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390836 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:46:26.402489 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390838 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:46:26.402489 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390841 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:46:26.402944 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390843 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:46:26.402944 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390846 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:46:26.402944 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390849 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:46:26.402944 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390851 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:46:26.402944 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390854 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:46:26.402944 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390858 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:46:26.402944 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390862 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:46:26.402944 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390866 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:46:26.402944 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390869 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:46:26.402944 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390871 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:46:26.402944 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390874 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:46:26.402944 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390877 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:46:26.402944 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390879 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:46:26.402944 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390882 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:46:26.402944 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390884 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:46:26.402944 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390887 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:46:26.402944 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390889 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:46:26.402944 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390892 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:46:26.402944 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390894 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:46:26.403394 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390897 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:46:26.403394 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390899 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:46:26.403394 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390902 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:46:26.403394 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390905 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:46:26.403394 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390907 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:46:26.403394 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390910 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:46:26.403394 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390912 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:46:26.403394 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390915 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:46:26.403394 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390917 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:46:26.403394 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390920 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:46:26.403394 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390922 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:46:26.403394 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390925 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:46:26.403394 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390927 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:46:26.403394 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390931 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:46:26.403394 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390934 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:46:26.403394 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390937 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:46:26.403394 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390940 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:46:26.403394 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390943 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:46:26.403394 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390946 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:46:26.403394 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390950 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:46:26.403881 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390954 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:46:26.403881 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390959 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:46:26.403881 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390963 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:46:26.403881 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390966 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:46:26.403881 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390970 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:46:26.403881 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390973 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:46:26.403881 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.390975 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:46:26.403881 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.397244 2580 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 16:46:26.403881 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.397259 2580 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 16:46:26.403881 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.397271 2580 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 16:46:26.403881 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.397276 2580 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 16:46:26.403881 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397349 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:46:26.403881 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397355 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:46:26.403881 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397358 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:46:26.403881 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397362 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:46:26.403881 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397365 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:46:26.403881 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397367 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:46:26.403881 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397370 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:46:26.404365 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397373 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:46:26.404365 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397375 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:46:26.404365 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397378 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:46:26.404365 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397381 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:46:26.404365 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397383 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:46:26.404365 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397386 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:46:26.404365 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397389 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:46:26.404365 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397391 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:46:26.404365 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397394 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:46:26.404365 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397398 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:46:26.404365 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397402 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:46:26.404365 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397405 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:46:26.404365 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397408 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:46:26.404365 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397412 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:46:26.404365 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397414 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:46:26.404365 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397417 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:46:26.404365 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397421 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:46:26.404365 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397423 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:46:26.404365 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397426 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:46:26.404365 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397429 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:46:26.404831 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397432 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:46:26.404831 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397434 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:46:26.404831 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397438 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:46:26.404831 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397441 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:46:26.404831 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397443 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:46:26.404831 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397446 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:46:26.404831 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397449 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:46:26.404831 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397452 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:46:26.404831 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397455 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:46:26.404831 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397457 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:46:26.404831 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397460 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:46:26.404831 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397462 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:46:26.404831 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397465 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:46:26.404831 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397467 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:46:26.404831 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397470 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:46:26.404831 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397472 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:46:26.404831 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397475 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:46:26.404831 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397478 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:46:26.404831 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397480 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:46:26.404831 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397483 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:46:26.405387 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397485 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:46:26.405387 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397488 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:46:26.405387 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397491 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:46:26.405387 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397494 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:46:26.405387 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397496 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:46:26.405387 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397499 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:46:26.405387 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397502 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:46:26.405387 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397504 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:46:26.405387 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397507 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:46:26.405387 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397509 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:46:26.405387 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397512 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:46:26.405387 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397514 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:46:26.405387 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397517 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:46:26.405387 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397519 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:46:26.405387 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397523 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:46:26.405387 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397525 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:46:26.405387 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397528 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:46:26.405387 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397531 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:46:26.405387 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397533 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:46:26.405387 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397536 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:46:26.405925 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397538 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:46:26.405925 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397541 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:46:26.405925 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397544 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:46:26.405925 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397547 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:46:26.405925 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397550 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:46:26.405925 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397553 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:46:26.405925 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397555 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:46:26.405925 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397558 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:46:26.405925 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397560 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:46:26.405925 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397563 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:46:26.405925 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397566 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:46:26.405925 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397568 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:46:26.405925 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397571 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:46:26.405925 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397573 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:46:26.405925 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397576 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:46:26.405925 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397578 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:46:26.405925 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397581 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:46:26.405925 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397583 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:46:26.405925 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397586 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:46:26.405925 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397589 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:46:26.406438 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397592 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:46:26.406438 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397594 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:46:26.406438 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397597 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:46:26.406438 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397599 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:46:26.406438 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397602 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:46:26.406438 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397604 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:46:26.406438 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397608 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:46:26.406438 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397611 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:46:26.406438 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397614 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:46:26.406438 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397616 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:46:26.406438 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397619 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:46:26.406438 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397621 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:46:26.406438 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397624 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:46:26.406438 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397626 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:46:26.406438 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397629 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:46:26.406438 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397633 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:46:26.406438 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397636 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:46:26.406438 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397639 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:46:26.406438 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397650 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:46:26.406877 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.397679 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 16:46:26.406877 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397819 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:46:26.406877 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397823 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:46:26.406877 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397826 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:46:26.406877 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397829 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:46:26.406877 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397831 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:46:26.406877 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397834 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:46:26.406877 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397836 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:46:26.406877 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397839 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:46:26.406877 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397842 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:46:26.406877 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397844 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:46:26.406877 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397847 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:46:26.406877 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397849 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:46:26.406877 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397852 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:46:26.406877 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397855 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:46:26.406877 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397857 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:46:26.407244 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397860 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:46:26.407244 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397863 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:46:26.407244 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397866 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:46:26.407244 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397869 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:46:26.407244 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397871 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:46:26.407244 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397874 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:46:26.407244 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397876 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:46:26.407244 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397879 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:46:26.407244 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397881 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:46:26.407244 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397884 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:46:26.407244 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397886 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:46:26.407244 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397889 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:46:26.407244 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397892 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:46:26.407244 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397894 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:46:26.407244 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397896 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:46:26.407244 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397899 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:46:26.407244 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397901 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:46:26.407244 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397904 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:46:26.407244 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397907 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:46:26.407244 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397909 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:46:26.407731 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397911 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:46:26.407731 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397914 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:46:26.407731 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397917 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:46:26.407731 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397919 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:46:26.407731 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397921 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:46:26.407731 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397924 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:46:26.407731 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397926 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:46:26.407731 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397929 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:46:26.407731 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397931 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:46:26.407731 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397934 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:46:26.407731 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397937 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:46:26.407731 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397939 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:46:26.407731 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397942 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:46:26.407731 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397944 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:46:26.407731 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397947 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:46:26.407731 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397949 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:46:26.407731 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397952 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:46:26.407731 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397954 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:46:26.407731 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397957 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:46:26.408324 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397960 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:46:26.408324 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397963 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:46:26.408324 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397965 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:46:26.408324 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397968 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:46:26.408324 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397970 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:46:26.408324 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397973 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:46:26.408324 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397976 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:46:26.408324 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397980 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:46:26.408324 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397983 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:46:26.408324 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397986 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:46:26.408324 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397989 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:46:26.408324 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397992 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:46:26.408324 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397994 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:46:26.408324 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397997 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:46:26.408324 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.397999 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:46:26.408324 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398002 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:46:26.408324 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398005 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:46:26.408324 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398007 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:46:26.408324 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398009 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:46:26.409528 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398012 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:46:26.409528 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398014 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:46:26.409528 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398017 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:46:26.409528 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398019 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:46:26.409528 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398022 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:46:26.409528 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398025 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:46:26.409528 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398027 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:46:26.409528 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398030 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:46:26.409528 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398033 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:46:26.409528 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398035 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:46:26.409528 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398038 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:46:26.409528 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398040 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:46:26.409528 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398043 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:46:26.409528 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398046 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:46:26.409528 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398048 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:46:26.409528 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398051 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:46:26.409528 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398053 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:46:26.409528 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398056 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:46:26.409528 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398058 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:46:26.409528 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398061 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:46:26.410079 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398063 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:46:26.410079 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398066 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:46:26.410079 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398068 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:46:26.410079 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398071 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:46:26.410079 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398073 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:46:26.410079 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398076 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:46:26.410079 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398079 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:46:26.410079 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398081 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:46:26.410079 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398084 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:46:26.410079 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398087 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:46:26.410079 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398091 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:46:26.410079 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398093 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:46:26.410079 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398096 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:46:26.410079 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.398118 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 16:46:26.410079 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398148 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:46:26.410079 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398153 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:46:26.410512 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398156 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:46:26.410512 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398159 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:46:26.410512 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398162 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:46:26.410512 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398164 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:46:26.410512 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398167 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:46:26.410512 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398169 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:46:26.410512 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398172 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:46:26.410512 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398174 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:46:26.410512 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398177 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:46:26.410512 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398179 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:46:26.410512 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398181 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:46:26.410512 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398184 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:46:26.410512 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398187 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:46:26.410512 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398189 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:46:26.410512 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398191 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:46:26.410512 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398196 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:46:26.410512 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398199 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:46:26.410512 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398201 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:46:26.410512 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398204 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:46:26.411031 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398207 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:46:26.411031 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398209 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:46:26.411031 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398212 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:46:26.411031 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398215 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:46:26.411031 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398217 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:46:26.411031 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398220 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:46:26.411031 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398222 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:46:26.411031 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398225 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:46:26.411031 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398227 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:46:26.411031 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398230 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:46:26.411031 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398232 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:46:26.411031 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398235 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:46:26.411031 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398238 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:46:26.411031 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398240 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:46:26.411031 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398243 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:46:26.411031 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398245 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:46:26.411031 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398248 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:46:26.411031 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398251 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:46:26.411031 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398253 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:46:26.411688 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398256 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:46:26.411688 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398258 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:46:26.411688 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398261 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:46:26.411688 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398263 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:46:26.411688 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398266 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:46:26.411688 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398268 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:46:26.411688 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398270 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:46:26.411688 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398273 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:46:26.411688 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398275 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:46:26.411688 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398280 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:46:26.411688 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398283 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:46:26.411688 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398286 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:46:26.411688 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398289 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:46:26.411688 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398291 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:46:26.411688 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398294 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:46:26.411688 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398296 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:46:26.411688 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398300 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:46:26.411688 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398316 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:46:26.411688 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398321 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:46:26.412158 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398324 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:46:26.412158 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398328 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:46:26.412158 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398330 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:46:26.412158 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398334 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:46:26.412158 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398336 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:46:26.412158 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398340 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:46:26.412158 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398342 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:46:26.412158 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398345 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:46:26.412158 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398347 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:46:26.412158 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398350 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:46:26.412158 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398352 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:46:26.412158 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398355 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:46:26.412158 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398358 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:46:26.412158 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398360 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:46:26.412158 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398363 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:46:26.412158 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398365 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:46:26.412158 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398368 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:46:26.412158 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398370 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:46:26.412158 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398373 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:46:26.412158 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398376 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:46:26.412673 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398378 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:46:26.412673 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398381 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:46:26.412673 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398386 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:46:26.412673 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398389 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:46:26.412673 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398391 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:46:26.412673 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398394 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:46:26.412673 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398396 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:46:26.412673 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398399 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:46:26.412673 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398401 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:46:26.412673 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398405 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:46:26.412673 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398408 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:46:26.412673 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398410 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:46:26.412673 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398413 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:46:26.412673 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398416 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:46:26.412673 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398418 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:46:26.412673 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398421 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:46:26.412673 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398424 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:46:26.412673 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398432 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:46:26.412673 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398436 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:46:26.412673 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398438 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:46:26.413138 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398441 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:46:26.413138 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398446 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:46:26.413138 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398449 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:46:26.413138 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398452 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:46:26.413138 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398456 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:46:26.413138 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398459 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:46:26.413138 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:26.398462 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:46:26.413138 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.398512 2580 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 16:46:26.413138 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.400476 2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 16:46:26.413138 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.401295 2580 server.go:1014] "Starting client certificate rotation" Oct 30 16:46:26.413138 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.401404 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 16:46:26.413138 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.401451 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 16:46:26.413138 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.403452 2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 16:46:26.413138 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.404985 2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 16:46:26.416012 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.415993 2580 log.go:25] "Validated CRI v1 runtime API" Oct 30 16:46:26.418864 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.418849 2580 log.go:25] "Validated CRI v1 image API" Oct 30 16:46:26.419674 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.419652 2580 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 16:46:26.425120 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.425095 2580 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a5440e34-6202-4b3f-9a67-2aacbb8464f3:/dev/nvme0n1p3 bb58242a-b906-465f-b426-fc8649b1174d:/dev/nvme0n1p4] Oct 30 16:46:26.425199 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.425120 2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 16:46:26.430333 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.430207 2580 manager.go:217] Machine: {Timestamp:2026-10-30 16:46:26.428850488 +0000 UTC m=+0.303827718 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100017 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d6784ae2daf86a731425ba0a818e0 SystemUUID:ec2d6784-ae2d-af86-a731-425ba0a818e0 BootID:e25712fa-9da9-46bd-8c13-27216cc8ebc5 Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d4:19:d5:a5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d4:19:d5:a5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:0e:a9:5c:d8:d5:fd Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 16:46:26.430421 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.430333 2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 16:46:26.430421 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.430410 2580 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 16:46:26.432457 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.432432 2580 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 16:46:26.432588 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.432458 2580 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-218.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 16:46:26.432633 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.432596 2580 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 16:46:26.432633 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.432603 2580 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 16:46:26.432633 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.432630 2580 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 16:46:26.432707 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.432643 2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 16:46:26.433343 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.433333 2580 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 16:46:26.436930 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.436918 2580 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 16:46:26.437046 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.437037 2580 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 16:46:26.437607 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.437598 2580 kubelet.go:493] "Attempting to sync node with API server" Oct 30 16:46:26.437642 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.437611 2580 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 16:46:26.437642 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.437623 2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 16:46:26.437642 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.437632 2580 kubelet.go:405] "Adding apiserver pod source" Oct 30 16:46:26.437642 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.437641 2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 16:46:26.437988 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.437974 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 16:46:26.440252 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.440235 2580 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 16:46:26.444130 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.444112 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 16:46:26.444778 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.444765 2580 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 16:46:26.444827 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.444797 2580 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 16:46:26.444967 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.444957 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 16:46:26.444996 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.444979 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 16:46:26.444996 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.444986 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 16:46:26.444996 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.444992 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 16:46:26.444996 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.444997 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 16:46:26.445095 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.445005 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 16:46:26.445485 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.445476 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 16:46:26.445557 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.445486 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 16:46:26.445557 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.445494 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 16:46:26.445557 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.445499 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 16:46:26.445557 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.445513 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 16:46:26.445557 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.445523 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 16:46:26.446369 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.446359 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 16:46:26.446406 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.446373 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 16:46:26.455279 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.455266 2580 server.go:1263] "Started kubelet" Oct 30 16:46:26.455438 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.455388 2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 16:46:26.455484 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.455456 2580 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 16:46:26.455884 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.455341 2580 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 16:46:26.456028 ip-10-0-128-218 systemd[1]: Started Kubernetes Kubelet. Oct 30 16:46:26.457422 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.457404 2580 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 16:46:26.460125 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.460107 2580 server.go:317] "Adding debug handlers to kubelet server" Oct 30 16:46:26.462077 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:26.462056 2580 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 16:46:26.462394 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.462378 2580 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 16:46:26.462460 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.462392 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 16:46:26.462510 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:26.462474 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-218.ec2.internal\" not found" Oct 30 16:46:26.462949 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.462909 2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 16:46:26.463183 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.463160 2580 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 16:46:26.463268 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.463186 2580 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 16:46:26.463394 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.463376 2580 reconstruct.go:97] "Volume reconstruction finished" Oct 30 16:46:26.463394 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.463389 2580 reconciler.go:29] "Reconciler: start to sync state" Oct 30 16:46:26.463556 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.463477 2580 factory.go:55] Registering systemd factory Oct 30 16:46:26.463556 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.463516 2580 factory.go:223] Registration of the systemd container factory successfully Oct 30 16:46:26.463762 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.463747 2580 factory.go:153] Registering CRI-O factory Oct 30 16:46:26.464010 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.463995 2580 factory.go:223] Registration of the crio container factory successfully Oct 30 16:46:26.464075 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.464043 2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 16:46:26.464075 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.464059 2580 factory.go:103] Registering Raw factory Oct 30 16:46:26.464075 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.464070 2580 manager.go:1199] Started watching for new ooms in manager Oct 30 16:46:26.464908 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.464886 2580 manager.go:319] Starting recovery of all containers Oct 30 16:46:26.476332 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.476299 2580 manager.go:324] Recovery completed Oct 30 16:46:26.481736 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.481720 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:46:26.485690 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.485670 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-218.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:46:26.485770 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.485712 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-218.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:46:26.485770 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.485731 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-218.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:46:26.486375 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.486357 2580 cpu_manager.go:226] "Starting" policy="none" Oct 30 16:46:26.486375 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.486369 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:46:26.486538 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.486372 2580 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 16:46:26.486538 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.486436 2580 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 16:46:26.487219 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.487202 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-218.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:46:26.487292 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.487234 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-218.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:46:26.487292 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.487257 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-218.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:46:26.489829 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.489805 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-218.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 16:46:26.489906 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:26.489804 2580 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-218.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 16:46:26.490637 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:26.489809 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-218.ec2.internal.18e35c812a62a45a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-218.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-218.ec2.internal,},FirstTimestamp:2026-10-30 16:46:26.455233626 +0000 UTC m=+0.330210855,LastTimestamp:2026-10-30 16:46:26.455233626 +0000 UTC m=+0.330210855,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-218.ec2.internal,}" Oct 30 16:46:26.503064 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.503036 2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 16:46:26.503165 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.503131 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gnwlf" Oct 30 16:46:26.510240 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:26.510172 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-218.ec2.internal.18e35c812c33639b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-218.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-218.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-218.ec2.internal,},FirstTimestamp:2026-10-30 16:46:26.485691291 +0000 UTC m=+0.360668528,LastTimestamp:2026-10-30 16:46:26.485691291 +0000 UTC m=+0.360668528,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-218.ec2.internal,}" Oct 30 16:46:26.510895 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:26.510860 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 16:46:26.510986 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:26.510857 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 16:46:26.511038 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.510989 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gnwlf" Oct 30 16:46:26.535868 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.535849 2580 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:46:26.563040 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:26.563016 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-218.ec2.internal\" not found" Oct 30 16:46:26.622627 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.622597 2580 policy_none.go:50] "Start" Oct 30 16:46:26.622627 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.622626 2580 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 16:46:26.622773 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.622638 2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 16:46:26.624647 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.624635 2580 policy_none.go:44] "Start" Oct 30 16:46:26.656895 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.656866 2580 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 16:46:26.659243 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.659224 2580 manager.go:341] "Starting Device Plugin manager" Oct 30 16:46:26.659373 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:26.659337 2580 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 16:46:26.659373 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.659352 2580 server.go:87] "Starting device plugin registration server" Oct 30 16:46:26.659632 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.659615 2580 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 16:46:26.659966 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.659634 2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 16:46:26.659966 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.659704 2580 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 16:46:26.659966 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.659820 2580 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 16:46:26.659966 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.659832 2580 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 16:46:26.660432 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:26.660407 2580 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 16:46:26.660540 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:26.660463 2580 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-218.ec2.internal\" not found" Oct 30 16:46:26.665206 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.665068 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 16:46:26.666340 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.666318 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 16:46:26.666406 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.666348 2580 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 16:46:26.666406 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.666372 2580 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 16:46:26.666481 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:26.666415 2580 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 16:46:26.680778 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.680758 2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:46:26.718214 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:26.718187 2580 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-128-218.ec2.internal\" not found" node="ip-10-0-128-218.ec2.internal" Oct 30 16:46:26.760335 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.760295 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:46:26.762213 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.762196 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-218.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:46:26.762323 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.762230 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-218.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:46:26.762323 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.762245 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-218.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:46:26.762323 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.762278 2580 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-218.ec2.internal" Oct 30 16:46:26.766980 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.766960 2580 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-218.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-218.ec2.internal"] Oct 30 16:46:26.767044 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.767013 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:46:26.767833 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.767816 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-218.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:46:26.767901 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.767846 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-218.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:46:26.767901 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.767868 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-218.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:46:26.769986 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.769973 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:46:26.770163 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.770125 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-218.ec2.internal" Oct 30 16:46:26.770163 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.770157 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:46:26.770716 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.770700 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-218.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:46:26.770716 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.770709 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-218.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:46:26.770824 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.770731 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-218.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:46:26.770824 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.770743 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-218.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:46:26.770824 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.770731 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-218.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:46:26.770824 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.770810 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-218.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:46:26.772948 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.772930 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-218.ec2.internal" Oct 30 16:46:26.773027 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.772961 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:46:26.773629 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.773612 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-218.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:46:26.773703 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.773646 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-218.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:46:26.773703 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.773660 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-218.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:46:26.781131 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.781116 2580 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-218.ec2.internal" Oct 30 16:46:26.781200 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:26.781142 2580 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-218.ec2.internal\": node \"ip-10-0-128-218.ec2.internal\" not found" Oct 30 16:46:26.803393 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.803373 2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-218.ec2.internal" Oct 30 16:46:26.805906 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.805892 2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-218.ec2.internal" Oct 30 16:46:26.839334 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.839296 2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 16:46:26.843837 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.843820 2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 16:46:26.862911 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.862890 2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-218.ec2.internal" Oct 30 16:46:26.866150 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.866129 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91966e7831263cb270802b19c1162280-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-218.ec2.internal\" (UID: \"91966e7831263cb270802b19c1162280\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-218.ec2.internal" Oct 30 16:46:26.866268 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.866155 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bc58f75ef247c6cbf4faad53957c9f43-config\") pod \"kube-apiserver-proxy-ip-10-0-128-218.ec2.internal\" (UID: \"bc58f75ef247c6cbf4faad53957c9f43\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-218.ec2.internal" Oct 30 16:46:26.866268 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.866174 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/91966e7831263cb270802b19c1162280-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-218.ec2.internal\" (UID: \"91966e7831263cb270802b19c1162280\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-218.ec2.internal" Oct 30 16:46:26.892337 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.892296 2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 16:46:26.892469 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:26.892361 2580 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-218.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-218.ec2.internal" Oct 30 16:46:26.892469 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.892377 2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-218.ec2.internal" Oct 30 16:46:26.916765 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.916704 2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 16:46:26.916765 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:26.916743 2580 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-218.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-218.ec2.internal" Oct 30 16:46:26.967178 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.967150 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/91966e7831263cb270802b19c1162280-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-218.ec2.internal\" (UID: \"91966e7831263cb270802b19c1162280\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-218.ec2.internal" Oct 30 16:46:26.967178 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.967186 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91966e7831263cb270802b19c1162280-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-218.ec2.internal\" (UID: \"91966e7831263cb270802b19c1162280\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-218.ec2.internal" Oct 30 16:46:26.967429 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.967209 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bc58f75ef247c6cbf4faad53957c9f43-config\") pod \"kube-apiserver-proxy-ip-10-0-128-218.ec2.internal\" (UID: \"bc58f75ef247c6cbf4faad53957c9f43\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-218.ec2.internal" Oct 30 16:46:26.967429 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.967267 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bc58f75ef247c6cbf4faad53957c9f43-config\") pod \"kube-apiserver-proxy-ip-10-0-128-218.ec2.internal\" (UID: \"bc58f75ef247c6cbf4faad53957c9f43\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-218.ec2.internal" Oct 30 16:46:26.967429 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.967272 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/91966e7831263cb270802b19c1162280-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-218.ec2.internal\" (UID: \"91966e7831263cb270802b19c1162280\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-218.ec2.internal" Oct 30 16:46:26.967429 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:26.967272 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91966e7831263cb270802b19c1162280-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-218.ec2.internal\" (UID: \"91966e7831263cb270802b19c1162280\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-218.ec2.internal" Oct 30 16:46:27.143038 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.143010 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-218.ec2.internal" Oct 30 16:46:27.146324 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.146290 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-218.ec2.internal" Oct 30 16:46:27.402602 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.402569 2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 16:46:27.403061 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.402758 2580 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 16:46:27.438690 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.438660 2580 apiserver.go:52] "Watching apiserver" Oct 30 16:46:27.462824 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.462796 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 16:46:27.511706 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.511668 2580 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 16:46:27.512043 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.512016 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-218.ec2.internal","kube-system/konnectivity-agent-mqhg8","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftqbf","openshift-dns/node-resolver-mbwdh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-218.ec2.internal","openshift-multus/multus-additional-cni-plugins-bll4b","openshift-multus/multus-n7lg2","openshift-multus/network-metrics-daemon-rsgvx","kube-system/global-pull-secret-syncer-2kbm7","openshift-cluster-node-tuning-operator/tuned-mlkjb","openshift-image-registry/node-ca-7dcvf","openshift-network-operator/iptables-alerter-v79c8","openshift-ovn-kubernetes/ovnkube-node-ns7rp","openshift-network-diagnostics/network-check-target-8zqlg"] Oct 30 16:46:27.514500 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.514473 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mqhg8" Oct 30 16:46:27.515783 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.515735 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 16:41:26 +0000 UTC" deadline="2028-07-16 02:29:22.27335678 +0000 UTC" Oct 30 16:46:27.515783 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.515783 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14985h42m54.757577616s" Oct 30 16:46:27.516971 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.516953 2580 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:46:27.518835 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.518816 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftqbf" Oct 30 16:46:27.521230 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.521214 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mbwdh" Oct 30 16:46:27.523406 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.523387 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bll4b" Oct 30 16:46:27.526283 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.526258 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n7lg2" Oct 30 16:46:27.528460 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.528443 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsgvx" Oct 30 16:46:27.528538 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:27.528518 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsgvx" podUID="1ba7d788-14ef-4b39-b6f6-3072ab0b3200" Oct 30 16:46:27.530791 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.530775 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kbm7" Oct 30 16:46:27.530849 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:27.530835 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kbm7" podUID="d52d0ddc-6bd1-4dcc-9d56-84213c328055" Oct 30 16:46:27.533125 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.533101 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:46:27.536221 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.536200 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 16:46:27.536548 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.536527 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-z7xnd\"" Oct 30 16:46:27.536622 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.536542 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 16:46:27.538362 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.538347 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7dcvf" Oct 30 16:46:27.541248 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.541218 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-v79c8" Oct 30 16:46:27.544151 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.544133 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:27.546767 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.546750 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8zqlg" Oct 30 16:46:27.546842 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:27.546803 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8zqlg" podUID="40e1c660-22c0-42e5-8511-4cbd0e170a5c" Oct 30 16:46:27.563918 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.563902 2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 16:46:27.569796 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.569773 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch8ct\" (UniqueName: \"kubernetes.io/projected/de08f19f-1712-4864-87b0-294e3d3e460d-kube-api-access-ch8ct\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:46:27.569937 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.569802 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-etc-kubernetes\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:46:27.569937 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.569818 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-sys\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:46:27.569937 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.569835 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-ovnkube-config\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:27.569937 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.569892 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e3fcf8dd-4ee4-4037-9952-9c54a49c6738-socket-dir\") pod \"aws-ebs-csi-driver-node-ftqbf\" (UID: \"e3fcf8dd-4ee4-4037-9952-9c54a49c6738\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftqbf" Oct 30 16:46:27.570088 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.569934 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c702aab4-18a3-4b3a-a629-7d87f15747e7-tmp-dir\") pod \"node-resolver-mbwdh\" (UID: \"c702aab4-18a3-4b3a-a629-7d87f15747e7\") " pod="openshift-dns/node-resolver-mbwdh" Oct 30 16:46:27.570088 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.569973 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d52d0ddc-6bd1-4dcc-9d56-84213c328055-original-pull-secret\") pod \"global-pull-secret-syncer-2kbm7\" (UID: \"d52d0ddc-6bd1-4dcc-9d56-84213c328055\") " pod="kube-system/global-pull-secret-syncer-2kbm7" Oct 30 16:46:27.570088 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.570006 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-systemd-units\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:27.570088 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.570023 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-var-lib-openvswitch\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:27.570209 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.570127 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-run-ovn\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:27.570209 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.570154 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-host-run-ovn-kubernetes\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:27.570209 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.570172 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfpbf\" (UniqueName: \"kubernetes.io/projected/e3fcf8dd-4ee4-4037-9952-9c54a49c6738-kube-api-access-mfpbf\") pod \"aws-ebs-csi-driver-node-ftqbf\" (UID: \"e3fcf8dd-4ee4-4037-9952-9c54a49c6738\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftqbf" Oct 30 16:46:27.570209 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.570188 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/24421640-34bc-4b15-a096-ffdaa7ae7bf4-cni-binary-copy\") pod \"multus-additional-cni-plugins-bll4b\" (UID: \"24421640-34bc-4b15-a096-ffdaa7ae7bf4\") " pod="openshift-multus/multus-additional-cni-plugins-bll4b" Oct 30 16:46:27.570334 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.570222 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-cnibin\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:46:27.570334 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.570250 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d52d0ddc-6bd1-4dcc-9d56-84213c328055-dbus\") pod \"global-pull-secret-syncer-2kbm7\" (UID: \"d52d0ddc-6bd1-4dcc-9d56-84213c328055\") " pod="kube-system/global-pull-secret-syncer-2kbm7" Oct 30 16:46:27.570334 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.570272 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-multus-cni-dir\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:46:27.570334 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.570305 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-ovn-node-metrics-cert\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:27.570478 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.570343 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-host-kubelet\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:27.570478 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.570369 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-run-systemd\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:27.570478 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.570388 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e3fcf8dd-4ee4-4037-9952-9c54a49c6738-etc-selinux\") pod \"aws-ebs-csi-driver-node-ftqbf\" (UID: \"e3fcf8dd-4ee4-4037-9952-9c54a49c6738\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftqbf" Oct 30 16:46:27.570478 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.570426 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d52d0ddc-6bd1-4dcc-9d56-84213c328055-kubelet-config\") pod \"global-pull-secret-syncer-2kbm7\" (UID: \"d52d0ddc-6bd1-4dcc-9d56-84213c328055\") " pod="kube-system/global-pull-secret-syncer-2kbm7" Oct 30 16:46:27.570478 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.570446 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-host-var-lib-kubelet\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:46:27.570630 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.570490 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/24421640-34bc-4b15-a096-ffdaa7ae7bf4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bll4b\" (UID: \"24421640-34bc-4b15-a096-ffdaa7ae7bf4\") " pod="openshift-multus/multus-additional-cni-plugins-bll4b" Oct 30 16:46:27.570630 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.570515 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jblxf\" (UniqueName: \"kubernetes.io/projected/bbdceb2b-73c7-4501-994c-4e3533fa47d0-kube-api-access-jblxf\") pod \"node-ca-7dcvf\" (UID: \"bbdceb2b-73c7-4501-994c-4e3533fa47d0\") " pod="openshift-image-registry/node-ca-7dcvf" Oct 30 16:46:27.570630 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.570532 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-host-run-multus-certs\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:46:27.570630 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.570571 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-etc-modprobe-d\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:46:27.570630 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.570596 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-etc-sysctl-conf\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:46:27.570630 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.570619 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-etc-systemd\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:46:27.570814 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.570643 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s2zv\" (UniqueName: \"kubernetes.io/projected/2434842e-d96c-4890-9b2a-2a9eb130394a-kube-api-access-2s2zv\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:46:27.570814 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.570666 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-etc-openvswitch\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:27.570814 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.570682 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhcd5\" (UniqueName: \"kubernetes.io/projected/24421640-34bc-4b15-a096-ffdaa7ae7bf4-kube-api-access-dhcd5\") pod \"multus-additional-cni-plugins-bll4b\" (UID: \"24421640-34bc-4b15-a096-ffdaa7ae7bf4\") " pod="openshift-multus/multus-additional-cni-plugins-bll4b" Oct 30 16:46:27.570814 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.570725 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjpvn\" (UniqueName: \"kubernetes.io/projected/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-kube-api-access-pjpvn\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:27.570814 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.570757 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-run-openvswitch\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:27.570814 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.570776 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c702aab4-18a3-4b3a-a629-7d87f15747e7-hosts-file\") pod \"node-resolver-mbwdh\" (UID: \"c702aab4-18a3-4b3a-a629-7d87f15747e7\") " pod="openshift-dns/node-resolver-mbwdh" Oct 30 16:46:27.570814 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.570799 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24421640-34bc-4b15-a096-ffdaa7ae7bf4-system-cni-dir\") pod \"multus-additional-cni-plugins-bll4b\" (UID: \"24421640-34bc-4b15-a096-ffdaa7ae7bf4\") " pod="openshift-multus/multus-additional-cni-plugins-bll4b" Oct 30 16:46:27.571031 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.570830 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/24421640-34bc-4b15-a096-ffdaa7ae7bf4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bll4b\" (UID: \"24421640-34bc-4b15-a096-ffdaa7ae7bf4\") " pod="openshift-multus/multus-additional-cni-plugins-bll4b" Oct 30 16:46:27.571031 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.570848 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d24bfb4d-652d-4e7d-bc67-43441164f652-agent-certs\") pod \"konnectivity-agent-mqhg8\" (UID: \"d24bfb4d-652d-4e7d-bc67-43441164f652\") " pod="kube-system/konnectivity-agent-mqhg8" Oct 30 16:46:27.571031 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.570891 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-host-run-k8s-cni-cncf-io\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:46:27.571031 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.570921 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-host\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:46:27.571031 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.570939 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/24421640-34bc-4b15-a096-ffdaa7ae7bf4-cnibin\") pod \"multus-additional-cni-plugins-bll4b\" (UID: \"24421640-34bc-4b15-a096-ffdaa7ae7bf4\") " pod="openshift-multus/multus-additional-cni-plugins-bll4b" Oct 30 16:46:27.571031 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.570991 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-hostroot\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:46:27.571031 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.571013 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8eab0b7c-14cc-42fa-993f-d52707e42d35-iptables-alerter-script\") pod \"iptables-alerter-v79c8\" (UID: \"8eab0b7c-14cc-42fa-993f-d52707e42d35\") " pod="openshift-network-operator/iptables-alerter-v79c8" Oct 30 16:46:27.571233 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.571036 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8eab0b7c-14cc-42fa-993f-d52707e42d35-host-slash\") pod \"iptables-alerter-v79c8\" (UID: \"8eab0b7c-14cc-42fa-993f-d52707e42d35\") " pod="openshift-network-operator/iptables-alerter-v79c8" Oct 30 16:46:27.571233 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.571066 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bbdceb2b-73c7-4501-994c-4e3533fa47d0-serviceca\") pod \"node-ca-7dcvf\" (UID: \"bbdceb2b-73c7-4501-994c-4e3533fa47d0\") " pod="openshift-image-registry/node-ca-7dcvf" Oct 30 16:46:27.571233 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.571089 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d24bfb4d-652d-4e7d-bc67-43441164f652-konnectivity-ca\") pod \"konnectivity-agent-mqhg8\" (UID: \"d24bfb4d-652d-4e7d-bc67-43441164f652\") " pod="kube-system/konnectivity-agent-mqhg8" Oct 30 16:46:27.571233 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.571113 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-ovnkube-script-lib\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:27.571233 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.571149 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e3fcf8dd-4ee4-4037-9952-9c54a49c6738-registration-dir\") pod \"aws-ebs-csi-driver-node-ftqbf\" (UID: \"e3fcf8dd-4ee4-4037-9952-9c54a49c6738\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftqbf" Oct 30 16:46:27.571233 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.571168 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-etc-sysconfig\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:46:27.571233 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.571212 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95mt2\" (UniqueName: \"kubernetes.io/projected/8eab0b7c-14cc-42fa-993f-d52707e42d35-kube-api-access-95mt2\") pod \"iptables-alerter-v79c8\" (UID: \"8eab0b7c-14cc-42fa-993f-d52707e42d35\") " pod="openshift-network-operator/iptables-alerter-v79c8" Oct 30 16:46:27.571459 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.571245 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-node-log\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:27.571459 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.571262 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-host-cni-netd\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:27.571459 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.571299 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ba7d788-14ef-4b39-b6f6-3072ab0b3200-metrics-certs\") pod \"network-metrics-daemon-rsgvx\" (UID: \"1ba7d788-14ef-4b39-b6f6-3072ab0b3200\") " pod="openshift-multus/network-metrics-daemon-rsgvx" Oct 30 16:46:27.571459 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.571349 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-multus-socket-dir-parent\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:46:27.571459 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.571368 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-host-var-lib-cni-bin\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:46:27.571459 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.571393 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-etc-kubernetes\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:46:27.571459 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.571416 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2434842e-d96c-4890-9b2a-2a9eb130394a-tmp\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:46:27.571459 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.571436 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-lib-modules\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:46:27.571459 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.571458 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2434842e-d96c-4890-9b2a-2a9eb130394a-etc-tuned\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:46:27.571719 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.571482 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5st2s\" (UniqueName: \"kubernetes.io/projected/40e1c660-22c0-42e5-8511-4cbd0e170a5c-kube-api-access-5st2s\") pod \"network-check-target-8zqlg\" (UID: \"40e1c660-22c0-42e5-8511-4cbd0e170a5c\") " pod="openshift-network-diagnostics/network-check-target-8zqlg" Oct 30 16:46:27.571719 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.571499 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e3fcf8dd-4ee4-4037-9952-9c54a49c6738-sys-fs\") pod \"aws-ebs-csi-driver-node-ftqbf\" (UID: \"e3fcf8dd-4ee4-4037-9952-9c54a49c6738\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftqbf" Oct 30 16:46:27.571719 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.571524 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/24421640-34bc-4b15-a096-ffdaa7ae7bf4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bll4b\" (UID: \"24421640-34bc-4b15-a096-ffdaa7ae7bf4\") " pod="openshift-multus/multus-additional-cni-plugins-bll4b" Oct 30 16:46:27.571719 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.571541 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bbdceb2b-73c7-4501-994c-4e3533fa47d0-host\") pod \"node-ca-7dcvf\" (UID: \"bbdceb2b-73c7-4501-994c-4e3533fa47d0\") " pod="openshift-image-registry/node-ca-7dcvf" Oct 30 16:46:27.571719 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.571558 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-etc-sysctl-d\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:46:27.571719 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.571616 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-host-cni-bin\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:27.571719 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.571639 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-multus-conf-dir\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:46:27.571719 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.571656 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-run\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:46:27.571719 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.571680 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-host-slash\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:27.571719 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.571702 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-log-socket\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:27.572028 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.571731 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:27.572028 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.571754 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-host-run-netns\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:46:27.572028 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.571791 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-host-var-lib-cni-multus\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:46:27.572028 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.571823 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-var-lib-kubelet\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:46:27.572028 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.571840 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-env-overrides\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:27.572028 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.571862 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e3fcf8dd-4ee4-4037-9952-9c54a49c6738-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ftqbf\" (UID: \"e3fcf8dd-4ee4-4037-9952-9c54a49c6738\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftqbf" Oct 30 16:46:27.572028 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.571884 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-host-run-netns\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:27.572028 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.571913 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e3fcf8dd-4ee4-4037-9952-9c54a49c6738-device-dir\") pod \"aws-ebs-csi-driver-node-ftqbf\" (UID: \"e3fcf8dd-4ee4-4037-9952-9c54a49c6738\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftqbf" Oct 30 16:46:27.572028 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.571938 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s78x6\" (UniqueName: \"kubernetes.io/projected/c702aab4-18a3-4b3a-a629-7d87f15747e7-kube-api-access-s78x6\") pod \"node-resolver-mbwdh\" (UID: \"c702aab4-18a3-4b3a-a629-7d87f15747e7\") " pod="openshift-dns/node-resolver-mbwdh" Oct 30 16:46:27.572028 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.571975 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pwzt\" (UniqueName: \"kubernetes.io/projected/1ba7d788-14ef-4b39-b6f6-3072ab0b3200-kube-api-access-4pwzt\") pod \"network-metrics-daemon-rsgvx\" (UID: \"1ba7d788-14ef-4b39-b6f6-3072ab0b3200\") " pod="openshift-multus/network-metrics-daemon-rsgvx" Oct 30 16:46:27.572028 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.572026 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/de08f19f-1712-4864-87b0-294e3d3e460d-cni-binary-copy\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:46:27.572354 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.572049 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/de08f19f-1712-4864-87b0-294e3d3e460d-multus-daemon-config\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:46:27.572354 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.572077 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-system-cni-dir\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:46:27.662652 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.662622 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 16:46:27.662838 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.662791 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 16:46:27.662838 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.662807 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 16:46:27.662838 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.662826 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 16:46:27.662838 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.662798 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 16:46:27.663036 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.662840 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 16:46:27.663036 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.662827 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 16:46:27.663036 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.662874 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 16:46:27.663036 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.662885 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 16:46:27.663036 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.662897 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 16:46:27.663036 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.662913 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 16:46:27.663036 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.662957 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 16:46:27.663036 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.663038 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 16:46:27.663464 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.663141 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 16:46:27.663464 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.663171 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:46:27.663464 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.663171 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:46:27.663464 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.663236 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 16:46:27.663464 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.663248 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 16:46:27.663464 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.663308 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 16:46:27.663464 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.663370 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 16:46:27.663464 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.663384 2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 16:46:27.663464 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.663416 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-j4x52\"" Oct 30 16:46:27.663464 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.663430 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8fhb8\"" Oct 30 16:46:27.663464 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.663436 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rbk5g" Oct 30 16:46:27.663464 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.663450 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6bljg\"" Oct 30 16:46:27.663464 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.663452 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 16:46:27.663464 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.663474 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 16:46:27.663909 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.663483 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 16:46:27.663909 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.663752 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 16:46:27.663909 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.663761 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-fxkbf\"" Oct 30 16:46:27.663909 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.663777 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-f7c68\"" Oct 30 16:46:27.663909 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.663784 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-pdblj\"" Oct 30 16:46:27.663909 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.663752 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wd4nn\"" Oct 30 16:46:27.663909 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.663797 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rbwcw\"" Oct 30 16:46:27.663909 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.663857 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 16:46:27.673258 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.673236 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-multus-conf-dir\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:46:27.673351 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.673264 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-run\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:46:27.673351 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.673281 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-host-slash\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:27.673351 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.673297 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-log-socket\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:27.673467 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.673416 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-run\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:46:27.673467 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.673414 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:27.673467 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.673452 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-host-run-netns\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:46:27.673561 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.673463 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:27.673561 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.673472 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-host-var-lib-cni-multus\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:46:27.673561 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.673488 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-var-lib-kubelet\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:46:27.673561 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.673505 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-env-overrides\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:27.673561 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.673508 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-log-socket\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:27.673561 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.673520 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e3fcf8dd-4ee4-4037-9952-9c54a49c6738-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ftqbf\" (UID: \"e3fcf8dd-4ee4-4037-9952-9c54a49c6738\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftqbf" Oct 30 16:46:27.673561 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.673535 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-host-run-netns\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:27.673561 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.673551 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e3fcf8dd-4ee4-4037-9952-9c54a49c6738-device-dir\") pod \"aws-ebs-csi-driver-node-ftqbf\" (UID: \"e3fcf8dd-4ee4-4037-9952-9c54a49c6738\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftqbf" Oct 30 16:46:27.673912 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.673563 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-host-slash\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:27.673912 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.673566 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s78x6\" (UniqueName: \"kubernetes.io/projected/c702aab4-18a3-4b3a-a629-7d87f15747e7-kube-api-access-s78x6\") pod \"node-resolver-mbwdh\" (UID: \"c702aab4-18a3-4b3a-a629-7d87f15747e7\") " pod="openshift-dns/node-resolver-mbwdh" Oct 30 16:46:27.673912 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.673537 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-multus-conf-dir\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:46:27.673912 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.673605 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pwzt\" (UniqueName: \"kubernetes.io/projected/1ba7d788-14ef-4b39-b6f6-3072ab0b3200-kube-api-access-4pwzt\") pod \"network-metrics-daemon-rsgvx\" (UID: \"1ba7d788-14ef-4b39-b6f6-3072ab0b3200\") " pod="openshift-multus/network-metrics-daemon-rsgvx" Oct 30 16:46:27.673912 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.673610 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-host-run-netns\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:46:27.673912 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.673626 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e3fcf8dd-4ee4-4037-9952-9c54a49c6738-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ftqbf\" (UID: \"e3fcf8dd-4ee4-4037-9952-9c54a49c6738\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftqbf" Oct 30 16:46:27.673912 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.673645 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/de08f19f-1712-4864-87b0-294e3d3e460d-cni-binary-copy\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:46:27.673912 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.673665 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/de08f19f-1712-4864-87b0-294e3d3e460d-multus-daemon-config\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:46:27.673912 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.673664 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-host-var-lib-cni-multus\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:46:27.673912 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.673680 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-system-cni-dir\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:46:27.673912 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.673664 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-host-run-netns\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:27.673912 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.673711 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch8ct\" (UniqueName: \"kubernetes.io/projected/de08f19f-1712-4864-87b0-294e3d3e460d-kube-api-access-ch8ct\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:46:27.673912 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.673746 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-etc-kubernetes\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:46:27.673912 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.673754 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e3fcf8dd-4ee4-4037-9952-9c54a49c6738-device-dir\") pod \"aws-ebs-csi-driver-node-ftqbf\" (UID: \"e3fcf8dd-4ee4-4037-9952-9c54a49c6738\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftqbf" Oct 30 16:46:27.673912 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.673771 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-sys\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:46:27.673912 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.673798 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-ovnkube-config\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:27.673912 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.673823 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e3fcf8dd-4ee4-4037-9952-9c54a49c6738-socket-dir\") pod \"aws-ebs-csi-driver-node-ftqbf\" (UID: \"e3fcf8dd-4ee4-4037-9952-9c54a49c6738\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftqbf" Oct 30 16:46:27.673912 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.673826 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-system-cni-dir\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:46:27.674618 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.673848 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-var-lib-kubelet\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:46:27.674618 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.673848 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c702aab4-18a3-4b3a-a629-7d87f15747e7-tmp-dir\") pod \"node-resolver-mbwdh\" (UID: \"c702aab4-18a3-4b3a-a629-7d87f15747e7\") " pod="openshift-dns/node-resolver-mbwdh" Oct 30 16:46:27.674618 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.673882 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-sys\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:46:27.674618 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.673916 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d52d0ddc-6bd1-4dcc-9d56-84213c328055-original-pull-secret\") pod \"global-pull-secret-syncer-2kbm7\" (UID: \"d52d0ddc-6bd1-4dcc-9d56-84213c328055\") " pod="kube-system/global-pull-secret-syncer-2kbm7" Oct 30 16:46:27.674618 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.673947 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-systemd-units\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:27.674618 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.673963 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e3fcf8dd-4ee4-4037-9952-9c54a49c6738-socket-dir\") pod \"aws-ebs-csi-driver-node-ftqbf\" (UID: \"e3fcf8dd-4ee4-4037-9952-9c54a49c6738\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftqbf" Oct 30 16:46:27.674618 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.673978 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-var-lib-openvswitch\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:27.674618 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:27.673984 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 16:46:27.674618 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.674005 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-run-ovn\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:27.674618 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.674024 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-etc-kubernetes\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:46:27.674618 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.674034 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-host-run-ovn-kubernetes\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:27.674618 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:27.674050 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d52d0ddc-6bd1-4dcc-9d56-84213c328055-original-pull-secret podName:d52d0ddc-6bd1-4dcc-9d56-84213c328055 nodeName:}" failed. No retries permitted until 2026-10-30 16:46:28.174029529 +0000 UTC m=+2.049006774 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d52d0ddc-6bd1-4dcc-9d56-84213c328055-original-pull-secret") pod "global-pull-secret-syncer-2kbm7" (UID: "d52d0ddc-6bd1-4dcc-9d56-84213c328055") : object "kube-system"/"original-pull-secret" not registered Oct 30 16:46:27.674618 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.674075 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-host-run-ovn-kubernetes\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:27.674618 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.674082 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-systemd-units\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:27.674618 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.674120 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-var-lib-openvswitch\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:27.674618 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.674133 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-run-ovn\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:27.674618 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.674172 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c702aab4-18a3-4b3a-a629-7d87f15747e7-tmp-dir\") pod \"node-resolver-mbwdh\" (UID: \"c702aab4-18a3-4b3a-a629-7d87f15747e7\") " pod="openshift-dns/node-resolver-mbwdh" Oct 30 16:46:27.675328 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.674172 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfpbf\" (UniqueName: \"kubernetes.io/projected/e3fcf8dd-4ee4-4037-9952-9c54a49c6738-kube-api-access-mfpbf\") pod \"aws-ebs-csi-driver-node-ftqbf\" (UID: \"e3fcf8dd-4ee4-4037-9952-9c54a49c6738\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftqbf" Oct 30 16:46:27.675328 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.674231 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/24421640-34bc-4b15-a096-ffdaa7ae7bf4-cni-binary-copy\") pod \"multus-additional-cni-plugins-bll4b\" (UID: \"24421640-34bc-4b15-a096-ffdaa7ae7bf4\") " pod="openshift-multus/multus-additional-cni-plugins-bll4b" Oct 30 16:46:27.675328 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.674258 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-cnibin\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:46:27.675328 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.674283 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d52d0ddc-6bd1-4dcc-9d56-84213c328055-dbus\") pod \"global-pull-secret-syncer-2kbm7\" (UID: \"d52d0ddc-6bd1-4dcc-9d56-84213c328055\") " pod="kube-system/global-pull-secret-syncer-2kbm7" Oct 30 16:46:27.675328 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.674310 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-multus-cni-dir\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:46:27.675328 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.674322 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/de08f19f-1712-4864-87b0-294e3d3e460d-cni-binary-copy\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:46:27.675328 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.674335 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-ovn-node-metrics-cert\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:27.675328 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.674382 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-host-kubelet\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:27.675328 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.674462 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-cnibin\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:46:27.675328 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.674503 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-host-kubelet\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:27.675328 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.674510 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-multus-cni-dir\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:46:27.675328 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.674526 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-run-systemd\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:27.675328 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.674548 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d52d0ddc-6bd1-4dcc-9d56-84213c328055-dbus\") pod \"global-pull-secret-syncer-2kbm7\" (UID: \"d52d0ddc-6bd1-4dcc-9d56-84213c328055\") " pod="kube-system/global-pull-secret-syncer-2kbm7" Oct 30 16:46:27.675328 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.674577 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e3fcf8dd-4ee4-4037-9952-9c54a49c6738-etc-selinux\") pod \"aws-ebs-csi-driver-node-ftqbf\" (UID: \"e3fcf8dd-4ee4-4037-9952-9c54a49c6738\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftqbf" Oct 30 16:46:27.675328 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.674605 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d52d0ddc-6bd1-4dcc-9d56-84213c328055-kubelet-config\") pod \"global-pull-secret-syncer-2kbm7\" (UID: \"d52d0ddc-6bd1-4dcc-9d56-84213c328055\") " pod="kube-system/global-pull-secret-syncer-2kbm7" Oct 30 16:46:27.675328 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.674639 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d52d0ddc-6bd1-4dcc-9d56-84213c328055-kubelet-config\") pod \"global-pull-secret-syncer-2kbm7\" (UID: \"d52d0ddc-6bd1-4dcc-9d56-84213c328055\") " pod="kube-system/global-pull-secret-syncer-2kbm7" Oct 30 16:46:27.675328 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.674664 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-run-systemd\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:27.675328 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.674670 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e3fcf8dd-4ee4-4037-9952-9c54a49c6738-etc-selinux\") pod \"aws-ebs-csi-driver-node-ftqbf\" (UID: \"e3fcf8dd-4ee4-4037-9952-9c54a49c6738\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftqbf" Oct 30 16:46:27.676058 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.674690 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-host-var-lib-kubelet\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:46:27.676058 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.674708 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/24421640-34bc-4b15-a096-ffdaa7ae7bf4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bll4b\" (UID: \"24421640-34bc-4b15-a096-ffdaa7ae7bf4\") " pod="openshift-multus/multus-additional-cni-plugins-bll4b" Oct 30 16:46:27.676058 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.674726 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jblxf\" (UniqueName: \"kubernetes.io/projected/bbdceb2b-73c7-4501-994c-4e3533fa47d0-kube-api-access-jblxf\") pod \"node-ca-7dcvf\" (UID: \"bbdceb2b-73c7-4501-994c-4e3533fa47d0\") " pod="openshift-image-registry/node-ca-7dcvf" Oct 30 16:46:27.676058 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.674743 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-host-run-multus-certs\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:46:27.676058 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.674759 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-etc-modprobe-d\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:46:27.676058 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.674774 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-etc-sysctl-conf\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:46:27.676058 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.674789 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-etc-systemd\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:46:27.676058 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.674805 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s2zv\" (UniqueName: \"kubernetes.io/projected/2434842e-d96c-4890-9b2a-2a9eb130394a-kube-api-access-2s2zv\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:46:27.676058 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.674820 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-etc-openvswitch\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:27.676058 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.674836 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhcd5\" (UniqueName: \"kubernetes.io/projected/24421640-34bc-4b15-a096-ffdaa7ae7bf4-kube-api-access-dhcd5\") pod \"multus-additional-cni-plugins-bll4b\" (UID: \"24421640-34bc-4b15-a096-ffdaa7ae7bf4\") " pod="openshift-multus/multus-additional-cni-plugins-bll4b" Oct 30 16:46:27.676058 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.674846 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/de08f19f-1712-4864-87b0-294e3d3e460d-multus-daemon-config\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:46:27.676058 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.674919 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-host-run-multus-certs\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:46:27.676058 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.674970 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-env-overrides\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:27.676058 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.674853 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjpvn\" (UniqueName: \"kubernetes.io/projected/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-kube-api-access-pjpvn\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:27.676058 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.675010 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-etc-modprobe-d\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:46:27.676058 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.675037 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-run-openvswitch\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:27.676058 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.675024 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-host-var-lib-kubelet\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:46:27.676901 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.675100 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-etc-systemd\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:46:27.676901 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.675116 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-etc-openvswitch\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:27.676901 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.675132 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c702aab4-18a3-4b3a-a629-7d87f15747e7-hosts-file\") pod \"node-resolver-mbwdh\" (UID: \"c702aab4-18a3-4b3a-a629-7d87f15747e7\") " pod="openshift-dns/node-resolver-mbwdh" Oct 30 16:46:27.676901 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.675145 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-run-openvswitch\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:27.676901 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.675067 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c702aab4-18a3-4b3a-a629-7d87f15747e7-hosts-file\") pod \"node-resolver-mbwdh\" (UID: \"c702aab4-18a3-4b3a-a629-7d87f15747e7\") " pod="openshift-dns/node-resolver-mbwdh" Oct 30 16:46:27.676901 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.675187 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24421640-34bc-4b15-a096-ffdaa7ae7bf4-system-cni-dir\") pod \"multus-additional-cni-plugins-bll4b\" (UID: \"24421640-34bc-4b15-a096-ffdaa7ae7bf4\") " pod="openshift-multus/multus-additional-cni-plugins-bll4b" Oct 30 16:46:27.676901 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.675218 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-ovnkube-config\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:27.676901 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.675219 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/24421640-34bc-4b15-a096-ffdaa7ae7bf4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bll4b\" (UID: \"24421640-34bc-4b15-a096-ffdaa7ae7bf4\") " pod="openshift-multus/multus-additional-cni-plugins-bll4b" Oct 30 16:46:27.676901 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.675268 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d24bfb4d-652d-4e7d-bc67-43441164f652-agent-certs\") pod \"konnectivity-agent-mqhg8\" (UID: \"d24bfb4d-652d-4e7d-bc67-43441164f652\") " pod="kube-system/konnectivity-agent-mqhg8" Oct 30 16:46:27.676901 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.675296 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-host-run-k8s-cni-cncf-io\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:46:27.676901 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.675304 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24421640-34bc-4b15-a096-ffdaa7ae7bf4-system-cni-dir\") pod \"multus-additional-cni-plugins-bll4b\" (UID: \"24421640-34bc-4b15-a096-ffdaa7ae7bf4\") " pod="openshift-multus/multus-additional-cni-plugins-bll4b" Oct 30 16:46:27.676901 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.675321 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-host\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:46:27.676901 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.675338 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-etc-sysctl-conf\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:46:27.676901 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.675348 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/24421640-34bc-4b15-a096-ffdaa7ae7bf4-cnibin\") pod \"multus-additional-cni-plugins-bll4b\" (UID: \"24421640-34bc-4b15-a096-ffdaa7ae7bf4\") " pod="openshift-multus/multus-additional-cni-plugins-bll4b" Oct 30 16:46:27.676901 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.675352 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-host-run-k8s-cni-cncf-io\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:46:27.676901 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.675401 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-hostroot\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:46:27.676901 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.675404 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/24421640-34bc-4b15-a096-ffdaa7ae7bf4-cni-binary-copy\") pod \"multus-additional-cni-plugins-bll4b\" (UID: \"24421640-34bc-4b15-a096-ffdaa7ae7bf4\") " pod="openshift-multus/multus-additional-cni-plugins-bll4b" Oct 30 16:46:27.676901 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.675418 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-host\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:46:27.677615 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.675420 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/24421640-34bc-4b15-a096-ffdaa7ae7bf4-cnibin\") pod \"multus-additional-cni-plugins-bll4b\" (UID: \"24421640-34bc-4b15-a096-ffdaa7ae7bf4\") " pod="openshift-multus/multus-additional-cni-plugins-bll4b" Oct 30 16:46:27.677615 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.675430 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8eab0b7c-14cc-42fa-993f-d52707e42d35-iptables-alerter-script\") pod \"iptables-alerter-v79c8\" (UID: \"8eab0b7c-14cc-42fa-993f-d52707e42d35\") " pod="openshift-network-operator/iptables-alerter-v79c8" Oct 30 16:46:27.677615 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.675475 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8eab0b7c-14cc-42fa-993f-d52707e42d35-host-slash\") pod \"iptables-alerter-v79c8\" (UID: \"8eab0b7c-14cc-42fa-993f-d52707e42d35\") " pod="openshift-network-operator/iptables-alerter-v79c8" Oct 30 16:46:27.677615 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.675506 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bbdceb2b-73c7-4501-994c-4e3533fa47d0-serviceca\") pod \"node-ca-7dcvf\" (UID: \"bbdceb2b-73c7-4501-994c-4e3533fa47d0\") " pod="openshift-image-registry/node-ca-7dcvf" Oct 30 16:46:27.677615 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.675519 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-hostroot\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:46:27.677615 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.675532 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d24bfb4d-652d-4e7d-bc67-43441164f652-konnectivity-ca\") pod \"konnectivity-agent-mqhg8\" (UID: \"d24bfb4d-652d-4e7d-bc67-43441164f652\") " pod="kube-system/konnectivity-agent-mqhg8" Oct 30 16:46:27.677615 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.675558 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-ovnkube-script-lib\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:27.677615 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.675562 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8eab0b7c-14cc-42fa-993f-d52707e42d35-host-slash\") pod \"iptables-alerter-v79c8\" (UID: \"8eab0b7c-14cc-42fa-993f-d52707e42d35\") " pod="openshift-network-operator/iptables-alerter-v79c8" Oct 30 16:46:27.677615 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.675585 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e3fcf8dd-4ee4-4037-9952-9c54a49c6738-registration-dir\") pod \"aws-ebs-csi-driver-node-ftqbf\" (UID: \"e3fcf8dd-4ee4-4037-9952-9c54a49c6738\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftqbf" Oct 30 16:46:27.677615 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.675591 2580 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 16:46:27.677615 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.675666 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-etc-sysconfig\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:46:27.677615 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.676210 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/24421640-34bc-4b15-a096-ffdaa7ae7bf4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bll4b\" (UID: \"24421640-34bc-4b15-a096-ffdaa7ae7bf4\") " pod="openshift-multus/multus-additional-cni-plugins-bll4b" Oct 30 16:46:27.677615 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.676249 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/24421640-34bc-4b15-a096-ffdaa7ae7bf4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bll4b\" (UID: \"24421640-34bc-4b15-a096-ffdaa7ae7bf4\") " pod="openshift-multus/multus-additional-cni-plugins-bll4b" Oct 30 16:46:27.677615 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.676275 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e3fcf8dd-4ee4-4037-9952-9c54a49c6738-registration-dir\") pod \"aws-ebs-csi-driver-node-ftqbf\" (UID: \"e3fcf8dd-4ee4-4037-9952-9c54a49c6738\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftqbf" Oct 30 16:46:27.677615 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.675614 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-etc-sysconfig\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:46:27.677615 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.676438 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95mt2\" (UniqueName: \"kubernetes.io/projected/8eab0b7c-14cc-42fa-993f-d52707e42d35-kube-api-access-95mt2\") pod \"iptables-alerter-v79c8\" (UID: \"8eab0b7c-14cc-42fa-993f-d52707e42d35\") " pod="openshift-network-operator/iptables-alerter-v79c8" Oct 30 16:46:27.677615 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.676468 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-node-log\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:27.678119 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.676476 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8eab0b7c-14cc-42fa-993f-d52707e42d35-iptables-alerter-script\") pod \"iptables-alerter-v79c8\" (UID: \"8eab0b7c-14cc-42fa-993f-d52707e42d35\") " pod="openshift-network-operator/iptables-alerter-v79c8" Oct 30 16:46:27.678119 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.676492 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-host-cni-netd\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:27.678119 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.676529 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ba7d788-14ef-4b39-b6f6-3072ab0b3200-metrics-certs\") pod \"network-metrics-daemon-rsgvx\" (UID: \"1ba7d788-14ef-4b39-b6f6-3072ab0b3200\") " pod="openshift-multus/network-metrics-daemon-rsgvx" Oct 30 16:46:27.678119 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.676536 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bbdceb2b-73c7-4501-994c-4e3533fa47d0-serviceca\") pod \"node-ca-7dcvf\" (UID: \"bbdceb2b-73c7-4501-994c-4e3533fa47d0\") " pod="openshift-image-registry/node-ca-7dcvf" Oct 30 16:46:27.678119 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.676556 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-multus-socket-dir-parent\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:46:27.678119 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.676584 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-host-var-lib-cni-bin\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:46:27.678119 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.676595 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-host-cni-netd\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:27.678119 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.676611 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-etc-kubernetes\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:46:27.678119 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.676621 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-node-log\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:27.678119 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.676583 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d24bfb4d-652d-4e7d-bc67-43441164f652-konnectivity-ca\") pod \"konnectivity-agent-mqhg8\" (UID: \"d24bfb4d-652d-4e7d-bc67-43441164f652\") " pod="kube-system/konnectivity-agent-mqhg8" Oct 30 16:46:27.678119 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.676637 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2434842e-d96c-4890-9b2a-2a9eb130394a-tmp\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:46:27.678119 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.676654 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-multus-socket-dir-parent\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:46:27.678119 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.676688 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-lib-modules\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:46:27.678119 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.676715 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2434842e-d96c-4890-9b2a-2a9eb130394a-etc-tuned\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:46:27.678119 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:27.676724 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:46:27.678119 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.676740 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5st2s\" (UniqueName: \"kubernetes.io/projected/40e1c660-22c0-42e5-8511-4cbd0e170a5c-kube-api-access-5st2s\") pod \"network-check-target-8zqlg\" (UID: \"40e1c660-22c0-42e5-8511-4cbd0e170a5c\") " pod="openshift-network-diagnostics/network-check-target-8zqlg" Oct 30 16:46:27.678119 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:27.676766 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ba7d788-14ef-4b39-b6f6-3072ab0b3200-metrics-certs podName:1ba7d788-14ef-4b39-b6f6-3072ab0b3200 nodeName:}" failed. No retries permitted until 2026-10-30 16:46:28.176753172 +0000 UTC m=+2.051730420 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1ba7d788-14ef-4b39-b6f6-3072ab0b3200-metrics-certs") pod "network-metrics-daemon-rsgvx" (UID: "1ba7d788-14ef-4b39-b6f6-3072ab0b3200") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:46:27.678636 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.676774 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-host-var-lib-cni-bin\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:46:27.678636 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.676784 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e3fcf8dd-4ee4-4037-9952-9c54a49c6738-sys-fs\") pod \"aws-ebs-csi-driver-node-ftqbf\" (UID: \"e3fcf8dd-4ee4-4037-9952-9c54a49c6738\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftqbf" Oct 30 16:46:27.678636 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.676789 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-ovnkube-script-lib\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:27.678636 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.676811 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/24421640-34bc-4b15-a096-ffdaa7ae7bf4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bll4b\" (UID: \"24421640-34bc-4b15-a096-ffdaa7ae7bf4\") " pod="openshift-multus/multus-additional-cni-plugins-bll4b" Oct 30 16:46:27.678636 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.676834 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bbdceb2b-73c7-4501-994c-4e3533fa47d0-host\") pod \"node-ca-7dcvf\" (UID: \"bbdceb2b-73c7-4501-994c-4e3533fa47d0\") " pod="openshift-image-registry/node-ca-7dcvf" Oct 30 16:46:27.678636 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.676854 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e3fcf8dd-4ee4-4037-9952-9c54a49c6738-sys-fs\") pod \"aws-ebs-csi-driver-node-ftqbf\" (UID: \"e3fcf8dd-4ee4-4037-9952-9c54a49c6738\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftqbf" Oct 30 16:46:27.678636 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.676856 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-etc-sysctl-d\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:46:27.678636 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.676885 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-host-cni-bin\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:27.678636 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.676909 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-lib-modules\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:46:27.678636 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.676816 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-etc-kubernetes\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:46:27.678636 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.676952 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-etc-sysctl-d\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:46:27.678636 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.676955 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-host-cni-bin\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:27.678636 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.676987 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bbdceb2b-73c7-4501-994c-4e3533fa47d0-host\") pod \"node-ca-7dcvf\" (UID: \"bbdceb2b-73c7-4501-994c-4e3533fa47d0\") " pod="openshift-image-registry/node-ca-7dcvf" Oct 30 16:46:27.678636 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.677891 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/24421640-34bc-4b15-a096-ffdaa7ae7bf4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bll4b\" (UID: \"24421640-34bc-4b15-a096-ffdaa7ae7bf4\") " pod="openshift-multus/multus-additional-cni-plugins-bll4b" Oct 30 16:46:27.679292 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.679267 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2434842e-d96c-4890-9b2a-2a9eb130394a-tmp\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:46:27.679417 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.679325 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2434842e-d96c-4890-9b2a-2a9eb130394a-etc-tuned\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:46:27.679487 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.679426 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-ovn-node-metrics-cert\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:27.679532 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.679508 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d24bfb4d-652d-4e7d-bc67-43441164f652-agent-certs\") pod \"konnectivity-agent-mqhg8\" (UID: \"d24bfb4d-652d-4e7d-bc67-43441164f652\") " pod="kube-system/konnectivity-agent-mqhg8" Oct 30 16:46:27.718175 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.718150 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pwzt\" (UniqueName: \"kubernetes.io/projected/1ba7d788-14ef-4b39-b6f6-3072ab0b3200-kube-api-access-4pwzt\") pod \"network-metrics-daemon-rsgvx\" (UID: \"1ba7d788-14ef-4b39-b6f6-3072ab0b3200\") " pod="openshift-multus/network-metrics-daemon-rsgvx" Oct 30 16:46:27.721897 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.721867 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s78x6\" (UniqueName: \"kubernetes.io/projected/c702aab4-18a3-4b3a-a629-7d87f15747e7-kube-api-access-s78x6\") pod \"node-resolver-mbwdh\" (UID: \"c702aab4-18a3-4b3a-a629-7d87f15747e7\") " pod="openshift-dns/node-resolver-mbwdh" Oct 30 16:46:27.722238 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.722208 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjpvn\" (UniqueName: \"kubernetes.io/projected/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-kube-api-access-pjpvn\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:27.722728 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.722706 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfpbf\" (UniqueName: \"kubernetes.io/projected/e3fcf8dd-4ee4-4037-9952-9c54a49c6738-kube-api-access-mfpbf\") pod \"aws-ebs-csi-driver-node-ftqbf\" (UID: \"e3fcf8dd-4ee4-4037-9952-9c54a49c6738\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftqbf" Oct 30 16:46:27.723817 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.723796 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch8ct\" (UniqueName: \"kubernetes.io/projected/de08f19f-1712-4864-87b0-294e3d3e460d-kube-api-access-ch8ct\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:46:27.723877 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.723848 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhcd5\" (UniqueName: \"kubernetes.io/projected/24421640-34bc-4b15-a096-ffdaa7ae7bf4-kube-api-access-dhcd5\") pod \"multus-additional-cni-plugins-bll4b\" (UID: \"24421640-34bc-4b15-a096-ffdaa7ae7bf4\") " pod="openshift-multus/multus-additional-cni-plugins-bll4b" Oct 30 16:46:27.725433 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:27.725408 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 16:46:27.725433 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:27.725434 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 16:46:27.725609 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:27.725459 2580 projected.go:196] Error preparing data for projected volume kube-api-access-5st2s for pod openshift-network-diagnostics/network-check-target-8zqlg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:46:27.725609 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:27.725530 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40e1c660-22c0-42e5-8511-4cbd0e170a5c-kube-api-access-5st2s podName:40e1c660-22c0-42e5-8511-4cbd0e170a5c nodeName:}" failed. No retries permitted until 2026-10-30 16:46:28.225509877 +0000 UTC m=+2.100487117 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5st2s" (UniqueName: "kubernetes.io/projected/40e1c660-22c0-42e5-8511-4cbd0e170a5c-kube-api-access-5st2s") pod "network-check-target-8zqlg" (UID: "40e1c660-22c0-42e5-8511-4cbd0e170a5c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:46:27.726922 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.726897 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jblxf\" (UniqueName: \"kubernetes.io/projected/bbdceb2b-73c7-4501-994c-4e3533fa47d0-kube-api-access-jblxf\") pod \"node-ca-7dcvf\" (UID: \"bbdceb2b-73c7-4501-994c-4e3533fa47d0\") " pod="openshift-image-registry/node-ca-7dcvf" Oct 30 16:46:27.727549 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.727528 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95mt2\" (UniqueName: \"kubernetes.io/projected/8eab0b7c-14cc-42fa-993f-d52707e42d35-kube-api-access-95mt2\") pod \"iptables-alerter-v79c8\" (UID: \"8eab0b7c-14cc-42fa-993f-d52707e42d35\") " pod="openshift-network-operator/iptables-alerter-v79c8" Oct 30 16:46:27.728948 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.728927 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s2zv\" (UniqueName: \"kubernetes.io/projected/2434842e-d96c-4890-9b2a-2a9eb130394a-kube-api-access-2s2zv\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:46:27.746038 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:27.746006 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc58f75ef247c6cbf4faad53957c9f43.slice/crio-77f86b5c5ce7f22475eac00b2a7033e5660a5b01cf17fe32625a4b783b19790a WatchSource:0}: Error finding container 77f86b5c5ce7f22475eac00b2a7033e5660a5b01cf17fe32625a4b783b19790a: Status 404 returned error can't find the container with id 77f86b5c5ce7f22475eac00b2a7033e5660a5b01cf17fe32625a4b783b19790a Oct 30 16:46:27.747096 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:27.747066 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91966e7831263cb270802b19c1162280.slice/crio-73cfaf3d1a1d8a84ba37bfd1f1c2358dfda8ce270736512c278c313aae6bbeb6 WatchSource:0}: Error finding container 73cfaf3d1a1d8a84ba37bfd1f1c2358dfda8ce270736512c278c313aae6bbeb6: Status 404 returned error can't find the container with id 73cfaf3d1a1d8a84ba37bfd1f1c2358dfda8ce270736512c278c313aae6bbeb6 Oct 30 16:46:27.752107 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.752092 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 16:46:27.790183 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.790158 2580 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:46:27.843788 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.843762 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mqhg8" Oct 30 16:46:27.854125 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:27.854100 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd24bfb4d_652d_4e7d_bc67_43441164f652.slice/crio-e1a02f18bc37c32c8c93383a529584ff2d952162e8341f80a4a99051ff438b85 WatchSource:0}: Error finding container e1a02f18bc37c32c8c93383a529584ff2d952162e8341f80a4a99051ff438b85: Status 404 returned error can't find the container with id e1a02f18bc37c32c8c93383a529584ff2d952162e8341f80a4a99051ff438b85 Oct 30 16:46:27.867418 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.867390 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftqbf" Oct 30 16:46:27.878010 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:27.877987 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3fcf8dd_4ee4_4037_9952_9c54a49c6738.slice/crio-5f90a5b2e8450d0fe35942286c1d9f96483f222fca4a6ec99a7a8061df7d6fc8 WatchSource:0}: Error finding container 5f90a5b2e8450d0fe35942286c1d9f96483f222fca4a6ec99a7a8061df7d6fc8: Status 404 returned error can't find the container with id 5f90a5b2e8450d0fe35942286c1d9f96483f222fca4a6ec99a7a8061df7d6fc8 Oct 30 16:46:27.878735 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.878714 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mbwdh" Oct 30 16:46:27.887676 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:27.887616 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc702aab4_18a3_4b3a_a629_7d87f15747e7.slice/crio-b1ebd56f579b03a320cb5b7cbc25f067b7169123bed80d2a34ae0af14f48283e WatchSource:0}: Error finding container b1ebd56f579b03a320cb5b7cbc25f067b7169123bed80d2a34ae0af14f48283e: Status 404 returned error can't find the container with id b1ebd56f579b03a320cb5b7cbc25f067b7169123bed80d2a34ae0af14f48283e Oct 30 16:46:27.897946 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.897922 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bll4b" Oct 30 16:46:27.905755 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.905726 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n7lg2" Oct 30 16:46:27.908051 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:27.908018 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24421640_34bc_4b15_a096_ffdaa7ae7bf4.slice/crio-95d6d3e2380f9668e7a87195cacc53233ec9bf359ee51a6eb1e66ce13dc91206 WatchSource:0}: Error finding container 95d6d3e2380f9668e7a87195cacc53233ec9bf359ee51a6eb1e66ce13dc91206: Status 404 returned error can't find the container with id 95d6d3e2380f9668e7a87195cacc53233ec9bf359ee51a6eb1e66ce13dc91206 Oct 30 16:46:27.911958 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.911933 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:46:27.915364 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:27.915344 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde08f19f_1712_4864_87b0_294e3d3e460d.slice/crio-e9cd4ba94b8ae75459f5b272d45e13d31c4bf152b15886757325f041f4147dc8 WatchSource:0}: Error finding container e9cd4ba94b8ae75459f5b272d45e13d31c4bf152b15886757325f041f4147dc8: Status 404 returned error can't find the container with id e9cd4ba94b8ae75459f5b272d45e13d31c4bf152b15886757325f041f4147dc8 Oct 30 16:46:27.921105 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:27.921079 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2434842e_d96c_4890_9b2a_2a9eb130394a.slice/crio-c8343a205bcb6580ade83510e94c4f2eec6fb35220099fd1b1922a356964a25d WatchSource:0}: Error finding container c8343a205bcb6580ade83510e94c4f2eec6fb35220099fd1b1922a356964a25d: Status 404 returned error can't find the container with id c8343a205bcb6580ade83510e94c4f2eec6fb35220099fd1b1922a356964a25d Oct 30 16:46:27.936447 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.936424 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7dcvf" Oct 30 16:46:27.946033 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:27.946010 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbdceb2b_73c7_4501_994c_4e3533fa47d0.slice/crio-962d4c9822b1abc27c25eb93a60baaf8beaf09e16b434e63802ff7732ec3e560 WatchSource:0}: Error finding container 962d4c9822b1abc27c25eb93a60baaf8beaf09e16b434e63802ff7732ec3e560: Status 404 returned error can't find the container with id 962d4c9822b1abc27c25eb93a60baaf8beaf09e16b434e63802ff7732ec3e560 Oct 30 16:46:27.951248 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.951229 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-v79c8" Oct 30 16:46:27.960407 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:27.960376 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eab0b7c_14cc_42fa_993f_d52707e42d35.slice/crio-6129a8fc729d736a455e6fc90d5fb7d979feaee41cf41276d3810ce84f8b900b WatchSource:0}: Error finding container 6129a8fc729d736a455e6fc90d5fb7d979feaee41cf41276d3810ce84f8b900b: Status 404 returned error can't find the container with id 6129a8fc729d736a455e6fc90d5fb7d979feaee41cf41276d3810ce84f8b900b Oct 30 16:46:27.975934 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:27.975910 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:27.985866 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:27.985842 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38c17ac6_93d3_4d38_83db_70d4d0fa3f30.slice/crio-11e310e8c78c4004c8aa0e76c03c4906cf266079403a7e0fda8db10f0e868c28 WatchSource:0}: Error finding container 11e310e8c78c4004c8aa0e76c03c4906cf266079403a7e0fda8db10f0e868c28: Status 404 returned error can't find the container with id 11e310e8c78c4004c8aa0e76c03c4906cf266079403a7e0fda8db10f0e868c28 Oct 30 16:46:28.179802 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:28.179710 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ba7d788-14ef-4b39-b6f6-3072ab0b3200-metrics-certs\") pod \"network-metrics-daemon-rsgvx\" (UID: \"1ba7d788-14ef-4b39-b6f6-3072ab0b3200\") " pod="openshift-multus/network-metrics-daemon-rsgvx" Oct 30 16:46:28.180032 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:28.179871 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d52d0ddc-6bd1-4dcc-9d56-84213c328055-original-pull-secret\") pod \"global-pull-secret-syncer-2kbm7\" (UID: \"d52d0ddc-6bd1-4dcc-9d56-84213c328055\") " pod="kube-system/global-pull-secret-syncer-2kbm7" Oct 30 16:46:28.180032 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:28.179886 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:46:28.180032 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:28.179957 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ba7d788-14ef-4b39-b6f6-3072ab0b3200-metrics-certs podName:1ba7d788-14ef-4b39-b6f6-3072ab0b3200 nodeName:}" failed. No retries permitted until 2026-10-30 16:46:29.17993591 +0000 UTC m=+3.054913134 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1ba7d788-14ef-4b39-b6f6-3072ab0b3200-metrics-certs") pod "network-metrics-daemon-rsgvx" (UID: "1ba7d788-14ef-4b39-b6f6-3072ab0b3200") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:46:28.180032 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:28.180009 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 16:46:28.180441 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:28.180062 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d52d0ddc-6bd1-4dcc-9d56-84213c328055-original-pull-secret podName:d52d0ddc-6bd1-4dcc-9d56-84213c328055 nodeName:}" failed. No retries permitted until 2026-10-30 16:46:29.180046735 +0000 UTC m=+3.055023968 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d52d0ddc-6bd1-4dcc-9d56-84213c328055-original-pull-secret") pod "global-pull-secret-syncer-2kbm7" (UID: "d52d0ddc-6bd1-4dcc-9d56-84213c328055") : object "kube-system"/"original-pull-secret" not registered Oct 30 16:46:28.280404 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:28.280356 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5st2s\" (UniqueName: \"kubernetes.io/projected/40e1c660-22c0-42e5-8511-4cbd0e170a5c-kube-api-access-5st2s\") pod \"network-check-target-8zqlg\" (UID: \"40e1c660-22c0-42e5-8511-4cbd0e170a5c\") " pod="openshift-network-diagnostics/network-check-target-8zqlg" Oct 30 16:46:28.280593 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:28.280573 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 16:46:28.280668 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:28.280597 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 16:46:28.280668 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:28.280610 2580 projected.go:196] Error preparing data for projected volume kube-api-access-5st2s for pod openshift-network-diagnostics/network-check-target-8zqlg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:46:28.280777 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:28.280670 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40e1c660-22c0-42e5-8511-4cbd0e170a5c-kube-api-access-5st2s podName:40e1c660-22c0-42e5-8511-4cbd0e170a5c nodeName:}" failed. No retries permitted until 2026-10-30 16:46:29.280650798 +0000 UTC m=+3.155628038 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-5st2s" (UniqueName: "kubernetes.io/projected/40e1c660-22c0-42e5-8511-4cbd0e170a5c-kube-api-access-5st2s") pod "network-check-target-8zqlg" (UID: "40e1c660-22c0-42e5-8511-4cbd0e170a5c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:46:28.667103 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:28.667029 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 16:41:27 +0000 UTC" deadline="2028-05-07 10:30:40.049619272 +0000 UTC" Oct 30 16:46:28.667103 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:28.667101 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13313h44m11.382522095s" Oct 30 16:46:28.667998 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:28.667500 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsgvx" Oct 30 16:46:28.667998 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:28.667633 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsgvx" podUID="1ba7d788-14ef-4b39-b6f6-3072ab0b3200" Oct 30 16:46:28.668108 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:28.668060 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8zqlg" Oct 30 16:46:28.668192 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:28.668168 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8zqlg" podUID="40e1c660-22c0-42e5-8511-4cbd0e170a5c" Oct 30 16:46:28.683130 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:28.683001 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" event={"ID":"38c17ac6-93d3-4d38-83db-70d4d0fa3f30","Type":"ContainerStarted","Data":"11e310e8c78c4004c8aa0e76c03c4906cf266079403a7e0fda8db10f0e868c28"} Oct 30 16:46:28.689409 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:28.689360 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-v79c8" event={"ID":"8eab0b7c-14cc-42fa-993f-d52707e42d35","Type":"ContainerStarted","Data":"6129a8fc729d736a455e6fc90d5fb7d979feaee41cf41276d3810ce84f8b900b"} Oct 30 16:46:28.692754 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:28.692723 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" event={"ID":"2434842e-d96c-4890-9b2a-2a9eb130394a","Type":"ContainerStarted","Data":"c8343a205bcb6580ade83510e94c4f2eec6fb35220099fd1b1922a356964a25d"} Oct 30 16:46:28.710270 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:28.710171 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n7lg2" event={"ID":"de08f19f-1712-4864-87b0-294e3d3e460d","Type":"ContainerStarted","Data":"e9cd4ba94b8ae75459f5b272d45e13d31c4bf152b15886757325f041f4147dc8"} Oct 30 16:46:28.719822 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:28.719789 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bll4b" event={"ID":"24421640-34bc-4b15-a096-ffdaa7ae7bf4","Type":"ContainerStarted","Data":"95d6d3e2380f9668e7a87195cacc53233ec9bf359ee51a6eb1e66ce13dc91206"} Oct 30 16:46:28.728083 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:28.728048 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mbwdh" event={"ID":"c702aab4-18a3-4b3a-a629-7d87f15747e7","Type":"ContainerStarted","Data":"b1ebd56f579b03a320cb5b7cbc25f067b7169123bed80d2a34ae0af14f48283e"} Oct 30 16:46:28.733193 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:28.733160 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftqbf" event={"ID":"e3fcf8dd-4ee4-4037-9952-9c54a49c6738","Type":"ContainerStarted","Data":"5f90a5b2e8450d0fe35942286c1d9f96483f222fca4a6ec99a7a8061df7d6fc8"} Oct 30 16:46:28.745033 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:28.745000 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mqhg8" event={"ID":"d24bfb4d-652d-4e7d-bc67-43441164f652","Type":"ContainerStarted","Data":"e1a02f18bc37c32c8c93383a529584ff2d952162e8341f80a4a99051ff438b85"} Oct 30 16:46:28.755604 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:28.755568 2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:46:28.763356 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:28.763326 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7dcvf" event={"ID":"bbdceb2b-73c7-4501-994c-4e3533fa47d0","Type":"ContainerStarted","Data":"962d4c9822b1abc27c25eb93a60baaf8beaf09e16b434e63802ff7732ec3e560"} Oct 30 16:46:28.767036 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:28.767006 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-218.ec2.internal" event={"ID":"bc58f75ef247c6cbf4faad53957c9f43","Type":"ContainerStarted","Data":"77f86b5c5ce7f22475eac00b2a7033e5660a5b01cf17fe32625a4b783b19790a"} Oct 30 16:46:28.769675 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:28.769644 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-218.ec2.internal" event={"ID":"91966e7831263cb270802b19c1162280","Type":"ContainerStarted","Data":"73cfaf3d1a1d8a84ba37bfd1f1c2358dfda8ce270736512c278c313aae6bbeb6"} Oct 30 16:46:29.189480 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:29.188631 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ba7d788-14ef-4b39-b6f6-3072ab0b3200-metrics-certs\") pod \"network-metrics-daemon-rsgvx\" (UID: \"1ba7d788-14ef-4b39-b6f6-3072ab0b3200\") " pod="openshift-multus/network-metrics-daemon-rsgvx" Oct 30 16:46:29.189480 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:29.188720 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d52d0ddc-6bd1-4dcc-9d56-84213c328055-original-pull-secret\") pod \"global-pull-secret-syncer-2kbm7\" (UID: \"d52d0ddc-6bd1-4dcc-9d56-84213c328055\") " pod="kube-system/global-pull-secret-syncer-2kbm7" Oct 30 16:46:29.189480 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:29.188879 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 16:46:29.189480 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:29.188945 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d52d0ddc-6bd1-4dcc-9d56-84213c328055-original-pull-secret podName:d52d0ddc-6bd1-4dcc-9d56-84213c328055 nodeName:}" failed. No retries permitted until 2026-10-30 16:46:31.188926288 +0000 UTC m=+5.063903528 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d52d0ddc-6bd1-4dcc-9d56-84213c328055-original-pull-secret") pod "global-pull-secret-syncer-2kbm7" (UID: "d52d0ddc-6bd1-4dcc-9d56-84213c328055") : object "kube-system"/"original-pull-secret" not registered Oct 30 16:46:29.189480 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:29.189402 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:46:29.189480 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:29.189453 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ba7d788-14ef-4b39-b6f6-3072ab0b3200-metrics-certs podName:1ba7d788-14ef-4b39-b6f6-3072ab0b3200 nodeName:}" failed. No retries permitted until 2026-10-30 16:46:31.189437365 +0000 UTC m=+5.064414598 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1ba7d788-14ef-4b39-b6f6-3072ab0b3200-metrics-certs") pod "network-metrics-daemon-rsgvx" (UID: "1ba7d788-14ef-4b39-b6f6-3072ab0b3200") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:46:29.289753 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:29.289709 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5st2s\" (UniqueName: \"kubernetes.io/projected/40e1c660-22c0-42e5-8511-4cbd0e170a5c-kube-api-access-5st2s\") pod \"network-check-target-8zqlg\" (UID: \"40e1c660-22c0-42e5-8511-4cbd0e170a5c\") " pod="openshift-network-diagnostics/network-check-target-8zqlg" Oct 30 16:46:29.290022 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:29.289936 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 16:46:29.290022 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:29.289961 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 16:46:29.290022 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:29.289976 2580 projected.go:196] Error preparing data for projected volume kube-api-access-5st2s for pod openshift-network-diagnostics/network-check-target-8zqlg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:46:29.290229 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:29.290047 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40e1c660-22c0-42e5-8511-4cbd0e170a5c-kube-api-access-5st2s podName:40e1c660-22c0-42e5-8511-4cbd0e170a5c nodeName:}" failed. No retries permitted until 2026-10-30 16:46:31.290020663 +0000 UTC m=+5.164997885 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-5st2s" (UniqueName: "kubernetes.io/projected/40e1c660-22c0-42e5-8511-4cbd0e170a5c-kube-api-access-5st2s") pod "network-check-target-8zqlg" (UID: "40e1c660-22c0-42e5-8511-4cbd0e170a5c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:46:29.667136 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:29.666582 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kbm7" Oct 30 16:46:29.667136 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:29.666730 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kbm7" podUID="d52d0ddc-6bd1-4dcc-9d56-84213c328055" Oct 30 16:46:29.667781 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:29.667470 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 16:41:27 +0000 UTC" deadline="2028-06-24 08:39:10.472404588 +0000 UTC" Oct 30 16:46:29.667781 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:29.667503 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14463h52m40.804906979s" Oct 30 16:46:30.668092 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:30.667490 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsgvx" Oct 30 16:46:30.668092 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:30.667641 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsgvx" podUID="1ba7d788-14ef-4b39-b6f6-3072ab0b3200" Oct 30 16:46:30.668936 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:30.668775 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8zqlg" Oct 30 16:46:30.668936 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:30.668887 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8zqlg" podUID="40e1c660-22c0-42e5-8511-4cbd0e170a5c" Oct 30 16:46:31.205955 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:31.205743 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ba7d788-14ef-4b39-b6f6-3072ab0b3200-metrics-certs\") pod \"network-metrics-daemon-rsgvx\" (UID: \"1ba7d788-14ef-4b39-b6f6-3072ab0b3200\") " pod="openshift-multus/network-metrics-daemon-rsgvx" Oct 30 16:46:31.205955 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:31.205828 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d52d0ddc-6bd1-4dcc-9d56-84213c328055-original-pull-secret\") pod \"global-pull-secret-syncer-2kbm7\" (UID: \"d52d0ddc-6bd1-4dcc-9d56-84213c328055\") " pod="kube-system/global-pull-secret-syncer-2kbm7" Oct 30 16:46:31.205955 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:31.205922 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:46:31.205955 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:31.205934 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 16:46:31.206249 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:31.205986 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d52d0ddc-6bd1-4dcc-9d56-84213c328055-original-pull-secret podName:d52d0ddc-6bd1-4dcc-9d56-84213c328055 nodeName:}" failed. No retries permitted until 2026-10-30 16:46:35.205973234 +0000 UTC m=+9.080950452 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d52d0ddc-6bd1-4dcc-9d56-84213c328055-original-pull-secret") pod "global-pull-secret-syncer-2kbm7" (UID: "d52d0ddc-6bd1-4dcc-9d56-84213c328055") : object "kube-system"/"original-pull-secret" not registered Oct 30 16:46:31.206249 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:31.206001 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ba7d788-14ef-4b39-b6f6-3072ab0b3200-metrics-certs podName:1ba7d788-14ef-4b39-b6f6-3072ab0b3200 nodeName:}" failed. No retries permitted until 2026-10-30 16:46:35.205993911 +0000 UTC m=+9.080971130 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1ba7d788-14ef-4b39-b6f6-3072ab0b3200-metrics-certs") pod "network-metrics-daemon-rsgvx" (UID: "1ba7d788-14ef-4b39-b6f6-3072ab0b3200") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:46:31.307481 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:31.306761 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5st2s\" (UniqueName: \"kubernetes.io/projected/40e1c660-22c0-42e5-8511-4cbd0e170a5c-kube-api-access-5st2s\") pod \"network-check-target-8zqlg\" (UID: \"40e1c660-22c0-42e5-8511-4cbd0e170a5c\") " pod="openshift-network-diagnostics/network-check-target-8zqlg" Oct 30 16:46:31.307481 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:31.306998 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 16:46:31.307481 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:31.307031 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 16:46:31.307481 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:31.307044 2580 projected.go:196] Error preparing data for projected volume kube-api-access-5st2s for pod openshift-network-diagnostics/network-check-target-8zqlg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:46:31.307481 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:31.307105 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40e1c660-22c0-42e5-8511-4cbd0e170a5c-kube-api-access-5st2s podName:40e1c660-22c0-42e5-8511-4cbd0e170a5c nodeName:}" failed. No retries permitted until 2026-10-30 16:46:35.307086549 +0000 UTC m=+9.182063771 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-5st2s" (UniqueName: "kubernetes.io/projected/40e1c660-22c0-42e5-8511-4cbd0e170a5c-kube-api-access-5st2s") pod "network-check-target-8zqlg" (UID: "40e1c660-22c0-42e5-8511-4cbd0e170a5c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:46:31.668896 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:31.667533 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kbm7" Oct 30 16:46:31.668896 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:31.667902 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kbm7" podUID="d52d0ddc-6bd1-4dcc-9d56-84213c328055" Oct 30 16:46:31.810409 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:31.809273 2580 generic.go:356] "Generic (PLEG): container finished" podID="91966e7831263cb270802b19c1162280" containerID="59b46fe6ab73bab7f86b93c2892e3ce465bd9d23699d52658fbe6373450e0b73" exitCode=0 Oct 30 16:46:31.810409 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:31.809406 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-218.ec2.internal" event={"ID":"91966e7831263cb270802b19c1162280","Type":"ContainerDied","Data":"59b46fe6ab73bab7f86b93c2892e3ce465bd9d23699d52658fbe6373450e0b73"} Oct 30 16:46:31.825734 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:31.825700 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bll4b" event={"ID":"24421640-34bc-4b15-a096-ffdaa7ae7bf4","Type":"ContainerStarted","Data":"223f5c9b874a8f0215a13fe5d178c0d4fccf81262403c832e3084c47b2e18a15"} Oct 30 16:46:32.667510 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:32.667472 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsgvx" Oct 30 16:46:32.667694 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:32.667616 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsgvx" podUID="1ba7d788-14ef-4b39-b6f6-3072ab0b3200" Oct 30 16:46:32.667694 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:32.667692 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8zqlg" Oct 30 16:46:32.667807 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:32.667768 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8zqlg" podUID="40e1c660-22c0-42e5-8511-4cbd0e170a5c" Oct 30 16:46:32.841465 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:32.840613 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mqhg8" event={"ID":"d24bfb4d-652d-4e7d-bc67-43441164f652","Type":"ContainerStarted","Data":"df7ca5b385efdb8e13933059be946b411d7132bc585b40cf712321cf1d2fbf91"} Oct 30 16:46:32.845319 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:32.844848 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-218.ec2.internal" event={"ID":"bc58f75ef247c6cbf4faad53957c9f43","Type":"ContainerStarted","Data":"f3a2d8f02c3a23239a13368fe6de19760eebb489b8e06bd3717faaf7eb7d40d8"} Oct 30 16:46:32.850981 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:32.850538 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-218.ec2.internal" event={"ID":"91966e7831263cb270802b19c1162280","Type":"ContainerStarted","Data":"7ee8f2b1b704ba67980d54a714ab497fac0f29487c6a5a655a9ea590a2813899"} Oct 30 16:46:32.853545 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:32.853519 2580 generic.go:356] "Generic (PLEG): container finished" podID="24421640-34bc-4b15-a096-ffdaa7ae7bf4" containerID="223f5c9b874a8f0215a13fe5d178c0d4fccf81262403c832e3084c47b2e18a15" exitCode=0 Oct 30 16:46:32.853652 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:32.853555 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bll4b" event={"ID":"24421640-34bc-4b15-a096-ffdaa7ae7bf4","Type":"ContainerDied","Data":"223f5c9b874a8f0215a13fe5d178c0d4fccf81262403c832e3084c47b2e18a15"} Oct 30 16:46:32.887818 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:32.887762 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mqhg8" podStartSLOduration=1.617410235 podStartE2EDuration="5.887733733s" totalImagesPullingTime="4.270323498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:46:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:32.887008405 +0000 UTC m=+6.761985667" watchObservedRunningTime="2026-10-30 16:46:32.887733733 +0000 UTC m=+6.762710973" Oct 30 16:46:32.932458 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:32.932365 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-218.ec2.internal" podStartSLOduration=5.932346332 podStartE2EDuration="5.932346332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:46:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:32.931524258 +0000 UTC m=+6.806501500" watchObservedRunningTime="2026-10-30 16:46:32.932346332 +0000 UTC m=+6.807323585" Oct 30 16:46:33.035455 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:33.035390 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-218.ec2.internal" podStartSLOduration=6.035368265 podStartE2EDuration="6.035368265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:46:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:32.978740509 +0000 UTC m=+6.853717750" watchObservedRunningTime="2026-10-30 16:46:33.035368265 +0000 UTC m=+6.910345505" Oct 30 16:46:33.667601 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:33.667566 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kbm7" Oct 30 16:46:33.667760 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:33.667705 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kbm7" podUID="d52d0ddc-6bd1-4dcc-9d56-84213c328055" Oct 30 16:46:34.667506 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:34.667428 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8zqlg" Oct 30 16:46:34.668105 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:34.667559 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8zqlg" podUID="40e1c660-22c0-42e5-8511-4cbd0e170a5c" Oct 30 16:46:34.668105 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:34.667959 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsgvx" Oct 30 16:46:34.668105 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:34.668056 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsgvx" podUID="1ba7d788-14ef-4b39-b6f6-3072ab0b3200" Oct 30 16:46:34.862322 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:34.862267 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftqbf" event={"ID":"e3fcf8dd-4ee4-4037-9952-9c54a49c6738","Type":"ContainerStarted","Data":"1893ce879f562d79f9092a5592e7b414453a29e68cb75f9345618e6636d9354c"} Oct 30 16:46:34.867491 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:34.867459 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7dcvf" event={"ID":"bbdceb2b-73c7-4501-994c-4e3533fa47d0","Type":"ContainerStarted","Data":"00a89e264ae561d4d83dc8309b6352bd7231b9698ca9fffc183612a0c29be7ed"} Oct 30 16:46:35.238993 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:35.238419 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ba7d788-14ef-4b39-b6f6-3072ab0b3200-metrics-certs\") pod \"network-metrics-daemon-rsgvx\" (UID: \"1ba7d788-14ef-4b39-b6f6-3072ab0b3200\") " pod="openshift-multus/network-metrics-daemon-rsgvx" Oct 30 16:46:35.238993 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:35.238494 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d52d0ddc-6bd1-4dcc-9d56-84213c328055-original-pull-secret\") pod \"global-pull-secret-syncer-2kbm7\" (UID: \"d52d0ddc-6bd1-4dcc-9d56-84213c328055\") " pod="kube-system/global-pull-secret-syncer-2kbm7" Oct 30 16:46:35.238993 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:35.238592 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 16:46:35.238993 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:35.238637 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d52d0ddc-6bd1-4dcc-9d56-84213c328055-original-pull-secret podName:d52d0ddc-6bd1-4dcc-9d56-84213c328055 nodeName:}" failed. No retries permitted until 2026-10-30 16:46:43.23862559 +0000 UTC m=+17.113602807 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d52d0ddc-6bd1-4dcc-9d56-84213c328055-original-pull-secret") pod "global-pull-secret-syncer-2kbm7" (UID: "d52d0ddc-6bd1-4dcc-9d56-84213c328055") : object "kube-system"/"original-pull-secret" not registered Oct 30 16:46:35.238993 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:35.238919 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:46:35.238993 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:35.238950 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ba7d788-14ef-4b39-b6f6-3072ab0b3200-metrics-certs podName:1ba7d788-14ef-4b39-b6f6-3072ab0b3200 nodeName:}" failed. No retries permitted until 2026-10-30 16:46:43.23894035 +0000 UTC m=+17.113917567 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1ba7d788-14ef-4b39-b6f6-3072ab0b3200-metrics-certs") pod "network-metrics-daemon-rsgvx" (UID: "1ba7d788-14ef-4b39-b6f6-3072ab0b3200") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:46:35.341325 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:35.339022 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5st2s\" (UniqueName: \"kubernetes.io/projected/40e1c660-22c0-42e5-8511-4cbd0e170a5c-kube-api-access-5st2s\") pod \"network-check-target-8zqlg\" (UID: \"40e1c660-22c0-42e5-8511-4cbd0e170a5c\") " pod="openshift-network-diagnostics/network-check-target-8zqlg" Oct 30 16:46:35.341325 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:35.339232 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 16:46:35.341325 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:35.339250 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 16:46:35.341325 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:35.339263 2580 projected.go:196] Error preparing data for projected volume kube-api-access-5st2s for pod openshift-network-diagnostics/network-check-target-8zqlg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:46:35.341325 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:35.339333 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40e1c660-22c0-42e5-8511-4cbd0e170a5c-kube-api-access-5st2s podName:40e1c660-22c0-42e5-8511-4cbd0e170a5c nodeName:}" failed. No retries permitted until 2026-10-30 16:46:43.339301674 +0000 UTC m=+17.214278897 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-5st2s" (UniqueName: "kubernetes.io/projected/40e1c660-22c0-42e5-8511-4cbd0e170a5c-kube-api-access-5st2s") pod "network-check-target-8zqlg" (UID: "40e1c660-22c0-42e5-8511-4cbd0e170a5c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:46:35.416513 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:35.416475 2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mqhg8" Oct 30 16:46:35.417359 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:35.417340 2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mqhg8" Oct 30 16:46:35.473114 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:35.473018 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7dcvf" podStartSLOduration=2.677315405 podStartE2EDuration="8.472998873s" totalImagesPullingTime="5.795683468s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:46:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:34.914137851 +0000 UTC m=+8.789115107" watchObservedRunningTime="2026-10-30 16:46:35.472998873 +0000 UTC m=+9.347976113" Oct 30 16:46:35.667529 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:35.667492 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kbm7" Oct 30 16:46:35.668011 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:35.667638 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kbm7" podUID="d52d0ddc-6bd1-4dcc-9d56-84213c328055" Oct 30 16:46:35.979034 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:35.978010 2580 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 16:46:36.664495 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:36.664371 2580 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T16:46:35.978035891Z","UUID":"60bf6d1f-0c12-411f-9af0-9329d9822434","Handler":null,"Name":"","Endpoint":""} Oct 30 16:46:36.667061 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:36.667039 2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 16:46:36.667426 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:36.667069 2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 16:46:36.668692 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:36.668672 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8zqlg" Oct 30 16:46:36.669072 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:36.668791 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8zqlg" podUID="40e1c660-22c0-42e5-8511-4cbd0e170a5c" Oct 30 16:46:36.669406 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:36.669389 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsgvx" Oct 30 16:46:36.669506 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:36.669487 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsgvx" podUID="1ba7d788-14ef-4b39-b6f6-3072ab0b3200" Oct 30 16:46:36.876229 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:36.876147 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mbwdh" event={"ID":"c702aab4-18a3-4b3a-a629-7d87f15747e7","Type":"ContainerStarted","Data":"fad923386ae50f2102c73099ee09cf026067efc2fe7f64d13884478c8f9b9c2d"} Oct 30 16:46:36.879385 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:36.879356 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftqbf" event={"ID":"e3fcf8dd-4ee4-4037-9952-9c54a49c6738","Type":"ContainerStarted","Data":"d512021141aae76edac88913e85d0af81b52045b45d679b99d9730da655d9d21"} Oct 30 16:46:36.881009 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:36.880956 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n7lg2" event={"ID":"de08f19f-1712-4864-87b0-294e3d3e460d","Type":"ContainerStarted","Data":"94fa518a8298162f4ef754ae56a32c1ff828f7e4166e0ca3e32b23699652c0e2"} Oct 30 16:46:36.972238 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:36.972177 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-mbwdh" podStartSLOduration=1.637231434 podStartE2EDuration="9.972158751s" totalImagesPullingTime="8.334927317s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:46:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:36.925230154 +0000 UTC m=+10.800207393" watchObservedRunningTime="2026-10-30 16:46:36.972158751 +0000 UTC m=+10.847135991" Oct 30 16:46:36.972498 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:36.972451 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-n7lg2" podStartSLOduration=1.53980896 podStartE2EDuration="9.972441531s" totalImagesPullingTime="8.432632571s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:46:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:36.971907064 +0000 UTC m=+10.846884305" watchObservedRunningTime="2026-10-30 16:46:36.972441531 +0000 UTC m=+10.847418773" Oct 30 16:46:37.666803 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:37.666583 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kbm7" Oct 30 16:46:37.666998 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:37.666893 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kbm7" podUID="d52d0ddc-6bd1-4dcc-9d56-84213c328055" Oct 30 16:46:37.884821 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:37.884788 2580 generic.go:356] "Generic (PLEG): container finished" podID="24421640-34bc-4b15-a096-ffdaa7ae7bf4" containerID="8fabfe8d24a91c7077d80f902fc2e6a09ca9502357b6e1346fca31bdfccb3b4b" exitCode=0 Oct 30 16:46:37.885283 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:37.884868 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bll4b" event={"ID":"24421640-34bc-4b15-a096-ffdaa7ae7bf4","Type":"ContainerDied","Data":"8fabfe8d24a91c7077d80f902fc2e6a09ca9502357b6e1346fca31bdfccb3b4b"} Oct 30 16:46:37.887661 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:37.887638 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftqbf" event={"ID":"e3fcf8dd-4ee4-4037-9952-9c54a49c6738","Type":"ContainerStarted","Data":"801bb3e125afdd3761fbe5de6cfb7b8fba94cd9b965b6182f584ef001da38b52"} Oct 30 16:46:38.060159 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:38.060106 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mqhg8" Oct 30 16:46:38.060337 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:38.060278 2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 16:46:38.060824 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:38.060803 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mqhg8" Oct 30 16:46:38.107006 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:38.106936 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftqbf" podStartSLOduration=2.258912149 podStartE2EDuration="11.10691736s" totalImagesPullingTime="8.848005211s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:46:27 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:37.992453317 +0000 UTC m=+11.867430558" watchObservedRunningTime="2026-10-30 16:46:38.10691736 +0000 UTC m=+11.981894601" Oct 30 16:46:38.666985 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:38.666953 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsgvx" Oct 30 16:46:38.667142 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:38.667102 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsgvx" podUID="1ba7d788-14ef-4b39-b6f6-3072ab0b3200" Oct 30 16:46:38.667223 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:38.667157 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8zqlg" Oct 30 16:46:38.667326 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:38.667284 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8zqlg" podUID="40e1c660-22c0-42e5-8511-4cbd0e170a5c" Oct 30 16:46:38.892025 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:38.891948 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-v79c8" event={"ID":"8eab0b7c-14cc-42fa-993f-d52707e42d35","Type":"ContainerStarted","Data":"ba1209ed960bce031bb699069f1272b427829d4ac0e5e0ea428c5bf870e5e394"} Oct 30 16:46:38.893931 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:38.893903 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" event={"ID":"2434842e-d96c-4890-9b2a-2a9eb130394a","Type":"ContainerStarted","Data":"193528923b32d29fc89b6494c9d123cc0f44f05a313f24c940c42d72302bbe1e"} Oct 30 16:46:38.896179 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:38.896141 2580 generic.go:356] "Generic (PLEG): container finished" podID="24421640-34bc-4b15-a096-ffdaa7ae7bf4" containerID="ea6af0b0df933cdd7c2fc7ef5aed2fb7e5d277ab3c819b7731f07dc7c995e2a8" exitCode=0 Oct 30 16:46:38.896276 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:38.896197 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bll4b" event={"ID":"24421640-34bc-4b15-a096-ffdaa7ae7bf4","Type":"ContainerDied","Data":"ea6af0b0df933cdd7c2fc7ef5aed2fb7e5d277ab3c819b7731f07dc7c995e2a8"} Oct 30 16:46:38.944383 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:38.944325 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-v79c8" podStartSLOduration=3.377202184 podStartE2EDuration="11.944291615s" totalImagesPullingTime="8.567089431s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:46:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:38.942983007 +0000 UTC m=+12.817960274" watchObservedRunningTime="2026-10-30 16:46:38.944291615 +0000 UTC m=+12.819268856" Oct 30 16:46:39.058847 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:39.058775 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" podStartSLOduration=1.421989906 podStartE2EDuration="12.058754727s" totalImagesPullingTime="10.636764821s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:46:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:39.054835705 +0000 UTC m=+12.929812940" watchObservedRunningTime="2026-10-30 16:46:39.058754727 +0000 UTC m=+12.933731967" Oct 30 16:46:39.667211 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:39.667181 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kbm7" Oct 30 16:46:39.667337 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:39.667322 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kbm7" podUID="d52d0ddc-6bd1-4dcc-9d56-84213c328055" Oct 30 16:46:39.901226 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:39.901144 2580 generic.go:356] "Generic (PLEG): container finished" podID="24421640-34bc-4b15-a096-ffdaa7ae7bf4" containerID="102a6ca36ff2eada3328d6a51c4bd45cb531a99576a95bb9c6826e0de3ff03eb" exitCode=0 Oct 30 16:46:39.901795 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:39.901236 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bll4b" event={"ID":"24421640-34bc-4b15-a096-ffdaa7ae7bf4","Type":"ContainerDied","Data":"102a6ca36ff2eada3328d6a51c4bd45cb531a99576a95bb9c6826e0de3ff03eb"} Oct 30 16:46:40.669262 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:40.669228 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8zqlg" Oct 30 16:46:40.669473 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:40.669359 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8zqlg" podUID="40e1c660-22c0-42e5-8511-4cbd0e170a5c" Oct 30 16:46:40.669473 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:40.669438 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsgvx" Oct 30 16:46:40.669585 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:40.669566 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsgvx" podUID="1ba7d788-14ef-4b39-b6f6-3072ab0b3200" Oct 30 16:46:41.666861 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:41.666831 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kbm7" Oct 30 16:46:41.667381 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:41.666939 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kbm7" podUID="d52d0ddc-6bd1-4dcc-9d56-84213c328055" Oct 30 16:46:41.907822 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:41.907573 2580 generic.go:356] "Generic (PLEG): container finished" podID="24421640-34bc-4b15-a096-ffdaa7ae7bf4" containerID="4a81656e1e3509ac5f1e018a9df075c4da66a4b6e867c1747e636bbc1743dbce" exitCode=0 Oct 30 16:46:41.907987 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:41.907649 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bll4b" event={"ID":"24421640-34bc-4b15-a096-ffdaa7ae7bf4","Type":"ContainerDied","Data":"4a81656e1e3509ac5f1e018a9df075c4da66a4b6e867c1747e636bbc1743dbce"} Oct 30 16:46:42.668502 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:42.668452 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8zqlg" Oct 30 16:46:42.668948 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:42.668516 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsgvx" Oct 30 16:46:42.668948 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:42.668599 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8zqlg" podUID="40e1c660-22c0-42e5-8511-4cbd0e170a5c" Oct 30 16:46:42.668948 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:42.668715 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsgvx" podUID="1ba7d788-14ef-4b39-b6f6-3072ab0b3200" Oct 30 16:46:42.912943 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:42.912908 2580 generic.go:356] "Generic (PLEG): container finished" podID="24421640-34bc-4b15-a096-ffdaa7ae7bf4" containerID="bc333c854d4125cddd920936057338307f57f80f6675cd3a5b584ac2806523c0" exitCode=0 Oct 30 16:46:42.913078 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:42.912952 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bll4b" event={"ID":"24421640-34bc-4b15-a096-ffdaa7ae7bf4","Type":"ContainerDied","Data":"bc333c854d4125cddd920936057338307f57f80f6675cd3a5b584ac2806523c0"} Oct 30 16:46:43.301920 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:43.301882 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d52d0ddc-6bd1-4dcc-9d56-84213c328055-original-pull-secret\") pod \"global-pull-secret-syncer-2kbm7\" (UID: \"d52d0ddc-6bd1-4dcc-9d56-84213c328055\") " pod="kube-system/global-pull-secret-syncer-2kbm7" Oct 30 16:46:43.302099 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:43.301931 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ba7d788-14ef-4b39-b6f6-3072ab0b3200-metrics-certs\") pod \"network-metrics-daemon-rsgvx\" (UID: \"1ba7d788-14ef-4b39-b6f6-3072ab0b3200\") " pod="openshift-multus/network-metrics-daemon-rsgvx" Oct 30 16:46:43.302099 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:43.302026 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:46:43.302099 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:43.302028 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 16:46:43.302099 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:43.302095 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ba7d788-14ef-4b39-b6f6-3072ab0b3200-metrics-certs podName:1ba7d788-14ef-4b39-b6f6-3072ab0b3200 nodeName:}" failed. No retries permitted until 2026-10-30 16:46:59.302080565 +0000 UTC m=+33.177057781 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1ba7d788-14ef-4b39-b6f6-3072ab0b3200-metrics-certs") pod "network-metrics-daemon-rsgvx" (UID: "1ba7d788-14ef-4b39-b6f6-3072ab0b3200") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:46:43.302244 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:43.302109 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d52d0ddc-6bd1-4dcc-9d56-84213c328055-original-pull-secret podName:d52d0ddc-6bd1-4dcc-9d56-84213c328055 nodeName:}" failed. No retries permitted until 2026-10-30 16:46:59.302102736 +0000 UTC m=+33.177079956 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d52d0ddc-6bd1-4dcc-9d56-84213c328055-original-pull-secret") pod "global-pull-secret-syncer-2kbm7" (UID: "d52d0ddc-6bd1-4dcc-9d56-84213c328055") : object "kube-system"/"original-pull-secret" not registered Oct 30 16:46:43.402384 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:43.402351 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5st2s\" (UniqueName: \"kubernetes.io/projected/40e1c660-22c0-42e5-8511-4cbd0e170a5c-kube-api-access-5st2s\") pod \"network-check-target-8zqlg\" (UID: \"40e1c660-22c0-42e5-8511-4cbd0e170a5c\") " pod="openshift-network-diagnostics/network-check-target-8zqlg" Oct 30 16:46:43.402523 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:43.402510 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 16:46:43.402559 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:43.402528 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 16:46:43.402559 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:43.402537 2580 projected.go:196] Error preparing data for projected volume kube-api-access-5st2s for pod openshift-network-diagnostics/network-check-target-8zqlg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:46:43.402642 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:43.402591 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40e1c660-22c0-42e5-8511-4cbd0e170a5c-kube-api-access-5st2s podName:40e1c660-22c0-42e5-8511-4cbd0e170a5c nodeName:}" failed. No retries permitted until 2026-10-30 16:46:59.402575709 +0000 UTC m=+33.277552939 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-5st2s" (UniqueName: "kubernetes.io/projected/40e1c660-22c0-42e5-8511-4cbd0e170a5c-kube-api-access-5st2s") pod "network-check-target-8zqlg" (UID: "40e1c660-22c0-42e5-8511-4cbd0e170a5c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:46:43.667240 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:43.667222 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kbm7" Oct 30 16:46:43.667346 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:43.667327 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kbm7" podUID="d52d0ddc-6bd1-4dcc-9d56-84213c328055" Oct 30 16:46:43.918352 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:43.918146 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" event={"ID":"38c17ac6-93d3-4d38-83db-70d4d0fa3f30","Type":"ContainerStarted","Data":"e4d8594f58b5c3611e07953c678d2871d50495df446f392ef6e04ab6735b0811"} Oct 30 16:46:43.918781 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:43.918412 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" event={"ID":"38c17ac6-93d3-4d38-83db-70d4d0fa3f30","Type":"ContainerStarted","Data":"adb4e35efdabc4c08a4b78a924ad09f3e6d0ab311eb39cfceb78c86d7cdcf8d7"} Oct 30 16:46:43.918781 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:43.918465 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" event={"ID":"38c17ac6-93d3-4d38-83db-70d4d0fa3f30","Type":"ContainerStarted","Data":"c53263b3e90f3d91bcf920c0a913c1052a4e7106b36d37db88b8651ae53f100c"} Oct 30 16:46:43.918781 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:43.918516 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" event={"ID":"38c17ac6-93d3-4d38-83db-70d4d0fa3f30","Type":"ContainerStarted","Data":"b91fded584da87286850e83acd49252c51911b29c857fa703a28f22c72d09609"} Oct 30 16:46:43.918781 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:43.918561 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" event={"ID":"38c17ac6-93d3-4d38-83db-70d4d0fa3f30","Type":"ContainerStarted","Data":"848f613a0ad3deedd2053fa0deaf9a48647d41c5fe4d9b68f935b0bf4af4f1f0"} Oct 30 16:46:43.918781 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:43.918606 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" event={"ID":"38c17ac6-93d3-4d38-83db-70d4d0fa3f30","Type":"ContainerStarted","Data":"f46edc535291269a2a338c2bcfd8f7a1e1a0f2fc3b96e2a7c9ae21b82629e0b8"} Oct 30 16:46:43.921779 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:43.921755 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bll4b" event={"ID":"24421640-34bc-4b15-a096-ffdaa7ae7bf4","Type":"ContainerStarted","Data":"d2fc2163ddb1af98d3b7f8c2cc6ee200866e226025286cfe38aec9ba1c9a0f5e"} Oct 30 16:46:44.667558 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:44.667522 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8zqlg" Oct 30 16:46:44.667558 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:44.667551 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsgvx" Oct 30 16:46:44.667735 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:44.667649 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8zqlg" podUID="40e1c660-22c0-42e5-8511-4cbd0e170a5c" Oct 30 16:46:44.667774 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:44.667759 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsgvx" podUID="1ba7d788-14ef-4b39-b6f6-3072ab0b3200" Oct 30 16:46:45.666699 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:45.666668 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kbm7" Oct 30 16:46:45.667051 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:45.666770 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kbm7" podUID="d52d0ddc-6bd1-4dcc-9d56-84213c328055" Oct 30 16:46:46.667464 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:46.667437 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsgvx" Oct 30 16:46:46.667865 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:46.667534 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8zqlg" Oct 30 16:46:46.667865 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:46.667535 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsgvx" podUID="1ba7d788-14ef-4b39-b6f6-3072ab0b3200" Oct 30 16:46:46.667865 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:46.667614 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8zqlg" podUID="40e1c660-22c0-42e5-8511-4cbd0e170a5c" Oct 30 16:46:46.930476 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:46.930384 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" event={"ID":"38c17ac6-93d3-4d38-83db-70d4d0fa3f30","Type":"ContainerStarted","Data":"66401940e25d11d1d13c8934c9cb9ec36138de7c1f1bad277c27df7d51616b6d"} Oct 30 16:46:47.667224 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:47.667196 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kbm7" Oct 30 16:46:47.667433 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:47.667324 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kbm7" podUID="d52d0ddc-6bd1-4dcc-9d56-84213c328055" Oct 30 16:46:48.666595 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:48.666567 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8zqlg" Oct 30 16:46:48.667391 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:48.666694 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsgvx" Oct 30 16:46:48.667391 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:48.666696 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8zqlg" podUID="40e1c660-22c0-42e5-8511-4cbd0e170a5c" Oct 30 16:46:48.667391 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:48.666771 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsgvx" podUID="1ba7d788-14ef-4b39-b6f6-3072ab0b3200" Oct 30 16:46:48.938318 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:48.938221 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" event={"ID":"38c17ac6-93d3-4d38-83db-70d4d0fa3f30","Type":"ContainerStarted","Data":"cf049565f12349652c6d6b5478470b42dbb054f179e62922f1f67e2850435bda"} Oct 30 16:46:48.938481 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:48.938452 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:48.938549 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:48.938515 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:48.953726 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:48.953702 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:49.032090 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:49.032041 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bll4b" podStartSLOduration=11.86730511 podStartE2EDuration="22.032012246s" totalImagesPullingTime="10.164707136s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:46:27 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:43.977061846 +0000 UTC m=+17.852039085" watchObservedRunningTime="2026-10-30 16:46:49.032012246 +0000 UTC m=+22.906989485" Oct 30 16:46:49.113556 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:49.113508 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" podStartSLOduration=6.54721962 podStartE2EDuration="22.113494633s" totalImagesPullingTime="15.566275013s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:46:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:49.046628392 +0000 UTC m=+22.921605631" watchObservedRunningTime="2026-10-30 16:46:49.113494633 +0000 UTC m=+22.988471872" Oct 30 16:46:49.667581 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:49.667543 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kbm7" Oct 30 16:46:49.668065 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:49.667680 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kbm7" podUID="d52d0ddc-6bd1-4dcc-9d56-84213c328055" Oct 30 16:46:49.941561 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:49.941237 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:49.956710 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:49.956687 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:46:50.186924 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:50.186897 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8zqlg"] Oct 30 16:46:50.187156 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:50.187027 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8zqlg" Oct 30 16:46:50.187156 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:50.187098 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8zqlg" podUID="40e1c660-22c0-42e5-8511-4cbd0e170a5c" Oct 30 16:46:50.187538 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:50.187519 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rsgvx"] Oct 30 16:46:50.187617 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:50.187607 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsgvx" Oct 30 16:46:50.187747 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:50.187729 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsgvx" podUID="1ba7d788-14ef-4b39-b6f6-3072ab0b3200" Oct 30 16:46:50.189009 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:50.188985 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2kbm7"] Oct 30 16:46:50.189126 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:50.189086 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kbm7" Oct 30 16:46:50.189193 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:50.189175 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kbm7" podUID="d52d0ddc-6bd1-4dcc-9d56-84213c328055" Oct 30 16:46:51.666543 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:51.666513 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kbm7" Oct 30 16:46:51.666927 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:51.666614 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8zqlg" Oct 30 16:46:51.666927 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:51.666639 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsgvx" Oct 30 16:46:51.666927 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:51.666636 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kbm7" podUID="d52d0ddc-6bd1-4dcc-9d56-84213c328055" Oct 30 16:46:51.666927 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:51.666714 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8zqlg" podUID="40e1c660-22c0-42e5-8511-4cbd0e170a5c" Oct 30 16:46:51.666927 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:51.666801 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsgvx" podUID="1ba7d788-14ef-4b39-b6f6-3072ab0b3200" Oct 30 16:46:53.667344 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:53.667298 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kbm7" Oct 30 16:46:53.667344 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:53.667338 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8zqlg" Oct 30 16:46:53.667344 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:53.667346 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsgvx" Oct 30 16:46:53.667815 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:53.667428 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kbm7" podUID="d52d0ddc-6bd1-4dcc-9d56-84213c328055" Oct 30 16:46:53.667815 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:53.667548 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8zqlg" podUID="40e1c660-22c0-42e5-8511-4cbd0e170a5c" Oct 30 16:46:53.667815 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:53.667633 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsgvx" podUID="1ba7d788-14ef-4b39-b6f6-3072ab0b3200" Oct 30 16:46:55.666662 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:55.666624 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kbm7" Oct 30 16:46:55.666662 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:55.666663 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8zqlg" Oct 30 16:46:55.667220 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:55.666627 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsgvx" Oct 30 16:46:55.667220 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:55.666746 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kbm7" podUID="d52d0ddc-6bd1-4dcc-9d56-84213c328055" Oct 30 16:46:55.667220 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:55.666823 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsgvx" podUID="1ba7d788-14ef-4b39-b6f6-3072ab0b3200" Oct 30 16:46:55.667220 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:55.666890 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8zqlg" podUID="40e1c660-22c0-42e5-8511-4cbd0e170a5c" Oct 30 16:46:55.871423 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:55.871396 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-218.ec2.internal" event="NodeReady" Oct 30 16:46:55.871563 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:55.871533 2580 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 16:46:55.941155 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:55.941079 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wc9zw"] Oct 30 16:46:55.948170 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:55.948151 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-cqjj8"] Oct 30 16:46:55.948284 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:55.948266 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wc9zw" Oct 30 16:46:55.950324 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:55.950294 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cqjj8" Oct 30 16:46:55.969998 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:55.969976 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cqjj8"] Oct 30 16:46:55.971597 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:55.970642 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 16:46:55.971597 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:55.970918 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 16:46:55.971597 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:55.970944 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 16:46:55.971597 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:55.970993 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 16:46:55.971597 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:55.971035 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ffzh4\"" Oct 30 16:46:55.971597 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:55.971060 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-f9wpv\"" Oct 30 16:46:55.971597 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:55.971552 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wc9zw"] Oct 30 16:46:55.984367 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:55.984349 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 16:46:56.093489 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:56.093459 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3-metrics-tls\") pod \"dns-default-wc9zw\" (UID: \"bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3\") " pod="openshift-dns/dns-default-wc9zw" Oct 30 16:46:56.093582 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:56.093494 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3-tmp-dir\") pod \"dns-default-wc9zw\" (UID: \"bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3\") " pod="openshift-dns/dns-default-wc9zw" Oct 30 16:46:56.093637 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:56.093588 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3-config-volume\") pod \"dns-default-wc9zw\" (UID: \"bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3\") " pod="openshift-dns/dns-default-wc9zw" Oct 30 16:46:56.093637 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:56.093612 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm26j\" (UniqueName: \"kubernetes.io/projected/e275e41d-9573-47e2-b923-173dfb4361a7-kube-api-access-vm26j\") pod \"ingress-canary-cqjj8\" (UID: \"e275e41d-9573-47e2-b923-173dfb4361a7\") " pod="openshift-ingress-canary/ingress-canary-cqjj8" Oct 30 16:46:56.093698 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:56.093638 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e275e41d-9573-47e2-b923-173dfb4361a7-cert\") pod \"ingress-canary-cqjj8\" (UID: \"e275e41d-9573-47e2-b923-173dfb4361a7\") " pod="openshift-ingress-canary/ingress-canary-cqjj8" Oct 30 16:46:56.093698 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:56.093665 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2whld\" (UniqueName: \"kubernetes.io/projected/bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3-kube-api-access-2whld\") pod \"dns-default-wc9zw\" (UID: \"bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3\") " pod="openshift-dns/dns-default-wc9zw" Oct 30 16:46:56.194736 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:56.194663 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3-metrics-tls\") pod \"dns-default-wc9zw\" (UID: \"bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3\") " pod="openshift-dns/dns-default-wc9zw" Oct 30 16:46:56.194736 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:56.194701 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3-tmp-dir\") pod \"dns-default-wc9zw\" (UID: \"bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3\") " pod="openshift-dns/dns-default-wc9zw" Oct 30 16:46:56.194896 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:56.194748 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3-config-volume\") pod \"dns-default-wc9zw\" (UID: \"bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3\") " pod="openshift-dns/dns-default-wc9zw" Oct 30 16:46:56.194896 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:56.194770 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm26j\" (UniqueName: \"kubernetes.io/projected/e275e41d-9573-47e2-b923-173dfb4361a7-kube-api-access-vm26j\") pod \"ingress-canary-cqjj8\" (UID: \"e275e41d-9573-47e2-b923-173dfb4361a7\") " pod="openshift-ingress-canary/ingress-canary-cqjj8" Oct 30 16:46:56.194896 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:56.194806 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e275e41d-9573-47e2-b923-173dfb4361a7-cert\") pod \"ingress-canary-cqjj8\" (UID: \"e275e41d-9573-47e2-b923-173dfb4361a7\") " pod="openshift-ingress-canary/ingress-canary-cqjj8" Oct 30 16:46:56.194896 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:56.194833 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 16:46:56.194896 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:56.194894 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3-metrics-tls podName:bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3 nodeName:}" failed. No retries permitted until 2026-10-30 16:46:56.694876276 +0000 UTC m=+30.569853509 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3-metrics-tls") pod "dns-default-wc9zw" (UID: "bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3") : secret "dns-default-metrics-tls" not found Oct 30 16:46:56.195094 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:56.194956 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2whld\" (UniqueName: \"kubernetes.io/projected/bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3-kube-api-access-2whld\") pod \"dns-default-wc9zw\" (UID: \"bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3\") " pod="openshift-dns/dns-default-wc9zw" Oct 30 16:46:56.195152 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:56.195135 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3-tmp-dir\") pod \"dns-default-wc9zw\" (UID: \"bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3\") " pod="openshift-dns/dns-default-wc9zw" Oct 30 16:46:56.195500 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:56.195480 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3-config-volume\") pod \"dns-default-wc9zw\" (UID: \"bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3\") " pod="openshift-dns/dns-default-wc9zw" Oct 30 16:46:56.198465 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:56.198444 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e275e41d-9573-47e2-b923-173dfb4361a7-cert\") pod \"ingress-canary-cqjj8\" (UID: \"e275e41d-9573-47e2-b923-173dfb4361a7\") " pod="openshift-ingress-canary/ingress-canary-cqjj8" Oct 30 16:46:56.225260 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:56.225236 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2whld\" (UniqueName: \"kubernetes.io/projected/bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3-kube-api-access-2whld\") pod \"dns-default-wc9zw\" (UID: \"bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3\") " pod="openshift-dns/dns-default-wc9zw" Oct 30 16:46:56.229059 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:56.229040 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm26j\" (UniqueName: \"kubernetes.io/projected/e275e41d-9573-47e2-b923-173dfb4361a7-kube-api-access-vm26j\") pod \"ingress-canary-cqjj8\" (UID: \"e275e41d-9573-47e2-b923-173dfb4361a7\") " pod="openshift-ingress-canary/ingress-canary-cqjj8" Oct 30 16:46:56.267062 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:56.267031 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cqjj8" Oct 30 16:46:56.580227 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:56.580202 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cqjj8"] Oct 30 16:46:56.581714 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:56.581689 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode275e41d_9573_47e2_b923_173dfb4361a7.slice/crio-d4ec855f9a431818213116d72030492004afb27e6e9a5f1fae58b7223a06cfe3 WatchSource:0}: Error finding container d4ec855f9a431818213116d72030492004afb27e6e9a5f1fae58b7223a06cfe3: Status 404 returned error can't find the container with id d4ec855f9a431818213116d72030492004afb27e6e9a5f1fae58b7223a06cfe3 Oct 30 16:46:56.697999 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:56.697968 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3-metrics-tls\") pod \"dns-default-wc9zw\" (UID: \"bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3\") " pod="openshift-dns/dns-default-wc9zw" Oct 30 16:46:56.698414 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:56.698107 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 16:46:56.698414 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:56.698164 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3-metrics-tls podName:bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3 nodeName:}" failed. No retries permitted until 2026-10-30 16:46:57.698149726 +0000 UTC m=+31.573126958 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3-metrics-tls") pod "dns-default-wc9zw" (UID: "bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3") : secret "dns-default-metrics-tls" not found Oct 30 16:46:56.957188 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:56.957152 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cqjj8" event={"ID":"e275e41d-9573-47e2-b923-173dfb4361a7","Type":"ContainerStarted","Data":"d4ec855f9a431818213116d72030492004afb27e6e9a5f1fae58b7223a06cfe3"} Oct 30 16:46:57.667401 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:57.667367 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8zqlg" Oct 30 16:46:57.667600 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:57.667395 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsgvx" Oct 30 16:46:57.667600 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:57.667544 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kbm7" Oct 30 16:46:57.687023 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:57.687002 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 16:46:57.687128 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:57.687102 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-8568q\"" Oct 30 16:46:57.701900 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:57.701876 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9bdrw\"" Oct 30 16:46:57.702250 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:57.701958 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 16:46:57.702250 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:57.702132 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-szhrj\"" Oct 30 16:46:57.702250 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:57.702198 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 16:46:57.702250 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:57.702210 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 16:46:57.706431 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:57.706409 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3-metrics-tls\") pod \"dns-default-wc9zw\" (UID: \"bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3\") " pod="openshift-dns/dns-default-wc9zw" Oct 30 16:46:57.706528 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:57.706514 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 16:46:57.706570 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:57.706563 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3-metrics-tls podName:bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3 nodeName:}" failed. No retries permitted until 2026-10-30 16:46:59.706549634 +0000 UTC m=+33.581526855 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3-metrics-tls") pod "dns-default-wc9zw" (UID: "bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3") : secret "dns-default-metrics-tls" not found Oct 30 16:46:58.320477 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:58.320433 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4wjcq"] Oct 30 16:46:58.338968 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:58.338940 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4wjcq" Oct 30 16:46:58.361650 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:58.361614 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4wjcq"] Oct 30 16:46:58.375417 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:58.375379 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 16:46:58.375535 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:58.375446 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 16:46:58.375535 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:58.375520 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 16:46:58.411085 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:58.411055 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a21d39c5-3813-4433-9307-855276f1a66e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4wjcq\" (UID: \"a21d39c5-3813-4433-9307-855276f1a66e\") " pod="openshift-insights/insights-runtime-extractor-4wjcq" Oct 30 16:46:58.411247 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:58.411105 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a21d39c5-3813-4433-9307-855276f1a66e-data-volume\") pod \"insights-runtime-extractor-4wjcq\" (UID: \"a21d39c5-3813-4433-9307-855276f1a66e\") " pod="openshift-insights/insights-runtime-extractor-4wjcq" Oct 30 16:46:58.411247 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:58.411132 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a21d39c5-3813-4433-9307-855276f1a66e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4wjcq\" (UID: \"a21d39c5-3813-4433-9307-855276f1a66e\") " pod="openshift-insights/insights-runtime-extractor-4wjcq" Oct 30 16:46:58.411350 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:58.411263 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a21d39c5-3813-4433-9307-855276f1a66e-crio-socket\") pod \"insights-runtime-extractor-4wjcq\" (UID: \"a21d39c5-3813-4433-9307-855276f1a66e\") " pod="openshift-insights/insights-runtime-extractor-4wjcq" Oct 30 16:46:58.411350 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:58.411335 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pjzv\" (UniqueName: \"kubernetes.io/projected/a21d39c5-3813-4433-9307-855276f1a66e-kube-api-access-4pjzv\") pod \"insights-runtime-extractor-4wjcq\" (UID: \"a21d39c5-3813-4433-9307-855276f1a66e\") " pod="openshift-insights/insights-runtime-extractor-4wjcq" Oct 30 16:46:58.420431 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:58.420406 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-f4j47-pgf7r"] Oct 30 16:46:58.429379 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:58.429356 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 16:46:58.429490 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:58.429442 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7stlv\"" Oct 30 16:46:58.443118 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:58.442789 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-f4j47-pgf7r" Oct 30 16:46:58.444561 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:58.444538 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-f4j47-pgf7r"] Oct 30 16:46:58.466694 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:58.466666 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-nwhz8\"" Oct 30 16:46:58.468111 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:58.468091 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 16:46:58.485176 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:58.485147 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 16:46:58.495280 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:58.495247 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-zm2rj"] Oct 30 16:46:58.511941 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:58.511912 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pjzv\" (UniqueName: \"kubernetes.io/projected/a21d39c5-3813-4433-9307-855276f1a66e-kube-api-access-4pjzv\") pod \"insights-runtime-extractor-4wjcq\" (UID: \"a21d39c5-3813-4433-9307-855276f1a66e\") " pod="openshift-insights/insights-runtime-extractor-4wjcq" Oct 30 16:46:58.511941 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:58.511945 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/252a7edc-651f-4c08-8fd7-f46ec29737c4-archives-path\") pod \"periodic-gathering-f4j47-pgf7r\" (UID: \"252a7edc-651f-4c08-8fd7-f46ec29737c4\") " pod="openshift-insights/periodic-gathering-f4j47-pgf7r" Oct 30 16:46:58.512132 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:58.511978 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a21d39c5-3813-4433-9307-855276f1a66e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4wjcq\" (UID: \"a21d39c5-3813-4433-9307-855276f1a66e\") " pod="openshift-insights/insights-runtime-extractor-4wjcq" Oct 30 16:46:58.512132 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:58.512079 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 16:46:58.512221 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:58.512137 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a21d39c5-3813-4433-9307-855276f1a66e-insights-runtime-extractor-tls podName:a21d39c5-3813-4433-9307-855276f1a66e nodeName:}" failed. No retries permitted until 2026-10-30 16:46:59.012115802 +0000 UTC m=+32.887093037 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a21d39c5-3813-4433-9307-855276f1a66e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4wjcq" (UID: "a21d39c5-3813-4433-9307-855276f1a66e") : secret "insights-runtime-extractor-tls" not found Oct 30 16:46:58.512321 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:58.512280 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a21d39c5-3813-4433-9307-855276f1a66e-data-volume\") pod \"insights-runtime-extractor-4wjcq\" (UID: \"a21d39c5-3813-4433-9307-855276f1a66e\") " pod="openshift-insights/insights-runtime-extractor-4wjcq" Oct 30 16:46:58.512387 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:58.512330 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a21d39c5-3813-4433-9307-855276f1a66e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4wjcq\" (UID: \"a21d39c5-3813-4433-9307-855276f1a66e\") " pod="openshift-insights/insights-runtime-extractor-4wjcq" Oct 30 16:46:58.512387 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:58.512355 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gxc2\" (UniqueName: \"kubernetes.io/projected/252a7edc-651f-4c08-8fd7-f46ec29737c4-kube-api-access-2gxc2\") pod \"periodic-gathering-f4j47-pgf7r\" (UID: \"252a7edc-651f-4c08-8fd7-f46ec29737c4\") " pod="openshift-insights/periodic-gathering-f4j47-pgf7r" Oct 30 16:46:58.512482 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:58.512436 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 16:46:58.512520 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:58.512505 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a21d39c5-3813-4433-9307-855276f1a66e-kube-rbac-proxy-cm podName:a21d39c5-3813-4433-9307-855276f1a66e nodeName:}" failed. No retries permitted until 2026-10-30 16:46:59.012489257 +0000 UTC m=+32.887466492 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a21d39c5-3813-4433-9307-855276f1a66e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4wjcq" (UID: "a21d39c5-3813-4433-9307-855276f1a66e") : configmap "kube-rbac-proxy" not found Oct 30 16:46:58.512557 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:58.512539 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a21d39c5-3813-4433-9307-855276f1a66e-crio-socket\") pod \"insights-runtime-extractor-4wjcq\" (UID: \"a21d39c5-3813-4433-9307-855276f1a66e\") " pod="openshift-insights/insights-runtime-extractor-4wjcq" Oct 30 16:46:58.512594 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:58.512569 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/252a7edc-651f-4c08-8fd7-f46ec29737c4-service-ca-bundle\") pod \"periodic-gathering-f4j47-pgf7r\" (UID: \"252a7edc-651f-4c08-8fd7-f46ec29737c4\") " pod="openshift-insights/periodic-gathering-f4j47-pgf7r" Oct 30 16:46:58.512640 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:58.512594 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/252a7edc-651f-4c08-8fd7-f46ec29737c4-trusted-ca-bundle\") pod \"periodic-gathering-f4j47-pgf7r\" (UID: \"252a7edc-651f-4c08-8fd7-f46ec29737c4\") " pod="openshift-insights/periodic-gathering-f4j47-pgf7r" Oct 30 16:46:58.512705 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:58.512687 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a21d39c5-3813-4433-9307-855276f1a66e-data-volume\") pod \"insights-runtime-extractor-4wjcq\" (UID: \"a21d39c5-3813-4433-9307-855276f1a66e\") " pod="openshift-insights/insights-runtime-extractor-4wjcq" Oct 30 16:46:58.512850 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:58.512834 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a21d39c5-3813-4433-9307-855276f1a66e-crio-socket\") pod \"insights-runtime-extractor-4wjcq\" (UID: \"a21d39c5-3813-4433-9307-855276f1a66e\") " pod="openshift-insights/insights-runtime-extractor-4wjcq" Oct 30 16:46:58.517548 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:58.517525 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zm2rj" Oct 30 16:46:58.522332 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:58.522285 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-zm2rj"] Oct 30 16:46:58.542886 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:58.542854 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pjzv\" (UniqueName: \"kubernetes.io/projected/a21d39c5-3813-4433-9307-855276f1a66e-kube-api-access-4pjzv\") pod \"insights-runtime-extractor-4wjcq\" (UID: \"a21d39c5-3813-4433-9307-855276f1a66e\") " pod="openshift-insights/insights-runtime-extractor-4wjcq" Oct 30 16:46:58.570653 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:58.570589 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-vmkvz\"" Oct 30 16:46:58.572586 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:58.572562 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 16:46:58.572695 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:58.572638 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 16:46:58.613767 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:58.613736 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gxc2\" (UniqueName: \"kubernetes.io/projected/252a7edc-651f-4c08-8fd7-f46ec29737c4-kube-api-access-2gxc2\") pod \"periodic-gathering-f4j47-pgf7r\" (UID: \"252a7edc-651f-4c08-8fd7-f46ec29737c4\") " pod="openshift-insights/periodic-gathering-f4j47-pgf7r" Oct 30 16:46:58.613935 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:58.613781 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/252a7edc-651f-4c08-8fd7-f46ec29737c4-service-ca-bundle\") pod \"periodic-gathering-f4j47-pgf7r\" (UID: \"252a7edc-651f-4c08-8fd7-f46ec29737c4\") " pod="openshift-insights/periodic-gathering-f4j47-pgf7r" Oct 30 16:46:58.613935 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:58.613800 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/252a7edc-651f-4c08-8fd7-f46ec29737c4-trusted-ca-bundle\") pod \"periodic-gathering-f4j47-pgf7r\" (UID: \"252a7edc-651f-4c08-8fd7-f46ec29737c4\") " pod="openshift-insights/periodic-gathering-f4j47-pgf7r" Oct 30 16:46:58.616338 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:58.614351 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dnnp\" (UniqueName: \"kubernetes.io/projected/cd83ecbc-117f-4890-bf7a-cab9282d4484-kube-api-access-7dnnp\") pod \"migrator-66df5d496b-zm2rj\" (UID: \"cd83ecbc-117f-4890-bf7a-cab9282d4484\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zm2rj" Oct 30 16:46:58.616338 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:58.614515 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/252a7edc-651f-4c08-8fd7-f46ec29737c4-archives-path\") pod \"periodic-gathering-f4j47-pgf7r\" (UID: \"252a7edc-651f-4c08-8fd7-f46ec29737c4\") " pod="openshift-insights/periodic-gathering-f4j47-pgf7r" Oct 30 16:46:58.616338 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:58.614945 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/252a7edc-651f-4c08-8fd7-f46ec29737c4-archives-path\") pod \"periodic-gathering-f4j47-pgf7r\" (UID: \"252a7edc-651f-4c08-8fd7-f46ec29737c4\") " pod="openshift-insights/periodic-gathering-f4j47-pgf7r" Oct 30 16:46:58.616338 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:58.615141 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/252a7edc-651f-4c08-8fd7-f46ec29737c4-service-ca-bundle\") pod \"periodic-gathering-f4j47-pgf7r\" (UID: \"252a7edc-651f-4c08-8fd7-f46ec29737c4\") " pod="openshift-insights/periodic-gathering-f4j47-pgf7r" Oct 30 16:46:58.618063 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:58.618042 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/252a7edc-651f-4c08-8fd7-f46ec29737c4-trusted-ca-bundle\") pod \"periodic-gathering-f4j47-pgf7r\" (UID: \"252a7edc-651f-4c08-8fd7-f46ec29737c4\") " pod="openshift-insights/periodic-gathering-f4j47-pgf7r" Oct 30 16:46:58.641697 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:58.641668 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gxc2\" (UniqueName: \"kubernetes.io/projected/252a7edc-651f-4c08-8fd7-f46ec29737c4-kube-api-access-2gxc2\") pod \"periodic-gathering-f4j47-pgf7r\" (UID: \"252a7edc-651f-4c08-8fd7-f46ec29737c4\") " pod="openshift-insights/periodic-gathering-f4j47-pgf7r" Oct 30 16:46:58.715959 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:58.715922 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dnnp\" (UniqueName: \"kubernetes.io/projected/cd83ecbc-117f-4890-bf7a-cab9282d4484-kube-api-access-7dnnp\") pod \"migrator-66df5d496b-zm2rj\" (UID: \"cd83ecbc-117f-4890-bf7a-cab9282d4484\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zm2rj" Oct 30 16:46:58.755920 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:58.755890 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dnnp\" (UniqueName: \"kubernetes.io/projected/cd83ecbc-117f-4890-bf7a-cab9282d4484-kube-api-access-7dnnp\") pod \"migrator-66df5d496b-zm2rj\" (UID: \"cd83ecbc-117f-4890-bf7a-cab9282d4484\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zm2rj" Oct 30 16:46:58.767820 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:58.767786 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-f4j47-pgf7r" Oct 30 16:46:58.846958 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:58.846650 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zm2rj" Oct 30 16:46:59.018853 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:59.018818 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a21d39c5-3813-4433-9307-855276f1a66e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4wjcq\" (UID: \"a21d39c5-3813-4433-9307-855276f1a66e\") " pod="openshift-insights/insights-runtime-extractor-4wjcq" Oct 30 16:46:59.019008 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:59.018863 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a21d39c5-3813-4433-9307-855276f1a66e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4wjcq\" (UID: \"a21d39c5-3813-4433-9307-855276f1a66e\") " pod="openshift-insights/insights-runtime-extractor-4wjcq" Oct 30 16:46:59.019008 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:59.018931 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 16:46:59.019008 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:59.018969 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 16:46:59.019100 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:59.018981 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a21d39c5-3813-4433-9307-855276f1a66e-kube-rbac-proxy-cm podName:a21d39c5-3813-4433-9307-855276f1a66e nodeName:}" failed. No retries permitted until 2026-10-30 16:47:00.01896693 +0000 UTC m=+33.893944160 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a21d39c5-3813-4433-9307-855276f1a66e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4wjcq" (UID: "a21d39c5-3813-4433-9307-855276f1a66e") : configmap "kube-rbac-proxy" not found Oct 30 16:46:59.019100 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:59.019076 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a21d39c5-3813-4433-9307-855276f1a66e-insights-runtime-extractor-tls podName:a21d39c5-3813-4433-9307-855276f1a66e nodeName:}" failed. No retries permitted until 2026-10-30 16:47:00.019062981 +0000 UTC m=+33.894040202 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a21d39c5-3813-4433-9307-855276f1a66e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4wjcq" (UID: "a21d39c5-3813-4433-9307-855276f1a66e") : secret "insights-runtime-extractor-tls" not found Oct 30 16:46:59.094961 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:59.094933 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-f4j47-pgf7r"] Oct 30 16:46:59.139818 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:59.139789 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod252a7edc_651f_4c08_8fd7_f46ec29737c4.slice/crio-6fc7bce1f36d0e8e13dc09c5275b70fe8ea2790f94977fde82f4fa803951bcfc WatchSource:0}: Error finding container 6fc7bce1f36d0e8e13dc09c5275b70fe8ea2790f94977fde82f4fa803951bcfc: Status 404 returned error can't find the container with id 6fc7bce1f36d0e8e13dc09c5275b70fe8ea2790f94977fde82f4fa803951bcfc Oct 30 16:46:59.157290 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:59.157263 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-zm2rj"] Oct 30 16:46:59.158529 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:59.158506 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd83ecbc_117f_4890_bf7a_cab9282d4484.slice/crio-a737e28cca11d4405dfef3adba95876fa07f4117d846e29ac7515304d0e46c62 WatchSource:0}: Error finding container a737e28cca11d4405dfef3adba95876fa07f4117d846e29ac7515304d0e46c62: Status 404 returned error can't find the container with id a737e28cca11d4405dfef3adba95876fa07f4117d846e29ac7515304d0e46c62 Oct 30 16:46:59.271014 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:59.270984 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-hng8x"] Oct 30 16:46:59.284588 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:59.284561 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-hng8x" Oct 30 16:46:59.294507 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:59.294474 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-hng8x"] Oct 30 16:46:59.306356 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:59.306326 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-l499l\"" Oct 30 16:46:59.306356 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:59.306330 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 16:46:59.306845 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:59.306762 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 16:46:59.306845 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:59.306792 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 16:46:59.306845 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:59.306823 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 16:46:59.321449 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:59.321426 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ba7d788-14ef-4b39-b6f6-3072ab0b3200-metrics-certs\") pod \"network-metrics-daemon-rsgvx\" (UID: \"1ba7d788-14ef-4b39-b6f6-3072ab0b3200\") " pod="openshift-multus/network-metrics-daemon-rsgvx" Oct 30 16:46:59.321577 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:59.321489 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d52d0ddc-6bd1-4dcc-9d56-84213c328055-original-pull-secret\") pod \"global-pull-secret-syncer-2kbm7\" (UID: \"d52d0ddc-6bd1-4dcc-9d56-84213c328055\") " pod="kube-system/global-pull-secret-syncer-2kbm7" Oct 30 16:46:59.321615 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:59.321583 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 16:46:59.321666 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:59.321654 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ba7d788-14ef-4b39-b6f6-3072ab0b3200-metrics-certs podName:1ba7d788-14ef-4b39-b6f6-3072ab0b3200 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:31.321637771 +0000 UTC m=+65.196614988 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1ba7d788-14ef-4b39-b6f6-3072ab0b3200-metrics-certs") pod "network-metrics-daemon-rsgvx" (UID: "1ba7d788-14ef-4b39-b6f6-3072ab0b3200") : secret "metrics-daemon-secret" not found Oct 30 16:46:59.323773 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:59.323751 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d52d0ddc-6bd1-4dcc-9d56-84213c328055-original-pull-secret\") pod \"global-pull-secret-syncer-2kbm7\" (UID: \"d52d0ddc-6bd1-4dcc-9d56-84213c328055\") " pod="kube-system/global-pull-secret-syncer-2kbm7" Oct 30 16:46:59.422197 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:59.422120 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs8qj\" (UniqueName: \"kubernetes.io/projected/bf581b63-6285-4ac9-aa45-cd8a367478f2-kube-api-access-hs8qj\") pod \"service-ca-5fffc97bdf-hng8x\" (UID: \"bf581b63-6285-4ac9-aa45-cd8a367478f2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hng8x" Oct 30 16:46:59.422197 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:59.422170 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/bf581b63-6285-4ac9-aa45-cd8a367478f2-signing-key\") pod \"service-ca-5fffc97bdf-hng8x\" (UID: \"bf581b63-6285-4ac9-aa45-cd8a367478f2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hng8x" Oct 30 16:46:59.422447 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:59.422273 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/bf581b63-6285-4ac9-aa45-cd8a367478f2-signing-cabundle\") pod \"service-ca-5fffc97bdf-hng8x\" (UID: \"bf581b63-6285-4ac9-aa45-cd8a367478f2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hng8x" Oct 30 16:46:59.422447 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:59.422347 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5st2s\" (UniqueName: \"kubernetes.io/projected/40e1c660-22c0-42e5-8511-4cbd0e170a5c-kube-api-access-5st2s\") pod \"network-check-target-8zqlg\" (UID: \"40e1c660-22c0-42e5-8511-4cbd0e170a5c\") " pod="openshift-network-diagnostics/network-check-target-8zqlg" Oct 30 16:46:59.424878 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:59.424859 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5st2s\" (UniqueName: \"kubernetes.io/projected/40e1c660-22c0-42e5-8511-4cbd0e170a5c-kube-api-access-5st2s\") pod \"network-check-target-8zqlg\" (UID: \"40e1c660-22c0-42e5-8511-4cbd0e170a5c\") " pod="openshift-network-diagnostics/network-check-target-8zqlg" Oct 30 16:46:59.507470 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:59.507429 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8zqlg" Oct 30 16:46:59.522685 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:59.522654 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/bf581b63-6285-4ac9-aa45-cd8a367478f2-signing-cabundle\") pod \"service-ca-5fffc97bdf-hng8x\" (UID: \"bf581b63-6285-4ac9-aa45-cd8a367478f2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hng8x" Oct 30 16:46:59.522823 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:59.522720 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs8qj\" (UniqueName: \"kubernetes.io/projected/bf581b63-6285-4ac9-aa45-cd8a367478f2-kube-api-access-hs8qj\") pod \"service-ca-5fffc97bdf-hng8x\" (UID: \"bf581b63-6285-4ac9-aa45-cd8a367478f2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hng8x" Oct 30 16:46:59.522823 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:59.522754 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/bf581b63-6285-4ac9-aa45-cd8a367478f2-signing-key\") pod \"service-ca-5fffc97bdf-hng8x\" (UID: \"bf581b63-6285-4ac9-aa45-cd8a367478f2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hng8x" Oct 30 16:46:59.523466 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:59.523445 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/bf581b63-6285-4ac9-aa45-cd8a367478f2-signing-cabundle\") pod \"service-ca-5fffc97bdf-hng8x\" (UID: \"bf581b63-6285-4ac9-aa45-cd8a367478f2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hng8x" Oct 30 16:46:59.525374 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:59.525351 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/bf581b63-6285-4ac9-aa45-cd8a367478f2-signing-key\") pod \"service-ca-5fffc97bdf-hng8x\" (UID: \"bf581b63-6285-4ac9-aa45-cd8a367478f2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hng8x" Oct 30 16:46:59.533230 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:59.533211 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kbm7" Oct 30 16:46:59.549676 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:59.549654 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs8qj\" (UniqueName: \"kubernetes.io/projected/bf581b63-6285-4ac9-aa45-cd8a367478f2-kube-api-access-hs8qj\") pod \"service-ca-5fffc97bdf-hng8x\" (UID: \"bf581b63-6285-4ac9-aa45-cd8a367478f2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hng8x" Oct 30 16:46:59.611576 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:59.611357 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-hng8x" Oct 30 16:46:59.725019 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:59.724984 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3-metrics-tls\") pod \"dns-default-wc9zw\" (UID: \"bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3\") " pod="openshift-dns/dns-default-wc9zw" Oct 30 16:46:59.725502 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:59.725156 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 16:46:59.725502 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:46:59.725234 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3-metrics-tls podName:bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:03.725214021 +0000 UTC m=+37.600191241 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3-metrics-tls") pod "dns-default-wc9zw" (UID: "bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3") : secret "dns-default-metrics-tls" not found Oct 30 16:46:59.852574 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:59.852542 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8zqlg"] Oct 30 16:46:59.853906 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:59.853862 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e1c660_22c0_42e5_8511_4cbd0e170a5c.slice/crio-0423717bb045b25f4e574b686e6564701d7df72979a3bfff1359674754052774 WatchSource:0}: Error finding container 0423717bb045b25f4e574b686e6564701d7df72979a3bfff1359674754052774: Status 404 returned error can't find the container with id 0423717bb045b25f4e574b686e6564701d7df72979a3bfff1359674754052774 Oct 30 16:46:59.883644 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:59.883608 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2kbm7"] Oct 30 16:46:59.893246 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:59.893215 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd52d0ddc_6bd1_4dcc_9d56_84213c328055.slice/crio-bc75e72cf1463bccb83e5933bc73e9cc0b3b99aed55f7cd0fe3371a4924f6271 WatchSource:0}: Error finding container bc75e72cf1463bccb83e5933bc73e9cc0b3b99aed55f7cd0fe3371a4924f6271: Status 404 returned error can't find the container with id bc75e72cf1463bccb83e5933bc73e9cc0b3b99aed55f7cd0fe3371a4924f6271 Oct 30 16:46:59.941708 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:59.941582 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-hng8x"] Oct 30 16:46:59.943862 ip-10-0-128-218 kubenswrapper[2580]: W1030 16:46:59.943834 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf581b63_6285_4ac9_aa45_cd8a367478f2.slice/crio-4d74c110ad70b58f43e0b1a60d4379a8287fdeefadb7e639e03c3e7f1459fcae WatchSource:0}: Error finding container 4d74c110ad70b58f43e0b1a60d4379a8287fdeefadb7e639e03c3e7f1459fcae: Status 404 returned error can't find the container with id 4d74c110ad70b58f43e0b1a60d4379a8287fdeefadb7e639e03c3e7f1459fcae Oct 30 16:46:59.966277 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:59.965468 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cqjj8" event={"ID":"e275e41d-9573-47e2-b923-173dfb4361a7","Type":"ContainerStarted","Data":"4d540fa69043e29d9f1c5e77a33332722e0d2c30a20ac4de8dcaec162f27c99d"} Oct 30 16:46:59.968013 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:59.967985 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-hng8x" event={"ID":"bf581b63-6285-4ac9-aa45-cd8a367478f2","Type":"ContainerStarted","Data":"4d74c110ad70b58f43e0b1a60d4379a8287fdeefadb7e639e03c3e7f1459fcae"} Oct 30 16:46:59.969378 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:59.969329 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8zqlg" event={"ID":"40e1c660-22c0-42e5-8511-4cbd0e170a5c","Type":"ContainerStarted","Data":"0423717bb045b25f4e574b686e6564701d7df72979a3bfff1359674754052774"} Oct 30 16:46:59.970895 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:59.970869 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zm2rj" event={"ID":"cd83ecbc-117f-4890-bf7a-cab9282d4484","Type":"ContainerStarted","Data":"a737e28cca11d4405dfef3adba95876fa07f4117d846e29ac7515304d0e46c62"} Oct 30 16:46:59.972362 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:59.972337 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2kbm7" event={"ID":"d52d0ddc-6bd1-4dcc-9d56-84213c328055","Type":"ContainerStarted","Data":"bc75e72cf1463bccb83e5933bc73e9cc0b3b99aed55f7cd0fe3371a4924f6271"} Oct 30 16:46:59.973846 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:46:59.973796 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-f4j47-pgf7r" event={"ID":"252a7edc-651f-4c08-8fd7-f46ec29737c4","Type":"ContainerStarted","Data":"6fc7bce1f36d0e8e13dc09c5275b70fe8ea2790f94977fde82f4fa803951bcfc"} Oct 30 16:47:00.013462 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:47:00.013406 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-cqjj8" podStartSLOduration=2.430353028 podStartE2EDuration="5.013393939s" totalImagesPullingTime="2.583040911s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:46:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:47:00.012804862 +0000 UTC m=+33.887782127" watchObservedRunningTime="2026-10-30 16:47:00.013393939 +0000 UTC m=+33.888371177" Oct 30 16:47:00.027866 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:47:00.027827 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a21d39c5-3813-4433-9307-855276f1a66e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4wjcq\" (UID: \"a21d39c5-3813-4433-9307-855276f1a66e\") " pod="openshift-insights/insights-runtime-extractor-4wjcq" Oct 30 16:47:00.028034 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:47:00.027883 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a21d39c5-3813-4433-9307-855276f1a66e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4wjcq\" (UID: \"a21d39c5-3813-4433-9307-855276f1a66e\") " pod="openshift-insights/insights-runtime-extractor-4wjcq" Oct 30 16:47:00.029590 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:47:00.028759 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 16:47:00.029590 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:47:00.028816 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a21d39c5-3813-4433-9307-855276f1a66e-insights-runtime-extractor-tls podName:a21d39c5-3813-4433-9307-855276f1a66e nodeName:}" failed. No retries permitted until 2026-10-30 16:47:02.028801361 +0000 UTC m=+35.903778577 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a21d39c5-3813-4433-9307-855276f1a66e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4wjcq" (UID: "a21d39c5-3813-4433-9307-855276f1a66e") : secret "insights-runtime-extractor-tls" not found Oct 30 16:47:00.029590 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:47:00.029197 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 16:47:00.029590 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:47:00.029242 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a21d39c5-3813-4433-9307-855276f1a66e-kube-rbac-proxy-cm podName:a21d39c5-3813-4433-9307-855276f1a66e nodeName:}" failed. No retries permitted until 2026-10-30 16:47:02.029231807 +0000 UTC m=+35.904209024 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a21d39c5-3813-4433-9307-855276f1a66e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4wjcq" (UID: "a21d39c5-3813-4433-9307-855276f1a66e") : configmap "kube-rbac-proxy" not found Oct 30 16:47:01.985202 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:47:01.984774 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zm2rj" event={"ID":"cd83ecbc-117f-4890-bf7a-cab9282d4484","Type":"ContainerStarted","Data":"bb83f518f542e3c4f8e32556e1bc7962750926855a2e11d33e240992012cac65"} Oct 30 16:47:02.044686 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:47:02.044644 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a21d39c5-3813-4433-9307-855276f1a66e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4wjcq\" (UID: \"a21d39c5-3813-4433-9307-855276f1a66e\") " pod="openshift-insights/insights-runtime-extractor-4wjcq" Oct 30 16:47:02.044883 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:47:02.044716 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a21d39c5-3813-4433-9307-855276f1a66e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4wjcq\" (UID: \"a21d39c5-3813-4433-9307-855276f1a66e\") " pod="openshift-insights/insights-runtime-extractor-4wjcq" Oct 30 16:47:02.044883 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:47:02.044809 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 16:47:02.044883 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:47:02.044825 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 16:47:02.044883 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:47:02.044883 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a21d39c5-3813-4433-9307-855276f1a66e-insights-runtime-extractor-tls podName:a21d39c5-3813-4433-9307-855276f1a66e nodeName:}" failed. No retries permitted until 2026-10-30 16:47:06.04486337 +0000 UTC m=+39.919840589 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a21d39c5-3813-4433-9307-855276f1a66e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4wjcq" (UID: "a21d39c5-3813-4433-9307-855276f1a66e") : secret "insights-runtime-extractor-tls" not found Oct 30 16:47:02.045048 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:47:02.044904 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a21d39c5-3813-4433-9307-855276f1a66e-kube-rbac-proxy-cm podName:a21d39c5-3813-4433-9307-855276f1a66e nodeName:}" failed. No retries permitted until 2026-10-30 16:47:06.044893986 +0000 UTC m=+39.919871205 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a21d39c5-3813-4433-9307-855276f1a66e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4wjcq" (UID: "a21d39c5-3813-4433-9307-855276f1a66e") : configmap "kube-rbac-proxy" not found Oct 30 16:47:02.887841 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:47:02.887739 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqhg8_d24bfb4d-652d-4e7d-bc67-43441164f652/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:02.990687 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:47:02.990650 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-f4j47-pgf7r" event={"ID":"252a7edc-651f-4c08-8fd7-f46ec29737c4","Type":"ContainerStarted","Data":"65e723da13fb308bc9833e0b7bb7da28ccdfc88cdd79098cdf01d48a88ecc1ff"} Oct 30 16:47:03.038902 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:47:03.038841 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zm2rj" podStartSLOduration=3.114115668 podStartE2EDuration="5.038820724s" totalImagesPullingTime="1.924705056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:46:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:47:02.046582355 +0000 UTC m=+35.921559594" watchObservedRunningTime="2026-10-30 16:47:03.038820724 +0000 UTC m=+36.913797966" Oct 30 16:47:03.039177 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:47:03.039151 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-f4j47-pgf7r" podStartSLOduration=2.090017092 podStartE2EDuration="5.039144736s" totalImagesPullingTime="2.949127644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:46:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:47:03.038493428 +0000 UTC m=+36.913470690" watchObservedRunningTime="2026-10-30 16:47:03.039144736 +0000 UTC m=+36.914121976" Oct 30 16:47:03.757779 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:47:03.757749 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3-metrics-tls\") pod \"dns-default-wc9zw\" (UID: \"bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3\") " pod="openshift-dns/dns-default-wc9zw" Oct 30 16:47:03.757956 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:47:03.757922 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 16:47:03.758009 ip-10-0-128-218 kubenswrapper[2580]: E1030 16:47:03.758001 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3-metrics-tls podName:bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:11.757981085 +0000 UTC m=+45.632958308 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3-metrics-tls") pod "dns-default-wc9zw" (UID: "bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3") : secret "dns-default-metrics-tls" not found Oct 30 16:47:03.995527 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:47:03.995447 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-hng8x" event={"ID":"bf581b63-6285-4ac9-aa45-cd8a367478f2","Type":"ContainerStarted","Data":"def07e63de73ccb121d22417be26e3cc91b173e217f342d9929b9c3f3cad3d66"} Oct 30 16:47:04.041706 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:47:04.041656 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-hng8x" podStartSLOduration=1.5369470779999999 podStartE2EDuration="5.041641603s" totalImagesPullingTime="3.504694525s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:46:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:47:04.041191837 +0000 UTC m=+37.916169077" watchObservedRunningTime="2026-10-30 16:47:04.041641603 +0000 UTC m=+37.916618841" Oct 30 16:47:04.998977 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:47:04.998942 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8zqlg" event={"ID":"40e1c660-22c0-42e5-8511-4cbd0e170a5c","Type":"ContainerStarted","Data":"6937ad3cb00cbe2a44515e39aa9629ecc38fda6619d9eae98f704fe034dfa73b"} Oct 30 16:47:04.999423 ip-10-0-128-218 kubenswrapper[2580]: I1030 16:47:04.999098 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8zqlg" Oct 30 16:47:05.932383 ip-10-0-128-218 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 16:47:05.947737 ip-10-0-128-218 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 16:47:05.947900 ip-10-0-128-218 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 16:47:05.947950 ip-10-0-128-218 systemd[1]: kubelet.service: Consumed 2.351s CPU time. Oct 30 16:47:05.960803 ip-10-0-128-218 systemd[1]: Starting Kubernetes Kubelet... Oct 30 16:47:06.054181 ip-10-0-128-218 kubenswrapper[4522]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 16:47:06.054181 ip-10-0-128-218 kubenswrapper[4522]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 16:47:06.054181 ip-10-0-128-218 kubenswrapper[4522]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 16:47:06.054181 ip-10-0-128-218 kubenswrapper[4522]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 16:47:06.056320 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056218 4522 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:47:06.056320 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056315 4522 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:47:06.056320 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056319 4522 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:47:06.056320 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056323 4522 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:47:06.056320 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056325 4522 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:47:06.056320 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056328 4522 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:47:06.056513 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056331 4522 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:47:06.056513 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056334 4522 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:47:06.056513 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056336 4522 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:47:06.056513 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056339 4522 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:47:06.056513 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056342 4522 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:47:06.056513 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056345 4522 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:47:06.056513 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056347 4522 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:47:06.056513 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056350 4522 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:47:06.056513 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056359 4522 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:47:06.056513 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056362 4522 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:47:06.056513 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056364 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:47:06.056513 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056367 4522 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:47:06.056513 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056369 4522 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:47:06.056513 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056372 4522 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:47:06.056513 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056374 4522 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:47:06.056513 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056377 4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:47:06.056513 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056380 4522 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:47:06.056513 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056382 4522 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:47:06.056513 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056385 4522 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:47:06.056513 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056388 4522 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:47:06.057027 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056390 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:47:06.057027 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056393 4522 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:47:06.057027 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056395 4522 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:47:06.057027 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056398 4522 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:47:06.057027 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056401 4522 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:47:06.057027 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056404 4522 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:47:06.057027 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056406 4522 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:47:06.057027 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056409 4522 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:47:06.057027 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056412 4522 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:47:06.057027 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056414 4522 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:47:06.057027 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056417 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:47:06.057027 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056419 4522 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:47:06.057027 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056422 4522 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:47:06.057027 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056425 4522 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:47:06.057027 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056428 4522 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:47:06.057027 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056431 4522 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:47:06.057027 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056434 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:47:06.057027 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056436 4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:47:06.057027 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056439 4522 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:47:06.057027 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056442 4522 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:47:06.057546 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056444 4522 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:47:06.057546 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056448 4522 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:47:06.057546 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056450 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:47:06.057546 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056459 4522 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:47:06.057546 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056462 4522 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:47:06.057546 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056465 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:47:06.057546 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056468 4522 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:47:06.057546 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056470 4522 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:47:06.057546 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056473 4522 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:47:06.057546 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056475 4522 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:47:06.057546 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056479 4522 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:47:06.057546 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056481 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:47:06.057546 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056484 4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:47:06.057546 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056488 4522 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:47:06.057546 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056492 4522 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:47:06.057546 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056495 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:47:06.057546 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056498 4522 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:47:06.057546 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056501 4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:47:06.057546 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056503 4522 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:47:06.058032 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056506 4522 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:47:06.058032 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056509 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:47:06.058032 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056511 4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:47:06.058032 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056514 4522 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:47:06.058032 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056517 4522 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:47:06.058032 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056521 4522 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:47:06.058032 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056527 4522 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:47:06.058032 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056530 4522 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:47:06.058032 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056532 4522 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:47:06.058032 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056535 4522 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:47:06.058032 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056538 4522 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:47:06.058032 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056540 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:47:06.058032 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056543 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:47:06.058032 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056546 4522 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:47:06.058032 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056548 4522 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:47:06.058032 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056552 4522 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:47:06.058032 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056555 4522 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:47:06.058032 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056558 4522 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:47:06.058032 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056561 4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:47:06.058523 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056563 4522 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:47:06.058523 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056566 4522 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:47:06.058523 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056568 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:47:06.058523 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056571 4522 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:47:06.058523 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056573 4522 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:47:06.058523 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056576 4522 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:47:06.058523 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056579 4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:47:06.058523 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056582 4522 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:47:06.058523 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056585 4522 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:47:06.058523 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056587 4522 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:47:06.058523 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056590 4522 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:47:06.058523 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056592 4522 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:47:06.058523 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056595 4522 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:47:06.058523 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056597 4522 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:47:06.058523 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056600 4522 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:47:06.058523 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056602 4522 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:47:06.058523 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056605 4522 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:47:06.058523 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056607 4522 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:47:06.058523 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056610 4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:47:06.058523 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056614 4522 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:47:06.059040 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056617 4522 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:47:06.059040 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.056620 4522 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:47:06.059040 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057445 4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:47:06.059040 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057453 4522 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:47:06.059040 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057457 4522 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:47:06.059040 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057460 4522 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:47:06.059040 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057463 4522 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:47:06.059040 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057466 4522 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:47:06.059040 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057470 4522 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:47:06.059040 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057473 4522 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:47:06.059040 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057476 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:47:06.059040 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057480 4522 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:47:06.059040 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057483 4522 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:47:06.059040 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057486 4522 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:47:06.059040 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057489 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:47:06.059040 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057492 4522 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:47:06.059040 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057495 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:47:06.059040 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057497 4522 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:47:06.059040 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057500 4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:47:06.059535 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057504 4522 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:47:06.059535 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057506 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:47:06.059535 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057509 4522 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:47:06.059535 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057512 4522 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:47:06.059535 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057514 4522 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:47:06.059535 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057517 4522 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:47:06.059535 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057520 4522 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:47:06.059535 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057522 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:47:06.059535 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057525 4522 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:47:06.059535 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057527 4522 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:47:06.059535 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057530 4522 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:47:06.059535 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057533 4522 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:47:06.059535 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057535 4522 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:47:06.059535 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057538 4522 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:47:06.059535 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057540 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:47:06.059535 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057543 4522 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:47:06.059535 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057546 4522 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:47:06.059535 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057548 4522 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:47:06.059535 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057551 4522 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:47:06.059994 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057554 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:47:06.059994 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057558 4522 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:47:06.059994 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057562 4522 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:47:06.059994 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057565 4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:47:06.059994 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057569 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:47:06.059994 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057572 4522 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:47:06.059994 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057574 4522 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:47:06.059994 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057577 4522 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:47:06.059994 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057580 4522 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:47:06.059994 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057583 4522 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:47:06.059994 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057585 4522 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:47:06.059994 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057588 4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:47:06.059994 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057590 4522 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:47:06.059994 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057593 4522 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:47:06.059994 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057596 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:47:06.059994 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057599 4522 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:47:06.059994 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057601 4522 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:47:06.059994 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057605 4522 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:47:06.059994 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057607 4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:47:06.059994 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057610 4522 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:47:06.060515 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057612 4522 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:47:06.060515 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057615 4522 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:47:06.060515 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057617 4522 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:47:06.060515 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057621 4522 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:47:06.060515 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057625 4522 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:47:06.060515 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057628 4522 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:47:06.060515 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057630 4522 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:47:06.060515 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057633 4522 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:47:06.060515 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057635 4522 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:47:06.060515 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057638 4522 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:47:06.060515 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057640 4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:47:06.060515 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057643 4522 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:47:06.060515 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057645 4522 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:47:06.060515 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057648 4522 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:47:06.060515 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057650 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:47:06.060515 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057653 4522 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:47:06.060515 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057655 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:47:06.060515 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057659 4522 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:47:06.060515 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057661 4522 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:47:06.060515 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057664 4522 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:47:06.061015 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057667 4522 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:47:06.061015 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057670 4522 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:47:06.061015 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057672 4522 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:47:06.061015 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057675 4522 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:47:06.061015 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057677 4522 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:47:06.061015 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057680 4522 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:47:06.061015 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057683 4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:47:06.061015 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057685 4522 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:47:06.061015 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057688 4522 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:47:06.061015 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057691 4522 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:47:06.061015 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057693 4522 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:47:06.061015 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057696 4522 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:47:06.061015 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057699 4522 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:47:06.061015 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057701 4522 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:47:06.061015 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057704 4522 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:47:06.061015 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057707 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:47:06.061015 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057710 4522 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:47:06.061015 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057713 4522 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:47:06.061015 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057715 4522 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:47:06.061015 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057718 4522 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:47:06.061533 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057720 4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:47:06.061533 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057723 4522 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:47:06.061533 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057725 4522 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:47:06.061533 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057728 4522 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:47:06.061533 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057730 4522 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:47:06.061533 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057733 4522 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:47:06.061533 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057735 4522 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:47:06.061533 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057738 4522 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:47:06.061533 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057740 4522 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:47:06.061533 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.057743 4522 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:47:06.061533 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058229 4522 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:47:06.061533 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058234 4522 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:47:06.061533 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058237 4522 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:47:06.061533 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058239 4522 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:47:06.061533 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058242 4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:47:06.061533 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058245 4522 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:47:06.061533 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058247 4522 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:47:06.061533 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058250 4522 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:47:06.061533 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058253 4522 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:47:06.062258 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058255 4522 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:47:06.062258 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058258 4522 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:47:06.062258 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058261 4522 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:47:06.062258 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058263 4522 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:47:06.062258 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058266 4522 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:47:06.062258 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058269 4522 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:47:06.062258 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058272 4522 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:47:06.062258 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058275 4522 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:47:06.062258 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058277 4522 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:47:06.062258 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058280 4522 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:47:06.062258 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058283 4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:47:06.062258 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058285 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:47:06.062258 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058288 4522 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:47:06.062258 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058291 4522 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:47:06.062258 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058293 4522 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:47:06.062258 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058296 4522 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:47:06.062258 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058298 4522 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:47:06.062258 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058301 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:47:06.062258 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058317 4522 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:47:06.062258 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058320 4522 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:47:06.062777 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058322 4522 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:47:06.062777 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058325 4522 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:47:06.062777 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058327 4522 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:47:06.062777 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058331 4522 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:47:06.062777 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058333 4522 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:47:06.062777 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058336 4522 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:47:06.062777 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058338 4522 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:47:06.062777 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058341 4522 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:47:06.062777 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058343 4522 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:47:06.062777 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058346 4522 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:47:06.062777 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058348 4522 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:47:06.062777 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058351 4522 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:47:06.062777 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058353 4522 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:47:06.062777 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058356 4522 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:47:06.062777 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058358 4522 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:47:06.062777 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058361 4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:47:06.062777 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058363 4522 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:47:06.062777 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058365 4522 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:47:06.062777 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058371 4522 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:47:06.062777 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058374 4522 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:47:06.063268 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058376 4522 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:47:06.063268 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058379 4522 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:47:06.063268 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058382 4522 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:47:06.063268 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058384 4522 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:47:06.063268 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058388 4522 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:47:06.063268 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058390 4522 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:47:06.063268 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058394 4522 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:47:06.063268 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058397 4522 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:47:06.063268 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058401 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:47:06.063268 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058404 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:47:06.063268 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058407 4522 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:47:06.063268 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058409 4522 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:47:06.063268 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058412 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:47:06.063268 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058414 4522 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:47:06.063268 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058417 4522 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:47:06.063268 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058420 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:47:06.063268 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058423 4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:47:06.063268 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058426 4522 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:47:06.063268 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058429 4522 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:47:06.063756 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058431 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:47:06.063756 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058435 4522 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:47:06.063756 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058439 4522 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:47:06.063756 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058441 4522 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:47:06.063756 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058444 4522 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:47:06.063756 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058446 4522 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:47:06.063756 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058449 4522 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:47:06.063756 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058451 4522 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:47:06.063756 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058454 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:47:06.063756 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058456 4522 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:47:06.063756 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058459 4522 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:47:06.063756 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058463 4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:47:06.063756 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058466 4522 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:47:06.063756 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058469 4522 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:47:06.063756 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058473 4522 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:47:06.063756 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058476 4522 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:47:06.063756 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058478 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:47:06.063756 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058481 4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:47:06.063756 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058483 4522 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:47:06.064212 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058486 4522 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:47:06.064212 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058488 4522 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:47:06.064212 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058491 4522 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:47:06.064212 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058494 4522 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:47:06.064212 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058496 4522 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:47:06.064212 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058499 4522 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:47:06.064212 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058501 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:47:06.064212 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058504 4522 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:47:06.064212 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058506 4522 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:47:06.064212 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058509 4522 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:47:06.064212 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058512 4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:47:06.064212 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058515 4522 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:47:06.064212 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058517 4522 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:47:06.064212 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058520 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:47:06.064212 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058523 4522 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:47:06.064212 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058526 4522 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:47:06.064212 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058528 4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:47:06.064212 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058531 4522 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:47:06.064212 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.058534 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:47:06.064714 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058631 4522 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 16:47:06.064714 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058639 4522 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 16:47:06.064714 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058645 4522 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 16:47:06.064714 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058650 4522 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 16:47:06.064714 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058655 4522 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 16:47:06.064714 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058660 4522 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 16:47:06.064714 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058664 4522 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 16:47:06.064714 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058669 4522 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 16:47:06.064714 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058675 4522 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 16:47:06.064714 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058678 4522 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 16:47:06.064714 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058682 4522 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 16:47:06.064714 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058685 4522 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 16:47:06.064714 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058688 4522 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 16:47:06.064714 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058691 4522 flags.go:64] FLAG: --cgroup-root="" Oct 30 16:47:06.064714 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058695 4522 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 16:47:06.064714 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058698 4522 flags.go:64] FLAG: --client-ca-file="" Oct 30 16:47:06.064714 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058701 4522 flags.go:64] FLAG: --cloud-provider="external" Oct 30 16:47:06.064714 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058704 4522 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 16:47:06.064714 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058708 4522 flags.go:64] FLAG: --cluster-domain="" Oct 30 16:47:06.064714 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058711 4522 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 16:47:06.064714 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058714 4522 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 16:47:06.064714 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058717 4522 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 16:47:06.064714 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058721 4522 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 16:47:06.064714 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058724 4522 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 16:47:06.065444 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058728 4522 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 16:47:06.065444 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058731 4522 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 16:47:06.065444 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058735 4522 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 16:47:06.065444 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058738 4522 flags.go:64] FLAG: --contention-profiling="false" Oct 30 16:47:06.065444 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058741 4522 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 16:47:06.065444 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058744 4522 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 16:47:06.065444 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058747 4522 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 16:47:06.065444 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058750 4522 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 16:47:06.065444 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058754 4522 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 16:47:06.065444 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058758 4522 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 16:47:06.065444 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058760 4522 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 16:47:06.065444 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058763 4522 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 16:47:06.065444 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058767 4522 flags.go:64] FLAG: --enable-server="true" Oct 30 16:47:06.065444 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058771 4522 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 16:47:06.065444 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058774 4522 flags.go:64] FLAG: --event-burst="100" Oct 30 16:47:06.065444 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058778 4522 flags.go:64] FLAG: --event-qps="50" Oct 30 16:47:06.065444 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058782 4522 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 16:47:06.065444 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058785 4522 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 16:47:06.065444 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058788 4522 flags.go:64] FLAG: --eviction-hard="" Oct 30 16:47:06.065444 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058792 4522 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 16:47:06.065444 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058795 4522 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 16:47:06.065444 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058799 4522 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 16:47:06.065444 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058802 4522 flags.go:64] FLAG: --eviction-soft="" Oct 30 16:47:06.065444 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058805 4522 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 16:47:06.065444 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058808 4522 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 16:47:06.066462 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058811 4522 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 16:47:06.066462 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058814 4522 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 16:47:06.066462 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058817 4522 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 16:47:06.066462 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058820 4522 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 16:47:06.066462 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058822 4522 flags.go:64] FLAG: --feature-gates="" Oct 30 16:47:06.066462 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058829 4522 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 16:47:06.066462 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058833 4522 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 16:47:06.066462 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058836 4522 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 16:47:06.066462 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058839 4522 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 16:47:06.066462 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058843 4522 flags.go:64] FLAG: --healthz-port="10248" Oct 30 16:47:06.066462 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058846 4522 flags.go:64] FLAG: --help="false" Oct 30 16:47:06.066462 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058849 4522 flags.go:64] FLAG: --hostname-override="ip-10-0-128-218.ec2.internal" Oct 30 16:47:06.066462 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058852 4522 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 16:47:06.066462 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058855 4522 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 16:47:06.066462 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058858 4522 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 16:47:06.066462 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058862 4522 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 16:47:06.066462 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058866 4522 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 16:47:06.066462 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058868 4522 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 16:47:06.066462 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058871 4522 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 16:47:06.066462 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058874 4522 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 16:47:06.066462 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058878 4522 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 16:47:06.066462 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058881 4522 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 16:47:06.066462 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058885 4522 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 16:47:06.066462 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058889 4522 flags.go:64] FLAG: --kube-reserved="" Oct 30 16:47:06.067260 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058892 4522 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 16:47:06.067260 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058895 4522 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 16:47:06.067260 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058898 4522 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 16:47:06.067260 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058901 4522 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 16:47:06.067260 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058904 4522 flags.go:64] FLAG: --lock-file="" Oct 30 16:47:06.067260 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058907 4522 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 16:47:06.067260 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058910 4522 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 16:47:06.067260 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058913 4522 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 16:47:06.067260 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058919 4522 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 16:47:06.067260 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058928 4522 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 16:47:06.067260 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058931 4522 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 16:47:06.067260 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058934 4522 flags.go:64] FLAG: --logging-format="text" Oct 30 16:47:06.067260 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058937 4522 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 16:47:06.067260 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058940 4522 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 16:47:06.067260 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058943 4522 flags.go:64] FLAG: --manifest-url="" Oct 30 16:47:06.067260 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058946 4522 flags.go:64] FLAG: --manifest-url-header="" Oct 30 16:47:06.067260 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058951 4522 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 16:47:06.067260 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058954 4522 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 16:47:06.067260 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058959 4522 flags.go:64] FLAG: --max-pods="110" Oct 30 16:47:06.067260 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058962 4522 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 16:47:06.067260 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058965 4522 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 16:47:06.067260 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058968 4522 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 16:47:06.067260 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058971 4522 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 16:47:06.067260 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058974 4522 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 16:47:06.067260 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058977 4522 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 16:47:06.067921 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058980 4522 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 16:47:06.067921 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058988 4522 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 16:47:06.067921 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058991 4522 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 16:47:06.067921 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058995 4522 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 16:47:06.067921 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.058999 4522 flags.go:64] FLAG: --pod-cidr="" Oct 30 16:47:06.067921 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.059002 4522 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 16:47:06.067921 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.059006 4522 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 16:47:06.067921 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.059009 4522 flags.go:64] FLAG: --pods-per-core="0" Oct 30 16:47:06.067921 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.059012 4522 flags.go:64] FLAG: --port="10250" Oct 30 16:47:06.067921 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.059015 4522 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 16:47:06.067921 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.059018 4522 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ce764e851bffdc03" Oct 30 16:47:06.067921 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.059021 4522 flags.go:64] FLAG: --qos-reserved="" Oct 30 16:47:06.067921 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.059025 4522 flags.go:64] FLAG: --read-only-port="10255" Oct 30 16:47:06.067921 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.059028 4522 flags.go:64] FLAG: --register-node="true" Oct 30 16:47:06.067921 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.059031 4522 flags.go:64] FLAG: --register-with-taints="" Oct 30 16:47:06.067921 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.059034 4522 flags.go:64] FLAG: --registry-burst="10" Oct 30 16:47:06.067921 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.059038 4522 flags.go:64] FLAG: --registry-qps="5" Oct 30 16:47:06.067921 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.059041 4522 flags.go:64] FLAG: --reserved-cpus="" Oct 30 16:47:06.067921 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.059044 4522 flags.go:64] FLAG: --reserved-memory="" Oct 30 16:47:06.067921 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.059049 4522 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 16:47:06.067921 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.059052 4522 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 16:47:06.067921 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.059055 4522 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 16:47:06.067921 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.059058 4522 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 16:47:06.067921 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.059061 4522 flags.go:64] FLAG: --runonce="false" Oct 30 16:47:06.067921 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.059064 4522 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 16:47:06.067921 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.059067 4522 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 16:47:06.068698 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.059071 4522 flags.go:64] FLAG: --seccomp-default="false" Oct 30 16:47:06.068698 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.059074 4522 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 16:47:06.068698 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.059077 4522 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 16:47:06.068698 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.059080 4522 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 16:47:06.068698 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.059083 4522 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 16:47:06.068698 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.059086 4522 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 16:47:06.068698 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.059089 4522 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 16:47:06.068698 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.059092 4522 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 16:47:06.068698 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.059095 4522 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 16:47:06.068698 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.059100 4522 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 16:47:06.068698 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.059103 4522 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 16:47:06.068698 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.059106 4522 flags.go:64] FLAG: --system-cgroups="" Oct 30 16:47:06.068698 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.059110 4522 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 16:47:06.068698 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.059115 4522 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 16:47:06.068698 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.059118 4522 flags.go:64] FLAG: --tls-cert-file="" Oct 30 16:47:06.068698 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.059121 4522 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 16:47:06.068698 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.059126 4522 flags.go:64] FLAG: --tls-min-version="" Oct 30 16:47:06.068698 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.059129 4522 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 16:47:06.068698 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.059132 4522 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 16:47:06.068698 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.059134 4522 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 16:47:06.068698 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.059137 4522 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 16:47:06.068698 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.059141 4522 flags.go:64] FLAG: --v="2" Oct 30 16:47:06.068698 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.059145 4522 flags.go:64] FLAG: --version="false" Oct 30 16:47:06.068698 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.059149 4522 flags.go:64] FLAG: --vmodule="" Oct 30 16:47:06.068698 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.059153 4522 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 16:47:06.069329 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.059156 4522 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 16:47:06.069329 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059289 4522 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:47:06.069329 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059293 4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:47:06.069329 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059296 4522 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:47:06.069329 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059300 4522 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:47:06.069329 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059317 4522 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:47:06.069329 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059321 4522 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:47:06.069329 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059324 4522 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:47:06.069329 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059328 4522 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:47:06.069329 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059330 4522 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:47:06.069329 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059333 4522 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:47:06.069329 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059336 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:47:06.069329 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059339 4522 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:47:06.069329 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059342 4522 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:47:06.069329 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059345 4522 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:47:06.069329 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059347 4522 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:47:06.069329 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059352 4522 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:47:06.069329 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059354 4522 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:47:06.069329 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059357 4522 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:47:06.069329 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059361 4522 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:47:06.069837 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059364 4522 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:47:06.069837 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059366 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:47:06.069837 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059369 4522 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:47:06.069837 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059371 4522 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:47:06.069837 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059374 4522 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:47:06.069837 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059377 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:47:06.069837 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059380 4522 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:47:06.069837 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059382 4522 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:47:06.069837 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059385 4522 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:47:06.069837 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059388 4522 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:47:06.069837 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059390 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:47:06.069837 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059393 4522 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:47:06.069837 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059395 4522 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:47:06.069837 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059398 4522 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:47:06.069837 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059400 4522 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:47:06.069837 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059403 4522 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:47:06.069837 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059405 4522 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:47:06.069837 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059408 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:47:06.069837 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059411 4522 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:47:06.070446 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059413 4522 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:47:06.070446 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059416 4522 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:47:06.070446 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059419 4522 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:47:06.070446 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059421 4522 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:47:06.070446 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059424 4522 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:47:06.070446 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059427 4522 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:47:06.070446 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059429 4522 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:47:06.070446 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059432 4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:47:06.070446 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059434 4522 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:47:06.070446 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059438 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:47:06.070446 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059441 4522 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:47:06.070446 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059443 4522 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:47:06.070446 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059447 4522 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:47:06.070446 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059450 4522 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:47:06.070446 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059452 4522 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:47:06.070446 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059455 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:47:06.070446 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059458 4522 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:47:06.070446 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059460 4522 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:47:06.070446 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059463 4522 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:47:06.071037 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059466 4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:47:06.071037 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059468 4522 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:47:06.071037 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059471 4522 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:47:06.071037 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059473 4522 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:47:06.071037 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059476 4522 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:47:06.071037 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059479 4522 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:47:06.071037 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059481 4522 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:47:06.071037 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059484 4522 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:47:06.071037 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059486 4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:47:06.071037 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059489 4522 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:47:06.071037 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059492 4522 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:47:06.071037 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059494 4522 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:47:06.071037 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059497 4522 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:47:06.071037 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059499 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:47:06.071037 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059502 4522 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:47:06.071037 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059504 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:47:06.071037 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059508 4522 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:47:06.071037 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059510 4522 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:47:06.071037 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059513 4522 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:47:06.071037 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059516 4522 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:47:06.071596 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059518 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:47:06.071596 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059521 4522 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:47:06.071596 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059526 4522 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:47:06.071596 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059531 4522 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:47:06.071596 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059534 4522 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:47:06.071596 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059539 4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:47:06.071596 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059542 4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:47:06.071596 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059545 4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:47:06.071596 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059548 4522 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:47:06.071596 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059551 4522 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:47:06.071596 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059554 4522 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:47:06.071596 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059556 4522 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:47:06.071596 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059559 4522 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:47:06.071596 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059562 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:47:06.071596 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059564 4522 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:47:06.071596 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059567 4522 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:47:06.071596 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059569 4522 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:47:06.071596 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059572 4522 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:47:06.071596 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059574 4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:47:06.072088 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059577 4522 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:47:06.072088 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059580 4522 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:47:06.072088 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059582 4522 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:47:06.072088 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059585 4522 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:47:06.072088 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059587 4522 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:47:06.072088 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059590 4522 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:47:06.072088 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059593 4522 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:47:06.072088 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059595 4522 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:47:06.072088 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059598 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:47:06.072088 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059600 4522 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:47:06.072088 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.059627 4522 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 16:47:06.072088 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059663 4522 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:47:06.072088 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059668 4522 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:47:06.072088 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059671 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:47:06.072088 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059674 4522 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:47:06.072088 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059677 4522 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:47:06.072510 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059681 4522 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:47:06.072510 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059683 4522 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:47:06.072510 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059687 4522 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:47:06.072510 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059689 4522 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:47:06.072510 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059692 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:47:06.072510 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059695 4522 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:47:06.072510 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059697 4522 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:47:06.072510 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059700 4522 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:47:06.072510 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059702 4522 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:47:06.072510 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059706 4522 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:47:06.072510 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059709 4522 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:47:06.072510 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059711 4522 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:47:06.072510 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059713 4522 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:47:06.072510 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059716 4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:47:06.072510 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059718 4522 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:47:06.072510 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059721 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:47:06.072510 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059724 4522 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:47:06.072510 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059726 4522 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:47:06.072510 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059729 4522 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:47:06.073024 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059731 4522 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:47:06.073024 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059734 4522 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:47:06.073024 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059737 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:47:06.073024 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059740 4522 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:47:06.073024 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059742 4522 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:47:06.073024 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059745 4522 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:47:06.073024 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059747 4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:47:06.073024 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059750 4522 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:47:06.073024 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059753 4522 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:47:06.073024 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059755 4522 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:47:06.073024 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059758 4522 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:47:06.073024 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059760 4522 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:47:06.073024 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059763 4522 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:47:06.073024 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059767 4522 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:47:06.073024 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059770 4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:47:06.073024 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059773 4522 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:47:06.073024 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059775 4522 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:47:06.073024 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059778 4522 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:47:06.073024 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059781 4522 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:47:06.073024 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059783 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:47:06.073797 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059786 4522 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:47:06.073797 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059788 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:47:06.073797 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059791 4522 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:47:06.073797 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059793 4522 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:47:06.073797 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059796 4522 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:47:06.073797 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059798 4522 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:47:06.073797 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059801 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:47:06.073797 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059804 4522 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:47:06.073797 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059807 4522 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:47:06.073797 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059811 4522 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:47:06.073797 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059814 4522 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:47:06.073797 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059817 4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:47:06.073797 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059820 4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:47:06.073797 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059822 4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:47:06.073797 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059825 4522 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:47:06.073797 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059827 4522 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:47:06.073797 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059830 4522 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:47:06.073797 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059832 4522 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:47:06.073797 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059835 4522 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:47:06.074393 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059838 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:47:06.074393 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059840 4522 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:47:06.074393 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059843 4522 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:47:06.074393 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059846 4522 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:47:06.074393 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059849 4522 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:47:06.074393 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059851 4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:47:06.074393 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059855 4522 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:47:06.074393 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059858 4522 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:47:06.074393 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059860 4522 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:47:06.074393 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059863 4522 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:47:06.074393 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059865 4522 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:47:06.074393 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059868 4522 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:47:06.074393 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059871 4522 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:47:06.074393 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059873 4522 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:47:06.074393 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059876 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:47:06.074393 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059878 4522 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:47:06.074393 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059880 4522 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:47:06.074393 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059883 4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:47:06.074393 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059886 4522 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:47:06.074854 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059890 4522 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:47:06.074854 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059893 4522 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:47:06.074854 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059896 4522 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:47:06.074854 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059900 4522 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:47:06.074854 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059903 4522 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:47:06.074854 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059906 4522 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:47:06.074854 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059908 4522 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:47:06.074854 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059911 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:47:06.074854 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059914 4522 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:47:06.074854 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059916 4522 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:47:06.074854 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059919 4522 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:47:06.074854 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059922 4522 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:47:06.074854 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059924 4522 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:47:06.074854 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059926 4522 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:47:06.074854 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059929 4522 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:47:06.074854 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059931 4522 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:47:06.074854 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059934 4522 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:47:06.074854 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059937 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:47:06.074854 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059939 4522 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:47:06.075320 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059943 4522 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:47:06.075320 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059946 4522 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:47:06.075320 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059949 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:47:06.075320 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059952 4522 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:47:06.075320 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.059954 4522 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:47:06.075320 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.064131 4522 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 16:47:06.075320 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.064147 4522 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 16:47:06.075320 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.064166 4522 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 16:47:06.075320 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.064171 4522 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 16:47:06.075320 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064254 4522 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:47:06.075320 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064259 4522 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:47:06.075320 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064262 4522 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:47:06.075320 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064265 4522 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:47:06.075320 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064268 4522 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:47:06.075320 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064271 4522 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:47:06.075320 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064274 4522 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:47:06.075320 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064276 4522 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:47:06.075320 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064279 4522 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:47:06.075320 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064282 4522 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:47:06.075320 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064284 4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:47:06.075823 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064287 4522 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:47:06.075823 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064289 4522 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:47:06.075823 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064292 4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:47:06.075823 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064295 4522 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:47:06.075823 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064298 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:47:06.075823 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064300 4522 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:47:06.075823 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064315 4522 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:47:06.075823 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064318 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:47:06.075823 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064320 4522 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:47:06.075823 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064323 4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:47:06.075823 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064332 4522 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:47:06.075823 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064335 4522 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:47:06.075823 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064338 4522 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:47:06.075823 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064341 4522 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:47:06.075823 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064344 4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:47:06.075823 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064347 4522 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:47:06.075823 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064349 4522 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:47:06.075823 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064352 4522 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:47:06.075823 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064356 4522 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:47:06.076360 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064360 4522 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:47:06.076360 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064364 4522 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:47:06.076360 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064366 4522 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:47:06.076360 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064369 4522 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:47:06.076360 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064372 4522 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:47:06.076360 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064374 4522 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:47:06.076360 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064377 4522 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:47:06.076360 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064379 4522 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:47:06.076360 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064382 4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:47:06.076360 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064385 4522 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:47:06.076360 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064387 4522 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:47:06.076360 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064390 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:47:06.076360 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064392 4522 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:47:06.076360 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064395 4522 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:47:06.076360 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064399 4522 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:47:06.076360 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064402 4522 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:47:06.076360 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064406 4522 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:47:06.076360 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064409 4522 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:47:06.076360 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064412 4522 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:47:06.076969 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064415 4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:47:06.076969 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064418 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:47:06.076969 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064421 4522 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:47:06.076969 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064424 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:47:06.076969 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064427 4522 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:47:06.076969 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064436 4522 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:47:06.076969 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064439 4522 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:47:06.076969 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064442 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:47:06.076969 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064445 4522 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:47:06.076969 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064448 4522 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:47:06.076969 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064455 4522 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:47:06.076969 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064460 4522 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:47:06.076969 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064463 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:47:06.076969 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064466 4522 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:47:06.076969 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064468 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:47:06.076969 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064471 4522 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:47:06.076969 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064474 4522 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:47:06.076969 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064477 4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:47:06.076969 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064479 4522 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:47:06.076969 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064482 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:47:06.077663 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064485 4522 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:47:06.077663 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064487 4522 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:47:06.077663 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064490 4522 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:47:06.077663 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064493 4522 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:47:06.077663 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064496 4522 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:47:06.077663 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064499 4522 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:47:06.077663 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064501 4522 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:47:06.077663 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064504 4522 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:47:06.077663 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064506 4522 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:47:06.077663 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064509 4522 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:47:06.077663 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064511 4522 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:47:06.077663 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064514 4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:47:06.077663 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064517 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:47:06.077663 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064519 4522 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:47:06.077663 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064522 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:47:06.077663 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064525 4522 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:47:06.077663 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064528 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:47:06.077663 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064530 4522 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:47:06.077663 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064539 4522 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:47:06.078151 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064542 4522 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:47:06.078151 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064545 4522 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:47:06.078151 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064548 4522 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:47:06.078151 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064550 4522 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:47:06.078151 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064553 4522 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:47:06.078151 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064555 4522 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:47:06.078151 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064558 4522 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:47:06.078151 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064560 4522 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:47:06.078151 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064563 4522 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:47:06.078151 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064566 4522 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:47:06.078151 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064568 4522 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:47:06.078151 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064571 4522 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:47:06.078151 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064574 4522 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:47:06.078151 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064576 4522 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:47:06.078151 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064579 4522 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:47:06.078151 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064581 4522 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:47:06.078151 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064584 4522 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:47:06.078151 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064586 4522 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:47:06.078607 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.064614 4522 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 16:47:06.078607 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064816 4522 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:47:06.078607 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064823 4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:47:06.078607 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064826 4522 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:47:06.078607 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064829 4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:47:06.078607 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064832 4522 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:47:06.078607 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064835 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:47:06.078607 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064838 4522 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:47:06.078607 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064840 4522 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:47:06.078607 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064843 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:47:06.078607 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064846 4522 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:47:06.078607 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064848 4522 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:47:06.078607 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064851 4522 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:47:06.078607 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064854 4522 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:47:06.078607 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064862 4522 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:47:06.078976 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064865 4522 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:47:06.078976 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064868 4522 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:47:06.078976 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064870 4522 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:47:06.078976 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064873 4522 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:47:06.078976 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064876 4522 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:47:06.078976 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064878 4522 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:47:06.078976 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064881 4522 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:47:06.078976 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064884 4522 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:47:06.078976 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064888 4522 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:47:06.078976 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064890 4522 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:47:06.078976 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064893 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:47:06.078976 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064895 4522 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:47:06.078976 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064898 4522 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:47:06.078976 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064901 4522 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:47:06.078976 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064903 4522 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:47:06.078976 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064905 4522 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:47:06.078976 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064908 4522 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:47:06.078976 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064910 4522 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:47:06.078976 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064913 4522 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:47:06.079569 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064916 4522 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:47:06.079569 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064918 4522 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:47:06.079569 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064921 4522 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:47:06.079569 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064923 4522 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:47:06.079569 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064926 4522 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:47:06.079569 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064928 4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:47:06.079569 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064930 4522 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:47:06.079569 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064933 4522 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:47:06.079569 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064936 4522 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:47:06.079569 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064938 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:47:06.079569 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064941 4522 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:47:06.079569 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064944 4522 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:47:06.079569 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064946 4522 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:47:06.079569 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064955 4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:47:06.079569 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064957 4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:47:06.079569 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064960 4522 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:47:06.079569 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064963 4522 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:47:06.079569 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064965 4522 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:47:06.079569 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064968 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:47:06.080027 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064970 4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:47:06.080027 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064974 4522 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:47:06.080027 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064978 4522 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:47:06.080027 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064981 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:47:06.080027 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064983 4522 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:47:06.080027 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064986 4522 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:47:06.080027 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064989 4522 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:47:06.080027 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064991 4522 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:47:06.080027 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064993 4522 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:47:06.080027 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064996 4522 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:47:06.080027 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.064998 4522 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:47:06.080027 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065001 4522 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:47:06.080027 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065003 4522 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:47:06.080027 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065006 4522 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:47:06.080027 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065009 4522 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:47:06.080027 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065011 4522 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:47:06.080027 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065014 4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:47:06.080027 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065016 4522 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:47:06.080027 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065019 4522 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:47:06.080507 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065021 4522 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:47:06.080507 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065024 4522 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:47:06.080507 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065026 4522 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:47:06.080507 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065029 4522 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:47:06.080507 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065031 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:47:06.080507 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065034 4522 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:47:06.080507 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065037 4522 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:47:06.080507 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065039 4522 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:47:06.080507 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065049 4522 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:47:06.080507 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065052 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:47:06.080507 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065054 4522 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:47:06.080507 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065057 4522 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:47:06.080507 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065059 4522 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:47:06.080507 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065062 4522 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:47:06.080507 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065064 4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:47:06.080507 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065067 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:47:06.080507 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065069 4522 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:47:06.080507 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065072 4522 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:47:06.080507 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065074 4522 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:47:06.080507 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065077 4522 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:47:06.081030 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065079 4522 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:47:06.081030 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065082 4522 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:47:06.081030 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065084 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:47:06.081030 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065087 4522 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:47:06.081030 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065089 4522 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:47:06.081030 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065092 4522 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:47:06.081030 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065094 4522 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:47:06.081030 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065097 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:47:06.081030 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065099 4522 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:47:06.081030 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065102 4522 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:47:06.081030 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065104 4522 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:47:06.081030 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065107 4522 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:47:06.081030 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065109 4522 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:47:06.081030 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065112 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:47:06.081030 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065118 4522 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:47:06.081030 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.065141 4522 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 16:47:06.081030 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065184 4522 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:47:06.081473 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065189 4522 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:47:06.081473 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065192 4522 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:47:06.081473 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065194 4522 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:47:06.081473 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065197 4522 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:47:06.081473 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065210 4522 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:47:06.081473 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065212 4522 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:47:06.081473 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065215 4522 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:47:06.081473 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065218 4522 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:47:06.081473 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065220 4522 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:47:06.081473 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065223 4522 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:47:06.081473 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065226 4522 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:47:06.081473 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065228 4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:47:06.081473 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065231 4522 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:47:06.081473 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065233 4522 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:47:06.081473 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065236 4522 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:47:06.081473 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065238 4522 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:47:06.081473 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065241 4522 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:47:06.081473 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065243 4522 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:47:06.081473 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065246 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:47:06.081925 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065248 4522 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:47:06.081925 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065251 4522 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:47:06.081925 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065253 4522 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:47:06.081925 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065256 4522 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:47:06.081925 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065258 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:47:06.081925 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065261 4522 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:47:06.081925 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065263 4522 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:47:06.081925 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065266 4522 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:47:06.081925 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065269 4522 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:47:06.081925 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065271 4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:47:06.081925 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065277 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:47:06.081925 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065280 4522 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:47:06.081925 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065282 4522 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:47:06.081925 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065285 4522 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:47:06.081925 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065287 4522 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:47:06.081925 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065290 4522 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:47:06.081925 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065292 4522 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:47:06.081925 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065295 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:47:06.081925 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065314 4522 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:47:06.081925 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065317 4522 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:47:06.082513 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065320 4522 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:47:06.082513 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065323 4522 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:47:06.082513 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065325 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:47:06.082513 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065328 4522 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:47:06.082513 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065330 4522 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:47:06.082513 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065333 4522 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:47:06.082513 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065335 4522 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:47:06.082513 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065338 4522 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:47:06.082513 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065341 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:47:06.082513 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065351 4522 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:47:06.082513 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065354 4522 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:47:06.082513 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065357 4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:47:06.082513 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065359 4522 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:47:06.082513 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065362 4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:47:06.082513 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065365 4522 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:47:06.082513 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065367 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:47:06.082513 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065370 4522 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:47:06.082513 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065372 4522 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:47:06.082513 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065375 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:47:06.083300 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065378 4522 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:47:06.083300 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065380 4522 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:47:06.083300 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065383 4522 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:47:06.083300 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065387 4522 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:47:06.083300 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065389 4522 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:47:06.083300 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065392 4522 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:47:06.083300 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065394 4522 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:47:06.083300 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065397 4522 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:47:06.083300 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065399 4522 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:47:06.083300 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065403 4522 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:47:06.083300 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065405 4522 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:47:06.083300 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065408 4522 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:47:06.083300 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065417 4522 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:47:06.083300 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065421 4522 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:47:06.083300 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065424 4522 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:47:06.083300 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065427 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:47:06.083300 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065430 4522 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:47:06.083300 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065433 4522 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:47:06.083300 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065437 4522 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:47:06.083300 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065439 4522 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:47:06.084094 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065442 4522 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:47:06.084094 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065444 4522 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:47:06.084094 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065447 4522 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:47:06.084094 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065450 4522 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:47:06.084094 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065453 4522 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:47:06.084094 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065456 4522 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:47:06.084094 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065458 4522 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:47:06.084094 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065461 4522 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:47:06.084094 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065464 4522 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:47:06.084094 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065466 4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:47:06.084094 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065469 4522 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:47:06.084094 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065471 4522 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:47:06.084094 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065474 4522 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:47:06.084094 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065476 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:47:06.084094 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065479 4522 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:47:06.084094 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065484 4522 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:47:06.084094 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065486 4522 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:47:06.084094 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065489 4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:47:06.084094 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065491 4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:47:06.084902 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065494 4522 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:47:06.084902 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065496 4522 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:47:06.084902 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065499 4522 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:47:06.084902 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065501 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:47:06.084902 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065504 4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:47:06.084902 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065508 4522 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:47:06.084902 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065520 4522 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:47:06.084902 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:06.065523 4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:47:06.084902 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.065580 4522 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 16:47:06.084902 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.067791 4522 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 16:47:06.084902 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.067841 4522 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 16:47:06.084902 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.068044 4522 server.go:1014] "Starting client certificate rotation" Oct 30 16:47:06.084902 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.068193 4522 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 16:47:06.084902 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.068245 4522 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 16:41:26 +0000 UTC" deadline="2028-06-24 00:55:28.559789282 +0000 UTC" Oct 30 16:47:06.084902 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.068292 4522 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14456h8m22.491500823s" Oct 30 16:47:06.085643 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.068548 4522 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 16:47:06.085643 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.069649 4522 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 16:47:06.085643 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.071166 4522 log.go:25] "Validated CRI v1 runtime API" Oct 30 16:47:06.085643 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.072839 4522 log.go:25] "Validated CRI v1 image API" Oct 30 16:47:06.085643 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.073589 4522 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 16:47:06.085643 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.075339 4522 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a5440e34-6202-4b3f-9a67-2aacbb8464f3:/dev/nvme0n1p3 bb58242a-b906-465f-b426-fc8649b1174d:/dev/nvme0n1p4] Oct 30 16:47:06.085914 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.075366 4522 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0423717bb045b25f4e574b686e6564701d7df72979a3bfff1359674754052774/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0423717bb045b25f4e574b686e6564701d7df72979a3bfff1359674754052774/userdata/shm major:0 minor:227 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/11e310e8c78c4004c8aa0e76c03c4906cf266079403a7e0fda8db10f0e868c28/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/11e310e8c78c4004c8aa0e76c03c4906cf266079403a7e0fda8db10f0e868c28/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4d74c110ad70b58f43e0b1a60d4379a8287fdeefadb7e639e03c3e7f1459fcae/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4d74c110ad70b58f43e0b1a60d4379a8287fdeefadb7e639e03c3e7f1459fcae/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5f90a5b2e8450d0fe35942286c1d9f96483f222fca4a6ec99a7a8061df7d6fc8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5f90a5b2e8450d0fe35942286c1d9f96483f222fca4a6ec99a7a8061df7d6fc8/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6129a8fc729d736a455e6fc90d5fb7d979feaee41cf41276d3810ce84f8b900b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6129a8fc729d736a455e6fc90d5fb7d979feaee41cf41276d3810ce84f8b900b/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6fc7bce1f36d0e8e13dc09c5275b70fe8ea2790f94977fde82f4fa803951bcfc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6fc7bce1f36d0e8e13dc09c5275b70fe8ea2790f94977fde82f4fa803951bcfc/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/73cfaf3d1a1d8a84ba37bfd1f1c2358dfda8ce270736512c278c313aae6bbeb6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/73cfaf3d1a1d8a84ba37bfd1f1c2358dfda8ce270736512c278c313aae6bbeb6/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/77f86b5c5ce7f22475eac00b2a7033e5660a5b01cf17fe32625a4b783b19790a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/77f86b5c5ce7f22475eac00b2a7033e5660a5b01cf17fe32625a4b783b19790a/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/95d6d3e2380f9668e7a87195cacc53233ec9bf359ee51a6eb1e66ce13dc91206/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/95d6d3e2380f9668e7a87195cacc53233ec9bf359ee51a6eb1e66ce13dc91206/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/962d4c9822b1abc27c25eb93a60baaf8beaf09e16b434e63802ff7732ec3e560/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/962d4c9822b1abc27c25eb93a60baaf8beaf09e16b434e63802ff7732ec3e560/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a737e28cca11d4405dfef3adba95876fa07f4117d846e29ac7515304d0e46c62/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a737e28cca11d4405dfef3adba95876fa07f4117d846e29ac7515304d0e46c62/userdata/shm major:0 minor:211 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b1ebd56f579b03a320cb5b7cbc25f067b7169123bed80d2a34ae0af14f48283e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b1ebd56f579b03a320cb5b7cbc25f067b7169123bed80d2a34ae0af14f48283e/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bc75e72cf1463bccb83e5933bc73e9cc0b3b99aed55f7cd0fe3371a4924f6271/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bc75e72cf1463bccb83e5933bc73e9cc0b3b99aed55f7cd0fe3371a4924f6271/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d4ec855f9a431818213116d72030492004afb27e6e9a5f1fae58b7223a06cfe3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d4ec855f9a431818213116d72030492004afb27e6e9a5f1fae58b7223a06cfe3/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e1a02f18bc37c32c8c93383a529584ff2d952162e8341f80a4a99051ff438b85/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e1a02f18bc37c32c8c93383a529584ff2d952162e8341f80a4a99051ff438b85/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e9cd4ba94b8ae75459f5b272d45e13d31c4bf152b15886757325f041f4147dc8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e9cd4ba94b8ae75459f5b272d45e13d31c4bf152b15886757325f041f4147dc8/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1ba7d788-14ef-4b39-b6f6-3072ab0b3200/volumes/kubernetes.io~projected/kube-api-access-4pwzt:{mountpoint:/var/lib/kubelet/pods/1ba7d788-14ef-4b39-b6f6-3072ab0b3200/volumes/kubernetes.io~projected/kube-api-access-4pwzt major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2434842e-d96c-4890-9b2a-2a9eb130394a/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/2434842e-d96c-4890-9b2a-2a9eb130394a/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2434842e-d96c-4890-9b2a-2a9eb130394a/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/2434842e-d96c-4890-9b2a-2a9eb130394a/volumes/kubernetes.io~empty-dir/tmp major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2434842e-d96c-4890-9b2a-2a9eb130394a/volumes/kubernetes.io~projected/kube-api-access-2s2zv:{mountpoint:/var/lib/kubelet/pods/2434842e-d96c-4890-9b2a-2a9eb130394a/volumes/kubernetes.io~projected/kube-api-access-2s2zv major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/24421640-34bc-4b15-a096-ffdaa7ae7bf4/volumes/kubernetes.io~projected/kube-api-access-dhcd5:{mountpoint:/var/lib/kubelet/pods/24421640-34bc-4b15-a096-ffdaa7ae7bf4/volumes/kubernetes.io~projected/kube-api-access-dhcd5 major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/252a7edc-651f-4c08-8fd7-f46ec29737c4/volumes/kubernetes.io~projected/kube-api-access-2gxc2:{mountpoint:/var/lib/kubelet/pods/252a7edc-651f-4c08-8fd7-f46ec29737c4/volumes/kubernetes.io~projected/kube-api-access-2gxc2 major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/38c17ac6-93d3-4d38-83db-70d4d0fa3f30/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/38c17ac6-93d3-4d38-83db-70d4d0fa3f30/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/38c17ac6-93d3-4d38-83db-70d4d0fa3f30/volumes/kubernetes.io~projected/kube-api-access-pjpvn:{mountpoint:/var/lib/kubelet/pods/38c17ac6-93d3-4d38-83db-70d4d0fa3f30/volumes/kubernetes.io~projected/kube-api-access-pjpvn major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/38c17ac6-93d3-4d38-83db-70d4d0fa3f30/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/38c17ac6-93d3-4d38-83db-70d4d0fa3f30/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40e1c660-22c0-42e5-8511-4cbd0e170a5c/volumes/kubernetes.io~projected/kube-api-access-5st2s:{mountpoint:/var/lib/kubelet/pods/40e1c660-22c0-42e5-8511-4cbd0e170a5c/volumes/kubernetes.io~projected/kube-api-access-5st2s major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8eab0b7c-14cc-42fa-993f-d52707e42d35/volumes/kubernetes.io~projected/kube-api-access-95mt2:{mountpoint:/var/lib/kubelet/pods/8eab0b7c-14cc-42fa-993f-d52707e42d35/volumes/kubernetes.io~projected/kube-api-access-95mt2 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a21d39c5-3813-4433-9307-855276f1a66e/volumes/kubernetes.io~projected/kube-api-access-4pjzv:{mountpoint:/var/lib/kubelet/pods/a21d39c5-3813-4433-9307-855276f1a66e/volumes/kubernetes.io~projected/kube-api-access-4pjzv major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bbdceb2b-73c7-4501-994c-4e3533fa47d0/volumes/kubernetes.io~projected/kube-api-access-jblxf:{mountpoint:/var/lib/kubelet/pods/bbdceb2b-73c7-4501-994c-4e3533fa47d0/volumes/kubernetes.io~projected/kube-api-access-jblxf major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3/volumes/kubernetes.io~projected/kube-api-access-2whld:{mountpoint:/var/lib/kubelet/pods/bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3/volumes/kubernetes.io~projected/kube-api-access-2whld major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf581b63-6285-4ac9-aa45-cd8a367478f2/volumes/kubernetes.io~projected/kube-api-access-hs8qj:{mountpoint:/var/lib/kubelet/pods/bf581b63-6285-4ac9-aa45-cd8a367478f2/volumes/kubernetes.io~projected/kube-api-access-hs8qj major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf581b63-6285-4ac9-aa45-cd8a367478f2/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/bf581b63-6285-4ac9-aa45-cd8a367478f2/volumes/kubernetes.io~secret/signing-key major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c702aab4-18a3-4b3a-a629-7d87f15747e7/volumes/kubernetes.io~projected/kube-api-access-s78x6:{mountpoint:/var/lib/kubelet/pods/c702aab4-18a3-4b3a-a629-7d87f15747e7/volumes/kubernetes.io~projected/kube-api-access-s78x6 major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd83ecbc-117f-4890-bf7a-cab9282d4484/volumes/kubernetes.io~projected/kube-api-access-7dnnp:{mountpoint:/var/lib/kubelet/pods/cd83ecbc-117f-4890-bf7a-cab9282d4484/volumes/kubernetes.io~projected/kube-api-access-7dnnp major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d24bfb4d-652d-4e7d-bc67-43441164f652/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/d24bfb4d-652d-4e7d-bc67-43441164f652/volumes/kubernetes.io~secret/agent-certs major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d52d0ddc-6bd1-4dcc-9d56-84213c328055/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/d52d0ddc-6bd1-4dcc-9d56-84213c328055/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de08f19f-1712-4864-87b0-294e3d3e460d/volumes/kubernetes.io~projected/kube-api-access-ch8ct:{mountpoint:/var/lib/kubelet/pods/de08f19f-1712-4864-87b0-294e3d3e460d/volumes/kubernetes.io~projected/kube-api-access-ch8ct major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e275e41d-9573-47e2-b923-173dfb4361a7/volumes/kubernetes.io~projected/kube-api-access-vm26j:{mountpoint:/var/lib/kubelet/pods/e275e41d-9573-47e2-b923-173dfb4361a7/volumes/kubernetes.io~projected/kube-api-access-vm26j major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e275e41d-9573-47e2-b923-173dfb4361a7/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/e275e41d-9573-47e2-b923-173dfb4361a7/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e3fcf8dd-4ee4-4037-9952-9c54a49c6738/volumes/kubernetes.io~projected/kube-api-access-mfpbf:{mountpoint:/var/lib/kubelet/pods/e3fcf8dd-4ee4-4037-9952-9c54a49c6738/volumes/kubernetes.io~projected/kube-api-access-mfpbf major:0 minor:54 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/21ef22d13b710897a2fa1eec6467864bece5d8dac618a160371055446007ffc9/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/5000784629af25106aaa63b84bea5cb73527d48a360136dd5283281869b4425d/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/add6d3b2c2432b8caca4954d9f0fd26f6a38b9f6b21c015c1f74906ee11309d0/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-108:{mountpoint:/var/lib/containers/storage/overlay/279f3e03f5c65cbde86178b56b1f9ec17c248dbe2365a6189eb9fd7949ed1e16/merged major:0 minor:108 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/2fab0d7c2eacea11a08f8e1defacfda920f4dd76c475bd269cf7883b6e0d37f9/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/97804e2400faf72f284009dc715eaded66e2845bcdced5390fb6965c0e34780a/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/93078bd646f056f4951f948e014f8c43fb97a74cf680d0252f708da7a4d93b4b/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/86f325eae296207cb4f67de6e4815742a84c8b286c1fe2b04f3e24740ab2e04b/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/110fea903725a837b732502319bbf55e7f37951f0d46618a0a726c771de9681b/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/77e2470ac0e98c0d4b395116bb90bbf4d9066050cd6a68a89e21aa7bff115669/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/a2a13cdd69ee1efca72fb86b846d4f85536993deb33339d359b70b7e1022db4e/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/b7c62064b6e6592ff7812167f35ba1af9a744263163fa4db2ffd3da6ce3f2be6/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/6c304b95c8ee7eca55b50e898402e0c4cfb534d0c645ff03942236231ab031cf/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/7f37d628ca67e133f4e07137d4fbedd127e61f55dd398c19a76e0c59f5309291/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/998ac6e5e00011d048b3148128e923263517b57b4a83360eb263d403a9e2fee7/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/b52be2c6d80a9904056a4bb91e9e98babc384f9aa656f12367d15992c206ea68/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/1cfadfc1dfddc760612da86612fa0e5259487e5ef2b6e1b06d77cc1f03dde8ff/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/f009dd0667c507cfee4589a7b69991d3ae5678a4ce407687adcc69ac5904cec3/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/76216415b8d205f8d256e3a8af36001a75e90617aa97b626f41f453c69988f1f/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/59e6d437a804906b5735f720650acb6142d604857b21b3fcc0e9ff7982b4ef08/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/c520f9e157bdb654b1d1b305a410b508d2da5d02f8c97adf7bc0d51ed58a74a8/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/6a3623f2842fab54c7da6c67fd091bdca21fe7cd8d0124fc3a1ce9b3f4a9f641/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/1deedea06d1dea752e971435899018530676b567b45b46eb87e77c01f06fa9fd/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/3466ae9b437ebc75d15209ebd81fd749a670452fe3e5d0896709f3021c8c3ddc/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/6631608608df932f427a9b8854a44deb0887e60c61ddb9c15271c3652d14f40f/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/ad60385638233a28fbcee5a6dcdae3a539842f7a08db6b913c6e81503f863093/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/3a0a43193010813f20dca6fcf94be5f8b6ab23317d646f7c7d363e227142911f/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/02f3a941cf624c92825f9247626fbb74881c645d8c2e9130ec144708c7cc7c96/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-239:{mountpoint:/var/lib/containers/storage/overlay/5974d6a9ac28e9a2275fa252684113ba1c20a42e80d9934e0a823b71ab79f240/merged major:0 minor:239 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/32566ecec391be2cdb34aa2651584675d679cb728418680d57d1921f06c71ec5/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/b6027ba2239c16a890b9b7465644bff02df96055af845cdfd1a73665c9e763f1/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/1c23c7e7304b5436ce9467e5c9360b7a877c8cc58552e2782eb1c8dff09505d2/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/b2669cb156200c261c1cd91c1cf5ea85619975784fa1633f8643ed7b46dd7a25/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/5d8e923a5ce4f7c520f8daea973a873ac83bbd14862d036437f4845ef1e81dd0/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/5b626e790e80c4eaaaf0df0f9f5cc28588345f416159bd4ed1e386016a2e901f/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/4028a720ea1de7a59e4d741905300de7a19e8dd55b4d4c4986d9b683a34a69e8/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/2881fda510af85a4111438cd104f22e1f1aef53c45ec1aacb74e701cb903076f/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/bd5e8e3213262f648f12167ad5025b907341c8313653489a7788691e32e402c4/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/70154e092b619a492403ea647123488dc738b14a917414b7635f9892952bceea/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/e86966220511553b64468933853924e13aceed5eef76dbb3943dfe0e372d3534/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/85c53ee0a535e15ef86e8c81284799d05015e546fb867921c0e579de65879e12/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/53fe52e506630ba0266d50b80afa27fd0eff3b1b5789fd1b06be3a2cc8f87d93/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/cb61b4ba4e72e068f7b337f25195f51f35a4582a17f7bd06c2d9476f1ec4fd7b/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 16:47:06.086076 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.082137 4522 manager.go:217] Machine: {Timestamp:2026-10-30 16:47:06.081669262 +0000 UTC m=+0.082424155 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3107805 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d6784ae2daf86a731425ba0a818e0 SystemUUID:ec2d6784-ae2d-af86-a731-425ba0a818e0 BootID:e25712fa-9da9-46bd-8c13-27216cc8ebc5 Filesystems:[{Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/38c17ac6-93d3-4d38-83db-70d4d0fa3f30/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:48 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/77f86b5c5ce7f22475eac00b2a7033e5660a5b01cf17fe32625a4b783b19790a/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/95d6d3e2380f9668e7a87195cacc53233ec9bf359ee51a6eb1e66ce13dc91206/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d52d0ddc-6bd1-4dcc-9d56-84213c328055/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:219 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c702aab4-18a3-4b3a-a629-7d87f15747e7/volumes/kubernetes.io~projected/kube-api-access-s78x6 DeviceMajor:0 DeviceMinor:52 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-108 DeviceMajor:0 DeviceMinor:108 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2434842e-d96c-4890-9b2a-2a9eb130394a/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:47 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a21d39c5-3813-4433-9307-855276f1a66e/volumes/kubernetes.io~projected/kube-api-access-4pjzv DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bf581b63-6285-4ac9-aa45-cd8a367478f2/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:225 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/de08f19f-1712-4864-87b0-294e3d3e460d/volumes/kubernetes.io~projected/kube-api-access-ch8ct DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/11e310e8c78c4004c8aa0e76c03c4906cf266079403a7e0fda8db10f0e868c28/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/38c17ac6-93d3-4d38-83db-70d4d0fa3f30/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bf581b63-6285-4ac9-aa45-cd8a367478f2/volumes/kubernetes.io~projected/kube-api-access-hs8qj DeviceMajor:0 DeviceMinor:226 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d24bfb4d-652d-4e7d-bc67-43441164f652/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:49 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2434842e-d96c-4890-9b2a-2a9eb130394a/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:50 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e275e41d-9573-47e2-b923-173dfb4361a7/volumes/kubernetes.io~projected/kube-api-access-vm26j DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d4ec855f9a431818213116d72030492004afb27e6e9a5f1fae58b7223a06cfe3/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e3fcf8dd-4ee4-4037-9952-9c54a49c6738/volumes/kubernetes.io~projected/kube-api-access-mfpbf DeviceMajor:0 DeviceMinor:54 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e1a02f18bc37c32c8c93383a529584ff2d952162e8341f80a4a99051ff438b85/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6129a8fc729d736a455e6fc90d5fb7d979feaee41cf41276d3810ce84f8b900b/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cd83ecbc-117f-4890-bf7a-cab9282d4484/volumes/kubernetes.io~projected/kube-api-access-7dnnp DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a737e28cca11d4405dfef3adba95876fa07f4117d846e29ac7515304d0e46c62/userdata/shm DeviceMajor:0 DeviceMinor:211 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3/volumes/kubernetes.io~projected/kube-api-access-2whld DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/252a7edc-651f-4c08-8fd7-f46ec29737c4/volumes/kubernetes.io~projected/kube-api-access-2gxc2 DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-239 DeviceMajor:0 DeviceMinor:239 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e9cd4ba94b8ae75459f5b272d45e13d31c4bf152b15886757325f041f4147dc8/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b1ebd56f579b03a320cb5b7cbc25f067b7169123bed80d2a34ae0af14f48283e/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1ba7d788-14ef-4b39-b6f6-3072ab0b3200/volumes/kubernetes.io~projected/kube-api-access-4pwzt DeviceMajor:0 DeviceMinor:51 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2434842e-d96c-4890-9b2a-2a9eb130394a/volumes/kubernetes.io~projected/kube-api-access-2s2zv DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6fc7bce1f36d0e8e13dc09c5275b70fe8ea2790f94977fde82f4fa803951bcfc/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/40e1c660-22c0-42e5-8511-4cbd0e170a5c/volumes/kubernetes.io~projected/kube-api-access-5st2s DeviceMajor:0 DeviceMinor:224 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/38c17ac6-93d3-4d38-83db-70d4d0fa3f30/volumes/kubernetes.io~projected/kube-api-access-pjpvn DeviceMajor:0 DeviceMinor:53 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0423717bb045b25f4e574b686e6564701d7df72979a3bfff1359674754052774/userdata/shm DeviceMajor:0 DeviceMinor:227 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/24421640-34bc-4b15-a096-ffdaa7ae7bf4/volumes/kubernetes.io~projected/kube-api-access-dhcd5 DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8eab0b7c-14cc-42fa-993f-d52707e42d35/volumes/kubernetes.io~projected/kube-api-access-95mt2 DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/73cfaf3d1a1d8a84ba37bfd1f1c2358dfda8ce270736512c278c313aae6bbeb6/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5f90a5b2e8450d0fe35942286c1d9f96483f222fca4a6ec99a7a8061df7d6fc8/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bc75e72cf1463bccb83e5933bc73e9cc0b3b99aed55f7cd0fe3371a4924f6271/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4d74c110ad70b58f43e0b1a60d4379a8287fdeefadb7e639e03c3e7f1459fcae/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e275e41d-9573-47e2-b923-173dfb4361a7/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bbdceb2b-73c7-4501-994c-4e3533fa47d0/volumes/kubernetes.io~projected/kube-api-access-jblxf DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/962d4c9822b1abc27c25eb93a60baaf8beaf09e16b434e63802ff7732ec3e560/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0423717bb045b25 MacAddress:b2:70:cc:48:56:a1 Speed:10000 Mtu:8901} {Name:4d74c110ad70b58 MacAddress:16:62:12:5b:19:10 Speed:10000 Mtu:8901} {Name:6fc7bce1f36d0e8 MacAddress:66:15:53:4b:2f:7a Speed:10000 Mtu:8901} {Name:a737e28cca11d44 MacAddress:7e:1e:47:ee:1f:cf Speed:10000 Mtu:8901} {Name:bc75e72cf1463bc MacAddress:72:f6:07:19:23:9e Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d4:19:d5:a5 Speed:0 Mtu:9001} {Name:br-int MacAddress:3a:00:6c:5d:0c:9f Speed:0 Mtu:8901} {Name:d4ec855f9a43181 MacAddress:4e:d7:6d:7e:74:85 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:d4:19:d5:a5 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:de:97:18:72:c7:b1 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:0e:a9:5c:d8:d5:fd Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 16:47:06.086076 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.082416 4522 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 16:47:06.086076 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.082463 4522 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 16:47:06.086076 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.082803 4522 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 16:47:06.086076 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.082827 4522 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-218.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 16:47:06.086076 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.083030 4522 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 16:47:06.086076 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.083042 4522 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 16:47:06.086076 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.083061 4522 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 16:47:06.086076 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.083080 4522 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 16:47:06.086076 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.083215 4522 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 16:47:06.086076 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.083464 4522 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 16:47:06.086076 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.083641 4522 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 16:47:06.086076 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.083686 4522 kubelet.go:493] "Attempting to sync node with API server" Oct 30 16:47:06.086076 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.083700 4522 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 16:47:06.086076 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.083713 4522 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 16:47:06.086076 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.083727 4522 kubelet.go:405] "Adding apiserver pod source" Oct 30 16:47:06.086076 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.083740 4522 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 16:47:06.086076 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.084874 4522 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 16:47:06.086076 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.084891 4522 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 16:47:06.086696 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.086084 4522 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 16:47:06.094861 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.094843 4522 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 16:47:06.094861 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.094858 4522 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 16:47:06.094981 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.094971 4522 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 16:47:06.095216 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.095205 4522 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 16:47:06.095251 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.095235 4522 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 16:47:06.095358 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.095348 4522 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 16:47:06.095389 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.095363 4522 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 16:47:06.095389 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.095370 4522 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 16:47:06.095389 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.095375 4522 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 16:47:06.095389 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.095381 4522 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 16:47:06.095389 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.095387 4522 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 16:47:06.095519 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.095394 4522 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 16:47:06.095519 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.095403 4522 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 16:47:06.095519 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.095410 4522 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 16:47:06.095519 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.095415 4522 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 16:47:06.095519 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.095429 4522 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 16:47:06.095519 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.095438 4522 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 16:47:06.095519 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.095462 4522 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 16:47:06.095519 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.095468 4522 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 16:47:06.098892 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.098880 4522 server.go:1263] "Started kubelet" Oct 30 16:47:06.099259 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.098985 4522 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 16:47:06.099259 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.099116 4522 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 16:47:06.099259 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.099128 4522 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 16:47:06.099474 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.099436 4522 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 16:47:06.099471 ip-10-0-128-218 systemd[1]: Started Kubernetes Kubelet. Oct 30 16:47:06.101346 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.100590 4522 server.go:317] "Adding debug handlers to kubelet server" Oct 30 16:47:06.101690 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.101673 4522 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 16:47:06.101796 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.101694 4522 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 16:47:06.102548 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.101773 4522 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 16:41:27 +0000 UTC" deadline="2028-06-30 07:24:42.892307181 +0000 UTC" Oct 30 16:47:06.102548 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.101896 4522 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14606h37m36.790415506s" Oct 30 16:47:06.102548 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.101832 4522 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 16:47:06.102548 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.101824 4522 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 16:47:06.102548 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.101924 4522 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 16:47:06.102548 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:47:06.102408 4522 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-218.ec2.internal\" not found" Oct 30 16:47:06.104214 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.103975 4522 factory.go:55] Registering systemd factory Oct 30 16:47:06.104214 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.103997 4522 factory.go:223] Registration of the systemd container factory successfully Oct 30 16:47:06.110286 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.104612 4522 factory.go:153] Registering CRI-O factory Oct 30 16:47:06.110286 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.104630 4522 factory.go:223] Registration of the crio container factory successfully Oct 30 16:47:06.110286 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.104697 4522 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 16:47:06.110286 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.104716 4522 factory.go:103] Registering Raw factory Oct 30 16:47:06.110286 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.104733 4522 manager.go:1199] Started watching for new ooms in manager Oct 30 16:47:06.110286 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.104708 4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf581b63-6285-4ac9-aa45-cd8a367478f2" volumeName="kubernetes.io/projected/bf581b63-6285-4ac9-aa45-cd8a367478f2-kube-api-access-hs8qj" seLinuxMountContext="" Oct 30 16:47:06.110286 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.104753 4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d24bfb4d-652d-4e7d-bc67-43441164f652" volumeName="kubernetes.io/configmap/d24bfb4d-652d-4e7d-bc67-43441164f652-konnectivity-ca" seLinuxMountContext="" Oct 30 16:47:06.110286 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.104766 4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d52d0ddc-6bd1-4dcc-9d56-84213c328055" volumeName="kubernetes.io/secret/d52d0ddc-6bd1-4dcc-9d56-84213c328055-original-pull-secret" seLinuxMountContext="" Oct 30 16:47:06.110286 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.104779 4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de08f19f-1712-4864-87b0-294e3d3e460d" volumeName="kubernetes.io/configmap/de08f19f-1712-4864-87b0-294e3d3e460d-cni-binary-copy" seLinuxMountContext="" Oct 30 16:47:06.110286 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.104790 4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a21d39c5-3813-4433-9307-855276f1a66e" volumeName="kubernetes.io/projected/a21d39c5-3813-4433-9307-855276f1a66e-kube-api-access-4pjzv" seLinuxMountContext="" Oct 30 16:47:06.110286 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.104802 4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2434842e-d96c-4890-9b2a-2a9eb130394a" volumeName="kubernetes.io/empty-dir/2434842e-d96c-4890-9b2a-2a9eb130394a-tmp" seLinuxMountContext="" Oct 30 16:47:06.110286 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.104817 4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24421640-34bc-4b15-a096-ffdaa7ae7bf4" volumeName="kubernetes.io/configmap/24421640-34bc-4b15-a096-ffdaa7ae7bf4-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 16:47:06.110286 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.104829 4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="252a7edc-651f-4c08-8fd7-f46ec29737c4" volumeName="kubernetes.io/configmap/252a7edc-651f-4c08-8fd7-f46ec29737c4-service-ca-bundle" seLinuxMountContext="" Oct 30 16:47:06.110286 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.104845 4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40e1c660-22c0-42e5-8511-4cbd0e170a5c" volumeName="kubernetes.io/projected/40e1c660-22c0-42e5-8511-4cbd0e170a5c-kube-api-access-5st2s" seLinuxMountContext="" Oct 30 16:47:06.110286 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.104860 4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf581b63-6285-4ac9-aa45-cd8a367478f2" volumeName="kubernetes.io/secret/bf581b63-6285-4ac9-aa45-cd8a367478f2-signing-key" seLinuxMountContext="" Oct 30 16:47:06.110286 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.104873 4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd83ecbc-117f-4890-bf7a-cab9282d4484" volumeName="kubernetes.io/projected/cd83ecbc-117f-4890-bf7a-cab9282d4484-kube-api-access-7dnnp" seLinuxMountContext="" Oct 30 16:47:06.110286 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.104884 4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de08f19f-1712-4864-87b0-294e3d3e460d" volumeName="kubernetes.io/configmap/de08f19f-1712-4864-87b0-294e3d3e460d-multus-daemon-config" seLinuxMountContext="" Oct 30 16:47:06.110286 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.104906 4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e275e41d-9573-47e2-b923-173dfb4361a7" volumeName="kubernetes.io/secret/e275e41d-9573-47e2-b923-173dfb4361a7-cert" seLinuxMountContext="" Oct 30 16:47:06.110286 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.104919 4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24421640-34bc-4b15-a096-ffdaa7ae7bf4" volumeName="kubernetes.io/configmap/24421640-34bc-4b15-a096-ffdaa7ae7bf4-cni-binary-copy" seLinuxMountContext="" Oct 30 16:47:06.110286 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.104950 4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="252a7edc-651f-4c08-8fd7-f46ec29737c4" volumeName="kubernetes.io/configmap/252a7edc-651f-4c08-8fd7-f46ec29737c4-trusted-ca-bundle" seLinuxMountContext="" Oct 30 16:47:06.110286 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.104965 4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38c17ac6-93d3-4d38-83db-70d4d0fa3f30" volumeName="kubernetes.io/secret/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 16:47:06.110286 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.104978 4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbdceb2b-73c7-4501-994c-4e3533fa47d0" volumeName="kubernetes.io/projected/bbdceb2b-73c7-4501-994c-4e3533fa47d0-kube-api-access-jblxf" seLinuxMountContext="" Oct 30 16:47:06.110286 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.104991 4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3" volumeName="kubernetes.io/projected/bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3-kube-api-access-2whld" seLinuxMountContext="" Oct 30 16:47:06.110286 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.105004 4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d24bfb4d-652d-4e7d-bc67-43441164f652" volumeName="kubernetes.io/secret/d24bfb4d-652d-4e7d-bc67-43441164f652-agent-certs" seLinuxMountContext="" Oct 30 16:47:06.110286 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.105017 4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24421640-34bc-4b15-a096-ffdaa7ae7bf4" volumeName="kubernetes.io/configmap/24421640-34bc-4b15-a096-ffdaa7ae7bf4-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 16:47:06.110286 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.105030 4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="252a7edc-651f-4c08-8fd7-f46ec29737c4" volumeName="kubernetes.io/projected/252a7edc-651f-4c08-8fd7-f46ec29737c4-kube-api-access-2gxc2" seLinuxMountContext="" Oct 30 16:47:06.110286 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.105044 4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38c17ac6-93d3-4d38-83db-70d4d0fa3f30" volumeName="kubernetes.io/configmap/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-ovnkube-config" seLinuxMountContext="" Oct 30 16:47:06.110286 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.105057 4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c702aab4-18a3-4b3a-a629-7d87f15747e7" volumeName="kubernetes.io/projected/c702aab4-18a3-4b3a-a629-7d87f15747e7-kube-api-access-s78x6" seLinuxMountContext="" Oct 30 16:47:06.110286 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.105069 4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2434842e-d96c-4890-9b2a-2a9eb130394a" volumeName="kubernetes.io/projected/2434842e-d96c-4890-9b2a-2a9eb130394a-kube-api-access-2s2zv" seLinuxMountContext="" Oct 30 16:47:06.110286 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.105090 4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38c17ac6-93d3-4d38-83db-70d4d0fa3f30" volumeName="kubernetes.io/configmap/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-env-overrides" seLinuxMountContext="" Oct 30 16:47:06.110286 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.105124 4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38c17ac6-93d3-4d38-83db-70d4d0fa3f30" volumeName="kubernetes.io/configmap/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-ovnkube-script-lib" seLinuxMountContext="" Oct 30 16:47:06.110286 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.105138 4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3" volumeName="kubernetes.io/configmap/bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3-config-volume" seLinuxMountContext="" Oct 30 16:47:06.110286 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.105149 4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c702aab4-18a3-4b3a-a629-7d87f15747e7" volumeName="kubernetes.io/empty-dir/c702aab4-18a3-4b3a-a629-7d87f15747e7-tmp-dir" seLinuxMountContext="" Oct 30 16:47:06.110286 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.105165 4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de08f19f-1712-4864-87b0-294e3d3e460d" volumeName="kubernetes.io/projected/de08f19f-1712-4864-87b0-294e3d3e460d-kube-api-access-ch8ct" seLinuxMountContext="" Oct 30 16:47:06.110286 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.105176 4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3fcf8dd-4ee4-4037-9952-9c54a49c6738" volumeName="kubernetes.io/projected/e3fcf8dd-4ee4-4037-9952-9c54a49c6738-kube-api-access-mfpbf" seLinuxMountContext="" Oct 30 16:47:06.110286 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.105209 4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24421640-34bc-4b15-a096-ffdaa7ae7bf4" volumeName="kubernetes.io/projected/24421640-34bc-4b15-a096-ffdaa7ae7bf4-kube-api-access-dhcd5" seLinuxMountContext="" Oct 30 16:47:06.110286 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.105221 4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38c17ac6-93d3-4d38-83db-70d4d0fa3f30" volumeName="kubernetes.io/projected/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-kube-api-access-pjpvn" seLinuxMountContext="" Oct 30 16:47:06.110286 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.105231 4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3" volumeName="kubernetes.io/empty-dir/bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3-tmp-dir" seLinuxMountContext="" Oct 30 16:47:06.110286 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.105241 4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf581b63-6285-4ac9-aa45-cd8a367478f2" volumeName="kubernetes.io/configmap/bf581b63-6285-4ac9-aa45-cd8a367478f2-signing-cabundle" seLinuxMountContext="" Oct 30 16:47:06.110286 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.105251 4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e275e41d-9573-47e2-b923-173dfb4361a7" volumeName="kubernetes.io/projected/e275e41d-9573-47e2-b923-173dfb4361a7-kube-api-access-vm26j" seLinuxMountContext="" Oct 30 16:47:06.110286 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.105261 4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ba7d788-14ef-4b39-b6f6-3072ab0b3200" volumeName="kubernetes.io/projected/1ba7d788-14ef-4b39-b6f6-3072ab0b3200-kube-api-access-4pwzt" seLinuxMountContext="" Oct 30 16:47:06.110286 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.105272 4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2434842e-d96c-4890-9b2a-2a9eb130394a" volumeName="kubernetes.io/empty-dir/2434842e-d96c-4890-9b2a-2a9eb130394a-etc-tuned" seLinuxMountContext="" Oct 30 16:47:06.110286 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.105282 4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8eab0b7c-14cc-42fa-993f-d52707e42d35" volumeName="kubernetes.io/configmap/8eab0b7c-14cc-42fa-993f-d52707e42d35-iptables-alerter-script" seLinuxMountContext="" Oct 30 16:47:06.110286 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.105294 4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a21d39c5-3813-4433-9307-855276f1a66e" volumeName="kubernetes.io/empty-dir/a21d39c5-3813-4433-9307-855276f1a66e-data-volume" seLinuxMountContext="" Oct 30 16:47:06.110286 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.105140 4522 manager.go:319] Starting recovery of all containers Oct 30 16:47:06.110286 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.105324 4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="252a7edc-651f-4c08-8fd7-f46ec29737c4" volumeName="kubernetes.io/empty-dir/252a7edc-651f-4c08-8fd7-f46ec29737c4-archives-path" seLinuxMountContext="" Oct 30 16:47:06.110286 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.106294 4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8eab0b7c-14cc-42fa-993f-d52707e42d35" volumeName="kubernetes.io/projected/8eab0b7c-14cc-42fa-993f-d52707e42d35-kube-api-access-95mt2" seLinuxMountContext="" Oct 30 16:47:06.110286 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.106344 4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbdceb2b-73c7-4501-994c-4e3533fa47d0" volumeName="kubernetes.io/configmap/bbdceb2b-73c7-4501-994c-4e3533fa47d0-serviceca" seLinuxMountContext="" Oct 30 16:47:06.110286 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.106358 4522 reconstruct.go:97] "Volume reconstruction finished" Oct 30 16:47:06.110286 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.106365 4522 reconciler.go:29] "Reconciler: start to sync state" Oct 30 16:47:06.117592 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.117551 4522 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 16:47:06.136621 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.136593 4522 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 16:47:06.136621 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.136615 4522 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 16:47:06.136621 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.136631 4522 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 16:47:06.136879 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:47:06.136673 4522 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 16:47:06.147212 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.147171 4522 generic.go:356] "Generic (PLEG): container finished" podID="24421640-34bc-4b15-a096-ffdaa7ae7bf4" containerID="bc333c854d4125cddd920936057338307f57f80f6675cd3a5b584ac2806523c0" exitCode=0 Oct 30 16:47:06.147212 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.147207 4522 generic.go:356] "Generic (PLEG): container finished" podID="24421640-34bc-4b15-a096-ffdaa7ae7bf4" containerID="4a81656e1e3509ac5f1e018a9df075c4da66a4b6e867c1747e636bbc1743dbce" exitCode=0 Oct 30 16:47:06.147212 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.147215 4522 generic.go:356] "Generic (PLEG): container finished" podID="24421640-34bc-4b15-a096-ffdaa7ae7bf4" containerID="102a6ca36ff2eada3328d6a51c4bd45cb531a99576a95bb9c6826e0de3ff03eb" exitCode=0 Oct 30 16:47:06.147435 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.147224 4522 generic.go:356] "Generic (PLEG): container finished" podID="24421640-34bc-4b15-a096-ffdaa7ae7bf4" containerID="ea6af0b0df933cdd7c2fc7ef5aed2fb7e5d277ab3c819b7731f07dc7c995e2a8" exitCode=0 Oct 30 16:47:06.147435 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.147235 4522 generic.go:356] "Generic (PLEG): container finished" podID="24421640-34bc-4b15-a096-ffdaa7ae7bf4" containerID="8fabfe8d24a91c7077d80f902fc2e6a09ca9502357b6e1346fca31bdfccb3b4b" exitCode=0 Oct 30 16:47:06.147435 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.147246 4522 generic.go:356] "Generic (PLEG): container finished" podID="24421640-34bc-4b15-a096-ffdaa7ae7bf4" containerID="223f5c9b874a8f0215a13fe5d178c0d4fccf81262403c832e3084c47b2e18a15" exitCode=0 Oct 30 16:47:06.147785 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.147767 4522 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:47:06.160599 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.160578 4522 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 16:47:06.163860 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.163832 4522 generic.go:356] "Generic (PLEG): container finished" podID="91966e7831263cb270802b19c1162280" containerID="59b46fe6ab73bab7f86b93c2892e3ce465bd9d23699d52658fbe6373450e0b73" exitCode=0 Oct 30 16:47:06.174209 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.174188 4522 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:47:06.174943 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.174924 4522 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:47:06.175040 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.175004 4522 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:47:06.184839 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.184793 4522 manager.go:324] Recovery completed Oct 30 16:47:06.199732 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.199708 4522 cpu_manager.go:226] "Starting" policy="none" Oct 30 16:47:06.199732 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.199721 4522 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 16:47:06.199732 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.199738 4522 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 16:47:06.199942 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.199855 4522 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 16:47:06.199942 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.199865 4522 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 16:47:06.199942 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.199882 4522 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 16:47:06.199942 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.199888 4522 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 16:47:06.199942 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.199893 4522 policy_none.go:50] "Start" Oct 30 16:47:06.199942 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.199901 4522 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 16:47:06.199942 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.199910 4522 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 16:47:06.200130 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.200064 4522 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 16:47:06.200130 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.200071 4522 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 16:47:06.200130 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.200077 4522 policy_none.go:44] "Start" Oct 30 16:47:06.204048 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.204024 4522 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 16:47:06.207395 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.207380 4522 manager.go:341] "Starting Device Plugin manager" Oct 30 16:47:06.207858 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:47:06.207412 4522 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 16:47:06.207927 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.207863 4522 server.go:87] "Starting device plugin registration server" Oct 30 16:47:06.208144 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.208131 4522 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 16:47:06.208216 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.208145 4522 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 16:47:06.208262 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.208240 4522 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 16:47:06.208405 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.208364 4522 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 16:47:06.208405 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.208410 4522 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 16:47:06.208559 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.208415 4522 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 16:47:06.208559 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.208471 4522 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T16:47:06.208398882Z","UUID":"a2d21241-e850-45eb-bb76-037f75c1528a","Handler":null,"Name":"","Endpoint":""} Oct 30 16:47:06.210023 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:47:06.209997 4522 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 16:47:06.210231 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.210213 4522 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 16:47:06.210335 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.210245 4522 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 16:47:06.237275 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.237228 4522 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-218.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-218.ec2.internal"] Oct 30 16:47:06.237776 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.237743 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-218.ec2.internal" event={"ID":"bc58f75ef247c6cbf4faad53957c9f43","Type":"ContainerStarted","Data":"f3a2d8f02c3a23239a13368fe6de19760eebb489b8e06bd3717faaf7eb7d40d8"} Oct 30 16:47:06.237830 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.237783 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-218.ec2.internal" event={"ID":"bc58f75ef247c6cbf4faad53957c9f43","Type":"ContainerStarted","Data":"77f86b5c5ce7f22475eac00b2a7033e5660a5b01cf17fe32625a4b783b19790a"} Oct 30 16:47:06.237830 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.237796 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-218.ec2.internal" event={"ID":"91966e7831263cb270802b19c1162280","Type":"ContainerStarted","Data":"7ee8f2b1b704ba67980d54a714ab497fac0f29487c6a5a655a9ea590a2813899"} Oct 30 16:47:06.237830 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.237809 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-218.ec2.internal" event={"ID":"91966e7831263cb270802b19c1162280","Type":"ContainerDied","Data":"59b46fe6ab73bab7f86b93c2892e3ce465bd9d23699d52658fbe6373450e0b73"} Oct 30 16:47:06.237830 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.237816 4522 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-218.ec2.internal" Oct 30 16:47:06.237830 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.237825 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-218.ec2.internal" event={"ID":"91966e7831263cb270802b19c1162280","Type":"ContainerStarted","Data":"73cfaf3d1a1d8a84ba37bfd1f1c2358dfda8ce270736512c278c313aae6bbeb6"} Oct 30 16:47:06.238070 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.237887 4522 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-218.ec2.internal" Oct 30 16:47:06.258625 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.258602 4522 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 16:47:06.258725 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:47:06.258651 4522 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-218.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-218.ec2.internal" Oct 30 16:47:06.269301 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.269279 4522 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 16:47:06.269395 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:47:06.269334 4522 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-218.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-218.ec2.internal" Oct 30 16:47:06.308398 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.308366 4522 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:47:06.309652 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.309634 4522 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-218.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:47:06.309732 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.309670 4522 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-218.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:47:06.309732 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.309682 4522 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-218.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:47:06.309732 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.309723 4522 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-218.ec2.internal" Oct 30 16:47:06.347440 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.347418 4522 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-128-218.ec2.internal" Oct 30 16:47:06.347567 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.347481 4522 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-218.ec2.internal" Oct 30 16:47:06.361343 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.361297 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bc58f75ef247c6cbf4faad53957c9f43-config\") pod \"kube-apiserver-proxy-ip-10-0-128-218.ec2.internal\" (UID: \"bc58f75ef247c6cbf4faad53957c9f43\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-218.ec2.internal" Oct 30 16:47:06.361446 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.361347 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/91966e7831263cb270802b19c1162280-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-218.ec2.internal\" (UID: \"91966e7831263cb270802b19c1162280\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-218.ec2.internal" Oct 30 16:47:06.361446 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.361375 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91966e7831263cb270802b19c1162280-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-218.ec2.internal\" (UID: \"91966e7831263cb270802b19c1162280\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-218.ec2.internal" Oct 30 16:47:06.461830 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.461799 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bc58f75ef247c6cbf4faad53957c9f43-config\") pod \"kube-apiserver-proxy-ip-10-0-128-218.ec2.internal\" (UID: \"bc58f75ef247c6cbf4faad53957c9f43\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-218.ec2.internal" Oct 30 16:47:06.461978 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.461834 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/91966e7831263cb270802b19c1162280-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-218.ec2.internal\" (UID: \"91966e7831263cb270802b19c1162280\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-218.ec2.internal" Oct 30 16:47:06.461978 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.461858 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91966e7831263cb270802b19c1162280-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-218.ec2.internal\" (UID: \"91966e7831263cb270802b19c1162280\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-218.ec2.internal" Oct 30 16:47:06.461978 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.461908 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91966e7831263cb270802b19c1162280-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-218.ec2.internal\" (UID: \"91966e7831263cb270802b19c1162280\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-218.ec2.internal" Oct 30 16:47:06.461978 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.461933 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/91966e7831263cb270802b19c1162280-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-218.ec2.internal\" (UID: \"91966e7831263cb270802b19c1162280\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-218.ec2.internal" Oct 30 16:47:06.461978 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:06.461908 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bc58f75ef247c6cbf4faad53957c9f43-config\") pod \"kube-apiserver-proxy-ip-10-0-128-218.ec2.internal\" (UID: \"bc58f75ef247c6cbf4faad53957c9f43\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-218.ec2.internal" Oct 30 16:47:07.084734 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.084696 4522 apiserver.go:52] "Watching apiserver" Oct 30 16:47:07.129888 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.129853 4522 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 16:47:07.130343 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.130302 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-218.ec2.internal","openshift-ovn-kubernetes/ovnkube-node-ns7rp","kube-system/global-pull-secret-syncer-2kbm7","openshift-ingress-canary/ingress-canary-cqjj8","openshift-dns/node-resolver-mbwdh","openshift-insights/periodic-gathering-f4j47-pgf7r","openshift-network-operator/iptables-alerter-v79c8","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftqbf","openshift-kube-storage-version-migrator/migrator-66df5d496b-zm2rj","openshift-multus/network-metrics-daemon-rsgvx","openshift-multus/multus-n7lg2","openshift-service-ca/service-ca-5fffc97bdf-hng8x","openshift-cluster-node-tuning-operator/tuned-mlkjb","kube-system/konnectivity-agent-mqhg8","openshift-image-registry/node-ca-7dcvf","openshift-insights/insights-runtime-extractor-4wjcq","openshift-multus/multus-additional-cni-plugins-bll4b","openshift-dns/dns-default-wc9zw","openshift-network-diagnostics/network-check-target-8zqlg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-218.ec2.internal"] Oct 30 16:47:07.130743 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.130729 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsgvx" Oct 30 16:47:07.138482 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.138456 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wc9zw" Oct 30 16:47:07.139285 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.139265 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4wjcq" Oct 30 16:47:07.149629 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.149608 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 16:47:07.149789 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.149775 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 16:47:07.149886 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.149820 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 16:47:07.149954 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.149928 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-fxkbf\"" Oct 30 16:47:07.149954 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.149940 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 16:47:07.150021 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.149945 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 16:47:07.150190 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.150177 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 16:47:07.150227 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.150205 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 16:47:07.150912 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.150898 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6bljg\"" Oct 30 16:47:07.150968 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.150929 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9bdrw\"" Oct 30 16:47:07.150968 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.150927 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 16:47:07.151061 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.151049 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 16:47:07.151134 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.151119 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 16:47:07.151176 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.151136 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-szhrj\"" Oct 30 16:47:07.151230 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.151209 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 16:47:07.151396 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.151382 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 16:47:07.151396 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.151390 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 16:47:07.151919 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.151903 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 16:47:07.151991 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.151946 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 16:47:07.151991 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.151977 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 16:47:07.162256 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.162239 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wd4nn\"" Oct 30 16:47:07.162390 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.162291 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-j4x52\"" Oct 30 16:47:07.162444 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.162404 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 16:47:07.163152 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.163132 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7stlv\"" Oct 30 16:47:07.163255 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.163153 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 16:47:07.163255 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.163221 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 16:47:07.163255 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.163226 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 16:47:07.163420 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.163271 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ffzh4\"" Oct 30 16:47:07.163420 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.163285 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-nwhz8\"" Oct 30 16:47:07.163420 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.163292 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-8568q\"" Oct 30 16:47:07.163420 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.163326 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-pdblj\"" Oct 30 16:47:07.163420 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.163340 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 16:47:07.163420 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.163355 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-vmkvz\"" Oct 30 16:47:07.163665 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.163425 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-z7xnd\"" Oct 30 16:47:07.163665 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.163426 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-f7c68\"" Oct 30 16:47:07.163665 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.163455 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-f9wpv\"" Oct 30 16:47:07.163665 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.163487 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 16:47:07.163665 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.163509 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 16:47:07.163665 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.163569 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-l499l\"" Oct 30 16:47:07.163974 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.163670 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8fhb8\"" Oct 30 16:47:07.163974 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.163724 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rbwcw\"" Oct 30 16:47:07.167533 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.167517 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 16:47:07.170150 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.170132 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 16:47:07.170235 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.170149 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 16:47:07.170235 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.170188 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 16:47:07.170235 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.170214 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 16:47:07.170235 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.170190 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 16:47:07.170402 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.170250 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:47:07.170402 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.170278 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 16:47:07.170587 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.170567 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 16:47:07.170675 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.170613 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 16:47:07.170675 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.170620 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 16:47:07.170675 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.170631 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:47:07.170675 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.170674 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 16:47:07.170849 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.170686 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 16:47:07.170849 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.170695 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 16:47:07.170849 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.170697 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 16:47:07.170849 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.170763 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 16:47:07.170849 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.170810 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 16:47:07.171069 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.170857 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 16:47:07.171069 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.170929 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 16:47:07.171205 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.171193 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 16:47:07.171360 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.171347 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 16:47:07.171431 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.171420 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 16:47:07.171637 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.171613 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 16:47:07.173472 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.173457 4522 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-218.ec2.internal" Oct 30 16:47:07.173606 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.173593 4522 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-218.ec2.internal" Oct 30 16:47:07.176982 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.176968 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 16:47:07.200078 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.200045 4522 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 16:47:07.200245 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:47:07.200125 4522 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-218.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-218.ec2.internal" Oct 30 16:47:07.201381 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.201360 4522 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 16:47:07.201475 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:47:07.201437 4522 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-218.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-218.ec2.internal" Oct 30 16:47:07.205979 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.205957 4522 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 16:47:07.265725 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.265689 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-host-cni-netd\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:47:07.265725 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.265729 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dnnp\" (UniqueName: \"kubernetes.io/projected/cd83ecbc-117f-4890-bf7a-cab9282d4484-kube-api-access-7dnnp\") pod \"migrator-66df5d496b-zm2rj\" (UID: \"cd83ecbc-117f-4890-bf7a-cab9282d4484\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zm2rj" Oct 30 16:47:07.265969 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.265758 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3-config-volume\") pod \"dns-default-wc9zw\" (UID: \"bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3\") " pod="openshift-dns/dns-default-wc9zw" Oct 30 16:47:07.265969 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.265816 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfpbf\" (UniqueName: \"kubernetes.io/projected/e3fcf8dd-4ee4-4037-9952-9c54a49c6738-kube-api-access-mfpbf\") pod \"aws-ebs-csi-driver-node-ftqbf\" (UID: \"e3fcf8dd-4ee4-4037-9952-9c54a49c6738\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftqbf" Oct 30 16:47:07.265969 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.265882 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s2zv\" (UniqueName: \"kubernetes.io/projected/2434842e-d96c-4890-9b2a-2a9eb130394a-kube-api-access-2s2zv\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:47:07.265969 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.265908 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d52d0ddc-6bd1-4dcc-9d56-84213c328055-dbus\") pod \"global-pull-secret-syncer-2kbm7\" (UID: \"d52d0ddc-6bd1-4dcc-9d56-84213c328055\") " pod="kube-system/global-pull-secret-syncer-2kbm7" Oct 30 16:47:07.265969 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.265933 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bbdceb2b-73c7-4501-994c-4e3533fa47d0-serviceca\") pod \"node-ca-7dcvf\" (UID: \"bbdceb2b-73c7-4501-994c-4e3533fa47d0\") " pod="openshift-image-registry/node-ca-7dcvf" Oct 30 16:47:07.265969 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.265958 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-hostroot\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:47:07.266291 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.265981 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/bf581b63-6285-4ac9-aa45-cd8a367478f2-signing-cabundle\") pod \"service-ca-5fffc97bdf-hng8x\" (UID: \"bf581b63-6285-4ac9-aa45-cd8a367478f2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hng8x" Oct 30 16:47:07.266291 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.266042 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/24421640-34bc-4b15-a096-ffdaa7ae7bf4-cnibin\") pod \"multus-additional-cni-plugins-bll4b\" (UID: \"24421640-34bc-4b15-a096-ffdaa7ae7bf4\") " pod="openshift-multus/multus-additional-cni-plugins-bll4b" Oct 30 16:47:07.266291 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.266078 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/24421640-34bc-4b15-a096-ffdaa7ae7bf4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bll4b\" (UID: \"24421640-34bc-4b15-a096-ffdaa7ae7bf4\") " pod="openshift-multus/multus-additional-cni-plugins-bll4b" Oct 30 16:47:07.266291 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.266103 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/24421640-34bc-4b15-a096-ffdaa7ae7bf4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bll4b\" (UID: \"24421640-34bc-4b15-a096-ffdaa7ae7bf4\") " pod="openshift-multus/multus-additional-cni-plugins-bll4b" Oct 30 16:47:07.266291 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.266124 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2whld\" (UniqueName: \"kubernetes.io/projected/bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3-kube-api-access-2whld\") pod \"dns-default-wc9zw\" (UID: \"bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3\") " pod="openshift-dns/dns-default-wc9zw" Oct 30 16:47:07.266291 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.266125 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3-config-volume\") pod \"dns-default-wc9zw\" (UID: \"bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3\") " pod="openshift-dns/dns-default-wc9zw" Oct 30 16:47:07.266291 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.266189 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-etc-sysctl-conf\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:47:07.266291 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.266264 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-lib-modules\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:47:07.266291 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.266289 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/bf581b63-6285-4ac9-aa45-cd8a367478f2-signing-cabundle\") pod \"service-ca-5fffc97bdf-hng8x\" (UID: \"bf581b63-6285-4ac9-aa45-cd8a367478f2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hng8x" Oct 30 16:47:07.266719 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.266329 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/252a7edc-651f-4c08-8fd7-f46ec29737c4-trusted-ca-bundle\") pod \"periodic-gathering-f4j47-pgf7r\" (UID: \"252a7edc-651f-4c08-8fd7-f46ec29737c4\") " pod="openshift-insights/periodic-gathering-f4j47-pgf7r" Oct 30 16:47:07.266719 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.266338 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bbdceb2b-73c7-4501-994c-4e3533fa47d0-serviceca\") pod \"node-ca-7dcvf\" (UID: \"bbdceb2b-73c7-4501-994c-4e3533fa47d0\") " pod="openshift-image-registry/node-ca-7dcvf" Oct 30 16:47:07.266719 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.266378 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d52d0ddc-6bd1-4dcc-9d56-84213c328055-original-pull-secret\") pod \"global-pull-secret-syncer-2kbm7\" (UID: \"d52d0ddc-6bd1-4dcc-9d56-84213c328055\") " pod="kube-system/global-pull-secret-syncer-2kbm7" Oct 30 16:47:07.266719 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.266452 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bbdceb2b-73c7-4501-994c-4e3533fa47d0-host\") pod \"node-ca-7dcvf\" (UID: \"bbdceb2b-73c7-4501-994c-4e3533fa47d0\") " pod="openshift-image-registry/node-ca-7dcvf" Oct 30 16:47:07.266719 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.266454 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/24421640-34bc-4b15-a096-ffdaa7ae7bf4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bll4b\" (UID: \"24421640-34bc-4b15-a096-ffdaa7ae7bf4\") " pod="openshift-multus/multus-additional-cni-plugins-bll4b" Oct 30 16:47:07.266719 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.266461 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/24421640-34bc-4b15-a096-ffdaa7ae7bf4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bll4b\" (UID: \"24421640-34bc-4b15-a096-ffdaa7ae7bf4\") " pod="openshift-multus/multus-additional-cni-plugins-bll4b" Oct 30 16:47:07.266719 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.266519 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-multus-cni-dir\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:47:07.266719 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.266570 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-multus-socket-dir-parent\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:47:07.266719 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.266601 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-host-var-lib-cni-multus\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:47:07.266719 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.266632 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-multus-conf-dir\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:47:07.266719 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.266672 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-host-run-multus-certs\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:47:07.266719 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.266670 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d52d0ddc-6bd1-4dcc-9d56-84213c328055-original-pull-secret\") pod \"global-pull-secret-syncer-2kbm7\" (UID: \"d52d0ddc-6bd1-4dcc-9d56-84213c328055\") " pod="kube-system/global-pull-secret-syncer-2kbm7" Oct 30 16:47:07.266719 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.266698 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch8ct\" (UniqueName: \"kubernetes.io/projected/de08f19f-1712-4864-87b0-294e3d3e460d-kube-api-access-ch8ct\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:47:07.266719 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.266708 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/252a7edc-651f-4c08-8fd7-f46ec29737c4-trusted-ca-bundle\") pod \"periodic-gathering-f4j47-pgf7r\" (UID: \"252a7edc-651f-4c08-8fd7-f46ec29737c4\") " pod="openshift-insights/periodic-gathering-f4j47-pgf7r" Oct 30 16:47:07.266719 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.266718 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/de08f19f-1712-4864-87b0-294e3d3e460d-cni-binary-copy\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:47:07.267271 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.266770 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/de08f19f-1712-4864-87b0-294e3d3e460d-multus-daemon-config\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:47:07.267271 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.266865 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a21d39c5-3813-4433-9307-855276f1a66e-data-volume\") pod \"insights-runtime-extractor-4wjcq\" (UID: \"a21d39c5-3813-4433-9307-855276f1a66e\") " pod="openshift-insights/insights-runtime-extractor-4wjcq" Oct 30 16:47:07.267271 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.266936 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/de08f19f-1712-4864-87b0-294e3d3e460d-cni-binary-copy\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:47:07.267271 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.266964 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/bf581b63-6285-4ac9-aa45-cd8a367478f2-signing-key\") pod \"service-ca-5fffc97bdf-hng8x\" (UID: \"bf581b63-6285-4ac9-aa45-cd8a367478f2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hng8x" Oct 30 16:47:07.267271 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.266989 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-run-openvswitch\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:47:07.267271 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.266996 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a21d39c5-3813-4433-9307-855276f1a66e-data-volume\") pod \"insights-runtime-extractor-4wjcq\" (UID: \"a21d39c5-3813-4433-9307-855276f1a66e\") " pod="openshift-insights/insights-runtime-extractor-4wjcq" Oct 30 16:47:07.267271 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.267023 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-node-log\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:47:07.267271 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.267037 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/de08f19f-1712-4864-87b0-294e3d3e460d-multus-daemon-config\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:47:07.267271 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.267051 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-log-socket\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:47:07.267271 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.267073 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-ovnkube-config\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:47:07.267271 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.267089 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8eab0b7c-14cc-42fa-993f-d52707e42d35-iptables-alerter-script\") pod \"iptables-alerter-v79c8\" (UID: \"8eab0b7c-14cc-42fa-993f-d52707e42d35\") " pod="openshift-network-operator/iptables-alerter-v79c8" Oct 30 16:47:07.267271 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.267118 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/bf581b63-6285-4ac9-aa45-cd8a367478f2-signing-key\") pod \"service-ca-5fffc97bdf-hng8x\" (UID: \"bf581b63-6285-4ac9-aa45-cd8a367478f2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hng8x" Oct 30 16:47:07.267271 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.267133 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e3fcf8dd-4ee4-4037-9952-9c54a49c6738-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ftqbf\" (UID: \"e3fcf8dd-4ee4-4037-9952-9c54a49c6738\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftqbf" Oct 30 16:47:07.267271 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.267222 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e3fcf8dd-4ee4-4037-9952-9c54a49c6738-registration-dir\") pod \"aws-ebs-csi-driver-node-ftqbf\" (UID: \"e3fcf8dd-4ee4-4037-9952-9c54a49c6738\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftqbf" Oct 30 16:47:07.267271 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.267243 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-var-lib-kubelet\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:47:07.267271 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.267261 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a21d39c5-3813-4433-9307-855276f1a66e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4wjcq\" (UID: \"a21d39c5-3813-4433-9307-855276f1a66e\") " pod="openshift-insights/insights-runtime-extractor-4wjcq" Oct 30 16:47:07.267972 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.267297 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a21d39c5-3813-4433-9307-855276f1a66e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4wjcq\" (UID: \"a21d39c5-3813-4433-9307-855276f1a66e\") " pod="openshift-insights/insights-runtime-extractor-4wjcq" Oct 30 16:47:07.267972 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.267296 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-ovnkube-config\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:47:07.267972 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.267322 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8eab0b7c-14cc-42fa-993f-d52707e42d35-iptables-alerter-script\") pod \"iptables-alerter-v79c8\" (UID: \"8eab0b7c-14cc-42fa-993f-d52707e42d35\") " pod="openshift-network-operator/iptables-alerter-v79c8" Oct 30 16:47:07.267972 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.267338 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-run-systemd\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:47:07.267972 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.267355 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-etc-openvswitch\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:47:07.267972 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.267380 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/24421640-34bc-4b15-a096-ffdaa7ae7bf4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bll4b\" (UID: \"24421640-34bc-4b15-a096-ffdaa7ae7bf4\") " pod="openshift-multus/multus-additional-cni-plugins-bll4b" Oct 30 16:47:07.267972 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.267408 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c702aab4-18a3-4b3a-a629-7d87f15747e7-hosts-file\") pod \"node-resolver-mbwdh\" (UID: \"c702aab4-18a3-4b3a-a629-7d87f15747e7\") " pod="openshift-dns/node-resolver-mbwdh" Oct 30 16:47:07.267972 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.267447 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c702aab4-18a3-4b3a-a629-7d87f15747e7-tmp-dir\") pod \"node-resolver-mbwdh\" (UID: \"c702aab4-18a3-4b3a-a629-7d87f15747e7\") " pod="openshift-dns/node-resolver-mbwdh" Oct 30 16:47:07.267972 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.267482 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s78x6\" (UniqueName: \"kubernetes.io/projected/c702aab4-18a3-4b3a-a629-7d87f15747e7-kube-api-access-s78x6\") pod \"node-resolver-mbwdh\" (UID: \"c702aab4-18a3-4b3a-a629-7d87f15747e7\") " pod="openshift-dns/node-resolver-mbwdh" Oct 30 16:47:07.267972 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.267555 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-etc-modprobe-d\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:47:07.267972 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.267557 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c702aab4-18a3-4b3a-a629-7d87f15747e7-tmp-dir\") pod \"node-resolver-mbwdh\" (UID: \"c702aab4-18a3-4b3a-a629-7d87f15747e7\") " pod="openshift-dns/node-resolver-mbwdh" Oct 30 16:47:07.267972 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.267591 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-etc-systemd\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:47:07.267972 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.267629 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d52d0ddc-6bd1-4dcc-9d56-84213c328055-kubelet-config\") pod \"global-pull-secret-syncer-2kbm7\" (UID: \"d52d0ddc-6bd1-4dcc-9d56-84213c328055\") " pod="kube-system/global-pull-secret-syncer-2kbm7" Oct 30 16:47:07.267972 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.267654 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-host-run-netns\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:47:07.267972 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.267686 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhcd5\" (UniqueName: \"kubernetes.io/projected/24421640-34bc-4b15-a096-ffdaa7ae7bf4-kube-api-access-dhcd5\") pod \"multus-additional-cni-plugins-bll4b\" (UID: \"24421640-34bc-4b15-a096-ffdaa7ae7bf4\") " pod="openshift-multus/multus-additional-cni-plugins-bll4b" Oct 30 16:47:07.267972 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.267720 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3-metrics-tls\") pod \"dns-default-wc9zw\" (UID: \"bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3\") " pod="openshift-dns/dns-default-wc9zw" Oct 30 16:47:07.267972 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.267750 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e3fcf8dd-4ee4-4037-9952-9c54a49c6738-socket-dir\") pod \"aws-ebs-csi-driver-node-ftqbf\" (UID: \"e3fcf8dd-4ee4-4037-9952-9c54a49c6738\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftqbf" Oct 30 16:47:07.267972 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.267766 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-run-ovn\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:47:07.267972 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.267794 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/24421640-34bc-4b15-a096-ffdaa7ae7bf4-cni-binary-copy\") pod \"multus-additional-cni-plugins-bll4b\" (UID: \"24421640-34bc-4b15-a096-ffdaa7ae7bf4\") " pod="openshift-multus/multus-additional-cni-plugins-bll4b" Oct 30 16:47:07.267972 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.267812 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pwzt\" (UniqueName: \"kubernetes.io/projected/1ba7d788-14ef-4b39-b6f6-3072ab0b3200-kube-api-access-4pwzt\") pod \"network-metrics-daemon-rsgvx\" (UID: \"1ba7d788-14ef-4b39-b6f6-3072ab0b3200\") " pod="openshift-multus/network-metrics-daemon-rsgvx" Oct 30 16:47:07.267972 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.267933 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2434842e-d96c-4890-9b2a-2a9eb130394a-tmp\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:47:07.268809 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.267997 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/24421640-34bc-4b15-a096-ffdaa7ae7bf4-cni-binary-copy\") pod \"multus-additional-cni-plugins-bll4b\" (UID: \"24421640-34bc-4b15-a096-ffdaa7ae7bf4\") " pod="openshift-multus/multus-additional-cni-plugins-bll4b" Oct 30 16:47:07.268809 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.268010 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-host-run-k8s-cni-cncf-io\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:47:07.268809 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.268073 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2434842e-d96c-4890-9b2a-2a9eb130394a-tmp\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:47:07.268809 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.268083 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-host-run-netns\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:47:07.268809 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.268118 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-env-overrides\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:47:07.268809 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.268144 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-ovnkube-script-lib\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:47:07.268809 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.268173 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ba7d788-14ef-4b39-b6f6-3072ab0b3200-metrics-certs\") pod \"network-metrics-daemon-rsgvx\" (UID: \"1ba7d788-14ef-4b39-b6f6-3072ab0b3200\") " pod="openshift-multus/network-metrics-daemon-rsgvx" Oct 30 16:47:07.268809 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.268200 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3-tmp-dir\") pod \"dns-default-wc9zw\" (UID: \"bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3\") " pod="openshift-dns/dns-default-wc9zw" Oct 30 16:47:07.268809 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.268228 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-var-lib-openvswitch\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:47:07.268809 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.268255 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjpvn\" (UniqueName: \"kubernetes.io/projected/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-kube-api-access-pjpvn\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:47:07.268809 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.268267 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-env-overrides\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:47:07.268809 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.268299 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3-tmp-dir\") pod \"dns-default-wc9zw\" (UID: \"bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3\") " pod="openshift-dns/dns-default-wc9zw" Oct 30 16:47:07.268809 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.268344 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-etc-sysconfig\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:47:07.268809 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.268394 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-ovnkube-script-lib\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:47:07.268809 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.268465 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jblxf\" (UniqueName: \"kubernetes.io/projected/bbdceb2b-73c7-4501-994c-4e3533fa47d0-kube-api-access-jblxf\") pod \"node-ca-7dcvf\" (UID: \"bbdceb2b-73c7-4501-994c-4e3533fa47d0\") " pod="openshift-image-registry/node-ca-7dcvf" Oct 30 16:47:07.268809 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.268493 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d24bfb4d-652d-4e7d-bc67-43441164f652-agent-certs\") pod \"konnectivity-agent-mqhg8\" (UID: \"d24bfb4d-652d-4e7d-bc67-43441164f652\") " pod="kube-system/konnectivity-agent-mqhg8" Oct 30 16:47:07.268809 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.268517 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d24bfb4d-652d-4e7d-bc67-43441164f652-konnectivity-ca\") pod \"konnectivity-agent-mqhg8\" (UID: \"d24bfb4d-652d-4e7d-bc67-43441164f652\") " pod="kube-system/konnectivity-agent-mqhg8" Oct 30 16:47:07.268809 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.268547 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e3fcf8dd-4ee4-4037-9952-9c54a49c6738-sys-fs\") pod \"aws-ebs-csi-driver-node-ftqbf\" (UID: \"e3fcf8dd-4ee4-4037-9952-9c54a49c6738\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftqbf" Oct 30 16:47:07.268809 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.268572 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5st2s\" (UniqueName: \"kubernetes.io/projected/40e1c660-22c0-42e5-8511-4cbd0e170a5c-kube-api-access-5st2s\") pod \"network-check-target-8zqlg\" (UID: \"40e1c660-22c0-42e5-8511-4cbd0e170a5c\") " pod="openshift-network-diagnostics/network-check-target-8zqlg" Oct 30 16:47:07.268809 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.268597 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-host-var-lib-cni-bin\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:47:07.268809 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.268649 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-host-kubelet\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:47:07.268809 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.268678 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-host-run-ovn-kubernetes\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:47:07.268809 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.268702 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8eab0b7c-14cc-42fa-993f-d52707e42d35-host-slash\") pod \"iptables-alerter-v79c8\" (UID: \"8eab0b7c-14cc-42fa-993f-d52707e42d35\") " pod="openshift-network-operator/iptables-alerter-v79c8" Oct 30 16:47:07.268809 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.268732 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d24bfb4d-652d-4e7d-bc67-43441164f652-konnectivity-ca\") pod \"konnectivity-agent-mqhg8\" (UID: \"d24bfb4d-652d-4e7d-bc67-43441164f652\") " pod="kube-system/konnectivity-agent-mqhg8" Oct 30 16:47:07.268809 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.268737 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95mt2\" (UniqueName: \"kubernetes.io/projected/8eab0b7c-14cc-42fa-993f-d52707e42d35-kube-api-access-95mt2\") pod \"iptables-alerter-v79c8\" (UID: \"8eab0b7c-14cc-42fa-993f-d52707e42d35\") " pod="openshift-network-operator/iptables-alerter-v79c8" Oct 30 16:47:07.268809 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.268761 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d24bfb4d-652d-4e7d-bc67-43441164f652-agent-certs\") pod \"konnectivity-agent-mqhg8\" (UID: \"d24bfb4d-652d-4e7d-bc67-43441164f652\") " pod="kube-system/konnectivity-agent-mqhg8" Oct 30 16:47:07.268809 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.268775 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e3fcf8dd-4ee4-4037-9952-9c54a49c6738-etc-selinux\") pod \"aws-ebs-csi-driver-node-ftqbf\" (UID: \"e3fcf8dd-4ee4-4037-9952-9c54a49c6738\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftqbf" Oct 30 16:47:07.268809 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.268791 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-run\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:47:07.268809 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.268809 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-etc-kubernetes\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:47:07.269787 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.268841 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs8qj\" (UniqueName: \"kubernetes.io/projected/bf581b63-6285-4ac9-aa45-cd8a367478f2-kube-api-access-hs8qj\") pod \"service-ca-5fffc97bdf-hng8x\" (UID: \"bf581b63-6285-4ac9-aa45-cd8a367478f2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hng8x" Oct 30 16:47:07.269787 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.268926 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-ovn-node-metrics-cert\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:47:07.269787 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.268942 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e3fcf8dd-4ee4-4037-9952-9c54a49c6738-device-dir\") pod \"aws-ebs-csi-driver-node-ftqbf\" (UID: \"e3fcf8dd-4ee4-4037-9952-9c54a49c6738\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftqbf" Oct 30 16:47:07.269787 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.268961 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e275e41d-9573-47e2-b923-173dfb4361a7-cert\") pod \"ingress-canary-cqjj8\" (UID: \"e275e41d-9573-47e2-b923-173dfb4361a7\") " pod="openshift-ingress-canary/ingress-canary-cqjj8" Oct 30 16:47:07.269787 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.268984 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm26j\" (UniqueName: \"kubernetes.io/projected/e275e41d-9573-47e2-b923-173dfb4361a7-kube-api-access-vm26j\") pod \"ingress-canary-cqjj8\" (UID: \"e275e41d-9573-47e2-b923-173dfb4361a7\") " pod="openshift-ingress-canary/ingress-canary-cqjj8" Oct 30 16:47:07.269787 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.269016 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/252a7edc-651f-4c08-8fd7-f46ec29737c4-service-ca-bundle\") pod \"periodic-gathering-f4j47-pgf7r\" (UID: \"252a7edc-651f-4c08-8fd7-f46ec29737c4\") " pod="openshift-insights/periodic-gathering-f4j47-pgf7r" Oct 30 16:47:07.269787 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.269080 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-cnibin\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:47:07.269787 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.269101 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2434842e-d96c-4890-9b2a-2a9eb130394a-etc-tuned\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:47:07.269787 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.269111 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-ovn-node-metrics-cert\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:47:07.269787 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.269117 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/252a7edc-651f-4c08-8fd7-f46ec29737c4-archives-path\") pod \"periodic-gathering-f4j47-pgf7r\" (UID: \"252a7edc-651f-4c08-8fd7-f46ec29737c4\") " pod="openshift-insights/periodic-gathering-f4j47-pgf7r" Oct 30 16:47:07.269787 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.269154 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/252a7edc-651f-4c08-8fd7-f46ec29737c4-service-ca-bundle\") pod \"periodic-gathering-f4j47-pgf7r\" (UID: \"252a7edc-651f-4c08-8fd7-f46ec29737c4\") " pod="openshift-insights/periodic-gathering-f4j47-pgf7r" Oct 30 16:47:07.269787 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.269156 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a21d39c5-3813-4433-9307-855276f1a66e-crio-socket\") pod \"insights-runtime-extractor-4wjcq\" (UID: \"a21d39c5-3813-4433-9307-855276f1a66e\") " pod="openshift-insights/insights-runtime-extractor-4wjcq" Oct 30 16:47:07.269787 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.269161 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/252a7edc-651f-4c08-8fd7-f46ec29737c4-archives-path\") pod \"periodic-gathering-f4j47-pgf7r\" (UID: \"252a7edc-651f-4c08-8fd7-f46ec29737c4\") " pod="openshift-insights/periodic-gathering-f4j47-pgf7r" Oct 30 16:47:07.269787 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.269197 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-host-slash\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:47:07.269787 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.269225 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2434842e-d96c-4890-9b2a-2a9eb130394a-etc-tuned\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:47:07.269787 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.269229 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e275e41d-9573-47e2-b923-173dfb4361a7-cert\") pod \"ingress-canary-cqjj8\" (UID: \"e275e41d-9573-47e2-b923-173dfb4361a7\") " pod="openshift-ingress-canary/ingress-canary-cqjj8" Oct 30 16:47:07.269787 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.269234 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-etc-sysctl-d\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:47:07.269787 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.269301 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-sys\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:47:07.269787 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.269345 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-host-var-lib-kubelet\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:47:07.269787 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.269370 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-host-cni-bin\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:47:07.269787 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.269398 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:47:07.269787 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.269421 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24421640-34bc-4b15-a096-ffdaa7ae7bf4-system-cni-dir\") pod \"multus-additional-cni-plugins-bll4b\" (UID: \"24421640-34bc-4b15-a096-ffdaa7ae7bf4\") " pod="openshift-multus/multus-additional-cni-plugins-bll4b" Oct 30 16:47:07.269787 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.269436 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-etc-kubernetes\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:47:07.269787 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.269460 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-host\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:47:07.269787 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.269491 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gxc2\" (UniqueName: \"kubernetes.io/projected/252a7edc-651f-4c08-8fd7-f46ec29737c4-kube-api-access-2gxc2\") pod \"periodic-gathering-f4j47-pgf7r\" (UID: \"252a7edc-651f-4c08-8fd7-f46ec29737c4\") " pod="openshift-insights/periodic-gathering-f4j47-pgf7r" Oct 30 16:47:07.269787 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.269539 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-system-cni-dir\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:47:07.269787 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.269605 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pjzv\" (UniqueName: \"kubernetes.io/projected/a21d39c5-3813-4433-9307-855276f1a66e-kube-api-access-4pjzv\") pod \"insights-runtime-extractor-4wjcq\" (UID: \"a21d39c5-3813-4433-9307-855276f1a66e\") " pod="openshift-insights/insights-runtime-extractor-4wjcq" Oct 30 16:47:07.269787 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.269621 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-systemd-units\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:47:07.291467 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.291434 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch8ct\" (UniqueName: \"kubernetes.io/projected/de08f19f-1712-4864-87b0-294e3d3e460d-kube-api-access-ch8ct\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:47:07.291467 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.291452 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dnnp\" (UniqueName: \"kubernetes.io/projected/cd83ecbc-117f-4890-bf7a-cab9282d4484-kube-api-access-7dnnp\") pod \"migrator-66df5d496b-zm2rj\" (UID: \"cd83ecbc-117f-4890-bf7a-cab9282d4484\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zm2rj" Oct 30 16:47:07.291666 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.291508 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfpbf\" (UniqueName: \"kubernetes.io/projected/e3fcf8dd-4ee4-4037-9952-9c54a49c6738-kube-api-access-mfpbf\") pod \"aws-ebs-csi-driver-node-ftqbf\" (UID: \"e3fcf8dd-4ee4-4037-9952-9c54a49c6738\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftqbf" Oct 30 16:47:07.291718 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.291680 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2whld\" (UniqueName: \"kubernetes.io/projected/bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3-kube-api-access-2whld\") pod \"dns-default-wc9zw\" (UID: \"bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3\") " pod="openshift-dns/dns-default-wc9zw" Oct 30 16:47:07.295546 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.295525 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s2zv\" (UniqueName: \"kubernetes.io/projected/2434842e-d96c-4890-9b2a-2a9eb130394a-kube-api-access-2s2zv\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:47:07.295686 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.295667 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pwzt\" (UniqueName: \"kubernetes.io/projected/1ba7d788-14ef-4b39-b6f6-3072ab0b3200-kube-api-access-4pwzt\") pod \"network-metrics-daemon-rsgvx\" (UID: \"1ba7d788-14ef-4b39-b6f6-3072ab0b3200\") " pod="openshift-multus/network-metrics-daemon-rsgvx" Oct 30 16:47:07.297366 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.297347 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs8qj\" (UniqueName: \"kubernetes.io/projected/bf581b63-6285-4ac9-aa45-cd8a367478f2-kube-api-access-hs8qj\") pod \"service-ca-5fffc97bdf-hng8x\" (UID: \"bf581b63-6285-4ac9-aa45-cd8a367478f2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hng8x" Oct 30 16:47:07.297614 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.297600 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjpvn\" (UniqueName: \"kubernetes.io/projected/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-kube-api-access-pjpvn\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:47:07.298895 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.298879 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5st2s\" (UniqueName: \"kubernetes.io/projected/40e1c660-22c0-42e5-8511-4cbd0e170a5c-kube-api-access-5st2s\") pod \"network-check-target-8zqlg\" (UID: \"40e1c660-22c0-42e5-8511-4cbd0e170a5c\") " pod="openshift-network-diagnostics/network-check-target-8zqlg" Oct 30 16:47:07.299952 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.299935 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95mt2\" (UniqueName: \"kubernetes.io/projected/8eab0b7c-14cc-42fa-993f-d52707e42d35-kube-api-access-95mt2\") pod \"iptables-alerter-v79c8\" (UID: \"8eab0b7c-14cc-42fa-993f-d52707e42d35\") " pod="openshift-network-operator/iptables-alerter-v79c8" Oct 30 16:47:07.301779 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.301761 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jblxf\" (UniqueName: \"kubernetes.io/projected/bbdceb2b-73c7-4501-994c-4e3533fa47d0-kube-api-access-jblxf\") pod \"node-ca-7dcvf\" (UID: \"bbdceb2b-73c7-4501-994c-4e3533fa47d0\") " pod="openshift-image-registry/node-ca-7dcvf" Oct 30 16:47:07.302091 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.302075 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s78x6\" (UniqueName: \"kubernetes.io/projected/c702aab4-18a3-4b3a-a629-7d87f15747e7-kube-api-access-s78x6\") pod \"node-resolver-mbwdh\" (UID: \"c702aab4-18a3-4b3a-a629-7d87f15747e7\") " pod="openshift-dns/node-resolver-mbwdh" Oct 30 16:47:07.303371 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.303352 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm26j\" (UniqueName: \"kubernetes.io/projected/e275e41d-9573-47e2-b923-173dfb4361a7-kube-api-access-vm26j\") pod \"ingress-canary-cqjj8\" (UID: \"e275e41d-9573-47e2-b923-173dfb4361a7\") " pod="openshift-ingress-canary/ingress-canary-cqjj8" Oct 30 16:47:07.303638 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.303619 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pjzv\" (UniqueName: \"kubernetes.io/projected/a21d39c5-3813-4433-9307-855276f1a66e-kube-api-access-4pjzv\") pod \"insights-runtime-extractor-4wjcq\" (UID: \"a21d39c5-3813-4433-9307-855276f1a66e\") " pod="openshift-insights/insights-runtime-extractor-4wjcq" Oct 30 16:47:07.309665 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.309642 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gxc2\" (UniqueName: \"kubernetes.io/projected/252a7edc-651f-4c08-8fd7-f46ec29737c4-kube-api-access-2gxc2\") pod \"periodic-gathering-f4j47-pgf7r\" (UID: \"252a7edc-651f-4c08-8fd7-f46ec29737c4\") " pod="openshift-insights/periodic-gathering-f4j47-pgf7r" Oct 30 16:47:07.311784 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.311767 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhcd5\" (UniqueName: \"kubernetes.io/projected/24421640-34bc-4b15-a096-ffdaa7ae7bf4-kube-api-access-dhcd5\") pod \"multus-additional-cni-plugins-bll4b\" (UID: \"24421640-34bc-4b15-a096-ffdaa7ae7bf4\") " pod="openshift-multus/multus-additional-cni-plugins-bll4b" Oct 30 16:47:07.370859 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.370777 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-etc-sysctl-d\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:47:07.370859 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.370811 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-sys\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:47:07.370859 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.370828 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-host-var-lib-kubelet\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:47:07.371109 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.370887 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-host-var-lib-kubelet\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:47:07.371109 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.370906 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-sys\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:47:07.371109 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.370947 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-host-cni-bin\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:47:07.371109 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.370975 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:47:07.371109 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.370993 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24421640-34bc-4b15-a096-ffdaa7ae7bf4-system-cni-dir\") pod \"multus-additional-cni-plugins-bll4b\" (UID: \"24421640-34bc-4b15-a096-ffdaa7ae7bf4\") " pod="openshift-multus/multus-additional-cni-plugins-bll4b" Oct 30 16:47:07.371109 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.370992 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-etc-sysctl-d\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:47:07.371109 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371021 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-host-cni-bin\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:47:07.371109 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371028 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:47:07.371109 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371041 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24421640-34bc-4b15-a096-ffdaa7ae7bf4-system-cni-dir\") pod \"multus-additional-cni-plugins-bll4b\" (UID: \"24421640-34bc-4b15-a096-ffdaa7ae7bf4\") " pod="openshift-multus/multus-additional-cni-plugins-bll4b" Oct 30 16:47:07.371109 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371040 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-etc-kubernetes\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:47:07.371109 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371071 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-host\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:47:07.371109 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371093 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-system-cni-dir\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:47:07.371109 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371108 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-systemd-units\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:47:07.371614 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371125 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-etc-kubernetes\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:47:07.371614 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371130 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-host-cni-netd\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:47:07.371614 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371152 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d52d0ddc-6bd1-4dcc-9d56-84213c328055-dbus\") pod \"global-pull-secret-syncer-2kbm7\" (UID: \"d52d0ddc-6bd1-4dcc-9d56-84213c328055\") " pod="kube-system/global-pull-secret-syncer-2kbm7" Oct 30 16:47:07.371614 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371159 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-system-cni-dir\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:47:07.371614 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371173 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-hostroot\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:47:07.371614 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371179 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-systemd-units\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:47:07.371614 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371192 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/24421640-34bc-4b15-a096-ffdaa7ae7bf4-cnibin\") pod \"multus-additional-cni-plugins-bll4b\" (UID: \"24421640-34bc-4b15-a096-ffdaa7ae7bf4\") " pod="openshift-multus/multus-additional-cni-plugins-bll4b" Oct 30 16:47:07.371614 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371175 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-host-cni-netd\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:47:07.371614 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371206 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-hostroot\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:47:07.371614 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371221 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/24421640-34bc-4b15-a096-ffdaa7ae7bf4-cnibin\") pod \"multus-additional-cni-plugins-bll4b\" (UID: \"24421640-34bc-4b15-a096-ffdaa7ae7bf4\") " pod="openshift-multus/multus-additional-cni-plugins-bll4b" Oct 30 16:47:07.371614 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371222 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-etc-sysctl-conf\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:47:07.371614 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371151 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-host\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:47:07.371614 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371249 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-lib-modules\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:47:07.371614 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371208 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d52d0ddc-6bd1-4dcc-9d56-84213c328055-dbus\") pod \"global-pull-secret-syncer-2kbm7\" (UID: \"d52d0ddc-6bd1-4dcc-9d56-84213c328055\") " pod="kube-system/global-pull-secret-syncer-2kbm7" Oct 30 16:47:07.371614 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371276 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bbdceb2b-73c7-4501-994c-4e3533fa47d0-host\") pod \"node-ca-7dcvf\" (UID: \"bbdceb2b-73c7-4501-994c-4e3533fa47d0\") " pod="openshift-image-registry/node-ca-7dcvf" Oct 30 16:47:07.371614 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371324 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-multus-cni-dir\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:47:07.371614 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371328 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bbdceb2b-73c7-4501-994c-4e3533fa47d0-host\") pod \"node-ca-7dcvf\" (UID: \"bbdceb2b-73c7-4501-994c-4e3533fa47d0\") " pod="openshift-image-registry/node-ca-7dcvf" Oct 30 16:47:07.371614 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371348 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-multus-socket-dir-parent\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:47:07.371614 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371370 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-host-var-lib-cni-multus\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:47:07.371614 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371376 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-lib-modules\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:47:07.371614 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371389 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-multus-conf-dir\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:47:07.371614 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371391 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-multus-cni-dir\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:47:07.371614 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371415 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-host-var-lib-cni-multus\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:47:07.371614 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371420 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-multus-conf-dir\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:47:07.371614 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371412 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-host-run-multus-certs\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:47:07.371614 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371430 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-etc-sysctl-conf\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:47:07.371614 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371441 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-host-run-multus-certs\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:47:07.371614 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371448 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-run-openvswitch\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:47:07.371614 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371468 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-run-openvswitch\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:47:07.371614 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371471 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-node-log\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:47:07.371614 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371424 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-multus-socket-dir-parent\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:47:07.371614 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371492 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-node-log\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:47:07.371614 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371495 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-log-socket\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:47:07.371614 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371532 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e3fcf8dd-4ee4-4037-9952-9c54a49c6738-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ftqbf\" (UID: \"e3fcf8dd-4ee4-4037-9952-9c54a49c6738\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftqbf" Oct 30 16:47:07.371614 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371557 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e3fcf8dd-4ee4-4037-9952-9c54a49c6738-registration-dir\") pod \"aws-ebs-csi-driver-node-ftqbf\" (UID: \"e3fcf8dd-4ee4-4037-9952-9c54a49c6738\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftqbf" Oct 30 16:47:07.371614 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371578 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-var-lib-kubelet\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:47:07.371614 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371592 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e3fcf8dd-4ee4-4037-9952-9c54a49c6738-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ftqbf\" (UID: \"e3fcf8dd-4ee4-4037-9952-9c54a49c6738\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftqbf" Oct 30 16:47:07.371614 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371593 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a21d39c5-3813-4433-9307-855276f1a66e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4wjcq\" (UID: \"a21d39c5-3813-4433-9307-855276f1a66e\") " pod="openshift-insights/insights-runtime-extractor-4wjcq" Oct 30 16:47:07.371614 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371620 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a21d39c5-3813-4433-9307-855276f1a66e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4wjcq\" (UID: \"a21d39c5-3813-4433-9307-855276f1a66e\") " pod="openshift-insights/insights-runtime-extractor-4wjcq" Oct 30 16:47:07.371614 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371511 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-log-socket\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:47:07.371614 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:47:07.371635 4522 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 16:47:07.373013 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371675 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-var-lib-kubelet\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:47:07.373013 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371675 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e3fcf8dd-4ee4-4037-9952-9c54a49c6738-registration-dir\") pod \"aws-ebs-csi-driver-node-ftqbf\" (UID: \"e3fcf8dd-4ee4-4037-9952-9c54a49c6738\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftqbf" Oct 30 16:47:07.373013 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:47:07.371714 4522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a21d39c5-3813-4433-9307-855276f1a66e-kube-rbac-proxy-cm podName:a21d39c5-3813-4433-9307-855276f1a66e nodeName:}" failed. No retries permitted until 2026-10-30 16:47:07.871694207 +0000 UTC m=+1.872449087 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a21d39c5-3813-4433-9307-855276f1a66e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4wjcq" (UID: "a21d39c5-3813-4433-9307-855276f1a66e") : configmap "kube-rbac-proxy" not found Oct 30 16:47:07.373013 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:47:07.371716 4522 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 16:47:07.373013 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371733 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-run-systemd\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:47:07.373013 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371756 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-etc-openvswitch\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:47:07.373013 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:47:07.371776 4522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a21d39c5-3813-4433-9307-855276f1a66e-insights-runtime-extractor-tls podName:a21d39c5-3813-4433-9307-855276f1a66e nodeName:}" failed. No retries permitted until 2026-10-30 16:47:07.871759791 +0000 UTC m=+1.872514669 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a21d39c5-3813-4433-9307-855276f1a66e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4wjcq" (UID: "a21d39c5-3813-4433-9307-855276f1a66e") : secret "insights-runtime-extractor-tls" not found Oct 30 16:47:07.373013 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371779 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-run-systemd\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:47:07.373013 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371786 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-etc-openvswitch\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:47:07.373013 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371804 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/24421640-34bc-4b15-a096-ffdaa7ae7bf4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bll4b\" (UID: \"24421640-34bc-4b15-a096-ffdaa7ae7bf4\") " pod="openshift-multus/multus-additional-cni-plugins-bll4b" Oct 30 16:47:07.373013 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371821 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c702aab4-18a3-4b3a-a629-7d87f15747e7-hosts-file\") pod \"node-resolver-mbwdh\" (UID: \"c702aab4-18a3-4b3a-a629-7d87f15747e7\") " pod="openshift-dns/node-resolver-mbwdh" Oct 30 16:47:07.373013 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371839 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-etc-modprobe-d\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:47:07.373013 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371855 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-etc-systemd\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:47:07.373013 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371896 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-etc-systemd\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:47:07.373013 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371913 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/24421640-34bc-4b15-a096-ffdaa7ae7bf4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bll4b\" (UID: \"24421640-34bc-4b15-a096-ffdaa7ae7bf4\") " pod="openshift-multus/multus-additional-cni-plugins-bll4b" Oct 30 16:47:07.373013 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371926 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c702aab4-18a3-4b3a-a629-7d87f15747e7-hosts-file\") pod \"node-resolver-mbwdh\" (UID: \"c702aab4-18a3-4b3a-a629-7d87f15747e7\") " pod="openshift-dns/node-resolver-mbwdh" Oct 30 16:47:07.373013 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371950 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d52d0ddc-6bd1-4dcc-9d56-84213c328055-kubelet-config\") pod \"global-pull-secret-syncer-2kbm7\" (UID: \"d52d0ddc-6bd1-4dcc-9d56-84213c328055\") " pod="kube-system/global-pull-secret-syncer-2kbm7" Oct 30 16:47:07.373013 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371953 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-etc-modprobe-d\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:47:07.373013 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371968 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-host-run-netns\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:47:07.373013 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371975 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d52d0ddc-6bd1-4dcc-9d56-84213c328055-kubelet-config\") pod \"global-pull-secret-syncer-2kbm7\" (UID: \"d52d0ddc-6bd1-4dcc-9d56-84213c328055\") " pod="kube-system/global-pull-secret-syncer-2kbm7" Oct 30 16:47:07.373013 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371989 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3-metrics-tls\") pod \"dns-default-wc9zw\" (UID: \"bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3\") " pod="openshift-dns/dns-default-wc9zw" Oct 30 16:47:07.373013 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.371999 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-host-run-netns\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:47:07.373013 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.372004 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e3fcf8dd-4ee4-4037-9952-9c54a49c6738-socket-dir\") pod \"aws-ebs-csi-driver-node-ftqbf\" (UID: \"e3fcf8dd-4ee4-4037-9952-9c54a49c6738\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftqbf" Oct 30 16:47:07.373013 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.372034 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e3fcf8dd-4ee4-4037-9952-9c54a49c6738-socket-dir\") pod \"aws-ebs-csi-driver-node-ftqbf\" (UID: \"e3fcf8dd-4ee4-4037-9952-9c54a49c6738\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftqbf" Oct 30 16:47:07.373013 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.372034 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-run-ovn\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:47:07.373013 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:47:07.372054 4522 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 16:47:07.373013 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.372070 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-host-run-k8s-cni-cncf-io\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:47:07.373013 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.372073 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-run-ovn\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:47:07.373013 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:47:07.372089 4522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3-metrics-tls podName:bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:07.872077908 +0000 UTC m=+1.872832788 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3-metrics-tls") pod "dns-default-wc9zw" (UID: "bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3") : secret "dns-default-metrics-tls" not found Oct 30 16:47:07.373013 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.372113 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-host-run-k8s-cni-cncf-io\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:47:07.373013 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.372122 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-host-run-netns\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:47:07.373013 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.372145 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ba7d788-14ef-4b39-b6f6-3072ab0b3200-metrics-certs\") pod \"network-metrics-daemon-rsgvx\" (UID: \"1ba7d788-14ef-4b39-b6f6-3072ab0b3200\") " pod="openshift-multus/network-metrics-daemon-rsgvx" Oct 30 16:47:07.373013 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.372161 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-var-lib-openvswitch\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:47:07.373013 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.372173 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-host-run-netns\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:47:07.373013 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.372179 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-etc-sysconfig\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:47:07.373013 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.372221 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e3fcf8dd-4ee4-4037-9952-9c54a49c6738-sys-fs\") pod \"aws-ebs-csi-driver-node-ftqbf\" (UID: \"e3fcf8dd-4ee4-4037-9952-9c54a49c6738\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftqbf" Oct 30 16:47:07.373013 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:47:07.372230 4522 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 16:47:07.373013 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.372251 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-host-var-lib-cni-bin\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:47:07.373013 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.372270 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e3fcf8dd-4ee4-4037-9952-9c54a49c6738-sys-fs\") pod \"aws-ebs-csi-driver-node-ftqbf\" (UID: \"e3fcf8dd-4ee4-4037-9952-9c54a49c6738\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftqbf" Oct 30 16:47:07.373013 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.372227 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-etc-sysconfig\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:47:07.373013 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.372227 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-var-lib-openvswitch\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:47:07.373013 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.372293 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-host-var-lib-cni-bin\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:47:07.373013 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:47:07.372279 4522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ba7d788-14ef-4b39-b6f6-3072ab0b3200-metrics-certs podName:1ba7d788-14ef-4b39-b6f6-3072ab0b3200 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:07.872267187 +0000 UTC m=+1.873022069 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1ba7d788-14ef-4b39-b6f6-3072ab0b3200-metrics-certs") pod "network-metrics-daemon-rsgvx" (UID: "1ba7d788-14ef-4b39-b6f6-3072ab0b3200") : secret "metrics-daemon-secret" not found Oct 30 16:47:07.373013 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.372341 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-host-kubelet\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:47:07.373013 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.372370 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-host-run-ovn-kubernetes\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:47:07.373013 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.372375 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-host-kubelet\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:47:07.373013 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.372394 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8eab0b7c-14cc-42fa-993f-d52707e42d35-host-slash\") pod \"iptables-alerter-v79c8\" (UID: \"8eab0b7c-14cc-42fa-993f-d52707e42d35\") " pod="openshift-network-operator/iptables-alerter-v79c8" Oct 30 16:47:07.373013 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.372401 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-host-run-ovn-kubernetes\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:47:07.373013 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.372419 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e3fcf8dd-4ee4-4037-9952-9c54a49c6738-etc-selinux\") pod \"aws-ebs-csi-driver-node-ftqbf\" (UID: \"e3fcf8dd-4ee4-4037-9952-9c54a49c6738\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftqbf" Oct 30 16:47:07.373013 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.372435 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8eab0b7c-14cc-42fa-993f-d52707e42d35-host-slash\") pod \"iptables-alerter-v79c8\" (UID: \"8eab0b7c-14cc-42fa-993f-d52707e42d35\") " pod="openshift-network-operator/iptables-alerter-v79c8" Oct 30 16:47:07.373013 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.372442 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-run\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:47:07.373013 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.372477 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-etc-kubernetes\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:47:07.373013 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.372503 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e3fcf8dd-4ee4-4037-9952-9c54a49c6738-device-dir\") pod \"aws-ebs-csi-driver-node-ftqbf\" (UID: \"e3fcf8dd-4ee4-4037-9952-9c54a49c6738\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftqbf" Oct 30 16:47:07.373013 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.372527 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-cnibin\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:47:07.373013 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.372536 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-etc-kubernetes\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:47:07.373013 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.372534 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e3fcf8dd-4ee4-4037-9952-9c54a49c6738-etc-selinux\") pod \"aws-ebs-csi-driver-node-ftqbf\" (UID: \"e3fcf8dd-4ee4-4037-9952-9c54a49c6738\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftqbf" Oct 30 16:47:07.373013 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.372530 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2434842e-d96c-4890-9b2a-2a9eb130394a-run\") pod \"tuned-mlkjb\" (UID: \"2434842e-d96c-4890-9b2a-2a9eb130394a\") " pod="openshift-cluster-node-tuning-operator/tuned-mlkjb" Oct 30 16:47:07.373013 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.372555 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e3fcf8dd-4ee4-4037-9952-9c54a49c6738-device-dir\") pod \"aws-ebs-csi-driver-node-ftqbf\" (UID: \"e3fcf8dd-4ee4-4037-9952-9c54a49c6738\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftqbf" Oct 30 16:47:07.373013 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.372554 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a21d39c5-3813-4433-9307-855276f1a66e-crio-socket\") pod \"insights-runtime-extractor-4wjcq\" (UID: \"a21d39c5-3813-4433-9307-855276f1a66e\") " pod="openshift-insights/insights-runtime-extractor-4wjcq" Oct 30 16:47:07.373013 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.372574 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/de08f19f-1712-4864-87b0-294e3d3e460d-cnibin\") pod \"multus-n7lg2\" (UID: \"de08f19f-1712-4864-87b0-294e3d3e460d\") " pod="openshift-multus/multus-n7lg2" Oct 30 16:47:07.373013 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.372585 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-host-slash\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:47:07.373013 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.372595 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a21d39c5-3813-4433-9307-855276f1a66e-crio-socket\") pod \"insights-runtime-extractor-4wjcq\" (UID: \"a21d39c5-3813-4433-9307-855276f1a66e\") " pod="openshift-insights/insights-runtime-extractor-4wjcq" Oct 30 16:47:07.373013 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.372625 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/38c17ac6-93d3-4d38-83db-70d4d0fa3f30-host-slash\") pod \"ovnkube-node-ns7rp\" (UID: \"38c17ac6-93d3-4d38-83db-70d4d0fa3f30\") " pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:47:07.615340 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.615292 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7dcvf_bbdceb2b-73c7-4501-994c-4e3533fa47d0/node-ca/0.log" Oct 30 16:47:07.875547 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.875504 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ba7d788-14ef-4b39-b6f6-3072ab0b3200-metrics-certs\") pod \"network-metrics-daemon-rsgvx\" (UID: \"1ba7d788-14ef-4b39-b6f6-3072ab0b3200\") " pod="openshift-multus/network-metrics-daemon-rsgvx" Oct 30 16:47:07.875708 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.875575 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a21d39c5-3813-4433-9307-855276f1a66e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4wjcq\" (UID: \"a21d39c5-3813-4433-9307-855276f1a66e\") " pod="openshift-insights/insights-runtime-extractor-4wjcq" Oct 30 16:47:07.875708 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.875609 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a21d39c5-3813-4433-9307-855276f1a66e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4wjcq\" (UID: \"a21d39c5-3813-4433-9307-855276f1a66e\") " pod="openshift-insights/insights-runtime-extractor-4wjcq" Oct 30 16:47:07.875708 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:07.875638 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3-metrics-tls\") pod \"dns-default-wc9zw\" (UID: \"bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3\") " pod="openshift-dns/dns-default-wc9zw" Oct 30 16:47:07.875708 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:47:07.875648 4522 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 16:47:07.875708 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:47:07.875660 4522 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 16:47:07.875883 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:47:07.875726 4522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ba7d788-14ef-4b39-b6f6-3072ab0b3200-metrics-certs podName:1ba7d788-14ef-4b39-b6f6-3072ab0b3200 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:08.875706919 +0000 UTC m=+2.876461815 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1ba7d788-14ef-4b39-b6f6-3072ab0b3200-metrics-certs") pod "network-metrics-daemon-rsgvx" (UID: "1ba7d788-14ef-4b39-b6f6-3072ab0b3200") : secret "metrics-daemon-secret" not found Oct 30 16:47:07.875883 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:47:07.875753 4522 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 16:47:07.875883 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:47:07.875802 4522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a21d39c5-3813-4433-9307-855276f1a66e-kube-rbac-proxy-cm podName:a21d39c5-3813-4433-9307-855276f1a66e nodeName:}" failed. No retries permitted until 2026-10-30 16:47:08.875779555 +0000 UTC m=+2.876534438 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a21d39c5-3813-4433-9307-855276f1a66e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4wjcq" (UID: "a21d39c5-3813-4433-9307-855276f1a66e") : configmap "kube-rbac-proxy" not found Oct 30 16:47:07.875883 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:47:07.875805 4522 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 16:47:07.875883 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:47:07.875858 4522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a21d39c5-3813-4433-9307-855276f1a66e-insights-runtime-extractor-tls podName:a21d39c5-3813-4433-9307-855276f1a66e nodeName:}" failed. No retries permitted until 2026-10-30 16:47:08.875836424 +0000 UTC m=+2.876591318 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a21d39c5-3813-4433-9307-855276f1a66e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4wjcq" (UID: "a21d39c5-3813-4433-9307-855276f1a66e") : secret "insights-runtime-extractor-tls" not found Oct 30 16:47:07.875883 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:47:07.875872 4522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3-metrics-tls podName:bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:08.87586605 +0000 UTC m=+2.876620929 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3-metrics-tls") pod "dns-default-wc9zw" (UID: "bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3") : secret "dns-default-metrics-tls" not found Oct 30 16:47:08.409253 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:08.409220 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cqjj8_e275e41d-9573-47e2-b923-173dfb4361a7/serve-healthcheck-canary/0.log" Oct 30 16:47:08.884341 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:08.884280 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a21d39c5-3813-4433-9307-855276f1a66e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4wjcq\" (UID: \"a21d39c5-3813-4433-9307-855276f1a66e\") " pod="openshift-insights/insights-runtime-extractor-4wjcq" Oct 30 16:47:08.884341 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:08.884341 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a21d39c5-3813-4433-9307-855276f1a66e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4wjcq\" (UID: \"a21d39c5-3813-4433-9307-855276f1a66e\") " pod="openshift-insights/insights-runtime-extractor-4wjcq" Oct 30 16:47:08.884574 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:08.884365 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3-metrics-tls\") pod \"dns-default-wc9zw\" (UID: \"bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3\") " pod="openshift-dns/dns-default-wc9zw" Oct 30 16:47:08.884574 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:08.884388 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ba7d788-14ef-4b39-b6f6-3072ab0b3200-metrics-certs\") pod \"network-metrics-daemon-rsgvx\" (UID: \"1ba7d788-14ef-4b39-b6f6-3072ab0b3200\") " pod="openshift-multus/network-metrics-daemon-rsgvx" Oct 30 16:47:08.884574 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:47:08.884401 4522 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 16:47:08.884574 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:47:08.884495 4522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a21d39c5-3813-4433-9307-855276f1a66e-kube-rbac-proxy-cm podName:a21d39c5-3813-4433-9307-855276f1a66e nodeName:}" failed. No retries permitted until 2026-10-30 16:47:10.884478446 +0000 UTC m=+4.885233343 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a21d39c5-3813-4433-9307-855276f1a66e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4wjcq" (UID: "a21d39c5-3813-4433-9307-855276f1a66e") : configmap "kube-rbac-proxy" not found Oct 30 16:47:08.884574 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:47:08.884502 4522 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 16:47:08.884574 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:47:08.884503 4522 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 16:47:08.884574 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:47:08.884507 4522 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 16:47:08.884574 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:47:08.884551 4522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ba7d788-14ef-4b39-b6f6-3072ab0b3200-metrics-certs podName:1ba7d788-14ef-4b39-b6f6-3072ab0b3200 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:10.884539694 +0000 UTC m=+4.885294573 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1ba7d788-14ef-4b39-b6f6-3072ab0b3200-metrics-certs") pod "network-metrics-daemon-rsgvx" (UID: "1ba7d788-14ef-4b39-b6f6-3072ab0b3200") : secret "metrics-daemon-secret" not found Oct 30 16:47:08.884831 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:47:08.884590 4522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3-metrics-tls podName:bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:10.884578582 +0000 UTC m=+4.885333460 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3-metrics-tls") pod "dns-default-wc9zw" (UID: "bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3") : secret "dns-default-metrics-tls" not found Oct 30 16:47:08.884831 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:47:08.884607 4522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a21d39c5-3813-4433-9307-855276f1a66e-insights-runtime-extractor-tls podName:a21d39c5-3813-4433-9307-855276f1a66e nodeName:}" failed. No retries permitted until 2026-10-30 16:47:10.884595537 +0000 UTC m=+4.885350416 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a21d39c5-3813-4433-9307-855276f1a66e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4wjcq" (UID: "a21d39c5-3813-4433-9307-855276f1a66e") : secret "insights-runtime-extractor-tls" not found Oct 30 16:47:09.214076 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:09.214046 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-zm2rj_cd83ecbc-117f-4890-bf7a-cab9282d4484/migrator/0.log" Oct 30 16:47:09.841092 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:09.841064 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:47:10.899708 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:10.899672 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a21d39c5-3813-4433-9307-855276f1a66e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4wjcq\" (UID: \"a21d39c5-3813-4433-9307-855276f1a66e\") " pod="openshift-insights/insights-runtime-extractor-4wjcq" Oct 30 16:47:10.899708 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:10.899712 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a21d39c5-3813-4433-9307-855276f1a66e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4wjcq\" (UID: \"a21d39c5-3813-4433-9307-855276f1a66e\") " pod="openshift-insights/insights-runtime-extractor-4wjcq" Oct 30 16:47:10.900189 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:10.899734 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3-metrics-tls\") pod \"dns-default-wc9zw\" (UID: \"bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3\") " pod="openshift-dns/dns-default-wc9zw" Oct 30 16:47:10.900189 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:10.899752 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ba7d788-14ef-4b39-b6f6-3072ab0b3200-metrics-certs\") pod \"network-metrics-daemon-rsgvx\" (UID: \"1ba7d788-14ef-4b39-b6f6-3072ab0b3200\") " pod="openshift-multus/network-metrics-daemon-rsgvx" Oct 30 16:47:10.900189 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:47:10.899803 4522 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 16:47:10.900189 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:47:10.899850 4522 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 16:47:10.900189 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:47:10.899859 4522 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 16:47:10.900189 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:47:10.899850 4522 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 16:47:10.900189 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:47:10.899892 4522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a21d39c5-3813-4433-9307-855276f1a66e-kube-rbac-proxy-cm podName:a21d39c5-3813-4433-9307-855276f1a66e nodeName:}" failed. No retries permitted until 2026-10-30 16:47:14.899878566 +0000 UTC m=+8.900633445 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a21d39c5-3813-4433-9307-855276f1a66e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4wjcq" (UID: "a21d39c5-3813-4433-9307-855276f1a66e") : configmap "kube-rbac-proxy" not found Oct 30 16:47:10.900189 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:47:10.899926 4522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ba7d788-14ef-4b39-b6f6-3072ab0b3200-metrics-certs podName:1ba7d788-14ef-4b39-b6f6-3072ab0b3200 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:14.899914801 +0000 UTC m=+8.900669683 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1ba7d788-14ef-4b39-b6f6-3072ab0b3200-metrics-certs") pod "network-metrics-daemon-rsgvx" (UID: "1ba7d788-14ef-4b39-b6f6-3072ab0b3200") : secret "metrics-daemon-secret" not found Oct 30 16:47:10.900189 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:47:10.899935 4522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3-metrics-tls podName:bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:14.89993024 +0000 UTC m=+8.900685123 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3-metrics-tls") pod "dns-default-wc9zw" (UID: "bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3") : secret "dns-default-metrics-tls" not found Oct 30 16:47:10.900189 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:47:10.899944 4522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a21d39c5-3813-4433-9307-855276f1a66e-insights-runtime-extractor-tls podName:a21d39c5-3813-4433-9307-855276f1a66e nodeName:}" failed. No retries permitted until 2026-10-30 16:47:14.899938979 +0000 UTC m=+8.900693857 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a21d39c5-3813-4433-9307-855276f1a66e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4wjcq" (UID: "a21d39c5-3813-4433-9307-855276f1a66e") : secret "insights-runtime-extractor-tls" not found Oct 30 16:47:11.607466 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:11.607437 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mqhg8" Oct 30 16:47:13.393346 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:13.393296 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8zqlg" Oct 30 16:47:14.930572 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:14.930519 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a21d39c5-3813-4433-9307-855276f1a66e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4wjcq\" (UID: \"a21d39c5-3813-4433-9307-855276f1a66e\") " pod="openshift-insights/insights-runtime-extractor-4wjcq" Oct 30 16:47:14.930572 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:14.930574 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a21d39c5-3813-4433-9307-855276f1a66e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4wjcq\" (UID: \"a21d39c5-3813-4433-9307-855276f1a66e\") " pod="openshift-insights/insights-runtime-extractor-4wjcq" Oct 30 16:47:14.931105 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:14.930596 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3-metrics-tls\") pod \"dns-default-wc9zw\" (UID: \"bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3\") " pod="openshift-dns/dns-default-wc9zw" Oct 30 16:47:14.931105 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:14.930616 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ba7d788-14ef-4b39-b6f6-3072ab0b3200-metrics-certs\") pod \"network-metrics-daemon-rsgvx\" (UID: \"1ba7d788-14ef-4b39-b6f6-3072ab0b3200\") " pod="openshift-multus/network-metrics-daemon-rsgvx" Oct 30 16:47:14.931105 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:14.930954 4522 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 16:47:14.931216 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:14.931114 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a21d39c5-3813-4433-9307-855276f1a66e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4wjcq\" (UID: \"a21d39c5-3813-4433-9307-855276f1a66e\") " pod="openshift-insights/insights-runtime-extractor-4wjcq" Oct 30 16:47:14.933110 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:14.933081 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3-metrics-tls\") pod \"dns-default-wc9zw\" (UID: \"bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3\") " pod="openshift-dns/dns-default-wc9zw" Oct 30 16:47:14.933231 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:14.933112 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ba7d788-14ef-4b39-b6f6-3072ab0b3200-metrics-certs\") pod \"network-metrics-daemon-rsgvx\" (UID: \"1ba7d788-14ef-4b39-b6f6-3072ab0b3200\") " pod="openshift-multus/network-metrics-daemon-rsgvx" Oct 30 16:47:14.933231 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:14.933079 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a21d39c5-3813-4433-9307-855276f1a66e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4wjcq\" (UID: \"a21d39c5-3813-4433-9307-855276f1a66e\") " pod="openshift-insights/insights-runtime-extractor-4wjcq" Oct 30 16:47:14.939030 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:14.939013 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wc9zw" Oct 30 16:47:14.940095 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:14.940077 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4wjcq" Oct 30 16:47:15.232145 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:15.232099 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsgvx" Oct 30 16:47:15.247828 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:15.247805 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4wjcq"] Oct 30 16:47:15.249203 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:15.249175 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda21d39c5_3813_4433_9307_855276f1a66e.slice/crio-288859dcc11b4a869a42719f542c652c02883a7861478a476b4a69ed88fef0c3 WatchSource:0}: Error finding container 288859dcc11b4a869a42719f542c652c02883a7861478a476b4a69ed88fef0c3: Status 404 returned error can't find the container with id 288859dcc11b4a869a42719f542c652c02883a7861478a476b4a69ed88fef0c3 Oct 30 16:47:15.259425 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:15.259402 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcdeb755_d77b_4d2c_95e0_aa2ddf7426d3.slice/crio-e3072faec743a214de650cafae8ec09672dc1a9509b2abfaf9e4171d1367d908 WatchSource:0}: Error finding container e3072faec743a214de650cafae8ec09672dc1a9509b2abfaf9e4171d1367d908: Status 404 returned error can't find the container with id e3072faec743a214de650cafae8ec09672dc1a9509b2abfaf9e4171d1367d908 Oct 30 16:47:15.261270 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:15.261245 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wc9zw"] Oct 30 16:47:15.261477 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:15.261458 4522 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 16:47:15.548375 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:15.548349 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rsgvx"] Oct 30 16:47:15.548506 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:15.548447 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ba7d788_14ef_4b39_b6f6_3072ab0b3200.slice/crio-8c1245c48d10085e8fb3fe8eb8e4cd29daec1a8b2860c9a80e4945de311b5d1b WatchSource:0}: Error finding container 8c1245c48d10085e8fb3fe8eb8e4cd29daec1a8b2860c9a80e4945de311b5d1b: Status 404 returned error can't find the container with id 8c1245c48d10085e8fb3fe8eb8e4cd29daec1a8b2860c9a80e4945de311b5d1b Oct 30 16:47:16.198792 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:16.198666 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wc9zw" event={"ID":"bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3","Type":"ContainerStarted","Data":"e3072faec743a214de650cafae8ec09672dc1a9509b2abfaf9e4171d1367d908"} Oct 30 16:47:16.200772 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:16.200739 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rsgvx" event={"ID":"1ba7d788-14ef-4b39-b6f6-3072ab0b3200","Type":"ContainerStarted","Data":"8c1245c48d10085e8fb3fe8eb8e4cd29daec1a8b2860c9a80e4945de311b5d1b"} Oct 30 16:47:16.203022 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:16.202993 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4wjcq" event={"ID":"a21d39c5-3813-4433-9307-855276f1a66e","Type":"ContainerStarted","Data":"6ff8e044cc9c1f2124015c34bf5ad6f23d9689da74bd5cf3066ab0a914a302fb"} Oct 30 16:47:16.203117 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:16.203049 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4wjcq" event={"ID":"a21d39c5-3813-4433-9307-855276f1a66e","Type":"ContainerStarted","Data":"46529ed2c0fa279e02591d0e9f8477e2ffbff7f65959f0196a4b36136a9d1310"} Oct 30 16:47:16.203117 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:16.203077 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4wjcq" event={"ID":"a21d39c5-3813-4433-9307-855276f1a66e","Type":"ContainerStarted","Data":"288859dcc11b4a869a42719f542c652c02883a7861478a476b4a69ed88fef0c3"} Oct 30 16:47:16.252465 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:16.251457 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:47:16.252465 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:16.251651 4522 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 16:47:17.210253 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:17.210220 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wc9zw" event={"ID":"bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3","Type":"ContainerStarted","Data":"3de862456bb877349efcc3b988b22b2395d4567c71971980b9f4f207335249cb"} Oct 30 16:47:17.212210 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:17.212183 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rsgvx" event={"ID":"1ba7d788-14ef-4b39-b6f6-3072ab0b3200","Type":"ContainerStarted","Data":"4c4696bc674de82690455741c62e9ba0bf33ae5d07e43673246216484997c2ca"} Oct 30 16:47:17.212301 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:17.212230 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rsgvx" event={"ID":"1ba7d788-14ef-4b39-b6f6-3072ab0b3200","Type":"ContainerStarted","Data":"97d119986234890b92e440cf3493f8e3ccb56a1a4cb9ec27a27b5a33d3b0fc64"} Oct 30 16:47:18.217125 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:18.217091 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wc9zw" event={"ID":"bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3","Type":"ContainerStarted","Data":"6fbef53d28e2d02885e8cef403b394c7122918655fb26540391fd6a7c1e661cd"} Oct 30 16:47:18.217556 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:18.217191 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wc9zw" Oct 30 16:47:18.219318 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:18.219286 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4wjcq" event={"ID":"a21d39c5-3813-4433-9307-855276f1a66e","Type":"ContainerStarted","Data":"d8038ed51de3bb95c826b6d3785bddb986f8dbd16460a901966b6b9ab2b730ad"} Oct 30 16:47:20.618438 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:20.618407 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-59dd87c5f7-xhqgq"] Oct 30 16:47:20.620264 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:20.620247 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59dd87c5f7-xhqgq" Oct 30 16:47:20.665931 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:20.665902 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 16:47:20.677935 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:20.677910 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 16:47:20.678063 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:20.677955 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-zs99n\"" Oct 30 16:47:20.678100 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:20.678051 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 16:47:20.678211 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:20.678196 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 16:47:20.678789 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:20.678770 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59dd87c5f7-xhqgq"] Oct 30 16:47:20.771324 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:20.771279 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/12b90e9c-b1ad-4b61-8372-8a3b5fb9e317-image-registry-private-configuration\") pod \"image-registry-59dd87c5f7-xhqgq\" (UID: \"12b90e9c-b1ad-4b61-8372-8a3b5fb9e317\") " pod="openshift-image-registry/image-registry-59dd87c5f7-xhqgq" Oct 30 16:47:20.771471 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:20.771343 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/12b90e9c-b1ad-4b61-8372-8a3b5fb9e317-bound-sa-token\") pod \"image-registry-59dd87c5f7-xhqgq\" (UID: \"12b90e9c-b1ad-4b61-8372-8a3b5fb9e317\") " pod="openshift-image-registry/image-registry-59dd87c5f7-xhqgq" Oct 30 16:47:20.771471 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:20.771417 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/12b90e9c-b1ad-4b61-8372-8a3b5fb9e317-registry-tls\") pod \"image-registry-59dd87c5f7-xhqgq\" (UID: \"12b90e9c-b1ad-4b61-8372-8a3b5fb9e317\") " pod="openshift-image-registry/image-registry-59dd87c5f7-xhqgq" Oct 30 16:47:20.771471 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:20.771439 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/12b90e9c-b1ad-4b61-8372-8a3b5fb9e317-installation-pull-secrets\") pod \"image-registry-59dd87c5f7-xhqgq\" (UID: \"12b90e9c-b1ad-4b61-8372-8a3b5fb9e317\") " pod="openshift-image-registry/image-registry-59dd87c5f7-xhqgq" Oct 30 16:47:20.771471 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:20.771458 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbrg2\" (UniqueName: \"kubernetes.io/projected/12b90e9c-b1ad-4b61-8372-8a3b5fb9e317-kube-api-access-hbrg2\") pod \"image-registry-59dd87c5f7-xhqgq\" (UID: \"12b90e9c-b1ad-4b61-8372-8a3b5fb9e317\") " pod="openshift-image-registry/image-registry-59dd87c5f7-xhqgq" Oct 30 16:47:20.771597 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:20.771481 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/12b90e9c-b1ad-4b61-8372-8a3b5fb9e317-trusted-ca\") pod \"image-registry-59dd87c5f7-xhqgq\" (UID: \"12b90e9c-b1ad-4b61-8372-8a3b5fb9e317\") " pod="openshift-image-registry/image-registry-59dd87c5f7-xhqgq" Oct 30 16:47:20.771597 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:20.771546 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/12b90e9c-b1ad-4b61-8372-8a3b5fb9e317-registry-certificates\") pod \"image-registry-59dd87c5f7-xhqgq\" (UID: \"12b90e9c-b1ad-4b61-8372-8a3b5fb9e317\") " pod="openshift-image-registry/image-registry-59dd87c5f7-xhqgq" Oct 30 16:47:20.771658 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:20.771614 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/12b90e9c-b1ad-4b61-8372-8a3b5fb9e317-ca-trust-extracted\") pod \"image-registry-59dd87c5f7-xhqgq\" (UID: \"12b90e9c-b1ad-4b61-8372-8a3b5fb9e317\") " pod="openshift-image-registry/image-registry-59dd87c5f7-xhqgq" Oct 30 16:47:20.872640 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:20.872567 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/12b90e9c-b1ad-4b61-8372-8a3b5fb9e317-trusted-ca\") pod \"image-registry-59dd87c5f7-xhqgq\" (UID: \"12b90e9c-b1ad-4b61-8372-8a3b5fb9e317\") " pod="openshift-image-registry/image-registry-59dd87c5f7-xhqgq" Oct 30 16:47:20.872640 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:20.872606 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/12b90e9c-b1ad-4b61-8372-8a3b5fb9e317-registry-certificates\") pod \"image-registry-59dd87c5f7-xhqgq\" (UID: \"12b90e9c-b1ad-4b61-8372-8a3b5fb9e317\") " pod="openshift-image-registry/image-registry-59dd87c5f7-xhqgq" Oct 30 16:47:20.872640 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:20.872634 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/12b90e9c-b1ad-4b61-8372-8a3b5fb9e317-ca-trust-extracted\") pod \"image-registry-59dd87c5f7-xhqgq\" (UID: \"12b90e9c-b1ad-4b61-8372-8a3b5fb9e317\") " pod="openshift-image-registry/image-registry-59dd87c5f7-xhqgq" Oct 30 16:47:20.872883 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:20.872672 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/12b90e9c-b1ad-4b61-8372-8a3b5fb9e317-image-registry-private-configuration\") pod \"image-registry-59dd87c5f7-xhqgq\" (UID: \"12b90e9c-b1ad-4b61-8372-8a3b5fb9e317\") " pod="openshift-image-registry/image-registry-59dd87c5f7-xhqgq" Oct 30 16:47:20.872883 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:20.872694 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/12b90e9c-b1ad-4b61-8372-8a3b5fb9e317-bound-sa-token\") pod \"image-registry-59dd87c5f7-xhqgq\" (UID: \"12b90e9c-b1ad-4b61-8372-8a3b5fb9e317\") " pod="openshift-image-registry/image-registry-59dd87c5f7-xhqgq" Oct 30 16:47:20.872883 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:20.872712 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/12b90e9c-b1ad-4b61-8372-8a3b5fb9e317-registry-tls\") pod \"image-registry-59dd87c5f7-xhqgq\" (UID: \"12b90e9c-b1ad-4b61-8372-8a3b5fb9e317\") " pod="openshift-image-registry/image-registry-59dd87c5f7-xhqgq" Oct 30 16:47:20.872883 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:20.872728 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/12b90e9c-b1ad-4b61-8372-8a3b5fb9e317-installation-pull-secrets\") pod \"image-registry-59dd87c5f7-xhqgq\" (UID: \"12b90e9c-b1ad-4b61-8372-8a3b5fb9e317\") " pod="openshift-image-registry/image-registry-59dd87c5f7-xhqgq" Oct 30 16:47:20.872883 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:20.872744 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbrg2\" (UniqueName: \"kubernetes.io/projected/12b90e9c-b1ad-4b61-8372-8a3b5fb9e317-kube-api-access-hbrg2\") pod \"image-registry-59dd87c5f7-xhqgq\" (UID: \"12b90e9c-b1ad-4b61-8372-8a3b5fb9e317\") " pod="openshift-image-registry/image-registry-59dd87c5f7-xhqgq" Oct 30 16:47:20.873218 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:20.873182 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/12b90e9c-b1ad-4b61-8372-8a3b5fb9e317-ca-trust-extracted\") pod \"image-registry-59dd87c5f7-xhqgq\" (UID: \"12b90e9c-b1ad-4b61-8372-8a3b5fb9e317\") " pod="openshift-image-registry/image-registry-59dd87c5f7-xhqgq" Oct 30 16:47:20.873878 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:20.873855 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/12b90e9c-b1ad-4b61-8372-8a3b5fb9e317-trusted-ca\") pod \"image-registry-59dd87c5f7-xhqgq\" (UID: \"12b90e9c-b1ad-4b61-8372-8a3b5fb9e317\") " pod="openshift-image-registry/image-registry-59dd87c5f7-xhqgq" Oct 30 16:47:20.873878 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:20.873860 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/12b90e9c-b1ad-4b61-8372-8a3b5fb9e317-registry-certificates\") pod \"image-registry-59dd87c5f7-xhqgq\" (UID: \"12b90e9c-b1ad-4b61-8372-8a3b5fb9e317\") " pod="openshift-image-registry/image-registry-59dd87c5f7-xhqgq" Oct 30 16:47:20.875296 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:20.875277 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/12b90e9c-b1ad-4b61-8372-8a3b5fb9e317-installation-pull-secrets\") pod \"image-registry-59dd87c5f7-xhqgq\" (UID: \"12b90e9c-b1ad-4b61-8372-8a3b5fb9e317\") " pod="openshift-image-registry/image-registry-59dd87c5f7-xhqgq" Oct 30 16:47:20.875550 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:20.875528 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/12b90e9c-b1ad-4b61-8372-8a3b5fb9e317-image-registry-private-configuration\") pod \"image-registry-59dd87c5f7-xhqgq\" (UID: \"12b90e9c-b1ad-4b61-8372-8a3b5fb9e317\") " pod="openshift-image-registry/image-registry-59dd87c5f7-xhqgq" Oct 30 16:47:20.875625 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:20.875608 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/12b90e9c-b1ad-4b61-8372-8a3b5fb9e317-registry-tls\") pod \"image-registry-59dd87c5f7-xhqgq\" (UID: \"12b90e9c-b1ad-4b61-8372-8a3b5fb9e317\") " pod="openshift-image-registry/image-registry-59dd87c5f7-xhqgq" Oct 30 16:47:20.900699 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:20.900674 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbrg2\" (UniqueName: \"kubernetes.io/projected/12b90e9c-b1ad-4b61-8372-8a3b5fb9e317-kube-api-access-hbrg2\") pod \"image-registry-59dd87c5f7-xhqgq\" (UID: \"12b90e9c-b1ad-4b61-8372-8a3b5fb9e317\") " pod="openshift-image-registry/image-registry-59dd87c5f7-xhqgq" Oct 30 16:47:20.905192 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:20.905174 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/12b90e9c-b1ad-4b61-8372-8a3b5fb9e317-bound-sa-token\") pod \"image-registry-59dd87c5f7-xhqgq\" (UID: \"12b90e9c-b1ad-4b61-8372-8a3b5fb9e317\") " pod="openshift-image-registry/image-registry-59dd87c5f7-xhqgq" Oct 30 16:47:20.931299 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:20.931270 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59dd87c5f7-xhqgq" Oct 30 16:47:21.255963 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:21.255930 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59dd87c5f7-xhqgq"] Oct 30 16:47:21.257295 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:21.257273 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12b90e9c_b1ad_4b61_8372_8a3b5fb9e317.slice/crio-202566f2a8cfad0df8c34ad9e09f45f4f1cc162c616d1d45be91aa10a79ac778 WatchSource:0}: Error finding container 202566f2a8cfad0df8c34ad9e09f45f4f1cc162c616d1d45be91aa10a79ac778: Status 404 returned error can't find the container with id 202566f2a8cfad0df8c34ad9e09f45f4f1cc162c616d1d45be91aa10a79ac778 Oct 30 16:47:22.232726 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:22.232690 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59dd87c5f7-xhqgq" event={"ID":"12b90e9c-b1ad-4b61-8372-8a3b5fb9e317","Type":"ContainerStarted","Data":"3cf9d26580bd541165df7c4f404488784090cd518e25e7a3a16fa8869913ef51"} Oct 30 16:47:22.233119 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:22.232746 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59dd87c5f7-xhqgq" event={"ID":"12b90e9c-b1ad-4b61-8372-8a3b5fb9e317","Type":"ContainerStarted","Data":"202566f2a8cfad0df8c34ad9e09f45f4f1cc162c616d1d45be91aa10a79ac778"} Oct 30 16:47:22.233119 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:22.232822 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-59dd87c5f7-xhqgq" Oct 30 16:47:22.308481 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:22.308405 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-59dd87c5f7-xhqgq" podStartSLOduration=2.308389506 podStartE2EDuration="2.308389506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:47:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:47:22.294622374 +0000 UTC m=+16.295377295" watchObservedRunningTime="2026-10-30 16:47:22.308389506 +0000 UTC m=+16.309144404" Oct 30 16:47:28.117794 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.117756 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-6ljvl"] Oct 30 16:47:28.121104 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.121084 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6ljvl" Oct 30 16:47:28.144696 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.144667 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 16:47:28.158131 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.158096 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 16:47:28.158282 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.158149 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-vgp4k\"" Oct 30 16:47:28.158374 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.158351 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 16:47:28.158456 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.158428 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 16:47:28.158606 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.158483 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 16:47:28.159156 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.159137 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-6ljvl"] Oct 30 16:47:28.173703 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.173673 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-c46xd"] Oct 30 16:47:28.176230 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.176211 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c46xd" Oct 30 16:47:28.177076 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.177055 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-cvp4x"] Oct 30 16:47:28.178872 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.178855 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cvp4x" Oct 30 16:47:28.204540 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.204519 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-c46xd"] Oct 30 16:47:28.214355 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.214333 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 16:47:28.214460 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.214378 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 16:47:28.216353 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.216334 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 16:47:28.216446 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.216351 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 16:47:28.216509 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.216494 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 16:47:28.216617 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.216602 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-8rklc\"" Oct 30 16:47:28.216911 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.216886 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 16:47:28.216911 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.216892 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rb7tp\"" Oct 30 16:47:28.221346 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.221328 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48ac41e9-e6af-4f7e-925a-c871db2be357-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-6ljvl\" (UID: \"48ac41e9-e6af-4f7e-925a-c871db2be357\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6ljvl" Oct 30 16:47:28.221419 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.221368 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvjc9\" (UniqueName: \"kubernetes.io/projected/48ac41e9-e6af-4f7e-925a-c871db2be357-kube-api-access-qvjc9\") pod \"openshift-state-metrics-7bd7489957-6ljvl\" (UID: \"48ac41e9-e6af-4f7e-925a-c871db2be357\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6ljvl" Oct 30 16:47:28.221419 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.221392 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/48ac41e9-e6af-4f7e-925a-c871db2be357-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-6ljvl\" (UID: \"48ac41e9-e6af-4f7e-925a-c871db2be357\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6ljvl" Oct 30 16:47:28.221510 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.221442 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48ac41e9-e6af-4f7e-925a-c871db2be357-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-6ljvl\" (UID: \"48ac41e9-e6af-4f7e-925a-c871db2be357\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6ljvl" Oct 30 16:47:28.223691 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.223673 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wc9zw" Oct 30 16:47:28.322532 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.322501 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvjc9\" (UniqueName: \"kubernetes.io/projected/48ac41e9-e6af-4f7e-925a-c871db2be357-kube-api-access-qvjc9\") pod \"openshift-state-metrics-7bd7489957-6ljvl\" (UID: \"48ac41e9-e6af-4f7e-925a-c871db2be357\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6ljvl" Oct 30 16:47:28.322532 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.322532 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/48ac41e9-e6af-4f7e-925a-c871db2be357-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-6ljvl\" (UID: \"48ac41e9-e6af-4f7e-925a-c871db2be357\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6ljvl" Oct 30 16:47:28.322774 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.322553 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48ac41e9-e6af-4f7e-925a-c871db2be357-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-6ljvl\" (UID: \"48ac41e9-e6af-4f7e-925a-c871db2be357\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6ljvl" Oct 30 16:47:28.323098 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.322781 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b0a9853e-3d0e-4385-954c-bf4e19c996b8-metrics-client-ca\") pod \"node-exporter-cvp4x\" (UID: \"b0a9853e-3d0e-4385-954c-bf4e19c996b8\") " pod="openshift-monitoring/node-exporter-cvp4x" Oct 30 16:47:28.323098 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.322844 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/01e543d8-5483-4cc6-8af5-aaf0fc93b069-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-c46xd\" (UID: \"01e543d8-5483-4cc6-8af5-aaf0fc93b069\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c46xd" Oct 30 16:47:28.323098 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.322888 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b0a9853e-3d0e-4385-954c-bf4e19c996b8-node-exporter-tls\") pod \"node-exporter-cvp4x\" (UID: \"b0a9853e-3d0e-4385-954c-bf4e19c996b8\") " pod="openshift-monitoring/node-exporter-cvp4x" Oct 30 16:47:28.323098 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.322920 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b0a9853e-3d0e-4385-954c-bf4e19c996b8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cvp4x\" (UID: \"b0a9853e-3d0e-4385-954c-bf4e19c996b8\") " pod="openshift-monitoring/node-exporter-cvp4x" Oct 30 16:47:28.323098 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.323030 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/01e543d8-5483-4cc6-8af5-aaf0fc93b069-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-c46xd\" (UID: \"01e543d8-5483-4cc6-8af5-aaf0fc93b069\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c46xd" Oct 30 16:47:28.323098 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.323068 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b0a9853e-3d0e-4385-954c-bf4e19c996b8-sys\") pod \"node-exporter-cvp4x\" (UID: \"b0a9853e-3d0e-4385-954c-bf4e19c996b8\") " pod="openshift-monitoring/node-exporter-cvp4x" Oct 30 16:47:28.323507 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.323144 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b0a9853e-3d0e-4385-954c-bf4e19c996b8-node-exporter-wtmp\") pod \"node-exporter-cvp4x\" (UID: \"b0a9853e-3d0e-4385-954c-bf4e19c996b8\") " pod="openshift-monitoring/node-exporter-cvp4x" Oct 30 16:47:28.323507 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.323169 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jggbz\" (UniqueName: \"kubernetes.io/projected/01e543d8-5483-4cc6-8af5-aaf0fc93b069-kube-api-access-jggbz\") pod \"kube-state-metrics-5dc8d7d89b-c46xd\" (UID: \"01e543d8-5483-4cc6-8af5-aaf0fc93b069\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c46xd" Oct 30 16:47:28.323507 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.323203 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01e543d8-5483-4cc6-8af5-aaf0fc93b069-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-c46xd\" (UID: \"01e543d8-5483-4cc6-8af5-aaf0fc93b069\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c46xd" Oct 30 16:47:28.323507 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.323230 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/01e543d8-5483-4cc6-8af5-aaf0fc93b069-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-c46xd\" (UID: \"01e543d8-5483-4cc6-8af5-aaf0fc93b069\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c46xd" Oct 30 16:47:28.323507 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.323356 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b0a9853e-3d0e-4385-954c-bf4e19c996b8-root\") pod \"node-exporter-cvp4x\" (UID: \"b0a9853e-3d0e-4385-954c-bf4e19c996b8\") " pod="openshift-monitoring/node-exporter-cvp4x" Oct 30 16:47:28.323507 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.323379 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b0a9853e-3d0e-4385-954c-bf4e19c996b8-node-exporter-textfile\") pod \"node-exporter-cvp4x\" (UID: \"b0a9853e-3d0e-4385-954c-bf4e19c996b8\") " pod="openshift-monitoring/node-exporter-cvp4x" Oct 30 16:47:28.323507 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.323420 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b0a9853e-3d0e-4385-954c-bf4e19c996b8-node-exporter-accelerators-collector-config\") pod \"node-exporter-cvp4x\" (UID: \"b0a9853e-3d0e-4385-954c-bf4e19c996b8\") " pod="openshift-monitoring/node-exporter-cvp4x" Oct 30 16:47:28.323507 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.323445 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48ac41e9-e6af-4f7e-925a-c871db2be357-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-6ljvl\" (UID: \"48ac41e9-e6af-4f7e-925a-c871db2be357\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6ljvl" Oct 30 16:47:28.323507 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.323463 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk7ln\" (UniqueName: \"kubernetes.io/projected/b0a9853e-3d0e-4385-954c-bf4e19c996b8-kube-api-access-rk7ln\") pod \"node-exporter-cvp4x\" (UID: \"b0a9853e-3d0e-4385-954c-bf4e19c996b8\") " pod="openshift-monitoring/node-exporter-cvp4x" Oct 30 16:47:28.323507 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.323487 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/01e543d8-5483-4cc6-8af5-aaf0fc93b069-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-c46xd\" (UID: \"01e543d8-5483-4cc6-8af5-aaf0fc93b069\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c46xd" Oct 30 16:47:28.324002 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.323987 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48ac41e9-e6af-4f7e-925a-c871db2be357-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-6ljvl\" (UID: \"48ac41e9-e6af-4f7e-925a-c871db2be357\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6ljvl" Oct 30 16:47:28.324969 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.324947 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48ac41e9-e6af-4f7e-925a-c871db2be357-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-6ljvl\" (UID: \"48ac41e9-e6af-4f7e-925a-c871db2be357\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6ljvl" Oct 30 16:47:28.325192 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.325174 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/48ac41e9-e6af-4f7e-925a-c871db2be357-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-6ljvl\" (UID: \"48ac41e9-e6af-4f7e-925a-c871db2be357\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6ljvl" Oct 30 16:47:28.351605 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.351579 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvjc9\" (UniqueName: \"kubernetes.io/projected/48ac41e9-e6af-4f7e-925a-c871db2be357-kube-api-access-qvjc9\") pod \"openshift-state-metrics-7bd7489957-6ljvl\" (UID: \"48ac41e9-e6af-4f7e-925a-c871db2be357\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6ljvl" Oct 30 16:47:28.424705 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.424609 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b0a9853e-3d0e-4385-954c-bf4e19c996b8-node-exporter-wtmp\") pod \"node-exporter-cvp4x\" (UID: \"b0a9853e-3d0e-4385-954c-bf4e19c996b8\") " pod="openshift-monitoring/node-exporter-cvp4x" Oct 30 16:47:28.424705 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.424659 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jggbz\" (UniqueName: \"kubernetes.io/projected/01e543d8-5483-4cc6-8af5-aaf0fc93b069-kube-api-access-jggbz\") pod \"kube-state-metrics-5dc8d7d89b-c46xd\" (UID: \"01e543d8-5483-4cc6-8af5-aaf0fc93b069\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c46xd" Oct 30 16:47:28.424705 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.424695 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01e543d8-5483-4cc6-8af5-aaf0fc93b069-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-c46xd\" (UID: \"01e543d8-5483-4cc6-8af5-aaf0fc93b069\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c46xd" Oct 30 16:47:28.424957 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.424720 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/01e543d8-5483-4cc6-8af5-aaf0fc93b069-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-c46xd\" (UID: \"01e543d8-5483-4cc6-8af5-aaf0fc93b069\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c46xd" Oct 30 16:47:28.424957 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.424744 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b0a9853e-3d0e-4385-954c-bf4e19c996b8-root\") pod \"node-exporter-cvp4x\" (UID: \"b0a9853e-3d0e-4385-954c-bf4e19c996b8\") " pod="openshift-monitoring/node-exporter-cvp4x" Oct 30 16:47:28.424957 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.424763 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b0a9853e-3d0e-4385-954c-bf4e19c996b8-node-exporter-textfile\") pod \"node-exporter-cvp4x\" (UID: \"b0a9853e-3d0e-4385-954c-bf4e19c996b8\") " pod="openshift-monitoring/node-exporter-cvp4x" Oct 30 16:47:28.424957 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.424790 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b0a9853e-3d0e-4385-954c-bf4e19c996b8-node-exporter-accelerators-collector-config\") pod \"node-exporter-cvp4x\" (UID: \"b0a9853e-3d0e-4385-954c-bf4e19c996b8\") " pod="openshift-monitoring/node-exporter-cvp4x" Oct 30 16:47:28.424957 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.424813 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b0a9853e-3d0e-4385-954c-bf4e19c996b8-node-exporter-wtmp\") pod \"node-exporter-cvp4x\" (UID: \"b0a9853e-3d0e-4385-954c-bf4e19c996b8\") " pod="openshift-monitoring/node-exporter-cvp4x" Oct 30 16:47:28.424957 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.424843 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b0a9853e-3d0e-4385-954c-bf4e19c996b8-root\") pod \"node-exporter-cvp4x\" (UID: \"b0a9853e-3d0e-4385-954c-bf4e19c996b8\") " pod="openshift-monitoring/node-exporter-cvp4x" Oct 30 16:47:28.424957 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.424819 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk7ln\" (UniqueName: \"kubernetes.io/projected/b0a9853e-3d0e-4385-954c-bf4e19c996b8-kube-api-access-rk7ln\") pod \"node-exporter-cvp4x\" (UID: \"b0a9853e-3d0e-4385-954c-bf4e19c996b8\") " pod="openshift-monitoring/node-exporter-cvp4x" Oct 30 16:47:28.424957 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.424916 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/01e543d8-5483-4cc6-8af5-aaf0fc93b069-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-c46xd\" (UID: \"01e543d8-5483-4cc6-8af5-aaf0fc93b069\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c46xd" Oct 30 16:47:28.425353 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.424976 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b0a9853e-3d0e-4385-954c-bf4e19c996b8-metrics-client-ca\") pod \"node-exporter-cvp4x\" (UID: \"b0a9853e-3d0e-4385-954c-bf4e19c996b8\") " pod="openshift-monitoring/node-exporter-cvp4x" Oct 30 16:47:28.425353 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.425009 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/01e543d8-5483-4cc6-8af5-aaf0fc93b069-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-c46xd\" (UID: \"01e543d8-5483-4cc6-8af5-aaf0fc93b069\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c46xd" Oct 30 16:47:28.425353 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.425033 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b0a9853e-3d0e-4385-954c-bf4e19c996b8-node-exporter-tls\") pod \"node-exporter-cvp4x\" (UID: \"b0a9853e-3d0e-4385-954c-bf4e19c996b8\") " pod="openshift-monitoring/node-exporter-cvp4x" Oct 30 16:47:28.425353 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.425064 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b0a9853e-3d0e-4385-954c-bf4e19c996b8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cvp4x\" (UID: \"b0a9853e-3d0e-4385-954c-bf4e19c996b8\") " pod="openshift-monitoring/node-exporter-cvp4x" Oct 30 16:47:28.425353 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.425092 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/01e543d8-5483-4cc6-8af5-aaf0fc93b069-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-c46xd\" (UID: \"01e543d8-5483-4cc6-8af5-aaf0fc93b069\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c46xd" Oct 30 16:47:28.425353 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.425116 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b0a9853e-3d0e-4385-954c-bf4e19c996b8-sys\") pod \"node-exporter-cvp4x\" (UID: \"b0a9853e-3d0e-4385-954c-bf4e19c996b8\") " pod="openshift-monitoring/node-exporter-cvp4x" Oct 30 16:47:28.425353 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.425187 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b0a9853e-3d0e-4385-954c-bf4e19c996b8-sys\") pod \"node-exporter-cvp4x\" (UID: \"b0a9853e-3d0e-4385-954c-bf4e19c996b8\") " pod="openshift-monitoring/node-exporter-cvp4x" Oct 30 16:47:28.425699 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.425394 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/01e543d8-5483-4cc6-8af5-aaf0fc93b069-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-c46xd\" (UID: \"01e543d8-5483-4cc6-8af5-aaf0fc93b069\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c46xd" Oct 30 16:47:28.425699 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.425612 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b0a9853e-3d0e-4385-954c-bf4e19c996b8-node-exporter-accelerators-collector-config\") pod \"node-exporter-cvp4x\" (UID: \"b0a9853e-3d0e-4385-954c-bf4e19c996b8\") " pod="openshift-monitoring/node-exporter-cvp4x" Oct 30 16:47:28.425699 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.425660 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b0a9853e-3d0e-4385-954c-bf4e19c996b8-metrics-client-ca\") pod \"node-exporter-cvp4x\" (UID: \"b0a9853e-3d0e-4385-954c-bf4e19c996b8\") " pod="openshift-monitoring/node-exporter-cvp4x" Oct 30 16:47:28.425844 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.425733 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b0a9853e-3d0e-4385-954c-bf4e19c996b8-node-exporter-textfile\") pod \"node-exporter-cvp4x\" (UID: \"b0a9853e-3d0e-4385-954c-bf4e19c996b8\") " pod="openshift-monitoring/node-exporter-cvp4x" Oct 30 16:47:28.425844 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.425738 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01e543d8-5483-4cc6-8af5-aaf0fc93b069-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-c46xd\" (UID: \"01e543d8-5483-4cc6-8af5-aaf0fc93b069\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c46xd" Oct 30 16:47:28.426009 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.425988 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/01e543d8-5483-4cc6-8af5-aaf0fc93b069-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-c46xd\" (UID: \"01e543d8-5483-4cc6-8af5-aaf0fc93b069\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c46xd" Oct 30 16:47:28.427599 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.427577 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b0a9853e-3d0e-4385-954c-bf4e19c996b8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cvp4x\" (UID: \"b0a9853e-3d0e-4385-954c-bf4e19c996b8\") " pod="openshift-monitoring/node-exporter-cvp4x" Oct 30 16:47:28.427834 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.427812 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/01e543d8-5483-4cc6-8af5-aaf0fc93b069-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-c46xd\" (UID: \"01e543d8-5483-4cc6-8af5-aaf0fc93b069\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c46xd" Oct 30 16:47:28.427886 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.427858 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b0a9853e-3d0e-4385-954c-bf4e19c996b8-node-exporter-tls\") pod \"node-exporter-cvp4x\" (UID: \"b0a9853e-3d0e-4385-954c-bf4e19c996b8\") " pod="openshift-monitoring/node-exporter-cvp4x" Oct 30 16:47:28.427954 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.427937 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/01e543d8-5483-4cc6-8af5-aaf0fc93b069-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-c46xd\" (UID: \"01e543d8-5483-4cc6-8af5-aaf0fc93b069\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c46xd" Oct 30 16:47:28.437832 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.437810 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6ljvl" Oct 30 16:47:28.452587 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.452555 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jggbz\" (UniqueName: \"kubernetes.io/projected/01e543d8-5483-4cc6-8af5-aaf0fc93b069-kube-api-access-jggbz\") pod \"kube-state-metrics-5dc8d7d89b-c46xd\" (UID: \"01e543d8-5483-4cc6-8af5-aaf0fc93b069\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c46xd" Oct 30 16:47:28.452837 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.452812 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk7ln\" (UniqueName: \"kubernetes.io/projected/b0a9853e-3d0e-4385-954c-bf4e19c996b8-kube-api-access-rk7ln\") pod \"node-exporter-cvp4x\" (UID: \"b0a9853e-3d0e-4385-954c-bf4e19c996b8\") " pod="openshift-monitoring/node-exporter-cvp4x" Oct 30 16:47:28.492330 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.492184 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c46xd" Oct 30 16:47:28.495755 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.495734 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cvp4x" Oct 30 16:47:28.513838 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:28.513804 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0a9853e_3d0e_4385_954c_bf4e19c996b8.slice/crio-c25688d5c8f9a86c16f38cd9d22779422c1b18d3c1d494dcd3a37212edb5c8e7 WatchSource:0}: Error finding container c25688d5c8f9a86c16f38cd9d22779422c1b18d3c1d494dcd3a37212edb5c8e7: Status 404 returned error can't find the container with id c25688d5c8f9a86c16f38cd9d22779422c1b18d3c1d494dcd3a37212edb5c8e7 Oct 30 16:47:28.746882 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.746855 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-6ljvl"] Oct 30 16:47:28.747605 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:28.747577 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48ac41e9_e6af_4f7e_925a_c871db2be357.slice/crio-5618b8184e68352560ebd56c7620781f2fe3b246058879e28e74fe3ab0f9c07c WatchSource:0}: Error finding container 5618b8184e68352560ebd56c7620781f2fe3b246058879e28e74fe3ab0f9c07c: Status 404 returned error can't find the container with id 5618b8184e68352560ebd56c7620781f2fe3b246058879e28e74fe3ab0f9c07c Oct 30 16:47:28.801332 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:28.801293 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-c46xd"] Oct 30 16:47:28.802619 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:28.802591 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01e543d8_5483_4cc6_8af5_aaf0fc93b069.slice/crio-f8636baef6fd1d2c32d5fbd2eee00317876faf3eae26aa453e4841461c8858ec WatchSource:0}: Error finding container f8636baef6fd1d2c32d5fbd2eee00317876faf3eae26aa453e4841461c8858ec: Status 404 returned error can't find the container with id f8636baef6fd1d2c32d5fbd2eee00317876faf3eae26aa453e4841461c8858ec Oct 30 16:47:29.255518 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:29.255478 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cvp4x" event={"ID":"b0a9853e-3d0e-4385-954c-bf4e19c996b8","Type":"ContainerStarted","Data":"c25688d5c8f9a86c16f38cd9d22779422c1b18d3c1d494dcd3a37212edb5c8e7"} Oct 30 16:47:29.257006 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:29.256975 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c46xd" event={"ID":"01e543d8-5483-4cc6-8af5-aaf0fc93b069","Type":"ContainerStarted","Data":"f8636baef6fd1d2c32d5fbd2eee00317876faf3eae26aa453e4841461c8858ec"} Oct 30 16:47:29.259294 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:29.259257 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6ljvl" event={"ID":"48ac41e9-e6af-4f7e-925a-c871db2be357","Type":"ContainerStarted","Data":"e40096f3247f0ac66dcd52d89bd00d1e17c54c1e1c0b78bee79428204821eec1"} Oct 30 16:47:29.259423 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:29.259330 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6ljvl" event={"ID":"48ac41e9-e6af-4f7e-925a-c871db2be357","Type":"ContainerStarted","Data":"c24ec16a471831197fe4bf80ffe90414f2d13df395ad6c3805e1c362c9224745"} Oct 30 16:47:29.259423 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:29.259358 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6ljvl" event={"ID":"48ac41e9-e6af-4f7e-925a-c871db2be357","Type":"ContainerStarted","Data":"5618b8184e68352560ebd56c7620781f2fe3b246058879e28e74fe3ab0f9c07c"} Oct 30 16:47:30.264873 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:30.264839 4522 generic.go:356] "Generic (PLEG): container finished" podID="b0a9853e-3d0e-4385-954c-bf4e19c996b8" containerID="5ad91c80117ae4a9311192f81965efd239d6a3236d12827cdf9865b08154a442" exitCode=0 Oct 30 16:47:30.265375 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:30.264926 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cvp4x" event={"ID":"b0a9853e-3d0e-4385-954c-bf4e19c996b8","Type":"ContainerDied","Data":"5ad91c80117ae4a9311192f81965efd239d6a3236d12827cdf9865b08154a442"} Oct 30 16:47:30.268438 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:30.268409 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c46xd" event={"ID":"01e543d8-5483-4cc6-8af5-aaf0fc93b069","Type":"ContainerStarted","Data":"eef9516918d599aa554f2fa13d27f80e04dbd240066741d7766beb0cf5aac339"} Oct 30 16:47:30.268588 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:30.268468 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c46xd" event={"ID":"01e543d8-5483-4cc6-8af5-aaf0fc93b069","Type":"ContainerStarted","Data":"627a26a7321c4e955820b7b77c8e5352f4bc2cff4e1f032f749b0966d8bf091b"} Oct 30 16:47:30.268588 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:30.268517 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c46xd" event={"ID":"01e543d8-5483-4cc6-8af5-aaf0fc93b069","Type":"ContainerStarted","Data":"dbb20aac17b049854e20aaebc8b330ce45eaf976b6911f1940938dd0403897d4"} Oct 30 16:47:30.271128 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:30.271107 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6ljvl" event={"ID":"48ac41e9-e6af-4f7e-925a-c871db2be357","Type":"ContainerStarted","Data":"0fc55d35596276a9f9c10b5d5705f8adaab2dba32adb72669a035ffd3a87ecb8"} Oct 30 16:47:30.373445 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:30.373276 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6ljvl" podStartSLOduration=1.332188471 podStartE2EDuration="2.373261494s" totalImagesPullingTime="1.041073023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:47:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:47:30.373211178 +0000 UTC m=+24.373966091" watchObservedRunningTime="2026-10-30 16:47:30.373261494 +0000 UTC m=+24.374016394" Oct 30 16:47:30.424525 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:30.424467 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c46xd" podStartSLOduration=1.156611712 podStartE2EDuration="2.424451298s" totalImagesPullingTime="1.267839586s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:47:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:47:30.423682425 +0000 UTC m=+24.424437317" watchObservedRunningTime="2026-10-30 16:47:30.424451298 +0000 UTC m=+24.425206198" Oct 30 16:47:31.276987 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:31.276949 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cvp4x" event={"ID":"b0a9853e-3d0e-4385-954c-bf4e19c996b8","Type":"ContainerStarted","Data":"3cbb59016c4a792a400987c720fe4e6b7a0bc5d9fe0451f129654feefc4e1b25"} Oct 30 16:47:31.277502 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:31.277003 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cvp4x" event={"ID":"b0a9853e-3d0e-4385-954c-bf4e19c996b8","Type":"ContainerStarted","Data":"a36bcac1689a5851cc32bd76973c65e052b317e36b44df0552d37c10bf3fc241"} Oct 30 16:47:31.355752 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:31.355704 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-cvp4x" podStartSLOduration=2.268463911 podStartE2EDuration="3.355690972s" totalImagesPullingTime="1.006954516s" totalInitContainerRuntime="80.272545ms" isStatefulPod=true podCreationTimestamp="2026-10-30 16:47:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:47:31.341969946 +0000 UTC m=+25.342724887" watchObservedRunningTime="2026-10-30 16:47:31.355690972 +0000 UTC m=+25.356445867" Oct 30 16:47:32.347588 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:32.347557 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ns7rp" Oct 30 16:47:33.465527 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:33.465495 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-79b7d6d8f-d5kcb"] Oct 30 16:47:33.468209 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:33.468190 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-79b7d6d8f-d5kcb" Oct 30 16:47:33.486007 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:33.485977 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-79b7d6d8f-d5kcb"] Oct 30 16:47:33.488915 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:33.488890 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 30 16:47:33.488915 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:33.488890 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 30 16:47:33.498563 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:33.498540 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-dccst\"" Oct 30 16:47:33.498563 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:33.498552 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 30 16:47:33.498563 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:33.498549 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 30 16:47:33.498750 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:33.498590 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 30 16:47:33.516462 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:33.516436 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 30 16:47:33.570402 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:33.570367 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxbjp\" (UniqueName: \"kubernetes.io/projected/89202ceb-ee62-4a1a-8756-54a870c7b7ab-kube-api-access-qxbjp\") pod \"telemeter-client-79b7d6d8f-d5kcb\" (UID: \"89202ceb-ee62-4a1a-8756-54a870c7b7ab\") " pod="openshift-monitoring/telemeter-client-79b7d6d8f-d5kcb" Oct 30 16:47:33.570593 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:33.570409 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/89202ceb-ee62-4a1a-8756-54a870c7b7ab-telemeter-client-tls\") pod \"telemeter-client-79b7d6d8f-d5kcb\" (UID: \"89202ceb-ee62-4a1a-8756-54a870c7b7ab\") " pod="openshift-monitoring/telemeter-client-79b7d6d8f-d5kcb" Oct 30 16:47:33.570593 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:33.570433 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/89202ceb-ee62-4a1a-8756-54a870c7b7ab-federate-client-tls\") pod \"telemeter-client-79b7d6d8f-d5kcb\" (UID: \"89202ceb-ee62-4a1a-8756-54a870c7b7ab\") " pod="openshift-monitoring/telemeter-client-79b7d6d8f-d5kcb" Oct 30 16:47:33.570593 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:33.570565 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/89202ceb-ee62-4a1a-8756-54a870c7b7ab-secret-telemeter-client\") pod \"telemeter-client-79b7d6d8f-d5kcb\" (UID: \"89202ceb-ee62-4a1a-8756-54a870c7b7ab\") " pod="openshift-monitoring/telemeter-client-79b7d6d8f-d5kcb" Oct 30 16:47:33.570593 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:33.570587 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89202ceb-ee62-4a1a-8756-54a870c7b7ab-telemeter-trusted-ca-bundle\") pod \"telemeter-client-79b7d6d8f-d5kcb\" (UID: \"89202ceb-ee62-4a1a-8756-54a870c7b7ab\") " pod="openshift-monitoring/telemeter-client-79b7d6d8f-d5kcb" Oct 30 16:47:33.570759 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:33.570641 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/89202ceb-ee62-4a1a-8756-54a870c7b7ab-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-79b7d6d8f-d5kcb\" (UID: \"89202ceb-ee62-4a1a-8756-54a870c7b7ab\") " pod="openshift-monitoring/telemeter-client-79b7d6d8f-d5kcb" Oct 30 16:47:33.570759 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:33.570661 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/89202ceb-ee62-4a1a-8756-54a870c7b7ab-metrics-client-ca\") pod \"telemeter-client-79b7d6d8f-d5kcb\" (UID: \"89202ceb-ee62-4a1a-8756-54a870c7b7ab\") " pod="openshift-monitoring/telemeter-client-79b7d6d8f-d5kcb" Oct 30 16:47:33.570759 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:33.570688 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89202ceb-ee62-4a1a-8756-54a870c7b7ab-serving-certs-ca-bundle\") pod \"telemeter-client-79b7d6d8f-d5kcb\" (UID: \"89202ceb-ee62-4a1a-8756-54a870c7b7ab\") " pod="openshift-monitoring/telemeter-client-79b7d6d8f-d5kcb" Oct 30 16:47:33.671863 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:33.671820 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/89202ceb-ee62-4a1a-8756-54a870c7b7ab-secret-telemeter-client\") pod \"telemeter-client-79b7d6d8f-d5kcb\" (UID: \"89202ceb-ee62-4a1a-8756-54a870c7b7ab\") " pod="openshift-monitoring/telemeter-client-79b7d6d8f-d5kcb" Oct 30 16:47:33.671863 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:33.671861 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89202ceb-ee62-4a1a-8756-54a870c7b7ab-telemeter-trusted-ca-bundle\") pod \"telemeter-client-79b7d6d8f-d5kcb\" (UID: \"89202ceb-ee62-4a1a-8756-54a870c7b7ab\") " pod="openshift-monitoring/telemeter-client-79b7d6d8f-d5kcb" Oct 30 16:47:33.672128 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:33.671887 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/89202ceb-ee62-4a1a-8756-54a870c7b7ab-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-79b7d6d8f-d5kcb\" (UID: \"89202ceb-ee62-4a1a-8756-54a870c7b7ab\") " pod="openshift-monitoring/telemeter-client-79b7d6d8f-d5kcb" Oct 30 16:47:33.672128 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:33.672087 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/89202ceb-ee62-4a1a-8756-54a870c7b7ab-metrics-client-ca\") pod \"telemeter-client-79b7d6d8f-d5kcb\" (UID: \"89202ceb-ee62-4a1a-8756-54a870c7b7ab\") " pod="openshift-monitoring/telemeter-client-79b7d6d8f-d5kcb" Oct 30 16:47:33.672240 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:33.672133 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89202ceb-ee62-4a1a-8756-54a870c7b7ab-serving-certs-ca-bundle\") pod \"telemeter-client-79b7d6d8f-d5kcb\" (UID: \"89202ceb-ee62-4a1a-8756-54a870c7b7ab\") " pod="openshift-monitoring/telemeter-client-79b7d6d8f-d5kcb" Oct 30 16:47:33.672300 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:33.672269 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxbjp\" (UniqueName: \"kubernetes.io/projected/89202ceb-ee62-4a1a-8756-54a870c7b7ab-kube-api-access-qxbjp\") pod \"telemeter-client-79b7d6d8f-d5kcb\" (UID: \"89202ceb-ee62-4a1a-8756-54a870c7b7ab\") " pod="openshift-monitoring/telemeter-client-79b7d6d8f-d5kcb" Oct 30 16:47:33.672388 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:33.672352 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/89202ceb-ee62-4a1a-8756-54a870c7b7ab-telemeter-client-tls\") pod \"telemeter-client-79b7d6d8f-d5kcb\" (UID: \"89202ceb-ee62-4a1a-8756-54a870c7b7ab\") " pod="openshift-monitoring/telemeter-client-79b7d6d8f-d5kcb" Oct 30 16:47:33.672446 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:33.672393 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/89202ceb-ee62-4a1a-8756-54a870c7b7ab-federate-client-tls\") pod \"telemeter-client-79b7d6d8f-d5kcb\" (UID: \"89202ceb-ee62-4a1a-8756-54a870c7b7ab\") " pod="openshift-monitoring/telemeter-client-79b7d6d8f-d5kcb" Oct 30 16:47:33.672928 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:33.672898 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/89202ceb-ee62-4a1a-8756-54a870c7b7ab-metrics-client-ca\") pod \"telemeter-client-79b7d6d8f-d5kcb\" (UID: \"89202ceb-ee62-4a1a-8756-54a870c7b7ab\") " pod="openshift-monitoring/telemeter-client-79b7d6d8f-d5kcb" Oct 30 16:47:33.673054 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:33.672928 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89202ceb-ee62-4a1a-8756-54a870c7b7ab-serving-certs-ca-bundle\") pod \"telemeter-client-79b7d6d8f-d5kcb\" (UID: \"89202ceb-ee62-4a1a-8756-54a870c7b7ab\") " pod="openshift-monitoring/telemeter-client-79b7d6d8f-d5kcb" Oct 30 16:47:33.673054 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:33.673014 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89202ceb-ee62-4a1a-8756-54a870c7b7ab-telemeter-trusted-ca-bundle\") pod \"telemeter-client-79b7d6d8f-d5kcb\" (UID: \"89202ceb-ee62-4a1a-8756-54a870c7b7ab\") " pod="openshift-monitoring/telemeter-client-79b7d6d8f-d5kcb" Oct 30 16:47:33.674580 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:33.674555 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/89202ceb-ee62-4a1a-8756-54a870c7b7ab-secret-telemeter-client\") pod \"telemeter-client-79b7d6d8f-d5kcb\" (UID: \"89202ceb-ee62-4a1a-8756-54a870c7b7ab\") " pod="openshift-monitoring/telemeter-client-79b7d6d8f-d5kcb" Oct 30 16:47:33.674762 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:33.674743 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/89202ceb-ee62-4a1a-8756-54a870c7b7ab-federate-client-tls\") pod \"telemeter-client-79b7d6d8f-d5kcb\" (UID: \"89202ceb-ee62-4a1a-8756-54a870c7b7ab\") " pod="openshift-monitoring/telemeter-client-79b7d6d8f-d5kcb" Oct 30 16:47:33.674924 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:33.674907 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/89202ceb-ee62-4a1a-8756-54a870c7b7ab-telemeter-client-tls\") pod \"telemeter-client-79b7d6d8f-d5kcb\" (UID: \"89202ceb-ee62-4a1a-8756-54a870c7b7ab\") " pod="openshift-monitoring/telemeter-client-79b7d6d8f-d5kcb" Oct 30 16:47:33.674964 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:33.674920 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/89202ceb-ee62-4a1a-8756-54a870c7b7ab-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-79b7d6d8f-d5kcb\" (UID: \"89202ceb-ee62-4a1a-8756-54a870c7b7ab\") " pod="openshift-monitoring/telemeter-client-79b7d6d8f-d5kcb" Oct 30 16:47:33.700056 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:33.700030 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxbjp\" (UniqueName: \"kubernetes.io/projected/89202ceb-ee62-4a1a-8756-54a870c7b7ab-kube-api-access-qxbjp\") pod \"telemeter-client-79b7d6d8f-d5kcb\" (UID: \"89202ceb-ee62-4a1a-8756-54a870c7b7ab\") " pod="openshift-monitoring/telemeter-client-79b7d6d8f-d5kcb" Oct 30 16:47:33.785035 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:33.784942 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-79b7d6d8f-d5kcb" Oct 30 16:47:34.080471 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:34.080400 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5f65d9f544-8l579"] Oct 30 16:47:34.082882 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:34.082865 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f65d9f544-8l579" Oct 30 16:47:34.102064 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:34.102041 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 16:47:34.102208 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:34.102067 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 16:47:34.102371 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:34.102299 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 16:47:34.102371 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:34.102356 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 16:47:34.102661 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:34.102636 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 16:47:34.102775 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:34.102668 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 16:47:34.116653 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:34.116621 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 16:47:34.116808 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:34.116788 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-hh9r6\"" Oct 30 16:47:34.118156 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:34.118128 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f65d9f544-8l579"] Oct 30 16:47:34.119211 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:34.119192 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-79b7d6d8f-d5kcb"] Oct 30 16:47:34.120621 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:34.120597 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89202ceb_ee62_4a1a_8756_54a870c7b7ab.slice/crio-cf0ab4e3b59a6408b34b7a33d2edb08efd032ba52db7da25dc4edf76abc97646 WatchSource:0}: Error finding container cf0ab4e3b59a6408b34b7a33d2edb08efd032ba52db7da25dc4edf76abc97646: Status 404 returned error can't find the container with id cf0ab4e3b59a6408b34b7a33d2edb08efd032ba52db7da25dc4edf76abc97646 Oct 30 16:47:34.202468 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:34.202435 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 16:47:34.276617 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:34.276581 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqz5c\" (UniqueName: \"kubernetes.io/projected/8162f70d-7176-4d81-b77e-14766cd9d50f-kube-api-access-mqz5c\") pod \"console-5f65d9f544-8l579\" (UID: \"8162f70d-7176-4d81-b77e-14766cd9d50f\") " pod="openshift-console/console-5f65d9f544-8l579" Oct 30 16:47:34.276617 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:34.276619 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8162f70d-7176-4d81-b77e-14766cd9d50f-oauth-serving-cert\") pod \"console-5f65d9f544-8l579\" (UID: \"8162f70d-7176-4d81-b77e-14766cd9d50f\") " pod="openshift-console/console-5f65d9f544-8l579" Oct 30 16:47:34.276836 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:34.276645 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8162f70d-7176-4d81-b77e-14766cd9d50f-tmp\") pod \"console-5f65d9f544-8l579\" (UID: \"8162f70d-7176-4d81-b77e-14766cd9d50f\") " pod="openshift-console/console-5f65d9f544-8l579" Oct 30 16:47:34.276836 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:34.276706 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8162f70d-7176-4d81-b77e-14766cd9d50f-console-serving-cert\") pod \"console-5f65d9f544-8l579\" (UID: \"8162f70d-7176-4d81-b77e-14766cd9d50f\") " pod="openshift-console/console-5f65d9f544-8l579" Oct 30 16:47:34.276836 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:34.276787 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8162f70d-7176-4d81-b77e-14766cd9d50f-console-config\") pod \"console-5f65d9f544-8l579\" (UID: \"8162f70d-7176-4d81-b77e-14766cd9d50f\") " pod="openshift-console/console-5f65d9f544-8l579" Oct 30 16:47:34.276836 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:34.276817 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8162f70d-7176-4d81-b77e-14766cd9d50f-service-ca\") pod \"console-5f65d9f544-8l579\" (UID: \"8162f70d-7176-4d81-b77e-14766cd9d50f\") " pod="openshift-console/console-5f65d9f544-8l579" Oct 30 16:47:34.276958 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:34.276843 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8162f70d-7176-4d81-b77e-14766cd9d50f-console-oauth-config\") pod \"console-5f65d9f544-8l579\" (UID: \"8162f70d-7176-4d81-b77e-14766cd9d50f\") " pod="openshift-console/console-5f65d9f544-8l579" Oct 30 16:47:34.276958 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:34.276888 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8162f70d-7176-4d81-b77e-14766cd9d50f-trusted-ca-bundle\") pod \"console-5f65d9f544-8l579\" (UID: \"8162f70d-7176-4d81-b77e-14766cd9d50f\") " pod="openshift-console/console-5f65d9f544-8l579" Oct 30 16:47:34.290840 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:34.290807 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-79b7d6d8f-d5kcb" event={"ID":"89202ceb-ee62-4a1a-8756-54a870c7b7ab","Type":"ContainerStarted","Data":"cf0ab4e3b59a6408b34b7a33d2edb08efd032ba52db7da25dc4edf76abc97646"} Oct 30 16:47:34.378155 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:34.378069 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqz5c\" (UniqueName: \"kubernetes.io/projected/8162f70d-7176-4d81-b77e-14766cd9d50f-kube-api-access-mqz5c\") pod \"console-5f65d9f544-8l579\" (UID: \"8162f70d-7176-4d81-b77e-14766cd9d50f\") " pod="openshift-console/console-5f65d9f544-8l579" Oct 30 16:47:34.378155 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:34.378109 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8162f70d-7176-4d81-b77e-14766cd9d50f-oauth-serving-cert\") pod \"console-5f65d9f544-8l579\" (UID: \"8162f70d-7176-4d81-b77e-14766cd9d50f\") " pod="openshift-console/console-5f65d9f544-8l579" Oct 30 16:47:34.378155 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:34.378129 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8162f70d-7176-4d81-b77e-14766cd9d50f-tmp\") pod \"console-5f65d9f544-8l579\" (UID: \"8162f70d-7176-4d81-b77e-14766cd9d50f\") " pod="openshift-console/console-5f65d9f544-8l579" Oct 30 16:47:34.378447 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:34.378164 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8162f70d-7176-4d81-b77e-14766cd9d50f-console-serving-cert\") pod \"console-5f65d9f544-8l579\" (UID: \"8162f70d-7176-4d81-b77e-14766cd9d50f\") " pod="openshift-console/console-5f65d9f544-8l579" Oct 30 16:47:34.378447 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:34.378186 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8162f70d-7176-4d81-b77e-14766cd9d50f-console-config\") pod \"console-5f65d9f544-8l579\" (UID: \"8162f70d-7176-4d81-b77e-14766cd9d50f\") " pod="openshift-console/console-5f65d9f544-8l579" Oct 30 16:47:34.378447 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:34.378202 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8162f70d-7176-4d81-b77e-14766cd9d50f-service-ca\") pod \"console-5f65d9f544-8l579\" (UID: \"8162f70d-7176-4d81-b77e-14766cd9d50f\") " pod="openshift-console/console-5f65d9f544-8l579" Oct 30 16:47:34.378447 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:34.378221 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8162f70d-7176-4d81-b77e-14766cd9d50f-console-oauth-config\") pod \"console-5f65d9f544-8l579\" (UID: \"8162f70d-7176-4d81-b77e-14766cd9d50f\") " pod="openshift-console/console-5f65d9f544-8l579" Oct 30 16:47:34.378447 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:34.378243 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8162f70d-7176-4d81-b77e-14766cd9d50f-trusted-ca-bundle\") pod \"console-5f65d9f544-8l579\" (UID: \"8162f70d-7176-4d81-b77e-14766cd9d50f\") " pod="openshift-console/console-5f65d9f544-8l579" Oct 30 16:47:34.378679 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:34.378594 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8162f70d-7176-4d81-b77e-14766cd9d50f-tmp\") pod \"console-5f65d9f544-8l579\" (UID: \"8162f70d-7176-4d81-b77e-14766cd9d50f\") " pod="openshift-console/console-5f65d9f544-8l579" Oct 30 16:47:34.379013 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:34.378978 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8162f70d-7176-4d81-b77e-14766cd9d50f-oauth-serving-cert\") pod \"console-5f65d9f544-8l579\" (UID: \"8162f70d-7176-4d81-b77e-14766cd9d50f\") " pod="openshift-console/console-5f65d9f544-8l579" Oct 30 16:47:34.379110 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:34.378992 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8162f70d-7176-4d81-b77e-14766cd9d50f-console-config\") pod \"console-5f65d9f544-8l579\" (UID: \"8162f70d-7176-4d81-b77e-14766cd9d50f\") " pod="openshift-console/console-5f65d9f544-8l579" Oct 30 16:47:34.379172 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:34.379131 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8162f70d-7176-4d81-b77e-14766cd9d50f-service-ca\") pod \"console-5f65d9f544-8l579\" (UID: \"8162f70d-7176-4d81-b77e-14766cd9d50f\") " pod="openshift-console/console-5f65d9f544-8l579" Oct 30 16:47:34.379249 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:34.379186 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8162f70d-7176-4d81-b77e-14766cd9d50f-trusted-ca-bundle\") pod \"console-5f65d9f544-8l579\" (UID: \"8162f70d-7176-4d81-b77e-14766cd9d50f\") " pod="openshift-console/console-5f65d9f544-8l579" Oct 30 16:47:34.380735 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:34.380714 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8162f70d-7176-4d81-b77e-14766cd9d50f-console-serving-cert\") pod \"console-5f65d9f544-8l579\" (UID: \"8162f70d-7176-4d81-b77e-14766cd9d50f\") " pod="openshift-console/console-5f65d9f544-8l579" Oct 30 16:47:34.380788 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:34.380738 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8162f70d-7176-4d81-b77e-14766cd9d50f-console-oauth-config\") pod \"console-5f65d9f544-8l579\" (UID: \"8162f70d-7176-4d81-b77e-14766cd9d50f\") " pod="openshift-console/console-5f65d9f544-8l579" Oct 30 16:47:34.403894 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:34.403867 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqz5c\" (UniqueName: \"kubernetes.io/projected/8162f70d-7176-4d81-b77e-14766cd9d50f-kube-api-access-mqz5c\") pod \"console-5f65d9f544-8l579\" (UID: \"8162f70d-7176-4d81-b77e-14766cd9d50f\") " pod="openshift-console/console-5f65d9f544-8l579" Oct 30 16:47:34.693403 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:34.693328 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f65d9f544-8l579" Oct 30 16:47:34.999237 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:34.999208 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f65d9f544-8l579"] Oct 30 16:47:35.001059 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:47:35.001029 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8162f70d_7176_4d81_b77e_14766cd9d50f.slice/crio-5a444ada4e42aff211ae46d39fb25893c6ddc9fe4a908592d358e9122698068b WatchSource:0}: Error finding container 5a444ada4e42aff211ae46d39fb25893c6ddc9fe4a908592d358e9122698068b: Status 404 returned error can't find the container with id 5a444ada4e42aff211ae46d39fb25893c6ddc9fe4a908592d358e9122698068b Oct 30 16:47:35.296475 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:35.296390 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f65d9f544-8l579" event={"ID":"8162f70d-7176-4d81-b77e-14766cd9d50f","Type":"ContainerStarted","Data":"5a444ada4e42aff211ae46d39fb25893c6ddc9fe4a908592d358e9122698068b"} Oct 30 16:47:36.311087 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:36.311043 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-79b7d6d8f-d5kcb" event={"ID":"89202ceb-ee62-4a1a-8756-54a870c7b7ab","Type":"ContainerStarted","Data":"2223a53cbb86aa020b5f9160aba2118899fcb8aeded616762f376d52dbd53cf9"} Oct 30 16:47:37.317323 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:37.317269 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-79b7d6d8f-d5kcb" event={"ID":"89202ceb-ee62-4a1a-8756-54a870c7b7ab","Type":"ContainerStarted","Data":"a9643cf0948cd97de36011bf39fd417c680c2ac497fb7bf0e8443f975a115940"} Oct 30 16:47:37.317797 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:37.317358 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-79b7d6d8f-d5kcb" event={"ID":"89202ceb-ee62-4a1a-8756-54a870c7b7ab","Type":"ContainerStarted","Data":"b3f95f6e91ccf9807adb754d3d220a5e6e36c680dbdc481be15f25a62806cdd2"} Oct 30 16:47:37.396948 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:37.396747 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-79b7d6d8f-d5kcb" podStartSLOduration=1.808252698 podStartE2EDuration="4.396731059s" totalImagesPullingTime="2.588478361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:47:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:47:37.382531592 +0000 UTC m=+31.383286515" watchObservedRunningTime="2026-10-30 16:47:37.396731059 +0000 UTC m=+31.397485960" Oct 30 16:47:37.699184 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:37.699109 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f65d9f544-8l579"] Oct 30 16:47:38.322807 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:38.322764 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f65d9f544-8l579" event={"ID":"8162f70d-7176-4d81-b77e-14766cd9d50f","Type":"ContainerStarted","Data":"ee593334d28376e72fa55d92f2355d3359b3c0051818f665d5cb77874589cf1c"} Oct 30 16:47:38.371067 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:38.370997 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5f65d9f544-8l579" podStartSLOduration=1.162692025 podStartE2EDuration="4.370982855s" totalImagesPullingTime="3.20829083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:47:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:47:38.369789979 +0000 UTC m=+32.370544904" watchObservedRunningTime="2026-10-30 16:47:38.370982855 +0000 UTC m=+32.371737755" Oct 30 16:47:42.946075 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:47:42.946041 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 16:47:43.240741 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:43.240711 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-59dd87c5f7-xhqgq" Oct 30 16:47:43.395933 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:43.395901 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8zqlg" Oct 30 16:47:44.693561 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:47:44.693526 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5f65d9f544-8l579" Oct 30 16:48:03.342742 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:03.342673 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5f65d9f544-8l579" podUID="8162f70d-7176-4d81-b77e-14766cd9d50f" containerName="console" containerID="cri-o://ee593334d28376e72fa55d92f2355d3359b3c0051818f665d5cb77874589cf1c" gracePeriod=15 Oct 30 16:48:03.587553 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:03.587528 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f65d9f544-8l579" Oct 30 16:48:03.692475 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:03.692384 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8162f70d-7176-4d81-b77e-14766cd9d50f-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8162f70d-7176-4d81-b77e-14766cd9d50f-oauth-serving-cert\") pod \"8162f70d-7176-4d81-b77e-14766cd9d50f\" (UID: \"8162f70d-7176-4d81-b77e-14766cd9d50f\") " Oct 30 16:48:03.692475 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:03.692434 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8162f70d-7176-4d81-b77e-14766cd9d50f-console-config\" (UniqueName: \"kubernetes.io/configmap/8162f70d-7176-4d81-b77e-14766cd9d50f-console-config\") pod \"8162f70d-7176-4d81-b77e-14766cd9d50f\" (UID: \"8162f70d-7176-4d81-b77e-14766cd9d50f\") " Oct 30 16:48:03.692475 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:03.692462 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8162f70d-7176-4d81-b77e-14766cd9d50f-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8162f70d-7176-4d81-b77e-14766cd9d50f-console-serving-cert\") pod \"8162f70d-7176-4d81-b77e-14766cd9d50f\" (UID: \"8162f70d-7176-4d81-b77e-14766cd9d50f\") " Oct 30 16:48:03.692755 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:03.692484 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8162f70d-7176-4d81-b77e-14766cd9d50f-kube-api-access-mqz5c\" (UniqueName: \"kubernetes.io/projected/8162f70d-7176-4d81-b77e-14766cd9d50f-kube-api-access-mqz5c\") pod \"8162f70d-7176-4d81-b77e-14766cd9d50f\" (UID: \"8162f70d-7176-4d81-b77e-14766cd9d50f\") " Oct 30 16:48:03.692755 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:03.692503 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8162f70d-7176-4d81-b77e-14766cd9d50f-tmp\" (UniqueName: \"kubernetes.io/empty-dir/8162f70d-7176-4d81-b77e-14766cd9d50f-tmp\") pod \"8162f70d-7176-4d81-b77e-14766cd9d50f\" (UID: \"8162f70d-7176-4d81-b77e-14766cd9d50f\") " Oct 30 16:48:03.692755 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:03.692531 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8162f70d-7176-4d81-b77e-14766cd9d50f-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8162f70d-7176-4d81-b77e-14766cd9d50f-trusted-ca-bundle\") pod \"8162f70d-7176-4d81-b77e-14766cd9d50f\" (UID: \"8162f70d-7176-4d81-b77e-14766cd9d50f\") " Oct 30 16:48:03.692755 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:03.692562 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8162f70d-7176-4d81-b77e-14766cd9d50f-service-ca\" (UniqueName: \"kubernetes.io/configmap/8162f70d-7176-4d81-b77e-14766cd9d50f-service-ca\") pod \"8162f70d-7176-4d81-b77e-14766cd9d50f\" (UID: \"8162f70d-7176-4d81-b77e-14766cd9d50f\") " Oct 30 16:48:03.692755 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:03.692600 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8162f70d-7176-4d81-b77e-14766cd9d50f-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8162f70d-7176-4d81-b77e-14766cd9d50f-console-oauth-config\") pod \"8162f70d-7176-4d81-b77e-14766cd9d50f\" (UID: \"8162f70d-7176-4d81-b77e-14766cd9d50f\") " Oct 30 16:48:03.692992 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:03.692920 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8162f70d-7176-4d81-b77e-14766cd9d50f-tmp" pod "8162f70d-7176-4d81-b77e-14766cd9d50f" (UID: "8162f70d-7176-4d81-b77e-14766cd9d50f"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:48:03.693219 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:03.693130 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8162f70d-7176-4d81-b77e-14766cd9d50f-console-config" pod "8162f70d-7176-4d81-b77e-14766cd9d50f" (UID: "8162f70d-7176-4d81-b77e-14766cd9d50f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:48:03.693282 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:03.693209 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8162f70d-7176-4d81-b77e-14766cd9d50f-oauth-serving-cert" pod "8162f70d-7176-4d81-b77e-14766cd9d50f" (UID: "8162f70d-7176-4d81-b77e-14766cd9d50f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:48:03.693282 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:03.693220 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8162f70d-7176-4d81-b77e-14766cd9d50f-trusted-ca-bundle" pod "8162f70d-7176-4d81-b77e-14766cd9d50f" (UID: "8162f70d-7176-4d81-b77e-14766cd9d50f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:48:03.693378 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:03.693298 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8162f70d-7176-4d81-b77e-14766cd9d50f-service-ca" pod "8162f70d-7176-4d81-b77e-14766cd9d50f" (UID: "8162f70d-7176-4d81-b77e-14766cd9d50f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:48:03.694913 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:03.694888 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8162f70d-7176-4d81-b77e-14766cd9d50f-kube-api-access-mqz5c" pod "8162f70d-7176-4d81-b77e-14766cd9d50f" (UID: "8162f70d-7176-4d81-b77e-14766cd9d50f"). InnerVolumeSpecName "kube-api-access-mqz5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:48:03.695248 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:03.695228 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8162f70d-7176-4d81-b77e-14766cd9d50f-console-oauth-config" pod "8162f70d-7176-4d81-b77e-14766cd9d50f" (UID: "8162f70d-7176-4d81-b77e-14766cd9d50f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:48:03.695321 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:03.695249 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8162f70d-7176-4d81-b77e-14766cd9d50f-console-serving-cert" pod "8162f70d-7176-4d81-b77e-14766cd9d50f" (UID: "8162f70d-7176-4d81-b77e-14766cd9d50f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:48:03.793792 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:03.793753 4522 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8162f70d-7176-4d81-b77e-14766cd9d50f-oauth-serving-cert\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 16:48:03.793792 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:03.793785 4522 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8162f70d-7176-4d81-b77e-14766cd9d50f-console-config\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 16:48:03.793792 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:03.793796 4522 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8162f70d-7176-4d81-b77e-14766cd9d50f-console-serving-cert\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 16:48:03.793792 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:03.793805 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqz5c\" (UniqueName: \"kubernetes.io/projected/8162f70d-7176-4d81-b77e-14766cd9d50f-kube-api-access-mqz5c\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 16:48:03.794047 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:03.793815 4522 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8162f70d-7176-4d81-b77e-14766cd9d50f-tmp\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 16:48:03.794047 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:03.793823 4522 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8162f70d-7176-4d81-b77e-14766cd9d50f-trusted-ca-bundle\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 16:48:03.794047 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:03.793832 4522 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8162f70d-7176-4d81-b77e-14766cd9d50f-service-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 16:48:03.794047 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:03.793840 4522 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8162f70d-7176-4d81-b77e-14766cd9d50f-console-oauth-config\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 16:48:04.408008 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:04.407932 4522 generic.go:356] "Generic (PLEG): container finished" podID="8162f70d-7176-4d81-b77e-14766cd9d50f" containerID="ee593334d28376e72fa55d92f2355d3359b3c0051818f665d5cb77874589cf1c" exitCode=0 Oct 30 16:48:04.408008 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:04.407993 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f65d9f544-8l579" Oct 30 16:48:04.408008 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:04.408005 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f65d9f544-8l579" event={"ID":"8162f70d-7176-4d81-b77e-14766cd9d50f","Type":"ContainerDied","Data":"ee593334d28376e72fa55d92f2355d3359b3c0051818f665d5cb77874589cf1c"} Oct 30 16:48:04.408527 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:04.408050 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f65d9f544-8l579" event={"ID":"8162f70d-7176-4d81-b77e-14766cd9d50f","Type":"ContainerDied","Data":"5a444ada4e42aff211ae46d39fb25893c6ddc9fe4a908592d358e9122698068b"} Oct 30 16:48:04.408527 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:04.408070 4522 scope.go:122] "RemoveContainer" containerID="ee593334d28376e72fa55d92f2355d3359b3c0051818f665d5cb77874589cf1c" Oct 30 16:48:04.416005 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:04.415987 4522 scope.go:122] "RemoveContainer" containerID="ee593334d28376e72fa55d92f2355d3359b3c0051818f665d5cb77874589cf1c" Oct 30 16:48:04.416274 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:48:04.416248 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee593334d28376e72fa55d92f2355d3359b3c0051818f665d5cb77874589cf1c\": container with ID starting with ee593334d28376e72fa55d92f2355d3359b3c0051818f665d5cb77874589cf1c not found: ID does not exist" containerID="ee593334d28376e72fa55d92f2355d3359b3c0051818f665d5cb77874589cf1c" Oct 30 16:48:04.416380 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:04.416287 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee593334d28376e72fa55d92f2355d3359b3c0051818f665d5cb77874589cf1c"} err="failed to get container status \"ee593334d28376e72fa55d92f2355d3359b3c0051818f665d5cb77874589cf1c\": rpc error: code = NotFound desc = could not find container \"ee593334d28376e72fa55d92f2355d3359b3c0051818f665d5cb77874589cf1c\": container with ID starting with ee593334d28376e72fa55d92f2355d3359b3c0051818f665d5cb77874589cf1c not found: ID does not exist" Oct 30 16:48:04.463551 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:04.463518 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f65d9f544-8l579"] Oct 30 16:48:04.477108 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:04.477079 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5f65d9f544-8l579"] Oct 30 16:48:06.141448 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:06.141420 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8162f70d-7176-4d81-b77e-14766cd9d50f" path="/var/lib/kubelet/pods/8162f70d-7176-4d81-b77e-14766cd9d50f/volumes" Oct 30 16:48:09.470792 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:09.470760 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-f4j47-pgf7r"] Oct 30 16:48:09.471181 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:09.470957 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-f4j47-pgf7r" podUID="252a7edc-651f-4c08-8fd7-f46ec29737c4" containerName="insights-gathering" containerID="cri-o://65e723da13fb308bc9833e0b7bb7da28ccdfc88cdd79098cdf01d48a88ecc1ff" gracePeriod=30 Oct 30 16:48:09.726081 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:09.726026 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-f4j47-pgf7r" Oct 30 16:48:09.839361 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:09.839327 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/252a7edc-651f-4c08-8fd7-f46ec29737c4-kube-api-access-2gxc2\" (UniqueName: \"kubernetes.io/projected/252a7edc-651f-4c08-8fd7-f46ec29737c4-kube-api-access-2gxc2\") pod \"252a7edc-651f-4c08-8fd7-f46ec29737c4\" (UID: \"252a7edc-651f-4c08-8fd7-f46ec29737c4\") " Oct 30 16:48:09.839517 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:09.839450 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/252a7edc-651f-4c08-8fd7-f46ec29737c4-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/252a7edc-651f-4c08-8fd7-f46ec29737c4-trusted-ca-bundle\") pod \"252a7edc-651f-4c08-8fd7-f46ec29737c4\" (UID: \"252a7edc-651f-4c08-8fd7-f46ec29737c4\") " Oct 30 16:48:09.839517 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:09.839490 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/252a7edc-651f-4c08-8fd7-f46ec29737c4-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/252a7edc-651f-4c08-8fd7-f46ec29737c4-archives-path\") pod \"252a7edc-651f-4c08-8fd7-f46ec29737c4\" (UID: \"252a7edc-651f-4c08-8fd7-f46ec29737c4\") " Oct 30 16:48:09.839628 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:09.839551 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/252a7edc-651f-4c08-8fd7-f46ec29737c4-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/252a7edc-651f-4c08-8fd7-f46ec29737c4-service-ca-bundle\") pod \"252a7edc-651f-4c08-8fd7-f46ec29737c4\" (UID: \"252a7edc-651f-4c08-8fd7-f46ec29737c4\") " Oct 30 16:48:09.839917 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:09.839886 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/252a7edc-651f-4c08-8fd7-f46ec29737c4-archives-path" pod "252a7edc-651f-4c08-8fd7-f46ec29737c4" (UID: "252a7edc-651f-4c08-8fd7-f46ec29737c4"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:48:09.840024 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:09.839919 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/252a7edc-651f-4c08-8fd7-f46ec29737c4-service-ca-bundle" pod "252a7edc-651f-4c08-8fd7-f46ec29737c4" (UID: "252a7edc-651f-4c08-8fd7-f46ec29737c4"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:48:09.840024 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:09.839898 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/252a7edc-651f-4c08-8fd7-f46ec29737c4-trusted-ca-bundle" pod "252a7edc-651f-4c08-8fd7-f46ec29737c4" (UID: "252a7edc-651f-4c08-8fd7-f46ec29737c4"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:48:09.841553 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:09.841530 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/252a7edc-651f-4c08-8fd7-f46ec29737c4-kube-api-access-2gxc2" pod "252a7edc-651f-4c08-8fd7-f46ec29737c4" (UID: "252a7edc-651f-4c08-8fd7-f46ec29737c4"). InnerVolumeSpecName "kube-api-access-2gxc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:48:09.940967 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:09.940935 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gxc2\" (UniqueName: \"kubernetes.io/projected/252a7edc-651f-4c08-8fd7-f46ec29737c4-kube-api-access-2gxc2\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 16:48:09.940967 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:09.940962 4522 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/252a7edc-651f-4c08-8fd7-f46ec29737c4-trusted-ca-bundle\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 16:48:09.940967 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:09.940972 4522 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/252a7edc-651f-4c08-8fd7-f46ec29737c4-archives-path\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 16:48:09.941194 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:09.940981 4522 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/252a7edc-651f-4c08-8fd7-f46ec29737c4-service-ca-bundle\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 16:48:10.429941 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:10.429853 4522 generic.go:356] "Generic (PLEG): container finished" podID="252a7edc-651f-4c08-8fd7-f46ec29737c4" containerID="65e723da13fb308bc9833e0b7bb7da28ccdfc88cdd79098cdf01d48a88ecc1ff" exitCode=2 Oct 30 16:48:10.429941 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:10.429897 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-f4j47-pgf7r" Oct 30 16:48:10.430109 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:10.429939 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-f4j47-pgf7r" event={"ID":"252a7edc-651f-4c08-8fd7-f46ec29737c4","Type":"ContainerDied","Data":"65e723da13fb308bc9833e0b7bb7da28ccdfc88cdd79098cdf01d48a88ecc1ff"} Oct 30 16:48:10.430109 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:10.429986 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-f4j47-pgf7r" event={"ID":"252a7edc-651f-4c08-8fd7-f46ec29737c4","Type":"ContainerDied","Data":"6fc7bce1f36d0e8e13dc09c5275b70fe8ea2790f94977fde82f4fa803951bcfc"} Oct 30 16:48:10.430109 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:10.430005 4522 scope.go:122] "RemoveContainer" containerID="65e723da13fb308bc9833e0b7bb7da28ccdfc88cdd79098cdf01d48a88ecc1ff" Oct 30 16:48:10.437430 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:10.437411 4522 scope.go:122] "RemoveContainer" containerID="65e723da13fb308bc9833e0b7bb7da28ccdfc88cdd79098cdf01d48a88ecc1ff" Oct 30 16:48:10.437657 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:48:10.437639 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65e723da13fb308bc9833e0b7bb7da28ccdfc88cdd79098cdf01d48a88ecc1ff\": container with ID starting with 65e723da13fb308bc9833e0b7bb7da28ccdfc88cdd79098cdf01d48a88ecc1ff not found: ID does not exist" containerID="65e723da13fb308bc9833e0b7bb7da28ccdfc88cdd79098cdf01d48a88ecc1ff" Oct 30 16:48:10.437727 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:10.437662 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65e723da13fb308bc9833e0b7bb7da28ccdfc88cdd79098cdf01d48a88ecc1ff"} err="failed to get container status \"65e723da13fb308bc9833e0b7bb7da28ccdfc88cdd79098cdf01d48a88ecc1ff\": rpc error: code = NotFound desc = could not find container \"65e723da13fb308bc9833e0b7bb7da28ccdfc88cdd79098cdf01d48a88ecc1ff\": container with ID starting with 65e723da13fb308bc9833e0b7bb7da28ccdfc88cdd79098cdf01d48a88ecc1ff not found: ID does not exist" Oct 30 16:48:10.484228 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:10.484199 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-f4j47-pgf7r"] Oct 30 16:48:10.497809 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:10.497777 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-f4j47-pgf7r"] Oct 30 16:48:12.141388 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:12.141355 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="252a7edc-651f-4c08-8fd7-f46ec29737c4" path="/var/lib/kubelet/pods/252a7edc-651f-4c08-8fd7-f46ec29737c4/volumes" Oct 30 16:48:12.946017 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:48:12.945981 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 16:48:14.611205 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:14.611178 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7d89df5bcc-d5wph"] Oct 30 16:48:14.611583 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:14.611449 4522 cpu_manager.go:414] "containerMap: removing container" podUID="252a7edc-651f-4c08-8fd7-f46ec29737c4" containerName="insights-gathering" Oct 30 16:48:14.611583 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:14.611461 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="252a7edc-651f-4c08-8fd7-f46ec29737c4" containerName="insights-gathering" Oct 30 16:48:14.611583 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:14.611472 4522 cpu_manager.go:414] "containerMap: removing container" podUID="8162f70d-7176-4d81-b77e-14766cd9d50f" containerName="console" Oct 30 16:48:14.611583 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:14.611477 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8162f70d-7176-4d81-b77e-14766cd9d50f" containerName="console" Oct 30 16:48:14.611583 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:14.611522 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="252a7edc-651f-4c08-8fd7-f46ec29737c4" containerName="insights-gathering" Oct 30 16:48:14.611583 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:14.611530 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="8162f70d-7176-4d81-b77e-14766cd9d50f" containerName="console" Oct 30 16:48:14.616148 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:14.616131 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d89df5bcc-d5wph" Oct 30 16:48:14.643023 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:14.642994 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d89df5bcc-d5wph"] Oct 30 16:48:14.654961 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:14.654931 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 16:48:14.654961 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:14.654951 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 16:48:14.654961 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:14.654946 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 16:48:14.655193 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:14.654992 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 16:48:14.655193 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:14.654994 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 16:48:14.655193 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:14.655035 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-hh9r6\"" Oct 30 16:48:14.655193 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:14.655057 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 16:48:14.655193 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:14.655130 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 16:48:14.677429 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:14.677404 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b18a897c-62ee-482f-8a3a-306c95b41bce-console-oauth-config\") pod \"console-7d89df5bcc-d5wph\" (UID: \"b18a897c-62ee-482f-8a3a-306c95b41bce\") " pod="openshift-console/console-7d89df5bcc-d5wph" Oct 30 16:48:14.677552 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:14.677433 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b18a897c-62ee-482f-8a3a-306c95b41bce-service-ca\") pod \"console-7d89df5bcc-d5wph\" (UID: \"b18a897c-62ee-482f-8a3a-306c95b41bce\") " pod="openshift-console/console-7d89df5bcc-d5wph" Oct 30 16:48:14.677552 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:14.677451 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b18a897c-62ee-482f-8a3a-306c95b41bce-console-config\") pod \"console-7d89df5bcc-d5wph\" (UID: \"b18a897c-62ee-482f-8a3a-306c95b41bce\") " pod="openshift-console/console-7d89df5bcc-d5wph" Oct 30 16:48:14.677552 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:14.677469 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b18a897c-62ee-482f-8a3a-306c95b41bce-tmp\") pod \"console-7d89df5bcc-d5wph\" (UID: \"b18a897c-62ee-482f-8a3a-306c95b41bce\") " pod="openshift-console/console-7d89df5bcc-d5wph" Oct 30 16:48:14.677663 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:14.677614 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b18a897c-62ee-482f-8a3a-306c95b41bce-oauth-serving-cert\") pod \"console-7d89df5bcc-d5wph\" (UID: \"b18a897c-62ee-482f-8a3a-306c95b41bce\") " pod="openshift-console/console-7d89df5bcc-d5wph" Oct 30 16:48:14.677663 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:14.677649 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b18a897c-62ee-482f-8a3a-306c95b41bce-console-serving-cert\") pod \"console-7d89df5bcc-d5wph\" (UID: \"b18a897c-62ee-482f-8a3a-306c95b41bce\") " pod="openshift-console/console-7d89df5bcc-d5wph" Oct 30 16:48:14.677740 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:14.677673 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b18a897c-62ee-482f-8a3a-306c95b41bce-trusted-ca-bundle\") pod \"console-7d89df5bcc-d5wph\" (UID: \"b18a897c-62ee-482f-8a3a-306c95b41bce\") " pod="openshift-console/console-7d89df5bcc-d5wph" Oct 30 16:48:14.677740 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:14.677735 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgt4n\" (UniqueName: \"kubernetes.io/projected/b18a897c-62ee-482f-8a3a-306c95b41bce-kube-api-access-kgt4n\") pod \"console-7d89df5bcc-d5wph\" (UID: \"b18a897c-62ee-482f-8a3a-306c95b41bce\") " pod="openshift-console/console-7d89df5bcc-d5wph" Oct 30 16:48:14.682845 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:14.682824 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 16:48:14.778886 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:14.778848 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b18a897c-62ee-482f-8a3a-306c95b41bce-console-oauth-config\") pod \"console-7d89df5bcc-d5wph\" (UID: \"b18a897c-62ee-482f-8a3a-306c95b41bce\") " pod="openshift-console/console-7d89df5bcc-d5wph" Oct 30 16:48:14.778886 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:14.778885 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b18a897c-62ee-482f-8a3a-306c95b41bce-service-ca\") pod \"console-7d89df5bcc-d5wph\" (UID: \"b18a897c-62ee-482f-8a3a-306c95b41bce\") " pod="openshift-console/console-7d89df5bcc-d5wph" Oct 30 16:48:14.779107 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:14.778912 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b18a897c-62ee-482f-8a3a-306c95b41bce-console-config\") pod \"console-7d89df5bcc-d5wph\" (UID: \"b18a897c-62ee-482f-8a3a-306c95b41bce\") " pod="openshift-console/console-7d89df5bcc-d5wph" Oct 30 16:48:14.779107 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:14.778934 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b18a897c-62ee-482f-8a3a-306c95b41bce-tmp\") pod \"console-7d89df5bcc-d5wph\" (UID: \"b18a897c-62ee-482f-8a3a-306c95b41bce\") " pod="openshift-console/console-7d89df5bcc-d5wph" Oct 30 16:48:14.779107 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:14.778963 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b18a897c-62ee-482f-8a3a-306c95b41bce-oauth-serving-cert\") pod \"console-7d89df5bcc-d5wph\" (UID: \"b18a897c-62ee-482f-8a3a-306c95b41bce\") " pod="openshift-console/console-7d89df5bcc-d5wph" Oct 30 16:48:14.779107 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:14.779000 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b18a897c-62ee-482f-8a3a-306c95b41bce-console-serving-cert\") pod \"console-7d89df5bcc-d5wph\" (UID: \"b18a897c-62ee-482f-8a3a-306c95b41bce\") " pod="openshift-console/console-7d89df5bcc-d5wph" Oct 30 16:48:14.779107 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:14.779033 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b18a897c-62ee-482f-8a3a-306c95b41bce-trusted-ca-bundle\") pod \"console-7d89df5bcc-d5wph\" (UID: \"b18a897c-62ee-482f-8a3a-306c95b41bce\") " pod="openshift-console/console-7d89df5bcc-d5wph" Oct 30 16:48:14.779107 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:14.779078 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgt4n\" (UniqueName: \"kubernetes.io/projected/b18a897c-62ee-482f-8a3a-306c95b41bce-kube-api-access-kgt4n\") pod \"console-7d89df5bcc-d5wph\" (UID: \"b18a897c-62ee-482f-8a3a-306c95b41bce\") " pod="openshift-console/console-7d89df5bcc-d5wph" Oct 30 16:48:14.779755 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:14.779719 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b18a897c-62ee-482f-8a3a-306c95b41bce-oauth-serving-cert\") pod \"console-7d89df5bcc-d5wph\" (UID: \"b18a897c-62ee-482f-8a3a-306c95b41bce\") " pod="openshift-console/console-7d89df5bcc-d5wph" Oct 30 16:48:14.779866 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:14.779760 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b18a897c-62ee-482f-8a3a-306c95b41bce-console-config\") pod \"console-7d89df5bcc-d5wph\" (UID: \"b18a897c-62ee-482f-8a3a-306c95b41bce\") " pod="openshift-console/console-7d89df5bcc-d5wph" Oct 30 16:48:14.779953 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:14.779934 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b18a897c-62ee-482f-8a3a-306c95b41bce-tmp\") pod \"console-7d89df5bcc-d5wph\" (UID: \"b18a897c-62ee-482f-8a3a-306c95b41bce\") " pod="openshift-console/console-7d89df5bcc-d5wph" Oct 30 16:48:14.780147 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:14.780129 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b18a897c-62ee-482f-8a3a-306c95b41bce-service-ca\") pod \"console-7d89df5bcc-d5wph\" (UID: \"b18a897c-62ee-482f-8a3a-306c95b41bce\") " pod="openshift-console/console-7d89df5bcc-d5wph" Oct 30 16:48:14.780394 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:14.780372 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b18a897c-62ee-482f-8a3a-306c95b41bce-trusted-ca-bundle\") pod \"console-7d89df5bcc-d5wph\" (UID: \"b18a897c-62ee-482f-8a3a-306c95b41bce\") " pod="openshift-console/console-7d89df5bcc-d5wph" Oct 30 16:48:14.781531 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:14.781514 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b18a897c-62ee-482f-8a3a-306c95b41bce-console-serving-cert\") pod \"console-7d89df5bcc-d5wph\" (UID: \"b18a897c-62ee-482f-8a3a-306c95b41bce\") " pod="openshift-console/console-7d89df5bcc-d5wph" Oct 30 16:48:14.781630 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:14.781603 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b18a897c-62ee-482f-8a3a-306c95b41bce-console-oauth-config\") pod \"console-7d89df5bcc-d5wph\" (UID: \"b18a897c-62ee-482f-8a3a-306c95b41bce\") " pod="openshift-console/console-7d89df5bcc-d5wph" Oct 30 16:48:14.804662 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:14.804641 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgt4n\" (UniqueName: \"kubernetes.io/projected/b18a897c-62ee-482f-8a3a-306c95b41bce-kube-api-access-kgt4n\") pod \"console-7d89df5bcc-d5wph\" (UID: \"b18a897c-62ee-482f-8a3a-306c95b41bce\") " pod="openshift-console/console-7d89df5bcc-d5wph" Oct 30 16:48:14.928877 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:14.928798 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d89df5bcc-d5wph" Oct 30 16:48:15.225837 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:15.225804 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d89df5bcc-d5wph"] Oct 30 16:48:15.226602 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:48:15.226576 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb18a897c_62ee_482f_8a3a_306c95b41bce.slice/crio-0b70c2780b230f1d060d7720f662ce120c94020d4919ef54009cc7eb32630991 WatchSource:0}: Error finding container 0b70c2780b230f1d060d7720f662ce120c94020d4919ef54009cc7eb32630991: Status 404 returned error can't find the container with id 0b70c2780b230f1d060d7720f662ce120c94020d4919ef54009cc7eb32630991 Oct 30 16:48:15.449119 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:15.449083 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d89df5bcc-d5wph" event={"ID":"b18a897c-62ee-482f-8a3a-306c95b41bce","Type":"ContainerStarted","Data":"832bacf65706f0bee75d9eb3be4e693a24330b6c858da1e4b5e6f2126b927584"} Oct 30 16:48:15.449281 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:15.449136 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d89df5bcc-d5wph" event={"ID":"b18a897c-62ee-482f-8a3a-306c95b41bce","Type":"ContainerStarted","Data":"0b70c2780b230f1d060d7720f662ce120c94020d4919ef54009cc7eb32630991"} Oct 30 16:48:21.264294 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:21.264242 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7d89df5bcc-d5wph" podStartSLOduration=7.264228214 podStartE2EDuration="7.264228214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:48:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:48:15.494960909 +0000 UTC m=+69.495715810" watchObservedRunningTime="2026-10-30 16:48:21.264228214 +0000 UTC m=+75.264983112" Oct 30 16:48:21.265255 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:21.265236 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 16:48:21.269019 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:21.269003 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:48:21.285041 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:21.285015 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 16:48:21.297671 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:21.297639 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 16:48:21.297799 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:21.297671 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 16:48:21.297799 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:21.297714 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 16:48:21.297799 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:21.297735 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 16:48:21.297799 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:21.297741 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 16:48:21.297799 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:21.297748 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 16:48:21.297799 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:21.297748 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 16:48:21.298019 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:21.297924 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-29g2c\"" Oct 30 16:48:21.298019 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:21.298001 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 16:48:21.312977 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:21.312957 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 16:48:21.334378 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:21.334353 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ed4044f-063d-4be8-a4fe-9410ac57c1ac-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5ed4044f-063d-4be8-a4fe-9410ac57c1ac\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:48:21.334567 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:21.334427 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nsft\" (UniqueName: \"kubernetes.io/projected/5ed4044f-063d-4be8-a4fe-9410ac57c1ac-kube-api-access-5nsft\") pod \"alertmanager-main-0\" (UID: \"5ed4044f-063d-4be8-a4fe-9410ac57c1ac\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:48:21.334567 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:21.334540 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5ed4044f-063d-4be8-a4fe-9410ac57c1ac-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5ed4044f-063d-4be8-a4fe-9410ac57c1ac\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:48:21.334756 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:21.334634 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5ed4044f-063d-4be8-a4fe-9410ac57c1ac-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5ed4044f-063d-4be8-a4fe-9410ac57c1ac\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:48:21.334756 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:21.334701 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5ed4044f-063d-4be8-a4fe-9410ac57c1ac-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5ed4044f-063d-4be8-a4fe-9410ac57c1ac\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:48:21.334756 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:21.334742 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5ed4044f-063d-4be8-a4fe-9410ac57c1ac-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5ed4044f-063d-4be8-a4fe-9410ac57c1ac\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:48:21.334947 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:21.334829 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5ed4044f-063d-4be8-a4fe-9410ac57c1ac-config-out\") pod \"alertmanager-main-0\" (UID: \"5ed4044f-063d-4be8-a4fe-9410ac57c1ac\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:48:21.334947 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:21.334861 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5ed4044f-063d-4be8-a4fe-9410ac57c1ac-web-config\") pod \"alertmanager-main-0\" (UID: \"5ed4044f-063d-4be8-a4fe-9410ac57c1ac\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:48:21.334947 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:21.334913 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5ed4044f-063d-4be8-a4fe-9410ac57c1ac-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5ed4044f-063d-4be8-a4fe-9410ac57c1ac\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:48:21.335051 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:21.334973 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5ed4044f-063d-4be8-a4fe-9410ac57c1ac-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5ed4044f-063d-4be8-a4fe-9410ac57c1ac\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:48:21.335051 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:21.334997 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5ed4044f-063d-4be8-a4fe-9410ac57c1ac-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5ed4044f-063d-4be8-a4fe-9410ac57c1ac\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:48:21.335051 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:21.335026 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ed4044f-063d-4be8-a4fe-9410ac57c1ac-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5ed4044f-063d-4be8-a4fe-9410ac57c1ac\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:48:21.335155 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:21.335058 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5ed4044f-063d-4be8-a4fe-9410ac57c1ac-config-volume\") pod \"alertmanager-main-0\" (UID: \"5ed4044f-063d-4be8-a4fe-9410ac57c1ac\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:48:21.435545 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:21.435510 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5ed4044f-063d-4be8-a4fe-9410ac57c1ac-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5ed4044f-063d-4be8-a4fe-9410ac57c1ac\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:48:21.435545 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:21.435547 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5ed4044f-063d-4be8-a4fe-9410ac57c1ac-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5ed4044f-063d-4be8-a4fe-9410ac57c1ac\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:48:21.435758 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:21.435574 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5ed4044f-063d-4be8-a4fe-9410ac57c1ac-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5ed4044f-063d-4be8-a4fe-9410ac57c1ac\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:48:21.435758 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:21.435599 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5ed4044f-063d-4be8-a4fe-9410ac57c1ac-config-out\") pod \"alertmanager-main-0\" (UID: \"5ed4044f-063d-4be8-a4fe-9410ac57c1ac\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:48:21.435758 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:21.435614 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5ed4044f-063d-4be8-a4fe-9410ac57c1ac-web-config\") pod \"alertmanager-main-0\" (UID: \"5ed4044f-063d-4be8-a4fe-9410ac57c1ac\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:48:21.435758 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:21.435637 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5ed4044f-063d-4be8-a4fe-9410ac57c1ac-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5ed4044f-063d-4be8-a4fe-9410ac57c1ac\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:48:21.435758 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:21.435656 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5ed4044f-063d-4be8-a4fe-9410ac57c1ac-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5ed4044f-063d-4be8-a4fe-9410ac57c1ac\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:48:21.435758 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:21.435676 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5ed4044f-063d-4be8-a4fe-9410ac57c1ac-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5ed4044f-063d-4be8-a4fe-9410ac57c1ac\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:48:21.435758 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:21.435691 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ed4044f-063d-4be8-a4fe-9410ac57c1ac-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5ed4044f-063d-4be8-a4fe-9410ac57c1ac\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:48:21.436074 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:21.435986 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5ed4044f-063d-4be8-a4fe-9410ac57c1ac-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5ed4044f-063d-4be8-a4fe-9410ac57c1ac\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:48:21.436074 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:21.436060 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5ed4044f-063d-4be8-a4fe-9410ac57c1ac-config-volume\") pod \"alertmanager-main-0\" (UID: \"5ed4044f-063d-4be8-a4fe-9410ac57c1ac\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:48:21.436172 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:21.436091 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ed4044f-063d-4be8-a4fe-9410ac57c1ac-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5ed4044f-063d-4be8-a4fe-9410ac57c1ac\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:48:21.438192 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:21.438089 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nsft\" (UniqueName: \"kubernetes.io/projected/5ed4044f-063d-4be8-a4fe-9410ac57c1ac-kube-api-access-5nsft\") pod \"alertmanager-main-0\" (UID: \"5ed4044f-063d-4be8-a4fe-9410ac57c1ac\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:48:21.438192 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:21.438157 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5ed4044f-063d-4be8-a4fe-9410ac57c1ac-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5ed4044f-063d-4be8-a4fe-9410ac57c1ac\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:48:21.438417 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:21.438359 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5ed4044f-063d-4be8-a4fe-9410ac57c1ac-config-out\") pod \"alertmanager-main-0\" (UID: \"5ed4044f-063d-4be8-a4fe-9410ac57c1ac\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:48:21.439204 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:21.438733 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5ed4044f-063d-4be8-a4fe-9410ac57c1ac-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5ed4044f-063d-4be8-a4fe-9410ac57c1ac\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:48:21.439204 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:21.439084 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5ed4044f-063d-4be8-a4fe-9410ac57c1ac-web-config\") pod \"alertmanager-main-0\" (UID: \"5ed4044f-063d-4be8-a4fe-9410ac57c1ac\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:48:21.439204 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:21.439117 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5ed4044f-063d-4be8-a4fe-9410ac57c1ac-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5ed4044f-063d-4be8-a4fe-9410ac57c1ac\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:48:21.439436 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:21.439344 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5ed4044f-063d-4be8-a4fe-9410ac57c1ac-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5ed4044f-063d-4be8-a4fe-9410ac57c1ac\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:48:21.440470 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:21.440186 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ed4044f-063d-4be8-a4fe-9410ac57c1ac-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5ed4044f-063d-4be8-a4fe-9410ac57c1ac\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:48:21.443950 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:21.440771 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5ed4044f-063d-4be8-a4fe-9410ac57c1ac-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5ed4044f-063d-4be8-a4fe-9410ac57c1ac\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:48:21.443950 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:21.441062 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5ed4044f-063d-4be8-a4fe-9410ac57c1ac-config-volume\") pod \"alertmanager-main-0\" (UID: \"5ed4044f-063d-4be8-a4fe-9410ac57c1ac\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:48:21.443950 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:21.441646 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5ed4044f-063d-4be8-a4fe-9410ac57c1ac-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5ed4044f-063d-4be8-a4fe-9410ac57c1ac\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:48:21.443950 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:21.441902 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ed4044f-063d-4be8-a4fe-9410ac57c1ac-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5ed4044f-063d-4be8-a4fe-9410ac57c1ac\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:48:21.444557 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:21.444534 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5ed4044f-063d-4be8-a4fe-9410ac57c1ac-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5ed4044f-063d-4be8-a4fe-9410ac57c1ac\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:48:21.464710 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:21.464682 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nsft\" (UniqueName: \"kubernetes.io/projected/5ed4044f-063d-4be8-a4fe-9410ac57c1ac-kube-api-access-5nsft\") pod \"alertmanager-main-0\" (UID: \"5ed4044f-063d-4be8-a4fe-9410ac57c1ac\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:48:21.593364 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:21.593268 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:48:21.925543 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:21.925510 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 16:48:21.925739 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:48:21.925718 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ed4044f_063d_4be8_a4fe_9410ac57c1ac.slice/crio-ca46c9eec3c8533f4ce0db6ee1d187532554a65d1113e985756a19c10f98f12c WatchSource:0}: Error finding container ca46c9eec3c8533f4ce0db6ee1d187532554a65d1113e985756a19c10f98f12c: Status 404 returned error can't find the container with id ca46c9eec3c8533f4ce0db6ee1d187532554a65d1113e985756a19c10f98f12c Oct 30 16:48:22.474198 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:22.474161 4522 generic.go:356] "Generic (PLEG): container finished" podID="5ed4044f-063d-4be8-a4fe-9410ac57c1ac" containerID="cc764489dbc552c1ac7df47e3d2f57f6a1dd8627b024a74122b47940197e9fbd" exitCode=0 Oct 30 16:48:22.474599 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:22.474245 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5ed4044f-063d-4be8-a4fe-9410ac57c1ac","Type":"ContainerDied","Data":"cc764489dbc552c1ac7df47e3d2f57f6a1dd8627b024a74122b47940197e9fbd"} Oct 30 16:48:22.474599 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:22.474324 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5ed4044f-063d-4be8-a4fe-9410ac57c1ac","Type":"ContainerStarted","Data":"ca46c9eec3c8533f4ce0db6ee1d187532554a65d1113e985756a19c10f98f12c"} Oct 30 16:48:24.485368 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:24.485338 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5ed4044f-063d-4be8-a4fe-9410ac57c1ac","Type":"ContainerStarted","Data":"75019514df4c686143381d5ff0a846cd8bd24303fe6b2aefaad8048ab569f82f"} Oct 30 16:48:24.485740 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:24.485398 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5ed4044f-063d-4be8-a4fe-9410ac57c1ac","Type":"ContainerStarted","Data":"f53ddef6df945a78614bbbdf5cc5e17bef26477397ac40a3f24a6a794ad3068b"} Oct 30 16:48:24.485740 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:24.485423 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5ed4044f-063d-4be8-a4fe-9410ac57c1ac","Type":"ContainerStarted","Data":"5f125076cd653fabbb9832faf545ff24b903f8ab04ebed3d456faf36057f6d30"} Oct 30 16:48:24.485740 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:24.485447 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5ed4044f-063d-4be8-a4fe-9410ac57c1ac","Type":"ContainerStarted","Data":"cbcfb351e4752db489a0fa076ba51835495f25c5a99d7e53ba82b763496fe650"} Oct 30 16:48:24.485740 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:24.485469 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5ed4044f-063d-4be8-a4fe-9410ac57c1ac","Type":"ContainerStarted","Data":"9a6dfc3bf7b60bfdd2abf927a71a0f084a1ad46878973855515597790d8c1a85"} Oct 30 16:48:24.929891 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:24.929852 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7d89df5bcc-d5wph" Oct 30 16:48:24.930068 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:24.929980 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7d89df5bcc-d5wph" Oct 30 16:48:24.934725 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:24.934701 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7d89df5bcc-d5wph" Oct 30 16:48:25.492007 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:25.491969 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5ed4044f-063d-4be8-a4fe-9410ac57c1ac","Type":"ContainerStarted","Data":"42e96bf77ad8052e6e617694f197a2927a6608c6637c5b89ff147da05f3a786e"} Oct 30 16:48:25.495908 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:25.495886 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7d89df5bcc-d5wph" Oct 30 16:48:25.579452 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:48:25.579349 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.018211849 podStartE2EDuration="4.579328361s" totalImagesPullingTime="2.547560527s" totalInitContainerRuntime="13.555985ms" isStatefulPod=false podCreationTimestamp="2026-10-30 16:48:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:48:25.564868758 +0000 UTC m=+79.565623674" watchObservedRunningTime="2026-10-30 16:48:25.579328361 +0000 UTC m=+79.580083263" Oct 30 16:48:42.946251 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:48:42.946212 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 16:49:12.947026 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:49:12.946985 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 16:49:42.945768 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:49:42.945728 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 16:50:12.946764 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:50:12.946729 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 16:50:42.947125 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:50:42.947077 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 16:50:52.756406 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:50:52.756295 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-5xzxd"] Oct 30 16:50:52.759764 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:50:52.759742 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5xzxd" Oct 30 16:50:52.777520 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:50:52.777499 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-vmzg7\"" Oct 30 16:50:52.777520 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:50:52.777510 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 16:50:52.777696 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:50:52.777580 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 16:50:52.792682 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:50:52.792651 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-5xzxd"] Oct 30 16:50:52.915652 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:50:52.915604 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb2v4\" (UniqueName: \"kubernetes.io/projected/41c942d3-0353-4db1-b96b-7730829982af-kube-api-access-wb2v4\") pod \"cert-manager-webhook-5dbbbf75cc-5xzxd\" (UID: \"41c942d3-0353-4db1-b96b-7730829982af\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5xzxd" Oct 30 16:50:52.915652 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:50:52.915655 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/41c942d3-0353-4db1-b96b-7730829982af-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-5xzxd\" (UID: \"41c942d3-0353-4db1-b96b-7730829982af\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5xzxd" Oct 30 16:50:53.016591 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:50:53.016510 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb2v4\" (UniqueName: \"kubernetes.io/projected/41c942d3-0353-4db1-b96b-7730829982af-kube-api-access-wb2v4\") pod \"cert-manager-webhook-5dbbbf75cc-5xzxd\" (UID: \"41c942d3-0353-4db1-b96b-7730829982af\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5xzxd" Oct 30 16:50:53.016591 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:50:53.016552 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/41c942d3-0353-4db1-b96b-7730829982af-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-5xzxd\" (UID: \"41c942d3-0353-4db1-b96b-7730829982af\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5xzxd" Oct 30 16:50:53.041693 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:50:53.041655 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/41c942d3-0353-4db1-b96b-7730829982af-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-5xzxd\" (UID: \"41c942d3-0353-4db1-b96b-7730829982af\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5xzxd" Oct 30 16:50:53.041843 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:50:53.041772 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb2v4\" (UniqueName: \"kubernetes.io/projected/41c942d3-0353-4db1-b96b-7730829982af-kube-api-access-wb2v4\") pod \"cert-manager-webhook-5dbbbf75cc-5xzxd\" (UID: \"41c942d3-0353-4db1-b96b-7730829982af\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5xzxd" Oct 30 16:50:53.078680 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:50:53.078644 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5xzxd" Oct 30 16:50:53.377615 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:50:53.377585 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-5xzxd"] Oct 30 16:50:53.378818 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:50:53.378792 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod41c942d3_0353_4db1_b96b_7730829982af.slice/crio-f5514409b8dc0d77405ab2b3d6c16b7bc6b2f51a7d1c1410960463281deb7a60 WatchSource:0}: Error finding container f5514409b8dc0d77405ab2b3d6c16b7bc6b2f51a7d1c1410960463281deb7a60: Status 404 returned error can't find the container with id f5514409b8dc0d77405ab2b3d6c16b7bc6b2f51a7d1c1410960463281deb7a60 Oct 30 16:50:54.000111 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:50:54.000076 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5xzxd" event={"ID":"41c942d3-0353-4db1-b96b-7730829982af","Type":"ContainerStarted","Data":"f5514409b8dc0d77405ab2b3d6c16b7bc6b2f51a7d1c1410960463281deb7a60"} Oct 30 16:50:57.013788 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:50:57.013754 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5xzxd" event={"ID":"41c942d3-0353-4db1-b96b-7730829982af","Type":"ContainerStarted","Data":"59232d744392e72edc0bc10b05b5ae2d55f1e54de920129783e120fb60545147"} Oct 30 16:50:57.014224 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:50:57.013868 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5xzxd" Oct 30 16:50:57.066167 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:50:57.066118 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5xzxd" podStartSLOduration=1.969176074 podStartE2EDuration="5.066105339s" totalImagesPullingTime="3.096929265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:50:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:50:57.065715755 +0000 UTC m=+231.066470658" watchObservedRunningTime="2026-10-30 16:50:57.066105339 +0000 UTC m=+231.066860240" Oct 30 16:51:03.019298 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:03.019260 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5xzxd" Oct 30 16:51:12.948712 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:51:12.948676 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 16:51:28.167321 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:28.167276 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68c46f97f5-bzljn"] Oct 30 16:51:28.170662 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:28.170646 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-bzljn" Oct 30 16:51:28.207883 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:28.207854 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68c46f97f5-bzljn"] Oct 30 16:51:28.267067 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:28.267030 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 16:51:28.267276 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:28.267262 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-vn468\"" Oct 30 16:51:28.286467 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:28.286442 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\"" Oct 30 16:51:28.286467 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:28.286458 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 16:51:28.286758 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:28.286745 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 16:51:28.293199 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:28.293184 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 16:51:28.296618 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:28.296585 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d8c68be0-0463-4240-ac31-bb221e85ba64-metrics-cert\") pod \"opendatahub-operator-controller-manager-68c46f97f5-bzljn\" (UID: \"d8c68be0-0463-4240-ac31-bb221e85ba64\") " pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-bzljn" Oct 30 16:51:28.296739 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:28.296624 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm6kc\" (UniqueName: \"kubernetes.io/projected/d8c68be0-0463-4240-ac31-bb221e85ba64-kube-api-access-qm6kc\") pod \"opendatahub-operator-controller-manager-68c46f97f5-bzljn\" (UID: \"d8c68be0-0463-4240-ac31-bb221e85ba64\") " pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-bzljn" Oct 30 16:51:28.296739 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:28.296650 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d8c68be0-0463-4240-ac31-bb221e85ba64-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68c46f97f5-bzljn\" (UID: \"d8c68be0-0463-4240-ac31-bb221e85ba64\") " pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-bzljn" Oct 30 16:51:28.296739 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:28.296676 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d8c68be0-0463-4240-ac31-bb221e85ba64-manifests\") pod \"opendatahub-operator-controller-manager-68c46f97f5-bzljn\" (UID: \"d8c68be0-0463-4240-ac31-bb221e85ba64\") " pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-bzljn" Oct 30 16:51:28.296892 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:28.296815 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8c68be0-0463-4240-ac31-bb221e85ba64-tmp\") pod \"opendatahub-operator-controller-manager-68c46f97f5-bzljn\" (UID: \"d8c68be0-0463-4240-ac31-bb221e85ba64\") " pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-bzljn" Oct 30 16:51:28.296943 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:28.296903 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d8c68be0-0463-4240-ac31-bb221e85ba64-webhook-cert\") pod \"opendatahub-operator-controller-manager-68c46f97f5-bzljn\" (UID: \"d8c68be0-0463-4240-ac31-bb221e85ba64\") " pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-bzljn" Oct 30 16:51:28.397517 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:28.397485 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d8c68be0-0463-4240-ac31-bb221e85ba64-webhook-cert\") pod \"opendatahub-operator-controller-manager-68c46f97f5-bzljn\" (UID: \"d8c68be0-0463-4240-ac31-bb221e85ba64\") " pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-bzljn" Oct 30 16:51:28.397706 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:28.397544 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d8c68be0-0463-4240-ac31-bb221e85ba64-metrics-cert\") pod \"opendatahub-operator-controller-manager-68c46f97f5-bzljn\" (UID: \"d8c68be0-0463-4240-ac31-bb221e85ba64\") " pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-bzljn" Oct 30 16:51:28.397706 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:28.397573 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm6kc\" (UniqueName: \"kubernetes.io/projected/d8c68be0-0463-4240-ac31-bb221e85ba64-kube-api-access-qm6kc\") pod \"opendatahub-operator-controller-manager-68c46f97f5-bzljn\" (UID: \"d8c68be0-0463-4240-ac31-bb221e85ba64\") " pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-bzljn" Oct 30 16:51:28.397706 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:28.397596 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d8c68be0-0463-4240-ac31-bb221e85ba64-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68c46f97f5-bzljn\" (UID: \"d8c68be0-0463-4240-ac31-bb221e85ba64\") " pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-bzljn" Oct 30 16:51:28.397706 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:28.397612 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d8c68be0-0463-4240-ac31-bb221e85ba64-manifests\") pod \"opendatahub-operator-controller-manager-68c46f97f5-bzljn\" (UID: \"d8c68be0-0463-4240-ac31-bb221e85ba64\") " pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-bzljn" Oct 30 16:51:28.397706 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:28.397640 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8c68be0-0463-4240-ac31-bb221e85ba64-tmp\") pod \"opendatahub-operator-controller-manager-68c46f97f5-bzljn\" (UID: \"d8c68be0-0463-4240-ac31-bb221e85ba64\") " pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-bzljn" Oct 30 16:51:28.398120 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:28.398089 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d8c68be0-0463-4240-ac31-bb221e85ba64-manifests\") pod \"opendatahub-operator-controller-manager-68c46f97f5-bzljn\" (UID: \"d8c68be0-0463-4240-ac31-bb221e85ba64\") " pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-bzljn" Oct 30 16:51:28.399990 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:28.399969 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8c68be0-0463-4240-ac31-bb221e85ba64-tmp\") pod \"opendatahub-operator-controller-manager-68c46f97f5-bzljn\" (UID: \"d8c68be0-0463-4240-ac31-bb221e85ba64\") " pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-bzljn" Oct 30 16:51:28.400190 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:28.400171 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d8c68be0-0463-4240-ac31-bb221e85ba64-metrics-cert\") pod \"opendatahub-operator-controller-manager-68c46f97f5-bzljn\" (UID: \"d8c68be0-0463-4240-ac31-bb221e85ba64\") " pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-bzljn" Oct 30 16:51:28.400290 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:28.400270 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d8c68be0-0463-4240-ac31-bb221e85ba64-webhook-cert\") pod \"opendatahub-operator-controller-manager-68c46f97f5-bzljn\" (UID: \"d8c68be0-0463-4240-ac31-bb221e85ba64\") " pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-bzljn" Oct 30 16:51:28.400347 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:28.400298 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d8c68be0-0463-4240-ac31-bb221e85ba64-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68c46f97f5-bzljn\" (UID: \"d8c68be0-0463-4240-ac31-bb221e85ba64\") " pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-bzljn" Oct 30 16:51:28.423578 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:28.423513 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm6kc\" (UniqueName: \"kubernetes.io/projected/d8c68be0-0463-4240-ac31-bb221e85ba64-kube-api-access-qm6kc\") pod \"opendatahub-operator-controller-manager-68c46f97f5-bzljn\" (UID: \"d8c68be0-0463-4240-ac31-bb221e85ba64\") " pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-bzljn" Oct 30 16:51:28.485377 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:28.485341 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-bzljn" Oct 30 16:51:28.845208 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:51:28.845177 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8c68be0_0463_4240_ac31_bb221e85ba64.slice/crio-8d593bba799efa7498b6990677ae0c9c6cb02c83cf9ed774e94f4843ac03abce WatchSource:0}: Error finding container 8d593bba799efa7498b6990677ae0c9c6cb02c83cf9ed774e94f4843ac03abce: Status 404 returned error can't find the container with id 8d593bba799efa7498b6990677ae0c9c6cb02c83cf9ed774e94f4843ac03abce Oct 30 16:51:28.845342 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:28.845285 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68c46f97f5-bzljn"] Oct 30 16:51:29.127015 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:29.126930 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-bzljn" event={"ID":"d8c68be0-0463-4240-ac31-bb221e85ba64","Type":"ContainerStarted","Data":"8d593bba799efa7498b6990677ae0c9c6cb02c83cf9ed774e94f4843ac03abce"} Oct 30 16:51:32.147170 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:32.147081 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-bzljn" event={"ID":"d8c68be0-0463-4240-ac31-bb221e85ba64","Type":"ContainerStarted","Data":"185ec1a1945d27aa0d5f1628396c6a984728d214a5169b7f2235df0a9e9165da"} Oct 30 16:51:33.152151 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:33.152116 4522 generic.go:356] "Generic (PLEG): container finished" podID="d8c68be0-0463-4240-ac31-bb221e85ba64" containerID="185ec1a1945d27aa0d5f1628396c6a984728d214a5169b7f2235df0a9e9165da" exitCode=0 Oct 30 16:51:33.152669 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:33.152175 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-bzljn" event={"ID":"d8c68be0-0463-4240-ac31-bb221e85ba64","Type":"ContainerDied","Data":"185ec1a1945d27aa0d5f1628396c6a984728d214a5169b7f2235df0a9e9165da"} Oct 30 16:51:34.158618 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:34.158585 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-bzljn" event={"ID":"d8c68be0-0463-4240-ac31-bb221e85ba64","Type":"ContainerStarted","Data":"5b895cad1ef86e93b3277c7b1982574492ea09ff49505dde0e404955fa5b2551"} Oct 30 16:51:34.158997 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:34.158740 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-bzljn" Oct 30 16:51:34.252277 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:34.252218 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-bzljn" podStartSLOduration=2.7037598369999998 podStartE2EDuration="6.252204125s" totalImagesPullingTime="3.126833615s" totalInitContainerRuntime="421.610673ms" isStatefulPod=false podCreationTimestamp="2026-10-30 16:51:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:51:34.238281551 +0000 UTC m=+268.239036443" watchObservedRunningTime="2026-10-30 16:51:34.252204125 +0000 UTC m=+268.252959025" Oct 30 16:51:42.948711 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:51:42.948672 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 16:51:45.165072 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:45.165039 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-bzljn" Oct 30 16:51:48.281820 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:48.281788 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-768ffddf88-qflf4"] Oct 30 16:51:48.288061 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:48.288036 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-768ffddf88-qflf4" Oct 30 16:51:48.314236 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:48.314202 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 16:51:48.314423 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:48.314274 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 16:51:48.317489 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:48.317467 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 16:51:48.317489 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:48.317480 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-bg78w\"" Oct 30 16:51:48.327877 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:48.327852 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 16:51:48.329395 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:48.329374 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-768ffddf88-qflf4"] Oct 30 16:51:48.471937 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:48.471897 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz9pd\" (UniqueName: \"kubernetes.io/projected/af45ce16-d0ac-4fdf-8623-a23d3e8240f4-kube-api-access-bz9pd\") pod \"kube-auth-proxy-768ffddf88-qflf4\" (UID: \"af45ce16-d0ac-4fdf-8623-a23d3e8240f4\") " pod="openshift-ingress/kube-auth-proxy-768ffddf88-qflf4" Oct 30 16:51:48.471937 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:48.471936 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af45ce16-d0ac-4fdf-8623-a23d3e8240f4-tls-certs\") pod \"kube-auth-proxy-768ffddf88-qflf4\" (UID: \"af45ce16-d0ac-4fdf-8623-a23d3e8240f4\") " pod="openshift-ingress/kube-auth-proxy-768ffddf88-qflf4" Oct 30 16:51:48.472175 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:48.471959 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af45ce16-d0ac-4fdf-8623-a23d3e8240f4-tmp\") pod \"kube-auth-proxy-768ffddf88-qflf4\" (UID: \"af45ce16-d0ac-4fdf-8623-a23d3e8240f4\") " pod="openshift-ingress/kube-auth-proxy-768ffddf88-qflf4" Oct 30 16:51:48.573383 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:48.573275 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz9pd\" (UniqueName: \"kubernetes.io/projected/af45ce16-d0ac-4fdf-8623-a23d3e8240f4-kube-api-access-bz9pd\") pod \"kube-auth-proxy-768ffddf88-qflf4\" (UID: \"af45ce16-d0ac-4fdf-8623-a23d3e8240f4\") " pod="openshift-ingress/kube-auth-proxy-768ffddf88-qflf4" Oct 30 16:51:48.573383 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:48.573337 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af45ce16-d0ac-4fdf-8623-a23d3e8240f4-tls-certs\") pod \"kube-auth-proxy-768ffddf88-qflf4\" (UID: \"af45ce16-d0ac-4fdf-8623-a23d3e8240f4\") " pod="openshift-ingress/kube-auth-proxy-768ffddf88-qflf4" Oct 30 16:51:48.573383 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:48.573365 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af45ce16-d0ac-4fdf-8623-a23d3e8240f4-tmp\") pod \"kube-auth-proxy-768ffddf88-qflf4\" (UID: \"af45ce16-d0ac-4fdf-8623-a23d3e8240f4\") " pod="openshift-ingress/kube-auth-proxy-768ffddf88-qflf4" Oct 30 16:51:48.575768 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:48.575745 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af45ce16-d0ac-4fdf-8623-a23d3e8240f4-tmp\") pod \"kube-auth-proxy-768ffddf88-qflf4\" (UID: \"af45ce16-d0ac-4fdf-8623-a23d3e8240f4\") " pod="openshift-ingress/kube-auth-proxy-768ffddf88-qflf4" Oct 30 16:51:48.575960 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:48.575943 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af45ce16-d0ac-4fdf-8623-a23d3e8240f4-tls-certs\") pod \"kube-auth-proxy-768ffddf88-qflf4\" (UID: \"af45ce16-d0ac-4fdf-8623-a23d3e8240f4\") " pod="openshift-ingress/kube-auth-proxy-768ffddf88-qflf4" Oct 30 16:51:48.598162 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:48.598142 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz9pd\" (UniqueName: \"kubernetes.io/projected/af45ce16-d0ac-4fdf-8623-a23d3e8240f4-kube-api-access-bz9pd\") pod \"kube-auth-proxy-768ffddf88-qflf4\" (UID: \"af45ce16-d0ac-4fdf-8623-a23d3e8240f4\") " pod="openshift-ingress/kube-auth-proxy-768ffddf88-qflf4" Oct 30 16:51:48.599883 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:48.599863 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-768ffddf88-qflf4" Oct 30 16:51:48.898280 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:48.898255 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-768ffddf88-qflf4"] Oct 30 16:51:48.899118 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:51:48.899093 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf45ce16_d0ac_4fdf_8623_a23d3e8240f4.slice/crio-f2ec66de908214ae0cdd8af2f7d9790dd871abcf85ab19ba3d1a19616931885f WatchSource:0}: Error finding container f2ec66de908214ae0cdd8af2f7d9790dd871abcf85ab19ba3d1a19616931885f: Status 404 returned error can't find the container with id f2ec66de908214ae0cdd8af2f7d9790dd871abcf85ab19ba3d1a19616931885f Oct 30 16:51:49.218899 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:49.218866 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-768ffddf88-qflf4" event={"ID":"af45ce16-d0ac-4fdf-8623-a23d3e8240f4","Type":"ContainerStarted","Data":"f2ec66de908214ae0cdd8af2f7d9790dd871abcf85ab19ba3d1a19616931885f"} Oct 30 16:51:49.938485 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:49.938445 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-fc9769c4f-cxt8g"] Oct 30 16:51:49.941758 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:49.941738 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-fc9769c4f-cxt8g" Oct 30 16:51:49.962712 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:49.962685 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-fc9769c4f-cxt8g"] Oct 30 16:51:50.049582 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:50.049543 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:51:50.049762 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:50.049627 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Oct 30 16:51:50.049762 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:50.049629 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Oct 30 16:51:50.049866 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:50.049826 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Oct 30 16:51:50.049866 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:50.049836 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Oct 30 16:51:50.049954 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:50.049894 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-htr64\"" Oct 30 16:51:50.086675 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:50.086637 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7aa8a04f-61c8-434e-9818-9eba817f1ec3-manager-config\") pod \"lws-controller-manager-fc9769c4f-cxt8g\" (UID: \"7aa8a04f-61c8-434e-9818-9eba817f1ec3\") " pod="openshift-lws-operator/lws-controller-manager-fc9769c4f-cxt8g" Oct 30 16:51:50.086842 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:50.086681 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7aa8a04f-61c8-434e-9818-9eba817f1ec3-cert\") pod \"lws-controller-manager-fc9769c4f-cxt8g\" (UID: \"7aa8a04f-61c8-434e-9818-9eba817f1ec3\") " pod="openshift-lws-operator/lws-controller-manager-fc9769c4f-cxt8g" Oct 30 16:51:50.086842 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:50.086742 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7aa8a04f-61c8-434e-9818-9eba817f1ec3-metrics-cert\") pod \"lws-controller-manager-fc9769c4f-cxt8g\" (UID: \"7aa8a04f-61c8-434e-9818-9eba817f1ec3\") " pod="openshift-lws-operator/lws-controller-manager-fc9769c4f-cxt8g" Oct 30 16:51:50.086842 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:50.086766 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m769q\" (UniqueName: \"kubernetes.io/projected/7aa8a04f-61c8-434e-9818-9eba817f1ec3-kube-api-access-m769q\") pod \"lws-controller-manager-fc9769c4f-cxt8g\" (UID: \"7aa8a04f-61c8-434e-9818-9eba817f1ec3\") " pod="openshift-lws-operator/lws-controller-manager-fc9769c4f-cxt8g" Oct 30 16:51:50.188195 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:50.188155 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7aa8a04f-61c8-434e-9818-9eba817f1ec3-metrics-cert\") pod \"lws-controller-manager-fc9769c4f-cxt8g\" (UID: \"7aa8a04f-61c8-434e-9818-9eba817f1ec3\") " pod="openshift-lws-operator/lws-controller-manager-fc9769c4f-cxt8g" Oct 30 16:51:50.188370 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:50.188205 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m769q\" (UniqueName: \"kubernetes.io/projected/7aa8a04f-61c8-434e-9818-9eba817f1ec3-kube-api-access-m769q\") pod \"lws-controller-manager-fc9769c4f-cxt8g\" (UID: \"7aa8a04f-61c8-434e-9818-9eba817f1ec3\") " pod="openshift-lws-operator/lws-controller-manager-fc9769c4f-cxt8g" Oct 30 16:51:50.188370 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:50.188315 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7aa8a04f-61c8-434e-9818-9eba817f1ec3-manager-config\") pod \"lws-controller-manager-fc9769c4f-cxt8g\" (UID: \"7aa8a04f-61c8-434e-9818-9eba817f1ec3\") " pod="openshift-lws-operator/lws-controller-manager-fc9769c4f-cxt8g" Oct 30 16:51:50.188370 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:50.188342 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7aa8a04f-61c8-434e-9818-9eba817f1ec3-cert\") pod \"lws-controller-manager-fc9769c4f-cxt8g\" (UID: \"7aa8a04f-61c8-434e-9818-9eba817f1ec3\") " pod="openshift-lws-operator/lws-controller-manager-fc9769c4f-cxt8g" Oct 30 16:51:50.189267 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:50.189195 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7aa8a04f-61c8-434e-9818-9eba817f1ec3-manager-config\") pod \"lws-controller-manager-fc9769c4f-cxt8g\" (UID: \"7aa8a04f-61c8-434e-9818-9eba817f1ec3\") " pod="openshift-lws-operator/lws-controller-manager-fc9769c4f-cxt8g" Oct 30 16:51:50.191058 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:50.191034 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7aa8a04f-61c8-434e-9818-9eba817f1ec3-metrics-cert\") pod \"lws-controller-manager-fc9769c4f-cxt8g\" (UID: \"7aa8a04f-61c8-434e-9818-9eba817f1ec3\") " pod="openshift-lws-operator/lws-controller-manager-fc9769c4f-cxt8g" Oct 30 16:51:50.191150 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:50.191039 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7aa8a04f-61c8-434e-9818-9eba817f1ec3-cert\") pod \"lws-controller-manager-fc9769c4f-cxt8g\" (UID: \"7aa8a04f-61c8-434e-9818-9eba817f1ec3\") " pod="openshift-lws-operator/lws-controller-manager-fc9769c4f-cxt8g" Oct 30 16:51:50.213642 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:50.213609 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m769q\" (UniqueName: \"kubernetes.io/projected/7aa8a04f-61c8-434e-9818-9eba817f1ec3-kube-api-access-m769q\") pod \"lws-controller-manager-fc9769c4f-cxt8g\" (UID: \"7aa8a04f-61c8-434e-9818-9eba817f1ec3\") " pod="openshift-lws-operator/lws-controller-manager-fc9769c4f-cxt8g" Oct 30 16:51:50.256162 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:50.256130 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-fc9769c4f-cxt8g" Oct 30 16:51:50.568714 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:50.568683 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-fc9769c4f-cxt8g"] Oct 30 16:51:50.569788 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:51:50.569755 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aa8a04f_61c8_434e_9818_9eba817f1ec3.slice/crio-63863d1b751302c0487d03c9c6b573ee3c679ef29edcfb157a8fec354d6390ec WatchSource:0}: Error finding container 63863d1b751302c0487d03c9c6b573ee3c679ef29edcfb157a8fec354d6390ec: Status 404 returned error can't find the container with id 63863d1b751302c0487d03c9c6b573ee3c679ef29edcfb157a8fec354d6390ec Oct 30 16:51:51.229232 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:51.229192 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-fc9769c4f-cxt8g" event={"ID":"7aa8a04f-61c8-434e-9818-9eba817f1ec3","Type":"ContainerStarted","Data":"63863d1b751302c0487d03c9c6b573ee3c679ef29edcfb157a8fec354d6390ec"} Oct 30 16:51:51.230622 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:51.230602 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-768ffddf88-qflf4" event={"ID":"af45ce16-d0ac-4fdf-8623-a23d3e8240f4","Type":"ContainerStarted","Data":"3e9e3f0a69e4cf6cce057a92d61b5810e54823f79b6cb12a86f7a24756b89ea8"} Oct 30 16:51:53.240630 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:53.240591 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-fc9769c4f-cxt8g" event={"ID":"7aa8a04f-61c8-434e-9818-9eba817f1ec3","Type":"ContainerStarted","Data":"457fe0444cdcad01cfad87002fce52897d1731856f7051da941e320afa964779"} Oct 30 16:51:53.241019 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:53.240671 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-fc9769c4f-cxt8g" Oct 30 16:51:53.285604 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:53.285551 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-fc9769c4f-cxt8g" podStartSLOduration=1.7468292810000001 podStartE2EDuration="4.285535851s" totalImagesPullingTime="2.53870657s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:51:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:51:53.285257388 +0000 UTC m=+287.286012307" watchObservedRunningTime="2026-10-30 16:51:53.285535851 +0000 UTC m=+287.286290749" Oct 30 16:51:53.286094 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:51:53.286050 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-768ffddf88-qflf4" podStartSLOduration=3.123515812 podStartE2EDuration="5.28604371s" totalImagesPullingTime="2.162527898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:51:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:51:51.280126393 +0000 UTC m=+285.280881311" watchObservedRunningTime="2026-10-30 16:51:53.28604371 +0000 UTC m=+287.286798607" Oct 30 16:52:04.247004 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:52:04.246972 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-fc9769c4f-cxt8g" Oct 30 16:52:12.947050 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:52:12.947013 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 16:52:42.946524 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:52:42.946487 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 16:53:12.946269 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:53:12.946235 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 16:53:42.946508 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:53:42.946468 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 16:54:04.874853 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:54:04.874817 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-klsm7"] Oct 30 16:54:04.878020 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:54:04.878002 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-klsm7" Oct 30 16:54:04.898857 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:54:04.898821 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Oct 30 16:54:04.900103 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:54:04.900082 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-klsm7"] Oct 30 16:54:04.909542 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:54:04.909515 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Oct 30 16:54:04.909667 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:54:04.909552 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 16:54:04.909667 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:54:04.909571 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-wt7tg\"" Oct 30 16:54:04.909667 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:54:04.909576 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 16:54:04.941760 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:54:04.941720 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d03e75f7-ffc9-4069-963a-f6676c7e8922-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-klsm7\" (UID: \"d03e75f7-ffc9-4069-963a-f6676c7e8922\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-klsm7" Oct 30 16:54:04.941894 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:54:04.941789 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7flz\" (UniqueName: \"kubernetes.io/projected/d03e75f7-ffc9-4069-963a-f6676c7e8922-kube-api-access-z7flz\") pod \"kuadrant-console-plugin-66c4fb575d-klsm7\" (UID: \"d03e75f7-ffc9-4069-963a-f6676c7e8922\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-klsm7" Oct 30 16:54:04.941894 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:54:04.941851 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d03e75f7-ffc9-4069-963a-f6676c7e8922-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-klsm7\" (UID: \"d03e75f7-ffc9-4069-963a-f6676c7e8922\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-klsm7" Oct 30 16:54:05.042683 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:54:05.042642 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d03e75f7-ffc9-4069-963a-f6676c7e8922-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-klsm7\" (UID: \"d03e75f7-ffc9-4069-963a-f6676c7e8922\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-klsm7" Oct 30 16:54:05.042852 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:54:05.042692 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d03e75f7-ffc9-4069-963a-f6676c7e8922-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-klsm7\" (UID: \"d03e75f7-ffc9-4069-963a-f6676c7e8922\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-klsm7" Oct 30 16:54:05.042852 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:54:05.042730 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7flz\" (UniqueName: \"kubernetes.io/projected/d03e75f7-ffc9-4069-963a-f6676c7e8922-kube-api-access-z7flz\") pod \"kuadrant-console-plugin-66c4fb575d-klsm7\" (UID: \"d03e75f7-ffc9-4069-963a-f6676c7e8922\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-klsm7" Oct 30 16:54:05.043539 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:54:05.043519 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d03e75f7-ffc9-4069-963a-f6676c7e8922-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-klsm7\" (UID: \"d03e75f7-ffc9-4069-963a-f6676c7e8922\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-klsm7" Oct 30 16:54:05.045162 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:54:05.045136 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d03e75f7-ffc9-4069-963a-f6676c7e8922-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-klsm7\" (UID: \"d03e75f7-ffc9-4069-963a-f6676c7e8922\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-klsm7" Oct 30 16:54:05.072161 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:54:05.072131 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7flz\" (UniqueName: \"kubernetes.io/projected/d03e75f7-ffc9-4069-963a-f6676c7e8922-kube-api-access-z7flz\") pod \"kuadrant-console-plugin-66c4fb575d-klsm7\" (UID: \"d03e75f7-ffc9-4069-963a-f6676c7e8922\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-klsm7" Oct 30 16:54:05.188849 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:54:05.188771 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-klsm7" Oct 30 16:54:05.490158 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:54:05.490128 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-klsm7"] Oct 30 16:54:05.491269 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:54:05.491239 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd03e75f7_ffc9_4069_963a_f6676c7e8922.slice/crio-f0dd412cb6faa2271a4daff2cbb31899ba78658d9af63fea3ba4edbec9205c66 WatchSource:0}: Error finding container f0dd412cb6faa2271a4daff2cbb31899ba78658d9af63fea3ba4edbec9205c66: Status 404 returned error can't find the container with id f0dd412cb6faa2271a4daff2cbb31899ba78658d9af63fea3ba4edbec9205c66 Oct 30 16:54:05.492734 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:54:05.492714 4522 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 16:54:05.764053 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:54:05.763971 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-klsm7" event={"ID":"d03e75f7-ffc9-4069-963a-f6676c7e8922","Type":"ContainerStarted","Data":"f0dd412cb6faa2271a4daff2cbb31899ba78658d9af63fea3ba4edbec9205c66"} Oct 30 16:54:10.789041 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:54:10.789005 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-klsm7" event={"ID":"d03e75f7-ffc9-4069-963a-f6676c7e8922","Type":"ContainerStarted","Data":"7d92f85f90a15232a55e01b55abf30687e986db5154e29e8024c05b02e81db16"} Oct 30 16:54:10.833987 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:54:10.833934 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-klsm7" podStartSLOduration=2.317789068 podStartE2EDuration="6.833920133s" totalImagesPullingTime="4.516131065s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:54:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:54:10.833156101 +0000 UTC m=+424.833911002" watchObservedRunningTime="2026-10-30 16:54:10.833920133 +0000 UTC m=+424.834675034" Oct 30 16:54:12.946740 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:54:12.946699 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 16:54:17.434240 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:54:17.434203 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-tt5f8"] Oct 30 16:54:17.438237 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:54:17.438217 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-tt5f8" Oct 30 16:54:17.453699 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:54:17.453667 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rdjz\" (UniqueName: \"kubernetes.io/projected/3244034b-650b-4f3f-a651-0bf35bd9c8f1-kube-api-access-5rdjz\") pod \"limitador-operator-controller-manager-856dd584c5-tt5f8\" (UID: \"3244034b-650b-4f3f-a651-0bf35bd9c8f1\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-tt5f8" Oct 30 16:54:17.469084 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:54:17.469057 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-tt5f8"] Oct 30 16:54:17.481601 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:54:17.481576 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-f788c\"" Oct 30 16:54:17.554501 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:54:17.554466 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rdjz\" (UniqueName: \"kubernetes.io/projected/3244034b-650b-4f3f-a651-0bf35bd9c8f1-kube-api-access-5rdjz\") pod \"limitador-operator-controller-manager-856dd584c5-tt5f8\" (UID: \"3244034b-650b-4f3f-a651-0bf35bd9c8f1\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-tt5f8" Oct 30 16:54:17.580868 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:54:17.580839 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rdjz\" (UniqueName: \"kubernetes.io/projected/3244034b-650b-4f3f-a651-0bf35bd9c8f1-kube-api-access-5rdjz\") pod \"limitador-operator-controller-manager-856dd584c5-tt5f8\" (UID: \"3244034b-650b-4f3f-a651-0bf35bd9c8f1\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-tt5f8" Oct 30 16:54:17.750887 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:54:17.750853 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-tt5f8" Oct 30 16:54:18.052180 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:54:18.052151 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-tt5f8"] Oct 30 16:54:18.052632 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:54:18.052607 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3244034b_650b_4f3f_a651_0bf35bd9c8f1.slice/crio-99e2e17ea2c281a599d329e17ebd6ee05201e0075f97dbc5408d671272a3281c WatchSource:0}: Error finding container 99e2e17ea2c281a599d329e17ebd6ee05201e0075f97dbc5408d671272a3281c: Status 404 returned error can't find the container with id 99e2e17ea2c281a599d329e17ebd6ee05201e0075f97dbc5408d671272a3281c Oct 30 16:54:18.827795 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:54:18.827758 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-tt5f8" event={"ID":"3244034b-650b-4f3f-a651-0bf35bd9c8f1","Type":"ContainerStarted","Data":"99e2e17ea2c281a599d329e17ebd6ee05201e0075f97dbc5408d671272a3281c"} Oct 30 16:54:20.838359 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:54:20.838323 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-tt5f8" event={"ID":"3244034b-650b-4f3f-a651-0bf35bd9c8f1","Type":"ContainerStarted","Data":"d37787a80e66a4b5920e8a46ef70a5ca0e81e503134fcc26a344e69ad0336b61"} Oct 30 16:54:20.838758 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:54:20.838391 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-tt5f8" Oct 30 16:54:20.885598 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:54:20.885545 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-tt5f8" podStartSLOduration=1.656215135 podStartE2EDuration="3.885531316s" totalImagesPullingTime="2.229316181s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:54:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:54:20.884649353 +0000 UTC m=+434.885404254" watchObservedRunningTime="2026-10-30 16:54:20.885531316 +0000 UTC m=+434.886286217" Oct 30 16:54:31.845106 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:54:31.845073 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-tt5f8" Oct 30 16:54:42.947034 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:54:42.946995 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 16:55:03.629635 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:03.629600 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-rmp5r"] Oct 30 16:55:03.633823 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:03.633805 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-rmp5r" Oct 30 16:55:03.651511 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:03.651488 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-rmp5r"] Oct 30 16:55:03.652032 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:03.652013 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-497hv\"" Oct 30 16:55:03.756712 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:03.756667 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5grtv\" (UniqueName: \"kubernetes.io/projected/0fc34d7a-fb05-4c79-8301-8f5cdd34646f-kube-api-access-5grtv\") pod \"authorino-57f7f8659f-rmp5r\" (UID: \"0fc34d7a-fb05-4c79-8301-8f5cdd34646f\") " pod="kuadrant-system/authorino-57f7f8659f-rmp5r" Oct 30 16:55:03.857506 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:03.857469 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5grtv\" (UniqueName: \"kubernetes.io/projected/0fc34d7a-fb05-4c79-8301-8f5cdd34646f-kube-api-access-5grtv\") pod \"authorino-57f7f8659f-rmp5r\" (UID: \"0fc34d7a-fb05-4c79-8301-8f5cdd34646f\") " pod="kuadrant-system/authorino-57f7f8659f-rmp5r" Oct 30 16:55:03.882267 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:03.882198 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5grtv\" (UniqueName: \"kubernetes.io/projected/0fc34d7a-fb05-4c79-8301-8f5cdd34646f-kube-api-access-5grtv\") pod \"authorino-57f7f8659f-rmp5r\" (UID: \"0fc34d7a-fb05-4c79-8301-8f5cdd34646f\") " pod="kuadrant-system/authorino-57f7f8659f-rmp5r" Oct 30 16:55:03.944067 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:03.944031 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-rmp5r" Oct 30 16:55:04.237754 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:04.237720 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-rmp5r"] Oct 30 16:55:04.238620 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:55:04.238586 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0fc34d7a_fb05_4c79_8301_8f5cdd34646f.slice/crio-29504dc1ec7db819d4841193f901b5d87497b1c3a3480ffb438da4924965c9fc WatchSource:0}: Error finding container 29504dc1ec7db819d4841193f901b5d87497b1c3a3480ffb438da4924965c9fc: Status 404 returned error can't find the container with id 29504dc1ec7db819d4841193f901b5d87497b1c3a3480ffb438da4924965c9fc Oct 30 16:55:04.240836 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:04.240752 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-d5r7h"] Oct 30 16:55:04.243899 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:04.243879 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-d5r7h" Oct 30 16:55:04.267648 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:04.267618 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-d5r7h"] Oct 30 16:55:04.361735 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:04.361688 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlttd\" (UniqueName: \"kubernetes.io/projected/908f7490-017a-4868-8803-552cf0ee72f7-kube-api-access-xlttd\") pod \"authorino-56c6c9ddd5-d5r7h\" (UID: \"908f7490-017a-4868-8803-552cf0ee72f7\") " pod="kuadrant-system/authorino-56c6c9ddd5-d5r7h" Oct 30 16:55:04.463236 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:04.463197 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlttd\" (UniqueName: \"kubernetes.io/projected/908f7490-017a-4868-8803-552cf0ee72f7-kube-api-access-xlttd\") pod \"authorino-56c6c9ddd5-d5r7h\" (UID: \"908f7490-017a-4868-8803-552cf0ee72f7\") " pod="kuadrant-system/authorino-56c6c9ddd5-d5r7h" Oct 30 16:55:04.489061 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:04.488987 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlttd\" (UniqueName: \"kubernetes.io/projected/908f7490-017a-4868-8803-552cf0ee72f7-kube-api-access-xlttd\") pod \"authorino-56c6c9ddd5-d5r7h\" (UID: \"908f7490-017a-4868-8803-552cf0ee72f7\") " pod="kuadrant-system/authorino-56c6c9ddd5-d5r7h" Oct 30 16:55:04.556189 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:04.556150 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-d5r7h" Oct 30 16:55:04.849781 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:04.849752 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-d5r7h"] Oct 30 16:55:04.850914 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:55:04.850887 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod908f7490_017a_4868_8803_552cf0ee72f7.slice/crio-35992dbb8349d7258f2411868f734c043506143319102b90ed89bf99204ef8f3 WatchSource:0}: Error finding container 35992dbb8349d7258f2411868f734c043506143319102b90ed89bf99204ef8f3: Status 404 returned error can't find the container with id 35992dbb8349d7258f2411868f734c043506143319102b90ed89bf99204ef8f3 Oct 30 16:55:05.017678 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:05.017636 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-rmp5r" event={"ID":"0fc34d7a-fb05-4c79-8301-8f5cdd34646f","Type":"ContainerStarted","Data":"29504dc1ec7db819d4841193f901b5d87497b1c3a3480ffb438da4924965c9fc"} Oct 30 16:55:05.018846 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:05.018821 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-d5r7h" event={"ID":"908f7490-017a-4868-8803-552cf0ee72f7","Type":"ContainerStarted","Data":"35992dbb8349d7258f2411868f734c043506143319102b90ed89bf99204ef8f3"} Oct 30 16:55:07.029088 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:07.029055 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-d5r7h" event={"ID":"908f7490-017a-4868-8803-552cf0ee72f7","Type":"ContainerStarted","Data":"449e47fd257426152ced755bb47a9032c3e5492779e0682d494def214b6e34f6"} Oct 30 16:55:07.030598 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:07.030574 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-rmp5r" event={"ID":"0fc34d7a-fb05-4c79-8301-8f5cdd34646f","Type":"ContainerStarted","Data":"dc360ffece835300e0e362e1d04ec3611373cedc38efb26437511d518624c1b6"} Oct 30 16:55:07.071426 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:07.071388 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-d5r7h" podStartSLOduration=1.452087564 podStartE2EDuration="3.071375174s" totalImagesPullingTime="1.61928761s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:55:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:55:07.070127557 +0000 UTC m=+481.070882469" watchObservedRunningTime="2026-10-30 16:55:07.071375174 +0000 UTC m=+481.072130075" Oct 30 16:55:07.107642 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:07.107615 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-rmp5r"] Oct 30 16:55:07.128685 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:07.128639 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-rmp5r" podStartSLOduration=2.241006993 podStartE2EDuration="4.128626407s" totalImagesPullingTime="1.887619414s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:55:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:55:07.128369947 +0000 UTC m=+481.129124850" watchObservedRunningTime="2026-10-30 16:55:07.128626407 +0000 UTC m=+481.129381307" Oct 30 16:55:09.038871 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:09.038810 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-rmp5r" podUID="0fc34d7a-fb05-4c79-8301-8f5cdd34646f" containerName="authorino" containerID="cri-o://dc360ffece835300e0e362e1d04ec3611373cedc38efb26437511d518624c1b6" gracePeriod=30 Oct 30 16:55:09.282201 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:09.282173 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-rmp5r" Oct 30 16:55:09.307922 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:09.307847 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fc34d7a-fb05-4c79-8301-8f5cdd34646f-kube-api-access-5grtv\" (UniqueName: \"kubernetes.io/projected/0fc34d7a-fb05-4c79-8301-8f5cdd34646f-kube-api-access-5grtv\") pod \"0fc34d7a-fb05-4c79-8301-8f5cdd34646f\" (UID: \"0fc34d7a-fb05-4c79-8301-8f5cdd34646f\") " Oct 30 16:55:09.310037 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:09.309994 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fc34d7a-fb05-4c79-8301-8f5cdd34646f-kube-api-access-5grtv" pod "0fc34d7a-fb05-4c79-8301-8f5cdd34646f" (UID: "0fc34d7a-fb05-4c79-8301-8f5cdd34646f"). InnerVolumeSpecName "kube-api-access-5grtv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:55:09.409054 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:09.409017 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5grtv\" (UniqueName: \"kubernetes.io/projected/0fc34d7a-fb05-4c79-8301-8f5cdd34646f-kube-api-access-5grtv\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 16:55:10.043988 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:10.043954 4522 generic.go:356] "Generic (PLEG): container finished" podID="0fc34d7a-fb05-4c79-8301-8f5cdd34646f" containerID="dc360ffece835300e0e362e1d04ec3611373cedc38efb26437511d518624c1b6" exitCode=0 Oct 30 16:55:10.044457 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:10.043996 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-rmp5r" Oct 30 16:55:10.044457 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:10.043993 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-rmp5r" event={"ID":"0fc34d7a-fb05-4c79-8301-8f5cdd34646f","Type":"ContainerDied","Data":"dc360ffece835300e0e362e1d04ec3611373cedc38efb26437511d518624c1b6"} Oct 30 16:55:10.044457 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:10.044096 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-rmp5r" event={"ID":"0fc34d7a-fb05-4c79-8301-8f5cdd34646f","Type":"ContainerDied","Data":"29504dc1ec7db819d4841193f901b5d87497b1c3a3480ffb438da4924965c9fc"} Oct 30 16:55:10.044457 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:10.044114 4522 scope.go:122] "RemoveContainer" containerID="dc360ffece835300e0e362e1d04ec3611373cedc38efb26437511d518624c1b6" Oct 30 16:55:10.052688 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:10.052671 4522 scope.go:122] "RemoveContainer" containerID="dc360ffece835300e0e362e1d04ec3611373cedc38efb26437511d518624c1b6" Oct 30 16:55:10.052958 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:55:10.052941 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc360ffece835300e0e362e1d04ec3611373cedc38efb26437511d518624c1b6\": container with ID starting with dc360ffece835300e0e362e1d04ec3611373cedc38efb26437511d518624c1b6 not found: ID does not exist" containerID="dc360ffece835300e0e362e1d04ec3611373cedc38efb26437511d518624c1b6" Oct 30 16:55:10.052997 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:10.052966 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc360ffece835300e0e362e1d04ec3611373cedc38efb26437511d518624c1b6"} err="failed to get container status \"dc360ffece835300e0e362e1d04ec3611373cedc38efb26437511d518624c1b6\": rpc error: code = NotFound desc = could not find container \"dc360ffece835300e0e362e1d04ec3611373cedc38efb26437511d518624c1b6\": container with ID starting with dc360ffece835300e0e362e1d04ec3611373cedc38efb26437511d518624c1b6 not found: ID does not exist" Oct 30 16:55:10.098299 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:10.098270 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-rmp5r"] Oct 30 16:55:10.104845 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:10.104820 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-rmp5r"] Oct 30 16:55:10.142629 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:10.142599 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fc34d7a-fb05-4c79-8301-8f5cdd34646f" path="/var/lib/kubelet/pods/0fc34d7a-fb05-4c79-8301-8f5cdd34646f/volumes" Oct 30 16:55:12.946351 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:55:12.946295 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 16:55:32.752988 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:32.752955 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-h6dk2"] Oct 30 16:55:32.753490 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:32.753287 4522 cpu_manager.go:414] "containerMap: removing container" podUID="0fc34d7a-fb05-4c79-8301-8f5cdd34646f" containerName="authorino" Oct 30 16:55:32.753490 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:32.753316 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fc34d7a-fb05-4c79-8301-8f5cdd34646f" containerName="authorino" Oct 30 16:55:32.753490 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:32.753412 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fc34d7a-fb05-4c79-8301-8f5cdd34646f" containerName="authorino" Oct 30 16:55:32.755445 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:32.755428 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-h6dk2" Oct 30 16:55:32.775765 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:32.775743 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-h6dk2"] Oct 30 16:55:32.916883 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:32.916851 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-h6dk2"] Oct 30 16:55:32.917069 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:55:32.917052 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-pxphq], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-h6dk2" podUID="2c6f0eb9-7433-47ef-80ce-75e59b7a5d38" Oct 30 16:55:32.918882 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:32.918858 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxphq\" (UniqueName: \"kubernetes.io/projected/2c6f0eb9-7433-47ef-80ce-75e59b7a5d38-kube-api-access-pxphq\") pod \"authorino-6f6944bcdd-h6dk2\" (UID: \"2c6f0eb9-7433-47ef-80ce-75e59b7a5d38\") " pod="kuadrant-system/authorino-6f6944bcdd-h6dk2" Oct 30 16:55:32.948178 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:32.948142 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-v7xdt"] Oct 30 16:55:32.950435 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:32.950416 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-v7xdt" Oct 30 16:55:32.970126 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:32.970098 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 30 16:55:32.971463 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:32.971440 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-v7xdt"] Oct 30 16:55:33.019754 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:33.019682 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/61eb0c0e-3ff2-4878-9cd2-d86bf41a5ea9-tls-cert\") pod \"authorino-6889cbd8c4-v7xdt\" (UID: \"61eb0c0e-3ff2-4878-9cd2-d86bf41a5ea9\") " pod="kuadrant-system/authorino-6889cbd8c4-v7xdt" Oct 30 16:55:33.019754 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:33.019734 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr6g9\" (UniqueName: \"kubernetes.io/projected/61eb0c0e-3ff2-4878-9cd2-d86bf41a5ea9-kube-api-access-kr6g9\") pod \"authorino-6889cbd8c4-v7xdt\" (UID: \"61eb0c0e-3ff2-4878-9cd2-d86bf41a5ea9\") " pod="kuadrant-system/authorino-6889cbd8c4-v7xdt" Oct 30 16:55:33.019899 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:33.019826 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxphq\" (UniqueName: \"kubernetes.io/projected/2c6f0eb9-7433-47ef-80ce-75e59b7a5d38-kube-api-access-pxphq\") pod \"authorino-6f6944bcdd-h6dk2\" (UID: \"2c6f0eb9-7433-47ef-80ce-75e59b7a5d38\") " pod="kuadrant-system/authorino-6f6944bcdd-h6dk2" Oct 30 16:55:33.045359 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:33.045325 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxphq\" (UniqueName: \"kubernetes.io/projected/2c6f0eb9-7433-47ef-80ce-75e59b7a5d38-kube-api-access-pxphq\") pod \"authorino-6f6944bcdd-h6dk2\" (UID: \"2c6f0eb9-7433-47ef-80ce-75e59b7a5d38\") " pod="kuadrant-system/authorino-6f6944bcdd-h6dk2" Oct 30 16:55:33.121302 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:33.121263 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/61eb0c0e-3ff2-4878-9cd2-d86bf41a5ea9-tls-cert\") pod \"authorino-6889cbd8c4-v7xdt\" (UID: \"61eb0c0e-3ff2-4878-9cd2-d86bf41a5ea9\") " pod="kuadrant-system/authorino-6889cbd8c4-v7xdt" Oct 30 16:55:33.121496 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:33.121339 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr6g9\" (UniqueName: \"kubernetes.io/projected/61eb0c0e-3ff2-4878-9cd2-d86bf41a5ea9-kube-api-access-kr6g9\") pod \"authorino-6889cbd8c4-v7xdt\" (UID: \"61eb0c0e-3ff2-4878-9cd2-d86bf41a5ea9\") " pod="kuadrant-system/authorino-6889cbd8c4-v7xdt" Oct 30 16:55:33.123715 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:33.123685 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/61eb0c0e-3ff2-4878-9cd2-d86bf41a5ea9-tls-cert\") pod \"authorino-6889cbd8c4-v7xdt\" (UID: \"61eb0c0e-3ff2-4878-9cd2-d86bf41a5ea9\") " pod="kuadrant-system/authorino-6889cbd8c4-v7xdt" Oct 30 16:55:33.127838 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:33.127814 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-v7xdt"] Oct 30 16:55:33.128015 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:55:33.127999 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-kr6g9], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6889cbd8c4-v7xdt" podUID="61eb0c0e-3ff2-4878-9cd2-d86bf41a5ea9" Oct 30 16:55:33.147056 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:33.147024 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-v7xdt" Oct 30 16:55:33.147537 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:33.147519 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-h6dk2" Oct 30 16:55:33.152400 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:33.152376 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr6g9\" (UniqueName: \"kubernetes.io/projected/61eb0c0e-3ff2-4878-9cd2-d86bf41a5ea9-kube-api-access-kr6g9\") pod \"authorino-6889cbd8c4-v7xdt\" (UID: \"61eb0c0e-3ff2-4878-9cd2-d86bf41a5ea9\") " pod="kuadrant-system/authorino-6889cbd8c4-v7xdt" Oct 30 16:55:33.160610 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:33.160588 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5764994858-wwf9h"] Oct 30 16:55:33.162930 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:33.162910 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5764994858-wwf9h" Oct 30 16:55:33.163638 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:33.163618 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-v7xdt" Oct 30 16:55:33.166894 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:33.166877 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-h6dk2" Oct 30 16:55:33.183807 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:33.183777 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5764994858-wwf9h"] Oct 30 16:55:33.323768 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:33.323688 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61eb0c0e-3ff2-4878-9cd2-d86bf41a5ea9-tls-cert\" (UniqueName: \"kubernetes.io/secret/61eb0c0e-3ff2-4878-9cd2-d86bf41a5ea9-tls-cert\") pod \"61eb0c0e-3ff2-4878-9cd2-d86bf41a5ea9\" (UID: \"61eb0c0e-3ff2-4878-9cd2-d86bf41a5ea9\") " Oct 30 16:55:33.323768 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:33.323734 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61eb0c0e-3ff2-4878-9cd2-d86bf41a5ea9-kube-api-access-kr6g9\" (UniqueName: \"kubernetes.io/projected/61eb0c0e-3ff2-4878-9cd2-d86bf41a5ea9-kube-api-access-kr6g9\") pod \"61eb0c0e-3ff2-4878-9cd2-d86bf41a5ea9\" (UID: \"61eb0c0e-3ff2-4878-9cd2-d86bf41a5ea9\") " Oct 30 16:55:33.323768 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:33.323756 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c6f0eb9-7433-47ef-80ce-75e59b7a5d38-kube-api-access-pxphq\" (UniqueName: \"kubernetes.io/projected/2c6f0eb9-7433-47ef-80ce-75e59b7a5d38-kube-api-access-pxphq\") pod \"2c6f0eb9-7433-47ef-80ce-75e59b7a5d38\" (UID: \"2c6f0eb9-7433-47ef-80ce-75e59b7a5d38\") " Oct 30 16:55:33.323980 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:33.323935 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5fc165af-809c-4990-9d62-d9f68d4bcbb2-tls-cert\") pod \"authorino-5764994858-wwf9h\" (UID: \"5fc165af-809c-4990-9d62-d9f68d4bcbb2\") " pod="kuadrant-system/authorino-5764994858-wwf9h" Oct 30 16:55:33.323980 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:33.323957 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49ssb\" (UniqueName: \"kubernetes.io/projected/5fc165af-809c-4990-9d62-d9f68d4bcbb2-kube-api-access-49ssb\") pod \"authorino-5764994858-wwf9h\" (UID: \"5fc165af-809c-4990-9d62-d9f68d4bcbb2\") " pod="kuadrant-system/authorino-5764994858-wwf9h" Oct 30 16:55:33.326011 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:33.325977 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61eb0c0e-3ff2-4878-9cd2-d86bf41a5ea9-kube-api-access-kr6g9" pod "61eb0c0e-3ff2-4878-9cd2-d86bf41a5ea9" (UID: "61eb0c0e-3ff2-4878-9cd2-d86bf41a5ea9"). InnerVolumeSpecName "kube-api-access-kr6g9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:55:33.326011 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:33.325982 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61eb0c0e-3ff2-4878-9cd2-d86bf41a5ea9-tls-cert" pod "61eb0c0e-3ff2-4878-9cd2-d86bf41a5ea9" (UID: "61eb0c0e-3ff2-4878-9cd2-d86bf41a5ea9"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:55:33.326153 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:33.325985 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c6f0eb9-7433-47ef-80ce-75e59b7a5d38-kube-api-access-pxphq" pod "2c6f0eb9-7433-47ef-80ce-75e59b7a5d38" (UID: "2c6f0eb9-7433-47ef-80ce-75e59b7a5d38"). InnerVolumeSpecName "kube-api-access-pxphq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:55:33.424565 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:33.424529 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5fc165af-809c-4990-9d62-d9f68d4bcbb2-tls-cert\") pod \"authorino-5764994858-wwf9h\" (UID: \"5fc165af-809c-4990-9d62-d9f68d4bcbb2\") " pod="kuadrant-system/authorino-5764994858-wwf9h" Oct 30 16:55:33.424565 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:33.424563 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49ssb\" (UniqueName: \"kubernetes.io/projected/5fc165af-809c-4990-9d62-d9f68d4bcbb2-kube-api-access-49ssb\") pod \"authorino-5764994858-wwf9h\" (UID: \"5fc165af-809c-4990-9d62-d9f68d4bcbb2\") " pod="kuadrant-system/authorino-5764994858-wwf9h" Oct 30 16:55:33.424738 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:33.424635 4522 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/61eb0c0e-3ff2-4878-9cd2-d86bf41a5ea9-tls-cert\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 16:55:33.424738 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:33.424649 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr6g9\" (UniqueName: \"kubernetes.io/projected/61eb0c0e-3ff2-4878-9cd2-d86bf41a5ea9-kube-api-access-kr6g9\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 16:55:33.424738 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:33.424662 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxphq\" (UniqueName: \"kubernetes.io/projected/2c6f0eb9-7433-47ef-80ce-75e59b7a5d38-kube-api-access-pxphq\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 16:55:33.426829 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:33.426807 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5fc165af-809c-4990-9d62-d9f68d4bcbb2-tls-cert\") pod \"authorino-5764994858-wwf9h\" (UID: \"5fc165af-809c-4990-9d62-d9f68d4bcbb2\") " pod="kuadrant-system/authorino-5764994858-wwf9h" Oct 30 16:55:33.450045 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:33.450018 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49ssb\" (UniqueName: \"kubernetes.io/projected/5fc165af-809c-4990-9d62-d9f68d4bcbb2-kube-api-access-49ssb\") pod \"authorino-5764994858-wwf9h\" (UID: \"5fc165af-809c-4990-9d62-d9f68d4bcbb2\") " pod="kuadrant-system/authorino-5764994858-wwf9h" Oct 30 16:55:33.473106 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:33.473087 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5764994858-wwf9h" Oct 30 16:55:33.767401 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:33.767375 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5764994858-wwf9h"] Oct 30 16:55:33.767993 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:55:33.767966 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5fc165af_809c_4990_9d62_d9f68d4bcbb2.slice/crio-53bd14e8b542ec8206f00fc5bd17c8a9e2af211ddbfe45d3ac512b571981c18e WatchSource:0}: Error finding container 53bd14e8b542ec8206f00fc5bd17c8a9e2af211ddbfe45d3ac512b571981c18e: Status 404 returned error can't find the container with id 53bd14e8b542ec8206f00fc5bd17c8a9e2af211ddbfe45d3ac512b571981c18e Oct 30 16:55:34.152634 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:34.152591 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5764994858-wwf9h" event={"ID":"5fc165af-809c-4990-9d62-d9f68d4bcbb2","Type":"ContainerStarted","Data":"53bd14e8b542ec8206f00fc5bd17c8a9e2af211ddbfe45d3ac512b571981c18e"} Oct 30 16:55:34.152634 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:34.152613 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-h6dk2" Oct 30 16:55:34.152791 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:34.152637 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-v7xdt" Oct 30 16:55:34.251394 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:34.251365 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-h6dk2"] Oct 30 16:55:34.263666 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:34.263641 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-h6dk2"] Oct 30 16:55:34.317560 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:34.317528 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-v7xdt"] Oct 30 16:55:34.322986 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:34.322961 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-v7xdt"] Oct 30 16:55:35.158476 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:35.158442 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5764994858-wwf9h" event={"ID":"5fc165af-809c-4990-9d62-d9f68d4bcbb2","Type":"ContainerStarted","Data":"2a54c8a153d8759a42cc899f795b9ec4bdb46b0cca4c004a3b979e8def27c0d2"} Oct 30 16:55:35.207787 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:35.207737 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5764994858-wwf9h" podStartSLOduration=1.745883283 podStartE2EDuration="2.207722228s" totalImagesPullingTime="461.838945ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:55:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:55:35.206363429 +0000 UTC m=+509.207118347" watchObservedRunningTime="2026-10-30 16:55:35.207722228 +0000 UTC m=+509.208477128" Oct 30 16:55:35.246337 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:35.246292 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-d5r7h"] Oct 30 16:55:35.246536 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:35.246512 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-d5r7h" podUID="908f7490-017a-4868-8803-552cf0ee72f7" containerName="authorino" containerID="cri-o://449e47fd257426152ced755bb47a9032c3e5492779e0682d494def214b6e34f6" gracePeriod=30 Oct 30 16:55:35.499114 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:35.499092 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-d5r7h" Oct 30 16:55:35.642767 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:35.642693 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/908f7490-017a-4868-8803-552cf0ee72f7-kube-api-access-xlttd\" (UniqueName: \"kubernetes.io/projected/908f7490-017a-4868-8803-552cf0ee72f7-kube-api-access-xlttd\") pod \"908f7490-017a-4868-8803-552cf0ee72f7\" (UID: \"908f7490-017a-4868-8803-552cf0ee72f7\") " Oct 30 16:55:35.644807 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:35.644776 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/908f7490-017a-4868-8803-552cf0ee72f7-kube-api-access-xlttd" pod "908f7490-017a-4868-8803-552cf0ee72f7" (UID: "908f7490-017a-4868-8803-552cf0ee72f7"). InnerVolumeSpecName "kube-api-access-xlttd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:55:35.743517 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:35.743474 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlttd\" (UniqueName: \"kubernetes.io/projected/908f7490-017a-4868-8803-552cf0ee72f7-kube-api-access-xlttd\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 16:55:36.142207 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:36.142137 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c6f0eb9-7433-47ef-80ce-75e59b7a5d38" path="/var/lib/kubelet/pods/2c6f0eb9-7433-47ef-80ce-75e59b7a5d38/volumes" Oct 30 16:55:36.142407 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:36.142395 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61eb0c0e-3ff2-4878-9cd2-d86bf41a5ea9" path="/var/lib/kubelet/pods/61eb0c0e-3ff2-4878-9cd2-d86bf41a5ea9/volumes" Oct 30 16:55:36.163320 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:36.163277 4522 generic.go:356] "Generic (PLEG): container finished" podID="908f7490-017a-4868-8803-552cf0ee72f7" containerID="449e47fd257426152ced755bb47a9032c3e5492779e0682d494def214b6e34f6" exitCode=0 Oct 30 16:55:36.163651 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:36.163340 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-d5r7h" Oct 30 16:55:36.163651 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:36.163337 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-d5r7h" event={"ID":"908f7490-017a-4868-8803-552cf0ee72f7","Type":"ContainerDied","Data":"449e47fd257426152ced755bb47a9032c3e5492779e0682d494def214b6e34f6"} Oct 30 16:55:36.163651 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:36.163442 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-d5r7h" event={"ID":"908f7490-017a-4868-8803-552cf0ee72f7","Type":"ContainerDied","Data":"35992dbb8349d7258f2411868f734c043506143319102b90ed89bf99204ef8f3"} Oct 30 16:55:36.163651 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:36.163463 4522 scope.go:122] "RemoveContainer" containerID="449e47fd257426152ced755bb47a9032c3e5492779e0682d494def214b6e34f6" Oct 30 16:55:36.171352 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:36.171334 4522 scope.go:122] "RemoveContainer" containerID="449e47fd257426152ced755bb47a9032c3e5492779e0682d494def214b6e34f6" Oct 30 16:55:36.171604 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:55:36.171584 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"449e47fd257426152ced755bb47a9032c3e5492779e0682d494def214b6e34f6\": container with ID starting with 449e47fd257426152ced755bb47a9032c3e5492779e0682d494def214b6e34f6 not found: ID does not exist" containerID="449e47fd257426152ced755bb47a9032c3e5492779e0682d494def214b6e34f6" Oct 30 16:55:36.171655 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:36.171611 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"449e47fd257426152ced755bb47a9032c3e5492779e0682d494def214b6e34f6"} err="failed to get container status \"449e47fd257426152ced755bb47a9032c3e5492779e0682d494def214b6e34f6\": rpc error: code = NotFound desc = could not find container \"449e47fd257426152ced755bb47a9032c3e5492779e0682d494def214b6e34f6\": container with ID starting with 449e47fd257426152ced755bb47a9032c3e5492779e0682d494def214b6e34f6 not found: ID does not exist" Oct 30 16:55:36.225116 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:36.225083 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-d5r7h"] Oct 30 16:55:36.227930 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:36.227906 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-d5r7h"] Oct 30 16:55:38.143017 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:55:38.142978 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="908f7490-017a-4868-8803-552cf0ee72f7" path="/var/lib/kubelet/pods/908f7490-017a-4868-8803-552cf0ee72f7/volumes" Oct 30 16:55:42.949574 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:55:42.949536 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 16:56:12.949064 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:56:12.949021 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 16:56:42.946043 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:56:42.946003 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 16:57:12.946820 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:57:12.946782 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 16:57:23.165279 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:23.165216 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-9452k"] Oct 30 16:57:23.165925 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:23.165909 4522 cpu_manager.go:414] "containerMap: removing container" podUID="908f7490-017a-4868-8803-552cf0ee72f7" containerName="authorino" Oct 30 16:57:23.165972 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:23.165926 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="908f7490-017a-4868-8803-552cf0ee72f7" containerName="authorino" Oct 30 16:57:23.166008 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:23.165995 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="908f7490-017a-4868-8803-552cf0ee72f7" containerName="authorino" Oct 30 16:57:23.168846 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:23.168828 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-9452k" Oct 30 16:57:23.187532 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:23.187508 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-9452k"] Oct 30 16:57:23.199735 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:23.199713 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 16:57:23.199858 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:23.199773 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 16:57:23.199917 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:23.199870 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-45tpm\"" Oct 30 16:57:23.199917 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:23.199900 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Oct 30 16:57:23.238680 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:23.238652 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99c2ec18-1d5d-4bb1-99ac-f16a0f753a2e-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-9452k\" (UID: \"99c2ec18-1d5d-4bb1-99ac-f16a0f753a2e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-9452k" Oct 30 16:57:23.238976 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:23.238954 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99c2ec18-1d5d-4bb1-99ac-f16a0f753a2e-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-9452k\" (UID: \"99c2ec18-1d5d-4bb1-99ac-f16a0f753a2e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-9452k" Oct 30 16:57:23.239122 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:23.239100 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vlr6\" (UniqueName: \"kubernetes.io/projected/99c2ec18-1d5d-4bb1-99ac-f16a0f753a2e-kube-api-access-6vlr6\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-9452k\" (UID: \"99c2ec18-1d5d-4bb1-99ac-f16a0f753a2e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-9452k" Oct 30 16:57:23.239248 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:23.239222 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99c2ec18-1d5d-4bb1-99ac-f16a0f753a2e-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-9452k\" (UID: \"99c2ec18-1d5d-4bb1-99ac-f16a0f753a2e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-9452k" Oct 30 16:57:23.239328 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:23.239296 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99c2ec18-1d5d-4bb1-99ac-f16a0f753a2e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-9452k\" (UID: \"99c2ec18-1d5d-4bb1-99ac-f16a0f753a2e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-9452k" Oct 30 16:57:23.239385 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:23.239350 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99c2ec18-1d5d-4bb1-99ac-f16a0f753a2e-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-9452k\" (UID: \"99c2ec18-1d5d-4bb1-99ac-f16a0f753a2e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-9452k" Oct 30 16:57:23.340609 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:23.340580 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99c2ec18-1d5d-4bb1-99ac-f16a0f753a2e-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-9452k\" (UID: \"99c2ec18-1d5d-4bb1-99ac-f16a0f753a2e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-9452k" Oct 30 16:57:23.340775 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:23.340627 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99c2ec18-1d5d-4bb1-99ac-f16a0f753a2e-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-9452k\" (UID: \"99c2ec18-1d5d-4bb1-99ac-f16a0f753a2e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-9452k" Oct 30 16:57:23.340775 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:23.340751 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99c2ec18-1d5d-4bb1-99ac-f16a0f753a2e-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-9452k\" (UID: \"99c2ec18-1d5d-4bb1-99ac-f16a0f753a2e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-9452k" Oct 30 16:57:23.340866 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:23.340837 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vlr6\" (UniqueName: \"kubernetes.io/projected/99c2ec18-1d5d-4bb1-99ac-f16a0f753a2e-kube-api-access-6vlr6\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-9452k\" (UID: \"99c2ec18-1d5d-4bb1-99ac-f16a0f753a2e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-9452k" Oct 30 16:57:23.340910 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:23.340886 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99c2ec18-1d5d-4bb1-99ac-f16a0f753a2e-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-9452k\" (UID: \"99c2ec18-1d5d-4bb1-99ac-f16a0f753a2e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-9452k" Oct 30 16:57:23.340961 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:23.340920 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99c2ec18-1d5d-4bb1-99ac-f16a0f753a2e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-9452k\" (UID: \"99c2ec18-1d5d-4bb1-99ac-f16a0f753a2e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-9452k" Oct 30 16:57:23.340961 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:23.340935 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99c2ec18-1d5d-4bb1-99ac-f16a0f753a2e-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-9452k\" (UID: \"99c2ec18-1d5d-4bb1-99ac-f16a0f753a2e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-9452k" Oct 30 16:57:23.341180 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:23.341151 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99c2ec18-1d5d-4bb1-99ac-f16a0f753a2e-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-9452k\" (UID: \"99c2ec18-1d5d-4bb1-99ac-f16a0f753a2e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-9452k" Oct 30 16:57:23.341247 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:23.341226 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99c2ec18-1d5d-4bb1-99ac-f16a0f753a2e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-9452k\" (UID: \"99c2ec18-1d5d-4bb1-99ac-f16a0f753a2e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-9452k" Oct 30 16:57:23.342805 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:23.342779 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99c2ec18-1d5d-4bb1-99ac-f16a0f753a2e-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-9452k\" (UID: \"99c2ec18-1d5d-4bb1-99ac-f16a0f753a2e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-9452k" Oct 30 16:57:23.343164 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:23.343146 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99c2ec18-1d5d-4bb1-99ac-f16a0f753a2e-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-9452k\" (UID: \"99c2ec18-1d5d-4bb1-99ac-f16a0f753a2e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-9452k" Oct 30 16:57:23.365958 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:23.365928 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vlr6\" (UniqueName: \"kubernetes.io/projected/99c2ec18-1d5d-4bb1-99ac-f16a0f753a2e-kube-api-access-6vlr6\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-9452k\" (UID: \"99c2ec18-1d5d-4bb1-99ac-f16a0f753a2e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-9452k" Oct 30 16:57:23.482085 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:23.482051 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-9452k" Oct 30 16:57:23.780233 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:57:23.780198 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99c2ec18_1d5d_4bb1_99ac_f16a0f753a2e.slice/crio-730ed373169db3ad9ca1ecca4909fd049910f53dcff4474897500a031a04387c WatchSource:0}: Error finding container 730ed373169db3ad9ca1ecca4909fd049910f53dcff4474897500a031a04387c: Status 404 returned error can't find the container with id 730ed373169db3ad9ca1ecca4909fd049910f53dcff4474897500a031a04387c Oct 30 16:57:23.780422 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:23.780390 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-9452k"] Oct 30 16:57:24.610226 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:24.610188 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-9452k" event={"ID":"99c2ec18-1d5d-4bb1-99ac-f16a0f753a2e","Type":"ContainerStarted","Data":"730ed373169db3ad9ca1ecca4909fd049910f53dcff4474897500a031a04387c"} Oct 30 16:57:25.616743 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:25.616706 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-9452k" event={"ID":"99c2ec18-1d5d-4bb1-99ac-f16a0f753a2e","Type":"ContainerStarted","Data":"8588587b9bab21455027eb6392cc7f7b22f4735abfcbaa6fba579893b053ca2a"} Oct 30 16:57:25.661873 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:25.661818 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-9452k" podStartSLOduration=1.551281672 podStartE2EDuration="2.661802821s" totalImagesPullingTime="1.110521149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:57:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:57:25.661280767 +0000 UTC m=+619.662035668" watchObservedRunningTime="2026-10-30 16:57:25.661802821 +0000 UTC m=+619.662557722" Oct 30 16:57:26.482646 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:26.482614 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-9452k" Oct 30 16:57:27.888469 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:27.888433 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-2xdvj"] Oct 30 16:57:27.901931 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:27.901896 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2xdvj" Oct 30 16:57:27.915142 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:27.915112 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-2xdvj"] Oct 30 16:57:27.926912 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:27.926889 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Oct 30 16:57:27.983732 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:27.983695 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce97dea1-392d-437d-bcb8-b1d73b3a7528-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2xdvj\" (UID: \"ce97dea1-392d-437d-bcb8-b1d73b3a7528\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2xdvj" Oct 30 16:57:27.983732 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:27.983732 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h2cq\" (UniqueName: \"kubernetes.io/projected/ce97dea1-392d-437d-bcb8-b1d73b3a7528-kube-api-access-8h2cq\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2xdvj\" (UID: \"ce97dea1-392d-437d-bcb8-b1d73b3a7528\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2xdvj" Oct 30 16:57:27.983949 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:27.983827 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce97dea1-392d-437d-bcb8-b1d73b3a7528-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2xdvj\" (UID: \"ce97dea1-392d-437d-bcb8-b1d73b3a7528\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2xdvj" Oct 30 16:57:27.983949 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:27.983887 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce97dea1-392d-437d-bcb8-b1d73b3a7528-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2xdvj\" (UID: \"ce97dea1-392d-437d-bcb8-b1d73b3a7528\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2xdvj" Oct 30 16:57:27.983949 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:27.983915 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce97dea1-392d-437d-bcb8-b1d73b3a7528-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2xdvj\" (UID: \"ce97dea1-392d-437d-bcb8-b1d73b3a7528\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2xdvj" Oct 30 16:57:27.984057 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:27.983950 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce97dea1-392d-437d-bcb8-b1d73b3a7528-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2xdvj\" (UID: \"ce97dea1-392d-437d-bcb8-b1d73b3a7528\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2xdvj" Oct 30 16:57:28.084919 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:28.084890 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce97dea1-392d-437d-bcb8-b1d73b3a7528-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2xdvj\" (UID: \"ce97dea1-392d-437d-bcb8-b1d73b3a7528\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2xdvj" Oct 30 16:57:28.085115 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:28.084939 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce97dea1-392d-437d-bcb8-b1d73b3a7528-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2xdvj\" (UID: \"ce97dea1-392d-437d-bcb8-b1d73b3a7528\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2xdvj" Oct 30 16:57:28.085115 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:28.084969 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce97dea1-392d-437d-bcb8-b1d73b3a7528-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2xdvj\" (UID: \"ce97dea1-392d-437d-bcb8-b1d73b3a7528\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2xdvj" Oct 30 16:57:28.085115 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:28.084986 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h2cq\" (UniqueName: \"kubernetes.io/projected/ce97dea1-392d-437d-bcb8-b1d73b3a7528-kube-api-access-8h2cq\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2xdvj\" (UID: \"ce97dea1-392d-437d-bcb8-b1d73b3a7528\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2xdvj" Oct 30 16:57:28.085115 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:28.085025 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce97dea1-392d-437d-bcb8-b1d73b3a7528-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2xdvj\" (UID: \"ce97dea1-392d-437d-bcb8-b1d73b3a7528\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2xdvj" Oct 30 16:57:28.085345 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:28.085146 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce97dea1-392d-437d-bcb8-b1d73b3a7528-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2xdvj\" (UID: \"ce97dea1-392d-437d-bcb8-b1d73b3a7528\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2xdvj" Oct 30 16:57:28.085408 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:28.085358 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce97dea1-392d-437d-bcb8-b1d73b3a7528-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2xdvj\" (UID: \"ce97dea1-392d-437d-bcb8-b1d73b3a7528\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2xdvj" Oct 30 16:57:28.085449 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:28.085410 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce97dea1-392d-437d-bcb8-b1d73b3a7528-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2xdvj\" (UID: \"ce97dea1-392d-437d-bcb8-b1d73b3a7528\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2xdvj" Oct 30 16:57:28.085504 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:28.085487 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce97dea1-392d-437d-bcb8-b1d73b3a7528-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2xdvj\" (UID: \"ce97dea1-392d-437d-bcb8-b1d73b3a7528\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2xdvj" Oct 30 16:57:28.087395 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:28.087371 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce97dea1-392d-437d-bcb8-b1d73b3a7528-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2xdvj\" (UID: \"ce97dea1-392d-437d-bcb8-b1d73b3a7528\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2xdvj" Oct 30 16:57:28.087574 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:28.087556 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce97dea1-392d-437d-bcb8-b1d73b3a7528-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2xdvj\" (UID: \"ce97dea1-392d-437d-bcb8-b1d73b3a7528\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2xdvj" Oct 30 16:57:28.109759 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:28.109730 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h2cq\" (UniqueName: \"kubernetes.io/projected/ce97dea1-392d-437d-bcb8-b1d73b3a7528-kube-api-access-8h2cq\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2xdvj\" (UID: \"ce97dea1-392d-437d-bcb8-b1d73b3a7528\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2xdvj" Oct 30 16:57:28.214751 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:28.214715 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2xdvj" Oct 30 16:57:28.514607 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:28.514581 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-2xdvj"] Oct 30 16:57:28.515860 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:57:28.515832 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce97dea1_392d_437d_bcb8_b1d73b3a7528.slice/crio-a73ca6ea7dab1f9ccafca29f8ed77b53444492c0957508ca614f1a40803476ad WatchSource:0}: Error finding container a73ca6ea7dab1f9ccafca29f8ed77b53444492c0957508ca614f1a40803476ad: Status 404 returned error can't find the container with id a73ca6ea7dab1f9ccafca29f8ed77b53444492c0957508ca614f1a40803476ad Oct 30 16:57:28.630649 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:28.630618 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2xdvj" event={"ID":"ce97dea1-392d-437d-bcb8-b1d73b3a7528","Type":"ContainerStarted","Data":"a73ca6ea7dab1f9ccafca29f8ed77b53444492c0957508ca614f1a40803476ad"} Oct 30 16:57:29.637442 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:29.637406 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2xdvj" event={"ID":"ce97dea1-392d-437d-bcb8-b1d73b3a7528","Type":"ContainerStarted","Data":"7bd756892efc31f1af1d221cf69e70f20e39328d3cf4cd9fd08c24de9760c481"} Oct 30 16:57:29.686364 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:29.686295 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2xdvj" podStartSLOduration=2.504461014 podStartE2EDuration="2.686281242s" totalImagesPullingTime="181.820228ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:57:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:57:29.685040899 +0000 UTC m=+623.685795810" watchObservedRunningTime="2026-10-30 16:57:29.686281242 +0000 UTC m=+623.687036162" Oct 30 16:57:30.215011 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:30.214984 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2xdvj" Oct 30 16:57:33.246062 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:33.246031 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zcm49"] Oct 30 16:57:33.249527 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:33.249508 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zcm49" Oct 30 16:57:33.272050 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:33.272020 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Oct 30 16:57:33.273681 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:33.273655 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zcm49"] Oct 30 16:57:33.337812 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:33.337771 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9116a9b-6aaf-4f77-8d89-8766d89b6048-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zcm49\" (UID: \"e9116a9b-6aaf-4f77-8d89-8766d89b6048\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zcm49" Oct 30 16:57:33.338005 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:33.337823 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9116a9b-6aaf-4f77-8d89-8766d89b6048-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zcm49\" (UID: \"e9116a9b-6aaf-4f77-8d89-8766d89b6048\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zcm49" Oct 30 16:57:33.338005 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:33.337937 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwdfg\" (UniqueName: \"kubernetes.io/projected/e9116a9b-6aaf-4f77-8d89-8766d89b6048-kube-api-access-jwdfg\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zcm49\" (UID: \"e9116a9b-6aaf-4f77-8d89-8766d89b6048\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zcm49" Oct 30 16:57:33.338106 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:33.337996 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9116a9b-6aaf-4f77-8d89-8766d89b6048-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zcm49\" (UID: \"e9116a9b-6aaf-4f77-8d89-8766d89b6048\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zcm49" Oct 30 16:57:33.338106 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:33.338023 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9116a9b-6aaf-4f77-8d89-8766d89b6048-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zcm49\" (UID: \"e9116a9b-6aaf-4f77-8d89-8766d89b6048\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zcm49" Oct 30 16:57:33.338210 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:33.338110 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9116a9b-6aaf-4f77-8d89-8766d89b6048-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zcm49\" (UID: \"e9116a9b-6aaf-4f77-8d89-8766d89b6048\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zcm49" Oct 30 16:57:33.438637 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:33.438601 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9116a9b-6aaf-4f77-8d89-8766d89b6048-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zcm49\" (UID: \"e9116a9b-6aaf-4f77-8d89-8766d89b6048\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zcm49" Oct 30 16:57:33.438783 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:33.438658 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9116a9b-6aaf-4f77-8d89-8766d89b6048-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zcm49\" (UID: \"e9116a9b-6aaf-4f77-8d89-8766d89b6048\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zcm49" Oct 30 16:57:33.438783 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:33.438702 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwdfg\" (UniqueName: \"kubernetes.io/projected/e9116a9b-6aaf-4f77-8d89-8766d89b6048-kube-api-access-jwdfg\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zcm49\" (UID: \"e9116a9b-6aaf-4f77-8d89-8766d89b6048\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zcm49" Oct 30 16:57:33.438783 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:33.438737 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9116a9b-6aaf-4f77-8d89-8766d89b6048-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zcm49\" (UID: \"e9116a9b-6aaf-4f77-8d89-8766d89b6048\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zcm49" Oct 30 16:57:33.438783 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:33.438759 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9116a9b-6aaf-4f77-8d89-8766d89b6048-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zcm49\" (UID: \"e9116a9b-6aaf-4f77-8d89-8766d89b6048\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zcm49" Oct 30 16:57:33.438955 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:33.438821 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9116a9b-6aaf-4f77-8d89-8766d89b6048-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zcm49\" (UID: \"e9116a9b-6aaf-4f77-8d89-8766d89b6048\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zcm49" Oct 30 16:57:33.439075 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:33.439052 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9116a9b-6aaf-4f77-8d89-8766d89b6048-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zcm49\" (UID: \"e9116a9b-6aaf-4f77-8d89-8766d89b6048\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zcm49" Oct 30 16:57:33.439127 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:33.439104 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9116a9b-6aaf-4f77-8d89-8766d89b6048-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zcm49\" (UID: \"e9116a9b-6aaf-4f77-8d89-8766d89b6048\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zcm49" Oct 30 16:57:33.439167 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:33.439129 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9116a9b-6aaf-4f77-8d89-8766d89b6048-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zcm49\" (UID: \"e9116a9b-6aaf-4f77-8d89-8766d89b6048\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zcm49" Oct 30 16:57:33.441025 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:33.441002 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9116a9b-6aaf-4f77-8d89-8766d89b6048-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zcm49\" (UID: \"e9116a9b-6aaf-4f77-8d89-8766d89b6048\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zcm49" Oct 30 16:57:33.441430 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:33.441408 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9116a9b-6aaf-4f77-8d89-8766d89b6048-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zcm49\" (UID: \"e9116a9b-6aaf-4f77-8d89-8766d89b6048\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zcm49" Oct 30 16:57:33.463506 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:33.463479 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwdfg\" (UniqueName: \"kubernetes.io/projected/e9116a9b-6aaf-4f77-8d89-8766d89b6048-kube-api-access-jwdfg\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zcm49\" (UID: \"e9116a9b-6aaf-4f77-8d89-8766d89b6048\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zcm49" Oct 30 16:57:33.482247 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:33.482223 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-9452k" Oct 30 16:57:33.494565 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:33.494543 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-9452k" Oct 30 16:57:33.564157 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:33.564060 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zcm49" Oct 30 16:57:33.666690 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:33.666661 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-9452k" Oct 30 16:57:33.865271 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:33.865244 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zcm49"] Oct 30 16:57:33.865677 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:57:33.865654 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9116a9b_6aaf_4f77_8d89_8766d89b6048.slice/crio-3d358f84bdfbca5f7cf1b9030704d22c86da4a9e13b8fc76121c8f289810128f WatchSource:0}: Error finding container 3d358f84bdfbca5f7cf1b9030704d22c86da4a9e13b8fc76121c8f289810128f: Status 404 returned error can't find the container with id 3d358f84bdfbca5f7cf1b9030704d22c86da4a9e13b8fc76121c8f289810128f Oct 30 16:57:34.661005 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:34.660963 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zcm49" event={"ID":"e9116a9b-6aaf-4f77-8d89-8766d89b6048","Type":"ContainerStarted","Data":"3fdb504f7453b9f5aaf63e91a6e0513ffde0a0b76d73b257cb74a8747021f5e4"} Oct 30 16:57:34.661460 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:34.661034 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zcm49" event={"ID":"e9116a9b-6aaf-4f77-8d89-8766d89b6048","Type":"ContainerStarted","Data":"3d358f84bdfbca5f7cf1b9030704d22c86da4a9e13b8fc76121c8f289810128f"} Oct 30 16:57:34.708539 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:34.708486 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zcm49" podStartSLOduration=1.525633906 podStartE2EDuration="1.708464961s" totalImagesPullingTime="182.831055ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:57:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:57:34.707464458 +0000 UTC m=+628.708219372" watchObservedRunningTime="2026-10-30 16:57:34.708464961 +0000 UTC m=+628.709219861" Oct 30 16:57:35.284074 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:35.284040 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5"] Oct 30 16:57:35.293688 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:35.293660 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" Oct 30 16:57:35.309097 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:35.309065 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5"] Oct 30 16:57:35.321077 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:35.321042 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 30 16:57:35.357167 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:35.357131 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4360e9dc-c001-4102-8af6-b55b692fcf2c-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5\" (UID: \"4360e9dc-c001-4102-8af6-b55b692fcf2c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" Oct 30 16:57:35.357375 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:35.357191 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4360e9dc-c001-4102-8af6-b55b692fcf2c-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5\" (UID: \"4360e9dc-c001-4102-8af6-b55b692fcf2c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" Oct 30 16:57:35.357375 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:35.357300 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4360e9dc-c001-4102-8af6-b55b692fcf2c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5\" (UID: \"4360e9dc-c001-4102-8af6-b55b692fcf2c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" Oct 30 16:57:35.357375 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:35.357352 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4tmv\" (UniqueName: \"kubernetes.io/projected/4360e9dc-c001-4102-8af6-b55b692fcf2c-kube-api-access-c4tmv\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5\" (UID: \"4360e9dc-c001-4102-8af6-b55b692fcf2c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" Oct 30 16:57:35.357509 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:35.357386 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4360e9dc-c001-4102-8af6-b55b692fcf2c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5\" (UID: \"4360e9dc-c001-4102-8af6-b55b692fcf2c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" Oct 30 16:57:35.357509 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:35.357416 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4360e9dc-c001-4102-8af6-b55b692fcf2c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5\" (UID: \"4360e9dc-c001-4102-8af6-b55b692fcf2c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" Oct 30 16:57:35.458886 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:35.458852 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4360e9dc-c001-4102-8af6-b55b692fcf2c-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5\" (UID: \"4360e9dc-c001-4102-8af6-b55b692fcf2c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" Oct 30 16:57:35.459097 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:35.458905 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4360e9dc-c001-4102-8af6-b55b692fcf2c-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5\" (UID: \"4360e9dc-c001-4102-8af6-b55b692fcf2c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" Oct 30 16:57:35.459097 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:35.458933 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4360e9dc-c001-4102-8af6-b55b692fcf2c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5\" (UID: \"4360e9dc-c001-4102-8af6-b55b692fcf2c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" Oct 30 16:57:35.459097 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:35.458951 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4tmv\" (UniqueName: \"kubernetes.io/projected/4360e9dc-c001-4102-8af6-b55b692fcf2c-kube-api-access-c4tmv\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5\" (UID: \"4360e9dc-c001-4102-8af6-b55b692fcf2c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" Oct 30 16:57:35.459097 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:35.458974 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4360e9dc-c001-4102-8af6-b55b692fcf2c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5\" (UID: \"4360e9dc-c001-4102-8af6-b55b692fcf2c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" Oct 30 16:57:35.459097 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:35.458989 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4360e9dc-c001-4102-8af6-b55b692fcf2c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5\" (UID: \"4360e9dc-c001-4102-8af6-b55b692fcf2c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" Oct 30 16:57:35.459459 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:35.459412 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4360e9dc-c001-4102-8af6-b55b692fcf2c-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5\" (UID: \"4360e9dc-c001-4102-8af6-b55b692fcf2c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" Oct 30 16:57:35.459593 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:35.459473 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4360e9dc-c001-4102-8af6-b55b692fcf2c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5\" (UID: \"4360e9dc-c001-4102-8af6-b55b692fcf2c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" Oct 30 16:57:35.459653 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:35.459608 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4360e9dc-c001-4102-8af6-b55b692fcf2c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5\" (UID: \"4360e9dc-c001-4102-8af6-b55b692fcf2c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" Oct 30 16:57:35.461246 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:35.461218 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4360e9dc-c001-4102-8af6-b55b692fcf2c-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5\" (UID: \"4360e9dc-c001-4102-8af6-b55b692fcf2c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" Oct 30 16:57:35.461837 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:35.461816 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4360e9dc-c001-4102-8af6-b55b692fcf2c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5\" (UID: \"4360e9dc-c001-4102-8af6-b55b692fcf2c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" Oct 30 16:57:35.491161 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:35.491131 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4tmv\" (UniqueName: \"kubernetes.io/projected/4360e9dc-c001-4102-8af6-b55b692fcf2c-kube-api-access-c4tmv\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5\" (UID: \"4360e9dc-c001-4102-8af6-b55b692fcf2c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" Oct 30 16:57:35.564794 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:35.564716 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zcm49" Oct 30 16:57:35.609511 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:35.609468 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" Oct 30 16:57:35.911017 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:35.910992 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5"] Oct 30 16:57:35.913026 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:57:35.912995 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4360e9dc_c001_4102_8af6_b55b692fcf2c.slice/crio-5144427c0cca87c4afcf0f888d65da3e9ece1541f2751b2e802376d88004c586 WatchSource:0}: Error finding container 5144427c0cca87c4afcf0f888d65da3e9ece1541f2751b2e802376d88004c586: Status 404 returned error can't find the container with id 5144427c0cca87c4afcf0f888d65da3e9ece1541f2751b2e802376d88004c586 Oct 30 16:57:36.671706 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:36.671671 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" event={"ID":"4360e9dc-c001-4102-8af6-b55b692fcf2c","Type":"ContainerStarted","Data":"701fd474a56316c48a71e9274dcd5b84caa98ca2dfbda607a493605b4eeaf78e"} Oct 30 16:57:36.671706 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:36.671712 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" event={"ID":"4360e9dc-c001-4102-8af6-b55b692fcf2c","Type":"ContainerStarted","Data":"5144427c0cca87c4afcf0f888d65da3e9ece1541f2751b2e802376d88004c586"} Oct 30 16:57:36.716661 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:36.716614 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" podStartSLOduration=1.516375155 podStartE2EDuration="1.716600605s" totalImagesPullingTime="200.22545ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:57:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:57:36.716161885 +0000 UTC m=+630.716916796" watchObservedRunningTime="2026-10-30 16:57:36.716600605 +0000 UTC m=+630.717355504" Oct 30 16:57:37.609802 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:37.609760 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" Oct 30 16:57:38.215656 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:38.215608 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2xdvj" Oct 30 16:57:38.228367 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:38.228340 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2xdvj" Oct 30 16:57:38.691380 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:38.691294 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2xdvj" Oct 30 16:57:42.946098 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:57:42.946060 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 16:57:43.564648 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:43.564607 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zcm49" Oct 30 16:57:43.577083 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:43.577050 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zcm49" Oct 30 16:57:43.712278 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:43.712250 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zcm49" Oct 30 16:57:45.610575 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:45.610537 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" Oct 30 16:57:45.622680 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:45.622651 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" Oct 30 16:57:45.721449 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:45.721420 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" Oct 30 16:57:48.128387 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:48.128353 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-245xx"] Oct 30 16:57:48.131955 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:48.131935 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-245xx" Oct 30 16:57:48.148448 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:48.148424 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 16:57:48.163736 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:48.163690 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-245xx"] Oct 30 16:57:48.235488 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:48.235462 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-fcls9\"" Oct 30 16:57:48.276938 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:48.276906 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb8adf71-af34-4715-ada0-42494da2219a-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-245xx\" (UID: \"fb8adf71-af34-4715-ada0-42494da2219a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-245xx" Oct 30 16:57:48.277073 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:48.276944 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb8adf71-af34-4715-ada0-42494da2219a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-245xx\" (UID: \"fb8adf71-af34-4715-ada0-42494da2219a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-245xx" Oct 30 16:57:48.277073 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:48.276967 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb8adf71-af34-4715-ada0-42494da2219a-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-245xx\" (UID: \"fb8adf71-af34-4715-ada0-42494da2219a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-245xx" Oct 30 16:57:48.277162 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:48.277068 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb8adf71-af34-4715-ada0-42494da2219a-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-245xx\" (UID: \"fb8adf71-af34-4715-ada0-42494da2219a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-245xx" Oct 30 16:57:48.277162 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:48.277092 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb8adf71-af34-4715-ada0-42494da2219a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-245xx\" (UID: \"fb8adf71-af34-4715-ada0-42494da2219a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-245xx" Oct 30 16:57:48.277162 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:48.277136 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb8adf71-af34-4715-ada0-42494da2219a-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-245xx\" (UID: \"fb8adf71-af34-4715-ada0-42494da2219a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-245xx" Oct 30 16:57:48.277296 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:48.277220 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb8adf71-af34-4715-ada0-42494da2219a-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-245xx\" (UID: \"fb8adf71-af34-4715-ada0-42494da2219a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-245xx" Oct 30 16:57:48.277296 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:48.277259 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb8adf71-af34-4715-ada0-42494da2219a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-245xx\" (UID: \"fb8adf71-af34-4715-ada0-42494da2219a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-245xx" Oct 30 16:57:48.277296 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:48.277281 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mptn6\" (UniqueName: \"kubernetes.io/projected/fb8adf71-af34-4715-ada0-42494da2219a-kube-api-access-mptn6\") pod \"maas-default-gateway-openshift-default-6db586dd6c-245xx\" (UID: \"fb8adf71-af34-4715-ada0-42494da2219a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-245xx" Oct 30 16:57:48.378258 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:48.378224 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb8adf71-af34-4715-ada0-42494da2219a-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-245xx\" (UID: \"fb8adf71-af34-4715-ada0-42494da2219a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-245xx" Oct 30 16:57:48.378258 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:48.378259 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb8adf71-af34-4715-ada0-42494da2219a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-245xx\" (UID: \"fb8adf71-af34-4715-ada0-42494da2219a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-245xx" Oct 30 16:57:48.378546 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:48.378276 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb8adf71-af34-4715-ada0-42494da2219a-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-245xx\" (UID: \"fb8adf71-af34-4715-ada0-42494da2219a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-245xx" Oct 30 16:57:48.378546 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:48.378327 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb8adf71-af34-4715-ada0-42494da2219a-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-245xx\" (UID: \"fb8adf71-af34-4715-ada0-42494da2219a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-245xx" Oct 30 16:57:48.378546 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:48.378445 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb8adf71-af34-4715-ada0-42494da2219a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-245xx\" (UID: \"fb8adf71-af34-4715-ada0-42494da2219a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-245xx" Oct 30 16:57:48.378723 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:48.378528 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb8adf71-af34-4715-ada0-42494da2219a-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-245xx\" (UID: \"fb8adf71-af34-4715-ada0-42494da2219a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-245xx" Oct 30 16:57:48.378723 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:48.378580 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb8adf71-af34-4715-ada0-42494da2219a-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-245xx\" (UID: \"fb8adf71-af34-4715-ada0-42494da2219a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-245xx" Oct 30 16:57:48.378723 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:48.378664 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb8adf71-af34-4715-ada0-42494da2219a-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-245xx\" (UID: \"fb8adf71-af34-4715-ada0-42494da2219a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-245xx" Oct 30 16:57:48.378866 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:48.378715 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb8adf71-af34-4715-ada0-42494da2219a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-245xx\" (UID: \"fb8adf71-af34-4715-ada0-42494da2219a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-245xx" Oct 30 16:57:48.378866 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:48.378727 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb8adf71-af34-4715-ada0-42494da2219a-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-245xx\" (UID: \"fb8adf71-af34-4715-ada0-42494da2219a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-245xx" Oct 30 16:57:48.378866 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:48.378767 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mptn6\" (UniqueName: \"kubernetes.io/projected/fb8adf71-af34-4715-ada0-42494da2219a-kube-api-access-mptn6\") pod \"maas-default-gateway-openshift-default-6db586dd6c-245xx\" (UID: \"fb8adf71-af34-4715-ada0-42494da2219a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-245xx" Oct 30 16:57:48.379024 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:48.378893 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb8adf71-af34-4715-ada0-42494da2219a-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-245xx\" (UID: \"fb8adf71-af34-4715-ada0-42494da2219a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-245xx" Oct 30 16:57:48.379024 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:48.378933 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb8adf71-af34-4715-ada0-42494da2219a-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-245xx\" (UID: \"fb8adf71-af34-4715-ada0-42494da2219a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-245xx" Oct 30 16:57:48.379290 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:48.379267 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb8adf71-af34-4715-ada0-42494da2219a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-245xx\" (UID: \"fb8adf71-af34-4715-ada0-42494da2219a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-245xx" Oct 30 16:57:48.380963 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:48.380943 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb8adf71-af34-4715-ada0-42494da2219a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-245xx\" (UID: \"fb8adf71-af34-4715-ada0-42494da2219a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-245xx" Oct 30 16:57:48.380963 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:48.380959 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb8adf71-af34-4715-ada0-42494da2219a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-245xx\" (UID: \"fb8adf71-af34-4715-ada0-42494da2219a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-245xx" Oct 30 16:57:48.403464 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:48.403437 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb8adf71-af34-4715-ada0-42494da2219a-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-245xx\" (UID: \"fb8adf71-af34-4715-ada0-42494da2219a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-245xx" Oct 30 16:57:48.403573 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:48.403562 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mptn6\" (UniqueName: \"kubernetes.io/projected/fb8adf71-af34-4715-ada0-42494da2219a-kube-api-access-mptn6\") pod \"maas-default-gateway-openshift-default-6db586dd6c-245xx\" (UID: \"fb8adf71-af34-4715-ada0-42494da2219a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-245xx" Oct 30 16:57:48.446441 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:48.446404 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-245xx" Oct 30 16:57:48.774178 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:57:48.774145 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb8adf71_af34_4715_ada0_42494da2219a.slice/crio-da8dc602476e2369454816607333811c1e519acc9894752834900105ebb5b156 WatchSource:0}: Error finding container da8dc602476e2369454816607333811c1e519acc9894752834900105ebb5b156: Status 404 returned error can't find the container with id da8dc602476e2369454816607333811c1e519acc9894752834900105ebb5b156 Oct 30 16:57:48.774456 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:48.774439 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-245xx"] Oct 30 16:57:49.731245 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:49.731210 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-245xx" event={"ID":"fb8adf71-af34-4715-ada0-42494da2219a","Type":"ContainerStarted","Data":"da8dc602476e2369454816607333811c1e519acc9894752834900105ebb5b156"} Oct 30 16:57:50.851859 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:50.851819 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-648b5d97c7-78fx4"] Oct 30 16:57:50.858692 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:50.858665 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-648b5d97c7-78fx4" Oct 30 16:57:50.875344 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:50.875300 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-648b5d97c7-78fx4"] Oct 30 16:57:51.002646 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:51.002609 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8f03ee2a-146e-4c54-b83a-35637ceda4bb-tls-cert\") pod \"authorino-648b5d97c7-78fx4\" (UID: \"8f03ee2a-146e-4c54-b83a-35637ceda4bb\") " pod="kuadrant-system/authorino-648b5d97c7-78fx4" Oct 30 16:57:51.002815 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:51.002665 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb8mc\" (UniqueName: \"kubernetes.io/projected/8f03ee2a-146e-4c54-b83a-35637ceda4bb-kube-api-access-hb8mc\") pod \"authorino-648b5d97c7-78fx4\" (UID: \"8f03ee2a-146e-4c54-b83a-35637ceda4bb\") " pod="kuadrant-system/authorino-648b5d97c7-78fx4" Oct 30 16:57:51.103520 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:51.103430 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb8mc\" (UniqueName: \"kubernetes.io/projected/8f03ee2a-146e-4c54-b83a-35637ceda4bb-kube-api-access-hb8mc\") pod \"authorino-648b5d97c7-78fx4\" (UID: \"8f03ee2a-146e-4c54-b83a-35637ceda4bb\") " pod="kuadrant-system/authorino-648b5d97c7-78fx4" Oct 30 16:57:51.103697 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:51.103561 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8f03ee2a-146e-4c54-b83a-35637ceda4bb-tls-cert\") pod \"authorino-648b5d97c7-78fx4\" (UID: \"8f03ee2a-146e-4c54-b83a-35637ceda4bb\") " pod="kuadrant-system/authorino-648b5d97c7-78fx4" Oct 30 16:57:51.106248 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:51.106218 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8f03ee2a-146e-4c54-b83a-35637ceda4bb-tls-cert\") pod \"authorino-648b5d97c7-78fx4\" (UID: \"8f03ee2a-146e-4c54-b83a-35637ceda4bb\") " pod="kuadrant-system/authorino-648b5d97c7-78fx4" Oct 30 16:57:51.129245 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:51.129216 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb8mc\" (UniqueName: \"kubernetes.io/projected/8f03ee2a-146e-4c54-b83a-35637ceda4bb-kube-api-access-hb8mc\") pod \"authorino-648b5d97c7-78fx4\" (UID: \"8f03ee2a-146e-4c54-b83a-35637ceda4bb\") " pod="kuadrant-system/authorino-648b5d97c7-78fx4" Oct 30 16:57:51.172150 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:51.172113 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-648b5d97c7-78fx4" Oct 30 16:57:51.476046 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:51.476020 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-648b5d97c7-78fx4"] Oct 30 16:57:51.476155 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:57:51.476127 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f03ee2a_146e_4c54_b83a_35637ceda4bb.slice/crio-821f1168e2f3f6189b3c3003fa113cd35502483838c926515aaa8d5e69d2a27e WatchSource:0}: Error finding container 821f1168e2f3f6189b3c3003fa113cd35502483838c926515aaa8d5e69d2a27e: Status 404 returned error can't find the container with id 821f1168e2f3f6189b3c3003fa113cd35502483838c926515aaa8d5e69d2a27e Oct 30 16:57:51.522346 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:51.522318 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:57:51.522441 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:51.522382 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:57:51.522441 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:51.522409 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:57:51.742287 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:51.742190 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-648b5d97c7-78fx4" event={"ID":"8f03ee2a-146e-4c54-b83a-35637ceda4bb","Type":"ContainerStarted","Data":"821f1168e2f3f6189b3c3003fa113cd35502483838c926515aaa8d5e69d2a27e"} Oct 30 16:57:51.744651 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:51.744625 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-245xx" event={"ID":"fb8adf71-af34-4715-ada0-42494da2219a","Type":"ContainerStarted","Data":"cd6b2934b6f9e5a3f4d400e3315c4ca9f00aea0bb0a256511a5a48b504646689"} Oct 30 16:57:51.794762 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:51.794706 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-245xx" podStartSLOduration=1.04933865 podStartE2EDuration="3.794692257s" totalImagesPullingTime="2.745353607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:57:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:57:51.792704561 +0000 UTC m=+645.793459484" watchObservedRunningTime="2026-10-30 16:57:51.794692257 +0000 UTC m=+645.795447158" Oct 30 16:57:52.446948 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:52.446583 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-245xx" Oct 30 16:57:52.448330 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:52.448277 4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-245xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Oct 30 16:57:52.448499 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:52.448346 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-245xx" podUID="fb8adf71-af34-4715-ada0-42494da2219a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Oct 30 16:57:52.758490 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:52.754524 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-648b5d97c7-78fx4" event={"ID":"8f03ee2a-146e-4c54-b83a-35637ceda4bb","Type":"ContainerStarted","Data":"fc77146ebbf8673d3926e8ca9f469b45315e2a70925c6e4692d0eae5165c16fa"} Oct 30 16:57:52.810332 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:52.805265 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-648b5d97c7-78fx4" podStartSLOduration=2.130980661 podStartE2EDuration="2.80524614s" totalImagesPullingTime="674.265479ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:57:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:57:52.804701184 +0000 UTC m=+646.805456086" watchObservedRunningTime="2026-10-30 16:57:52.80524614 +0000 UTC m=+646.806001043" Oct 30 16:57:52.829337 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:52.828536 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5764994858-wwf9h"] Oct 30 16:57:52.829337 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:52.828761 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-5764994858-wwf9h" podUID="5fc165af-809c-4990-9d62-d9f68d4bcbb2" containerName="authorino" containerID="cri-o://2a54c8a153d8759a42cc899f795b9ec4bdb46b0cca4c004a3b979e8def27c0d2" gracePeriod=30 Oct 30 16:57:53.085375 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:53.085349 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5764994858-wwf9h" Oct 30 16:57:53.229349 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:53.229291 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fc165af-809c-4990-9d62-d9f68d4bcbb2-tls-cert\" (UniqueName: \"kubernetes.io/secret/5fc165af-809c-4990-9d62-d9f68d4bcbb2-tls-cert\") pod \"5fc165af-809c-4990-9d62-d9f68d4bcbb2\" (UID: \"5fc165af-809c-4990-9d62-d9f68d4bcbb2\") " Oct 30 16:57:53.229543 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:53.229378 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fc165af-809c-4990-9d62-d9f68d4bcbb2-kube-api-access-49ssb\" (UniqueName: \"kubernetes.io/projected/5fc165af-809c-4990-9d62-d9f68d4bcbb2-kube-api-access-49ssb\") pod \"5fc165af-809c-4990-9d62-d9f68d4bcbb2\" (UID: \"5fc165af-809c-4990-9d62-d9f68d4bcbb2\") " Oct 30 16:57:53.231524 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:53.231493 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fc165af-809c-4990-9d62-d9f68d4bcbb2-kube-api-access-49ssb" pod "5fc165af-809c-4990-9d62-d9f68d4bcbb2" (UID: "5fc165af-809c-4990-9d62-d9f68d4bcbb2"). InnerVolumeSpecName "kube-api-access-49ssb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:57:53.240234 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:53.240182 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fc165af-809c-4990-9d62-d9f68d4bcbb2-tls-cert" pod "5fc165af-809c-4990-9d62-d9f68d4bcbb2" (UID: "5fc165af-809c-4990-9d62-d9f68d4bcbb2"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:57:53.331084 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:53.330983 4522 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5fc165af-809c-4990-9d62-d9f68d4bcbb2-tls-cert\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 16:57:53.331084 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:53.331023 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49ssb\" (UniqueName: \"kubernetes.io/projected/5fc165af-809c-4990-9d62-d9f68d4bcbb2-kube-api-access-49ssb\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 16:57:53.446804 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:53.446768 4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-245xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Oct 30 16:57:53.446998 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:53.446830 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-245xx" podUID="fb8adf71-af34-4715-ada0-42494da2219a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Oct 30 16:57:53.759808 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:53.759773 4522 generic.go:356] "Generic (PLEG): container finished" podID="5fc165af-809c-4990-9d62-d9f68d4bcbb2" containerID="2a54c8a153d8759a42cc899f795b9ec4bdb46b0cca4c004a3b979e8def27c0d2" exitCode=0 Oct 30 16:57:53.759984 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:53.759814 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5764994858-wwf9h" Oct 30 16:57:53.759984 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:53.759852 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5764994858-wwf9h" event={"ID":"5fc165af-809c-4990-9d62-d9f68d4bcbb2","Type":"ContainerDied","Data":"2a54c8a153d8759a42cc899f795b9ec4bdb46b0cca4c004a3b979e8def27c0d2"} Oct 30 16:57:53.759984 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:53.759896 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5764994858-wwf9h" event={"ID":"5fc165af-809c-4990-9d62-d9f68d4bcbb2","Type":"ContainerDied","Data":"53bd14e8b542ec8206f00fc5bd17c8a9e2af211ddbfe45d3ac512b571981c18e"} Oct 30 16:57:53.759984 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:53.759919 4522 scope.go:122] "RemoveContainer" containerID="2a54c8a153d8759a42cc899f795b9ec4bdb46b0cca4c004a3b979e8def27c0d2" Oct 30 16:57:53.769567 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:53.769548 4522 scope.go:122] "RemoveContainer" containerID="2a54c8a153d8759a42cc899f795b9ec4bdb46b0cca4c004a3b979e8def27c0d2" Oct 30 16:57:53.769872 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:57:53.769852 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a54c8a153d8759a42cc899f795b9ec4bdb46b0cca4c004a3b979e8def27c0d2\": container with ID starting with 2a54c8a153d8759a42cc899f795b9ec4bdb46b0cca4c004a3b979e8def27c0d2 not found: ID does not exist" containerID="2a54c8a153d8759a42cc899f795b9ec4bdb46b0cca4c004a3b979e8def27c0d2" Oct 30 16:57:53.769923 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:53.769882 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a54c8a153d8759a42cc899f795b9ec4bdb46b0cca4c004a3b979e8def27c0d2"} err="failed to get container status \"2a54c8a153d8759a42cc899f795b9ec4bdb46b0cca4c004a3b979e8def27c0d2\": rpc error: code = NotFound desc = could not find container \"2a54c8a153d8759a42cc899f795b9ec4bdb46b0cca4c004a3b979e8def27c0d2\": container with ID starting with 2a54c8a153d8759a42cc899f795b9ec4bdb46b0cca4c004a3b979e8def27c0d2 not found: ID does not exist" Oct 30 16:57:53.822476 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:53.822441 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5764994858-wwf9h"] Oct 30 16:57:53.826493 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:53.826462 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5764994858-wwf9h"] Oct 30 16:57:54.142672 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:54.142594 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fc165af-809c-4990-9d62-d9f68d4bcbb2" path="/var/lib/kubelet/pods/5fc165af-809c-4990-9d62-d9f68d4bcbb2/volumes" Oct 30 16:57:54.447791 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:54.447700 4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-245xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Oct 30 16:57:54.448571 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:54.448540 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-245xx" podUID="fb8adf71-af34-4715-ada0-42494da2219a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Oct 30 16:57:55.447323 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:55.447265 4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-245xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Oct 30 16:57:55.447489 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:55.447341 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-245xx" podUID="fb8adf71-af34-4715-ada0-42494da2219a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Oct 30 16:57:56.447651 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:56.447614 4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-245xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Oct 30 16:57:56.448085 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:56.447665 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-245xx" podUID="fb8adf71-af34-4715-ada0-42494da2219a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Oct 30 16:57:57.447416 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:57.447378 4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-245xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Oct 30 16:57:57.447589 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:57.447433 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-245xx" podUID="fb8adf71-af34-4715-ada0-42494da2219a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Oct 30 16:57:58.447373 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:58.447340 4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-245xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Oct 30 16:57:58.447798 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:57:58.447395 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-245xx" podUID="fb8adf71-af34-4715-ada0-42494da2219a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Oct 30 16:58:00.448519 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:00.448464 4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-245xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:58:00.449050 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:00.448535 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-245xx" podUID="fb8adf71-af34-4715-ada0-42494da2219a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:58:01.489524 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:01.489482 4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-245xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:58:01.489947 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:01.489551 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-245xx" podUID="fb8adf71-af34-4715-ada0-42494da2219a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:58:01.953123 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:01.953096 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-245xx" Oct 30 16:58:01.953403 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:01.953381 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-245xx" Oct 30 16:58:01.954121 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:01.954105 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-245xx" Oct 30 16:58:03.148077 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:03.148042 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h5k2r"] Oct 30 16:58:03.148504 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:03.148491 4522 cpu_manager.go:414] "containerMap: removing container" podUID="5fc165af-809c-4990-9d62-d9f68d4bcbb2" containerName="authorino" Oct 30 16:58:03.148548 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:03.148507 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fc165af-809c-4990-9d62-d9f68d4bcbb2" containerName="authorino" Oct 30 16:58:03.148600 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:03.148589 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fc165af-809c-4990-9d62-d9f68d4bcbb2" containerName="authorino" Oct 30 16:58:03.154971 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:03.154954 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h5k2r" Oct 30 16:58:03.177121 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:03.177087 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h5k2r"] Oct 30 16:58:03.215552 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:03.215515 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23c04722-0747-49fe-a166-0969959367bd-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-h5k2r\" (UID: \"23c04722-0747-49fe-a166-0969959367bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h5k2r" Oct 30 16:58:03.215552 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:03.215551 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcgcd\" (UniqueName: \"kubernetes.io/projected/23c04722-0747-49fe-a166-0969959367bd-kube-api-access-wcgcd\") pod \"maas-default-gateway-openshift-default-6db586dd6c-h5k2r\" (UID: \"23c04722-0747-49fe-a166-0969959367bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h5k2r" Oct 30 16:58:03.215813 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:03.215738 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23c04722-0747-49fe-a166-0969959367bd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-h5k2r\" (UID: \"23c04722-0747-49fe-a166-0969959367bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h5k2r" Oct 30 16:58:03.215879 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:03.215825 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23c04722-0747-49fe-a166-0969959367bd-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-h5k2r\" (UID: \"23c04722-0747-49fe-a166-0969959367bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h5k2r" Oct 30 16:58:03.215969 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:03.215946 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23c04722-0747-49fe-a166-0969959367bd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-h5k2r\" (UID: \"23c04722-0747-49fe-a166-0969959367bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h5k2r" Oct 30 16:58:03.216024 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:03.215983 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23c04722-0747-49fe-a166-0969959367bd-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-h5k2r\" (UID: \"23c04722-0747-49fe-a166-0969959367bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h5k2r" Oct 30 16:58:03.216089 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:03.216031 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23c04722-0747-49fe-a166-0969959367bd-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-h5k2r\" (UID: \"23c04722-0747-49fe-a166-0969959367bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h5k2r" Oct 30 16:58:03.216089 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:03.216070 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23c04722-0747-49fe-a166-0969959367bd-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-h5k2r\" (UID: \"23c04722-0747-49fe-a166-0969959367bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h5k2r" Oct 30 16:58:03.216153 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:03.216104 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23c04722-0747-49fe-a166-0969959367bd-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-h5k2r\" (UID: \"23c04722-0747-49fe-a166-0969959367bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h5k2r" Oct 30 16:58:03.317569 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:03.317522 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23c04722-0747-49fe-a166-0969959367bd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-h5k2r\" (UID: \"23c04722-0747-49fe-a166-0969959367bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h5k2r" Oct 30 16:58:03.317569 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:03.317574 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23c04722-0747-49fe-a166-0969959367bd-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-h5k2r\" (UID: \"23c04722-0747-49fe-a166-0969959367bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h5k2r" Oct 30 16:58:03.317905 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:03.317604 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23c04722-0747-49fe-a166-0969959367bd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-h5k2r\" (UID: \"23c04722-0747-49fe-a166-0969959367bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h5k2r" Oct 30 16:58:03.317905 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:03.317623 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23c04722-0747-49fe-a166-0969959367bd-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-h5k2r\" (UID: \"23c04722-0747-49fe-a166-0969959367bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h5k2r" Oct 30 16:58:03.317905 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:03.317638 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23c04722-0747-49fe-a166-0969959367bd-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-h5k2r\" (UID: \"23c04722-0747-49fe-a166-0969959367bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h5k2r" Oct 30 16:58:03.317905 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:03.317657 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23c04722-0747-49fe-a166-0969959367bd-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-h5k2r\" (UID: \"23c04722-0747-49fe-a166-0969959367bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h5k2r" Oct 30 16:58:03.317905 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:03.317822 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23c04722-0747-49fe-a166-0969959367bd-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-h5k2r\" (UID: \"23c04722-0747-49fe-a166-0969959367bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h5k2r" Oct 30 16:58:03.318204 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:03.317949 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23c04722-0747-49fe-a166-0969959367bd-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-h5k2r\" (UID: \"23c04722-0747-49fe-a166-0969959367bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h5k2r" Oct 30 16:58:03.318204 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:03.317980 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcgcd\" (UniqueName: \"kubernetes.io/projected/23c04722-0747-49fe-a166-0969959367bd-kube-api-access-wcgcd\") pod \"maas-default-gateway-openshift-default-6db586dd6c-h5k2r\" (UID: \"23c04722-0747-49fe-a166-0969959367bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h5k2r" Oct 30 16:58:03.318204 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:03.318132 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23c04722-0747-49fe-a166-0969959367bd-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-h5k2r\" (UID: \"23c04722-0747-49fe-a166-0969959367bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h5k2r" Oct 30 16:58:03.318204 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:03.318190 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23c04722-0747-49fe-a166-0969959367bd-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-h5k2r\" (UID: \"23c04722-0747-49fe-a166-0969959367bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h5k2r" Oct 30 16:58:03.318487 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:03.318219 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23c04722-0747-49fe-a166-0969959367bd-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-h5k2r\" (UID: \"23c04722-0747-49fe-a166-0969959367bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h5k2r" Oct 30 16:58:03.318487 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:03.318357 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23c04722-0747-49fe-a166-0969959367bd-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-h5k2r\" (UID: \"23c04722-0747-49fe-a166-0969959367bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h5k2r" Oct 30 16:58:03.318566 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:03.318478 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23c04722-0747-49fe-a166-0969959367bd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-h5k2r\" (UID: \"23c04722-0747-49fe-a166-0969959367bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h5k2r" Oct 30 16:58:03.320116 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:03.320094 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23c04722-0747-49fe-a166-0969959367bd-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-h5k2r\" (UID: \"23c04722-0747-49fe-a166-0969959367bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h5k2r" Oct 30 16:58:03.320349 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:03.320333 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23c04722-0747-49fe-a166-0969959367bd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-h5k2r\" (UID: \"23c04722-0747-49fe-a166-0969959367bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h5k2r" Oct 30 16:58:03.342738 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:03.342703 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23c04722-0747-49fe-a166-0969959367bd-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-h5k2r\" (UID: \"23c04722-0747-49fe-a166-0969959367bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h5k2r" Oct 30 16:58:03.342867 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:03.342821 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcgcd\" (UniqueName: \"kubernetes.io/projected/23c04722-0747-49fe-a166-0969959367bd-kube-api-access-wcgcd\") pod \"maas-default-gateway-openshift-default-6db586dd6c-h5k2r\" (UID: \"23c04722-0747-49fe-a166-0969959367bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h5k2r" Oct 30 16:58:03.468904 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:03.468870 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h5k2r" Oct 30 16:58:03.788771 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:03.788740 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h5k2r"] Oct 30 16:58:03.789150 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:58:03.789125 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23c04722_0747_49fe_a166_0969959367bd.slice/crio-8f5fb68eab981f952b22ed8ee0925ef5e722c37d7aef92ae3cee46db700df2fd WatchSource:0}: Error finding container 8f5fb68eab981f952b22ed8ee0925ef5e722c37d7aef92ae3cee46db700df2fd: Status 404 returned error can't find the container with id 8f5fb68eab981f952b22ed8ee0925ef5e722c37d7aef92ae3cee46db700df2fd Oct 30 16:58:03.792282 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:03.792251 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:58:03.792399 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:03.792336 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:58:03.792399 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:03.792381 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:58:03.821254 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:03.821216 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h5k2r" event={"ID":"23c04722-0747-49fe-a166-0969959367bd","Type":"ContainerStarted","Data":"8f5fb68eab981f952b22ed8ee0925ef5e722c37d7aef92ae3cee46db700df2fd"} Oct 30 16:58:04.826995 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:04.826957 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h5k2r" event={"ID":"23c04722-0747-49fe-a166-0969959367bd","Type":"ContainerStarted","Data":"44d81bdc413b0e8823bda116b585b0611ebd70226426347fa26fbbbd78dfae77"} Oct 30 16:58:04.912397 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:04.892281 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h5k2r" podStartSLOduration=1.892261929 podStartE2EDuration="1.892261929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:58:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:58:04.889227361 +0000 UTC m=+658.889982281" watchObservedRunningTime="2026-10-30 16:58:04.892261929 +0000 UTC m=+658.893016831" Oct 30 16:58:05.469942 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:05.469905 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h5k2r" Oct 30 16:58:05.471428 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:05.471400 4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-h5k2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 16:58:05.471568 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:05.471451 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h5k2r" podUID="23c04722-0747-49fe-a166-0969959367bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 16:58:06.470168 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:06.470090 4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-h5k2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 16:58:06.470168 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:06.470147 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h5k2r" podUID="23c04722-0747-49fe-a166-0969959367bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 16:58:07.470149 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:07.470117 4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-h5k2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 16:58:07.470333 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:07.470168 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h5k2r" podUID="23c04722-0747-49fe-a166-0969959367bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 16:58:08.469902 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:08.469865 4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-h5k2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 16:58:08.470096 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:08.469928 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h5k2r" podUID="23c04722-0747-49fe-a166-0969959367bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 16:58:09.469931 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:09.469896 4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-h5k2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 16:58:09.470326 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:09.469956 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h5k2r" podUID="23c04722-0747-49fe-a166-0969959367bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 16:58:10.469984 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:10.469942 4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-h5k2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 16:58:10.470484 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:10.470007 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h5k2r" podUID="23c04722-0747-49fe-a166-0969959367bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 16:58:12.169825 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:12.169792 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h5k2r" Oct 30 16:58:12.170291 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:12.170185 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h5k2r" Oct 30 16:58:12.170880 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:12.170859 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h5k2r" Oct 30 16:58:12.946627 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:58:12.946591 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 16:58:42.946785 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:58:42.946746 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 16:58:51.666564 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:51.666530 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf"] Oct 30 16:58:51.670072 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:51.670048 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf" Oct 30 16:58:51.693441 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:51.693412 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf"] Oct 30 16:58:51.750732 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:51.750702 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-d384dc-openshift-default-dockercfg-r4qzb\"" Oct 30 16:58:51.769297 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:51.769255 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b590c8f6-9d91-49bb-8e0f-5e65eb24e631-istio-podinfo\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf\" (UID: \"b590c8f6-9d91-49bb-8e0f-5e65eb24e631\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf" Oct 30 16:58:51.769502 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:51.769401 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b590c8f6-9d91-49bb-8e0f-5e65eb24e631-workload-socket\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf\" (UID: \"b590c8f6-9d91-49bb-8e0f-5e65eb24e631\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf" Oct 30 16:58:51.769502 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:51.769458 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b590c8f6-9d91-49bb-8e0f-5e65eb24e631-istiod-ca-cert\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf\" (UID: \"b590c8f6-9d91-49bb-8e0f-5e65eb24e631\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf" Oct 30 16:58:51.769502 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:51.769487 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b590c8f6-9d91-49bb-8e0f-5e65eb24e631-istio-envoy\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf\" (UID: \"b590c8f6-9d91-49bb-8e0f-5e65eb24e631\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf" Oct 30 16:58:51.769642 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:51.769517 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b590c8f6-9d91-49bb-8e0f-5e65eb24e631-istio-data\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf\" (UID: \"b590c8f6-9d91-49bb-8e0f-5e65eb24e631\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf" Oct 30 16:58:51.769642 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:51.769580 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nzsc\" (UniqueName: \"kubernetes.io/projected/b590c8f6-9d91-49bb-8e0f-5e65eb24e631-kube-api-access-6nzsc\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf\" (UID: \"b590c8f6-9d91-49bb-8e0f-5e65eb24e631\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf" Oct 30 16:58:51.769642 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:51.769613 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b590c8f6-9d91-49bb-8e0f-5e65eb24e631-credential-socket\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf\" (UID: \"b590c8f6-9d91-49bb-8e0f-5e65eb24e631\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf" Oct 30 16:58:51.769800 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:51.769733 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b590c8f6-9d91-49bb-8e0f-5e65eb24e631-workload-certs\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf\" (UID: \"b590c8f6-9d91-49bb-8e0f-5e65eb24e631\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf" Oct 30 16:58:51.769800 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:51.769776 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b590c8f6-9d91-49bb-8e0f-5e65eb24e631-istio-token\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf\" (UID: \"b590c8f6-9d91-49bb-8e0f-5e65eb24e631\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf" Oct 30 16:58:51.871255 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:51.871208 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b590c8f6-9d91-49bb-8e0f-5e65eb24e631-istio-envoy\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf\" (UID: \"b590c8f6-9d91-49bb-8e0f-5e65eb24e631\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf" Oct 30 16:58:51.871255 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:51.871261 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b590c8f6-9d91-49bb-8e0f-5e65eb24e631-istio-data\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf\" (UID: \"b590c8f6-9d91-49bb-8e0f-5e65eb24e631\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf" Oct 30 16:58:51.871530 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:51.871278 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nzsc\" (UniqueName: \"kubernetes.io/projected/b590c8f6-9d91-49bb-8e0f-5e65eb24e631-kube-api-access-6nzsc\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf\" (UID: \"b590c8f6-9d91-49bb-8e0f-5e65eb24e631\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf" Oct 30 16:58:51.871530 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:51.871298 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b590c8f6-9d91-49bb-8e0f-5e65eb24e631-credential-socket\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf\" (UID: \"b590c8f6-9d91-49bb-8e0f-5e65eb24e631\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf" Oct 30 16:58:51.871530 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:51.871340 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b590c8f6-9d91-49bb-8e0f-5e65eb24e631-workload-certs\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf\" (UID: \"b590c8f6-9d91-49bb-8e0f-5e65eb24e631\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf" Oct 30 16:58:51.871530 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:51.871364 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b590c8f6-9d91-49bb-8e0f-5e65eb24e631-istio-token\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf\" (UID: \"b590c8f6-9d91-49bb-8e0f-5e65eb24e631\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf" Oct 30 16:58:51.871530 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:51.871411 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b590c8f6-9d91-49bb-8e0f-5e65eb24e631-istio-podinfo\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf\" (UID: \"b590c8f6-9d91-49bb-8e0f-5e65eb24e631\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf" Oct 30 16:58:51.871530 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:51.871493 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b590c8f6-9d91-49bb-8e0f-5e65eb24e631-workload-socket\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf\" (UID: \"b590c8f6-9d91-49bb-8e0f-5e65eb24e631\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf" Oct 30 16:58:51.871530 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:51.871520 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b590c8f6-9d91-49bb-8e0f-5e65eb24e631-istiod-ca-cert\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf\" (UID: \"b590c8f6-9d91-49bb-8e0f-5e65eb24e631\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf" Oct 30 16:58:51.871899 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:51.871718 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b590c8f6-9d91-49bb-8e0f-5e65eb24e631-istio-data\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf\" (UID: \"b590c8f6-9d91-49bb-8e0f-5e65eb24e631\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf" Oct 30 16:58:51.871899 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:51.871775 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b590c8f6-9d91-49bb-8e0f-5e65eb24e631-credential-socket\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf\" (UID: \"b590c8f6-9d91-49bb-8e0f-5e65eb24e631\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf" Oct 30 16:58:51.872043 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:51.872016 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b590c8f6-9d91-49bb-8e0f-5e65eb24e631-workload-socket\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf\" (UID: \"b590c8f6-9d91-49bb-8e0f-5e65eb24e631\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf" Oct 30 16:58:51.872109 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:51.872060 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b590c8f6-9d91-49bb-8e0f-5e65eb24e631-workload-certs\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf\" (UID: \"b590c8f6-9d91-49bb-8e0f-5e65eb24e631\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf" Oct 30 16:58:51.872216 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:51.872192 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b590c8f6-9d91-49bb-8e0f-5e65eb24e631-istiod-ca-cert\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf\" (UID: \"b590c8f6-9d91-49bb-8e0f-5e65eb24e631\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf" Oct 30 16:58:51.873719 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:51.873694 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b590c8f6-9d91-49bb-8e0f-5e65eb24e631-istio-envoy\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf\" (UID: \"b590c8f6-9d91-49bb-8e0f-5e65eb24e631\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf" Oct 30 16:58:51.874129 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:51.874110 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b590c8f6-9d91-49bb-8e0f-5e65eb24e631-istio-podinfo\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf\" (UID: \"b590c8f6-9d91-49bb-8e0f-5e65eb24e631\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf" Oct 30 16:58:51.897419 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:51.897386 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b590c8f6-9d91-49bb-8e0f-5e65eb24e631-istio-token\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf\" (UID: \"b590c8f6-9d91-49bb-8e0f-5e65eb24e631\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf" Oct 30 16:58:51.897586 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:51.897474 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nzsc\" (UniqueName: \"kubernetes.io/projected/b590c8f6-9d91-49bb-8e0f-5e65eb24e631-kube-api-access-6nzsc\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf\" (UID: \"b590c8f6-9d91-49bb-8e0f-5e65eb24e631\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf" Oct 30 16:58:51.966323 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:51.966271 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7"] Oct 30 16:58:51.970978 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:51.970952 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7" Oct 30 16:58:51.983928 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:51.983899 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf" Oct 30 16:58:51.990901 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:51.990876 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-86f50a-openshift-default-dockercfg-n29ph\"" Oct 30 16:58:52.001786 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.001756 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7"] Oct 30 16:58:52.073485 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.073456 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ffa9d7-4de9-4b98-8e7e-016db7370f08-credential-socket\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7\" (UID: \"c6ffa9d7-4de9-4b98-8e7e-016db7370f08\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7" Oct 30 16:58:52.073644 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.073491 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6ffa9d7-4de9-4b98-8e7e-016db7370f08-istio-podinfo\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7\" (UID: \"c6ffa9d7-4de9-4b98-8e7e-016db7370f08\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7" Oct 30 16:58:52.073644 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.073509 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6ffa9d7-4de9-4b98-8e7e-016db7370f08-workload-certs\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7\" (UID: \"c6ffa9d7-4de9-4b98-8e7e-016db7370f08\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7" Oct 30 16:58:52.073764 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.073649 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ffa9d7-4de9-4b98-8e7e-016db7370f08-workload-socket\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7\" (UID: \"c6ffa9d7-4de9-4b98-8e7e-016db7370f08\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7" Oct 30 16:58:52.073764 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.073747 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6ffa9d7-4de9-4b98-8e7e-016db7370f08-istio-token\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7\" (UID: \"c6ffa9d7-4de9-4b98-8e7e-016db7370f08\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7" Oct 30 16:58:52.073875 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.073777 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6ffa9d7-4de9-4b98-8e7e-016db7370f08-istio-envoy\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7\" (UID: \"c6ffa9d7-4de9-4b98-8e7e-016db7370f08\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7" Oct 30 16:58:52.073875 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.073813 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc6v5\" (UniqueName: \"kubernetes.io/projected/c6ffa9d7-4de9-4b98-8e7e-016db7370f08-kube-api-access-rc6v5\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7\" (UID: \"c6ffa9d7-4de9-4b98-8e7e-016db7370f08\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7" Oct 30 16:58:52.073875 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.073850 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6ffa9d7-4de9-4b98-8e7e-016db7370f08-istio-data\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7\" (UID: \"c6ffa9d7-4de9-4b98-8e7e-016db7370f08\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7" Oct 30 16:58:52.074009 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.073878 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6ffa9d7-4de9-4b98-8e7e-016db7370f08-istiod-ca-cert\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7\" (UID: \"c6ffa9d7-4de9-4b98-8e7e-016db7370f08\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7" Oct 30 16:58:52.174685 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.174650 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6ffa9d7-4de9-4b98-8e7e-016db7370f08-istio-token\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7\" (UID: \"c6ffa9d7-4de9-4b98-8e7e-016db7370f08\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7" Oct 30 16:58:52.174685 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.174686 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6ffa9d7-4de9-4b98-8e7e-016db7370f08-istio-envoy\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7\" (UID: \"c6ffa9d7-4de9-4b98-8e7e-016db7370f08\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7" Oct 30 16:58:52.174923 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.174714 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc6v5\" (UniqueName: \"kubernetes.io/projected/c6ffa9d7-4de9-4b98-8e7e-016db7370f08-kube-api-access-rc6v5\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7\" (UID: \"c6ffa9d7-4de9-4b98-8e7e-016db7370f08\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7" Oct 30 16:58:52.174923 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.174746 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6ffa9d7-4de9-4b98-8e7e-016db7370f08-istio-data\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7\" (UID: \"c6ffa9d7-4de9-4b98-8e7e-016db7370f08\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7" Oct 30 16:58:52.174923 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.174866 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6ffa9d7-4de9-4b98-8e7e-016db7370f08-istiod-ca-cert\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7\" (UID: \"c6ffa9d7-4de9-4b98-8e7e-016db7370f08\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7" Oct 30 16:58:52.175083 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.174978 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ffa9d7-4de9-4b98-8e7e-016db7370f08-credential-socket\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7\" (UID: \"c6ffa9d7-4de9-4b98-8e7e-016db7370f08\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7" Oct 30 16:58:52.175083 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.175015 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6ffa9d7-4de9-4b98-8e7e-016db7370f08-istio-podinfo\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7\" (UID: \"c6ffa9d7-4de9-4b98-8e7e-016db7370f08\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7" Oct 30 16:58:52.175083 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.175045 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6ffa9d7-4de9-4b98-8e7e-016db7370f08-workload-certs\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7\" (UID: \"c6ffa9d7-4de9-4b98-8e7e-016db7370f08\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7" Oct 30 16:58:52.175226 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.175084 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ffa9d7-4de9-4b98-8e7e-016db7370f08-workload-socket\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7\" (UID: \"c6ffa9d7-4de9-4b98-8e7e-016db7370f08\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7" Oct 30 16:58:52.175226 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.175163 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6ffa9d7-4de9-4b98-8e7e-016db7370f08-istio-data\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7\" (UID: \"c6ffa9d7-4de9-4b98-8e7e-016db7370f08\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7" Oct 30 16:58:52.175400 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.175384 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ffa9d7-4de9-4b98-8e7e-016db7370f08-credential-socket\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7\" (UID: \"c6ffa9d7-4de9-4b98-8e7e-016db7370f08\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7" Oct 30 16:58:52.175490 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.175469 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ffa9d7-4de9-4b98-8e7e-016db7370f08-workload-socket\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7\" (UID: \"c6ffa9d7-4de9-4b98-8e7e-016db7370f08\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7" Oct 30 16:58:52.175543 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.175496 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6ffa9d7-4de9-4b98-8e7e-016db7370f08-workload-certs\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7\" (UID: \"c6ffa9d7-4de9-4b98-8e7e-016db7370f08\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7" Oct 30 16:58:52.175851 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.175826 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6ffa9d7-4de9-4b98-8e7e-016db7370f08-istiod-ca-cert\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7\" (UID: \"c6ffa9d7-4de9-4b98-8e7e-016db7370f08\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7" Oct 30 16:58:52.177411 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.177392 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6ffa9d7-4de9-4b98-8e7e-016db7370f08-istio-envoy\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7\" (UID: \"c6ffa9d7-4de9-4b98-8e7e-016db7370f08\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7" Oct 30 16:58:52.177626 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.177609 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6ffa9d7-4de9-4b98-8e7e-016db7370f08-istio-podinfo\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7\" (UID: \"c6ffa9d7-4de9-4b98-8e7e-016db7370f08\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7" Oct 30 16:58:52.203559 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.203532 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6ffa9d7-4de9-4b98-8e7e-016db7370f08-istio-token\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7\" (UID: \"c6ffa9d7-4de9-4b98-8e7e-016db7370f08\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7" Oct 30 16:58:52.204723 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.204703 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc6v5\" (UniqueName: \"kubernetes.io/projected/c6ffa9d7-4de9-4b98-8e7e-016db7370f08-kube-api-access-rc6v5\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7\" (UID: \"c6ffa9d7-4de9-4b98-8e7e-016db7370f08\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7" Oct 30 16:58:52.285735 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.285635 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk"] Oct 30 16:58:52.287951 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.287911 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7" Oct 30 16:58:52.290870 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.290843 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk" Oct 30 16:58:52.313240 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.313214 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf"] Oct 30 16:58:52.315092 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.314609 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-096d75-openshift-default-dockercfg-twcrb\"" Oct 30 16:58:52.316887 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.316863 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk"] Oct 30 16:58:52.325099 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:58:52.325067 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb590c8f6_9d91_49bb_8e0f_5e65eb24e631.slice/crio-0419c901e7364525b1adff66cb8e77a30b904632c755cebe76966148b7cf5224 WatchSource:0}: Error finding container 0419c901e7364525b1adff66cb8e77a30b904632c755cebe76966148b7cf5224: Status 404 returned error can't find the container with id 0419c901e7364525b1adff66cb8e77a30b904632c755cebe76966148b7cf5224 Oct 30 16:58:52.328054 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.328024 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:58:52.328137 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.328086 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:58:52.328137 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.328133 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:58:52.376642 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.376616 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93447f05-2d76-440f-b704-5e37d09eb5b9-workload-socket\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk\" (UID: \"93447f05-2d76-440f-b704-5e37d09eb5b9\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk" Oct 30 16:58:52.376794 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.376647 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st6r5\" (UniqueName: \"kubernetes.io/projected/93447f05-2d76-440f-b704-5e37d09eb5b9-kube-api-access-st6r5\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk\" (UID: \"93447f05-2d76-440f-b704-5e37d09eb5b9\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk" Oct 30 16:58:52.376794 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.376669 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93447f05-2d76-440f-b704-5e37d09eb5b9-istio-data\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk\" (UID: \"93447f05-2d76-440f-b704-5e37d09eb5b9\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk" Oct 30 16:58:52.376794 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.376689 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93447f05-2d76-440f-b704-5e37d09eb5b9-credential-socket\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk\" (UID: \"93447f05-2d76-440f-b704-5e37d09eb5b9\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk" Oct 30 16:58:52.376947 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.376837 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93447f05-2d76-440f-b704-5e37d09eb5b9-workload-certs\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk\" (UID: \"93447f05-2d76-440f-b704-5e37d09eb5b9\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk" Oct 30 16:58:52.376947 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.376937 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93447f05-2d76-440f-b704-5e37d09eb5b9-istio-podinfo\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk\" (UID: \"93447f05-2d76-440f-b704-5e37d09eb5b9\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk" Oct 30 16:58:52.377029 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.376967 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93447f05-2d76-440f-b704-5e37d09eb5b9-istiod-ca-cert\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk\" (UID: \"93447f05-2d76-440f-b704-5e37d09eb5b9\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk" Oct 30 16:58:52.377069 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.377032 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93447f05-2d76-440f-b704-5e37d09eb5b9-istio-token\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk\" (UID: \"93447f05-2d76-440f-b704-5e37d09eb5b9\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk" Oct 30 16:58:52.377104 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.377087 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93447f05-2d76-440f-b704-5e37d09eb5b9-istio-envoy\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk\" (UID: \"93447f05-2d76-440f-b704-5e37d09eb5b9\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk" Oct 30 16:58:52.478207 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.478169 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93447f05-2d76-440f-b704-5e37d09eb5b9-workload-socket\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk\" (UID: \"93447f05-2d76-440f-b704-5e37d09eb5b9\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk" Oct 30 16:58:52.478207 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.478218 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st6r5\" (UniqueName: \"kubernetes.io/projected/93447f05-2d76-440f-b704-5e37d09eb5b9-kube-api-access-st6r5\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk\" (UID: \"93447f05-2d76-440f-b704-5e37d09eb5b9\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk" Oct 30 16:58:52.478504 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.478249 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93447f05-2d76-440f-b704-5e37d09eb5b9-istio-data\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk\" (UID: \"93447f05-2d76-440f-b704-5e37d09eb5b9\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk" Oct 30 16:58:52.478504 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.478284 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93447f05-2d76-440f-b704-5e37d09eb5b9-credential-socket\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk\" (UID: \"93447f05-2d76-440f-b704-5e37d09eb5b9\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk" Oct 30 16:58:52.478504 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.478380 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93447f05-2d76-440f-b704-5e37d09eb5b9-workload-certs\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk\" (UID: \"93447f05-2d76-440f-b704-5e37d09eb5b9\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk" Oct 30 16:58:52.478504 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.478446 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93447f05-2d76-440f-b704-5e37d09eb5b9-istio-podinfo\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk\" (UID: \"93447f05-2d76-440f-b704-5e37d09eb5b9\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk" Oct 30 16:58:52.478504 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.478478 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93447f05-2d76-440f-b704-5e37d09eb5b9-istiod-ca-cert\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk\" (UID: \"93447f05-2d76-440f-b704-5e37d09eb5b9\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk" Oct 30 16:58:52.478764 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.478530 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93447f05-2d76-440f-b704-5e37d09eb5b9-istio-token\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk\" (UID: \"93447f05-2d76-440f-b704-5e37d09eb5b9\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk" Oct 30 16:58:52.478764 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.478586 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93447f05-2d76-440f-b704-5e37d09eb5b9-istio-envoy\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk\" (UID: \"93447f05-2d76-440f-b704-5e37d09eb5b9\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk" Oct 30 16:58:52.478764 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.478668 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93447f05-2d76-440f-b704-5e37d09eb5b9-workload-socket\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk\" (UID: \"93447f05-2d76-440f-b704-5e37d09eb5b9\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk" Oct 30 16:58:52.478918 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.478890 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93447f05-2d76-440f-b704-5e37d09eb5b9-workload-certs\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk\" (UID: \"93447f05-2d76-440f-b704-5e37d09eb5b9\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk" Oct 30 16:58:52.479196 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.479169 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93447f05-2d76-440f-b704-5e37d09eb5b9-istio-data\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk\" (UID: \"93447f05-2d76-440f-b704-5e37d09eb5b9\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk" Oct 30 16:58:52.479721 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.479338 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93447f05-2d76-440f-b704-5e37d09eb5b9-credential-socket\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk\" (UID: \"93447f05-2d76-440f-b704-5e37d09eb5b9\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk" Oct 30 16:58:52.479721 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.479356 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93447f05-2d76-440f-b704-5e37d09eb5b9-istiod-ca-cert\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk\" (UID: \"93447f05-2d76-440f-b704-5e37d09eb5b9\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk" Oct 30 16:58:52.481617 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.481592 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93447f05-2d76-440f-b704-5e37d09eb5b9-istio-envoy\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk\" (UID: \"93447f05-2d76-440f-b704-5e37d09eb5b9\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk" Oct 30 16:58:52.482807 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.482784 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93447f05-2d76-440f-b704-5e37d09eb5b9-istio-podinfo\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk\" (UID: \"93447f05-2d76-440f-b704-5e37d09eb5b9\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk" Oct 30 16:58:52.511282 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.511249 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93447f05-2d76-440f-b704-5e37d09eb5b9-istio-token\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk\" (UID: \"93447f05-2d76-440f-b704-5e37d09eb5b9\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk" Oct 30 16:58:52.512290 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.512267 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-st6r5\" (UniqueName: \"kubernetes.io/projected/93447f05-2d76-440f-b704-5e37d09eb5b9-kube-api-access-st6r5\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk\" (UID: \"93447f05-2d76-440f-b704-5e37d09eb5b9\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk" Oct 30 16:58:52.605354 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.605317 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk" Oct 30 16:58:52.605572 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:58:52.605540 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6ffa9d7_4de9_4b98_8e7e_016db7370f08.slice/crio-80a40a456651254920b6a1016e1ac5993359853d0a2258e25a886abceef83a52 WatchSource:0}: Error finding container 80a40a456651254920b6a1016e1ac5993359853d0a2258e25a886abceef83a52: Status 404 returned error can't find the container with id 80a40a456651254920b6a1016e1ac5993359853d0a2258e25a886abceef83a52 Oct 30 16:58:52.605852 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.605830 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7"] Oct 30 16:58:52.617648 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.617607 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:58:52.617762 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.617718 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:58:52.617825 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.617757 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:58:52.925614 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.925587 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk"] Oct 30 16:58:52.926025 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:58:52.925997 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93447f05_2d76_440f_b704_5e37d09eb5b9.slice/crio-78d7bab050147b824a53c070a69eca1e90f2d1574e9beaee8f864d06d90a6639 WatchSource:0}: Error finding container 78d7bab050147b824a53c070a69eca1e90f2d1574e9beaee8f864d06d90a6639: Status 404 returned error can't find the container with id 78d7bab050147b824a53c070a69eca1e90f2d1574e9beaee8f864d06d90a6639 Oct 30 16:58:52.928680 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.928646 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:58:52.928748 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.928725 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:58:52.928790 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:52.928763 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:58:53.054209 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:53.054168 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7" event={"ID":"c6ffa9d7-4de9-4b98-8e7e-016db7370f08","Type":"ContainerStarted","Data":"325d2d8270cefdb0a5d94cf6e729d4271779e426379459d5f6628641cf644b40"} Oct 30 16:58:53.054594 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:53.054233 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7" event={"ID":"c6ffa9d7-4de9-4b98-8e7e-016db7370f08","Type":"ContainerStarted","Data":"80a40a456651254920b6a1016e1ac5993359853d0a2258e25a886abceef83a52"} Oct 30 16:58:53.055870 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:53.055839 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf" event={"ID":"b590c8f6-9d91-49bb-8e0f-5e65eb24e631","Type":"ContainerStarted","Data":"29549c3eb9f4e617a2608e54db8fa9e328f8a3843ab700753cb96f333c593ef8"} Oct 30 16:58:53.055990 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:53.055898 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf" event={"ID":"b590c8f6-9d91-49bb-8e0f-5e65eb24e631","Type":"ContainerStarted","Data":"0419c901e7364525b1adff66cb8e77a30b904632c755cebe76966148b7cf5224"} Oct 30 16:58:53.057459 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:53.057433 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk" event={"ID":"93447f05-2d76-440f-b704-5e37d09eb5b9","Type":"ContainerStarted","Data":"388ee59b65c02b896fabf0d8713ff08db363affa92507f67f24e34bd21ed3bcb"} Oct 30 16:58:53.057568 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:53.057482 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk" event={"ID":"93447f05-2d76-440f-b704-5e37d09eb5b9","Type":"ContainerStarted","Data":"78d7bab050147b824a53c070a69eca1e90f2d1574e9beaee8f864d06d90a6639"} Oct 30 16:58:53.108263 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:53.108210 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7" podStartSLOduration=2.108196037 podStartE2EDuration="2.108196037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:58:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:58:53.107557264 +0000 UTC m=+707.108312192" watchObservedRunningTime="2026-10-30 16:58:53.108196037 +0000 UTC m=+707.108950941" Oct 30 16:58:53.156706 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:53.156655 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf" podStartSLOduration=2.156643017 podStartE2EDuration="2.156643017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:58:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:58:53.155549755 +0000 UTC m=+707.156304668" watchObservedRunningTime="2026-10-30 16:58:53.156643017 +0000 UTC m=+707.157397918" Oct 30 16:58:53.206220 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:53.206166 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk" podStartSLOduration=1.206150781 podStartE2EDuration="1.206150781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:58:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:58:53.204215766 +0000 UTC m=+707.204970667" watchObservedRunningTime="2026-10-30 16:58:53.206150781 +0000 UTC m=+707.206905682" Oct 30 16:58:53.288398 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:53.288359 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7" Oct 30 16:58:53.293098 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:53.293071 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7" Oct 30 16:58:53.605532 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:53.605432 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk" Oct 30 16:58:53.610773 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:53.610748 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk" Oct 30 16:58:53.985380 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:53.985345 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf" Oct 30 16:58:53.990686 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:53.990651 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf" Oct 30 16:58:54.062110 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:54.062077 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf" Oct 30 16:58:54.062270 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:54.062137 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7" Oct 30 16:58:54.062270 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:54.062164 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk" Oct 30 16:58:54.062786 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:54.062764 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk" Oct 30 16:58:54.062875 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:54.062814 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7" Oct 30 16:58:54.062875 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:54.062849 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf" Oct 30 16:58:58.137016 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:58.136979 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-b80d83-58f8df6ff4-xt4dg"] Oct 30 16:58:58.141298 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:58.141268 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-b80d83-58f8df6ff4-xt4dg" Oct 30 16:58:58.163606 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:58.163572 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-b80d83-58f8df6ff4-xt4dg"] Oct 30 16:58:58.176771 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:58.176738 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-b80d83-dockercfg-2m7lr\"" Oct 30 16:58:58.176939 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:58.176924 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-b80d83\"" Oct 30 16:58:58.238055 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:58.238026 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n94df\" (UniqueName: \"kubernetes.io/projected/bfdddc9b-c6c6-4cb1-bdd6-1634e76c4825-kube-api-access-n94df\") pod \"payload-processing-e2e-worker-w3-b80d83-58f8df6ff4-xt4dg\" (UID: \"bfdddc9b-c6c6-4cb1-bdd6-1634e76c4825\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-b80d83-58f8df6ff4-xt4dg" Oct 30 16:58:58.238211 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:58.238090 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfdddc9b-c6c6-4cb1-bdd6-1634e76c4825-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-b80d83-58f8df6ff4-xt4dg\" (UID: \"bfdddc9b-c6c6-4cb1-bdd6-1634e76c4825\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-b80d83-58f8df6ff4-xt4dg" Oct 30 16:58:58.339236 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:58.339204 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfdddc9b-c6c6-4cb1-bdd6-1634e76c4825-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-b80d83-58f8df6ff4-xt4dg\" (UID: \"bfdddc9b-c6c6-4cb1-bdd6-1634e76c4825\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-b80d83-58f8df6ff4-xt4dg" Oct 30 16:58:58.339432 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:58.339278 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n94df\" (UniqueName: \"kubernetes.io/projected/bfdddc9b-c6c6-4cb1-bdd6-1634e76c4825-kube-api-access-n94df\") pod \"payload-processing-e2e-worker-w3-b80d83-58f8df6ff4-xt4dg\" (UID: \"bfdddc9b-c6c6-4cb1-bdd6-1634e76c4825\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-b80d83-58f8df6ff4-xt4dg" Oct 30 16:58:58.339955 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:58.339933 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfdddc9b-c6c6-4cb1-bdd6-1634e76c4825-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-b80d83-58f8df6ff4-xt4dg\" (UID: \"bfdddc9b-c6c6-4cb1-bdd6-1634e76c4825\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-b80d83-58f8df6ff4-xt4dg" Oct 30 16:58:58.366832 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:58.366807 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n94df\" (UniqueName: \"kubernetes.io/projected/bfdddc9b-c6c6-4cb1-bdd6-1634e76c4825-kube-api-access-n94df\") pod \"payload-processing-e2e-worker-w3-b80d83-58f8df6ff4-xt4dg\" (UID: \"bfdddc9b-c6c6-4cb1-bdd6-1634e76c4825\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-b80d83-58f8df6ff4-xt4dg" Oct 30 16:58:58.456035 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:58.456008 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-b80d83-58f8df6ff4-xt4dg" Oct 30 16:58:58.755852 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:58:58.755817 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfdddc9b_c6c6_4cb1_bdd6_1634e76c4825.slice/crio-6085afd8e31317b1fe94941995f246a5493eb2a7966a00d9f2302ade8bef3dab WatchSource:0}: Error finding container 6085afd8e31317b1fe94941995f246a5493eb2a7966a00d9f2302ade8bef3dab: Status 404 returned error can't find the container with id 6085afd8e31317b1fe94941995f246a5493eb2a7966a00d9f2302ade8bef3dab Oct 30 16:58:58.756325 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:58.756288 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-b80d83-58f8df6ff4-xt4dg"] Oct 30 16:58:59.088503 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:58:59.088423 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-b80d83-58f8df6ff4-xt4dg" event={"ID":"bfdddc9b-c6c6-4cb1-bdd6-1634e76c4825","Type":"ContainerStarted","Data":"6085afd8e31317b1fe94941995f246a5493eb2a7966a00d9f2302ade8bef3dab"} Oct 30 16:59:00.058922 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:00.058364 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-096d75-67ff8648-hmszh"] Oct 30 16:59:00.066086 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:00.065218 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-096d75-67ff8648-hmszh" Oct 30 16:59:00.080300 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:00.079034 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-096d75-67ff8648-hmszh"] Oct 30 16:59:00.089649 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:00.089618 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-096d75\"" Oct 30 16:59:00.089812 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:00.089749 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-096d75-dockercfg-q7rzx\"" Oct 30 16:59:00.160079 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:00.160042 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp7lf\" (UniqueName: \"kubernetes.io/projected/0e716095-6ba5-49f8-a79b-5ec7a226336b-kube-api-access-lp7lf\") pod \"payload-processing-e2e-worker-w0-096d75-67ff8648-hmszh\" (UID: \"0e716095-6ba5-49f8-a79b-5ec7a226336b\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-096d75-67ff8648-hmszh" Oct 30 16:59:00.160293 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:00.160096 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e716095-6ba5-49f8-a79b-5ec7a226336b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-096d75-67ff8648-hmszh\" (UID: \"0e716095-6ba5-49f8-a79b-5ec7a226336b\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-096d75-67ff8648-hmszh" Oct 30 16:59:00.261118 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:00.261081 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp7lf\" (UniqueName: \"kubernetes.io/projected/0e716095-6ba5-49f8-a79b-5ec7a226336b-kube-api-access-lp7lf\") pod \"payload-processing-e2e-worker-w0-096d75-67ff8648-hmszh\" (UID: \"0e716095-6ba5-49f8-a79b-5ec7a226336b\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-096d75-67ff8648-hmszh" Oct 30 16:59:00.261280 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:00.261138 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e716095-6ba5-49f8-a79b-5ec7a226336b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-096d75-67ff8648-hmszh\" (UID: \"0e716095-6ba5-49f8-a79b-5ec7a226336b\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-096d75-67ff8648-hmszh" Oct 30 16:59:00.261954 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:00.261922 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e716095-6ba5-49f8-a79b-5ec7a226336b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-096d75-67ff8648-hmszh\" (UID: \"0e716095-6ba5-49f8-a79b-5ec7a226336b\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-096d75-67ff8648-hmszh" Oct 30 16:59:00.291110 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:00.291083 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp7lf\" (UniqueName: \"kubernetes.io/projected/0e716095-6ba5-49f8-a79b-5ec7a226336b-kube-api-access-lp7lf\") pod \"payload-processing-e2e-worker-w0-096d75-67ff8648-hmszh\" (UID: \"0e716095-6ba5-49f8-a79b-5ec7a226336b\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-096d75-67ff8648-hmszh" Oct 30 16:59:00.385824 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:00.385740 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-096d75-67ff8648-hmszh" Oct 30 16:59:00.707679 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:59:00.707648 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e716095_6ba5_49f8_a79b_5ec7a226336b.slice/crio-1b3dfe4e6b9028d0a5dbeb284e32c332004fa1e558f73c618846e2f36b7adbd2 WatchSource:0}: Error finding container 1b3dfe4e6b9028d0a5dbeb284e32c332004fa1e558f73c618846e2f36b7adbd2: Status 404 returned error can't find the container with id 1b3dfe4e6b9028d0a5dbeb284e32c332004fa1e558f73c618846e2f36b7adbd2 Oct 30 16:59:00.708341 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:00.708300 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-096d75-67ff8648-hmszh"] Oct 30 16:59:00.954254 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:00.954229 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-b4f406-786855ccd7-ns4j8"] Oct 30 16:59:00.959217 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:00.959196 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b4f406-786855ccd7-ns4j8" Oct 30 16:59:00.981733 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:00.981710 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-b4f406-dockercfg-mrk9k\"" Oct 30 16:59:00.982996 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:00.982971 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-b4f406-786855ccd7-ns4j8"] Oct 30 16:59:00.993847 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:00.993823 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-b4f406\"" Oct 30 16:59:01.046470 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:01.046431 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-b4f406-6d66f86b4b-bzcnr"] Oct 30 16:59:01.050703 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:01.050680 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-b4f406-6d66f86b4b-bzcnr" Oct 30 16:59:01.069634 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:01.069602 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzwnq\" (UniqueName: \"kubernetes.io/projected/246bcd4d-bddf-466e-bf5b-9f18b81fa5fd-kube-api-access-qzwnq\") pod \"payload-pre-processing-e2e-worker-w2-b4f406-786855ccd7-ns4j8\" (UID: \"246bcd4d-bddf-466e-bf5b-9f18b81fa5fd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b4f406-786855ccd7-ns4j8" Oct 30 16:59:01.070093 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:01.069727 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/246bcd4d-bddf-466e-bf5b-9f18b81fa5fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-b4f406-786855ccd7-ns4j8\" (UID: \"246bcd4d-bddf-466e-bf5b-9f18b81fa5fd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b4f406-786855ccd7-ns4j8" Oct 30 16:59:01.073870 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:01.073843 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-b4f406-6d66f86b4b-bzcnr"] Oct 30 16:59:01.102410 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:01.102380 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-b80d83-58f8df6ff4-xt4dg" event={"ID":"bfdddc9b-c6c6-4cb1-bdd6-1634e76c4825","Type":"ContainerStarted","Data":"dc92488ee977f9f3b44786c79d31b230099d2240d2a2a8f8c60abf0e04eb98e1"} Oct 30 16:59:01.102532 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:01.102445 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-b80d83-58f8df6ff4-xt4dg" Oct 30 16:59:01.103707 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:01.103684 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-096d75-67ff8648-hmszh" event={"ID":"0e716095-6ba5-49f8-a79b-5ec7a226336b","Type":"ContainerStarted","Data":"1b3dfe4e6b9028d0a5dbeb284e32c332004fa1e558f73c618846e2f36b7adbd2"} Oct 30 16:59:01.157594 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:01.157551 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-b80d83-58f8df6ff4-xt4dg" podStartSLOduration=0.973057539 podStartE2EDuration="3.157537685s" totalImagesPullingTime="2.184480146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:58:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:59:01.154361679 +0000 UTC m=+715.155116581" watchObservedRunningTime="2026-10-30 16:59:01.157537685 +0000 UTC m=+715.158292587" Oct 30 16:59:01.170484 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:01.170458 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/519e4a59-cb60-4a1a-949b-56018ea6b89c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-b4f406-6d66f86b4b-bzcnr\" (UID: \"519e4a59-cb60-4a1a-949b-56018ea6b89c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-b4f406-6d66f86b4b-bzcnr" Oct 30 16:59:01.170484 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:01.170486 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4w72\" (UniqueName: \"kubernetes.io/projected/519e4a59-cb60-4a1a-949b-56018ea6b89c-kube-api-access-h4w72\") pod \"payload-processing-e2e-worker-w2-b4f406-6d66f86b4b-bzcnr\" (UID: \"519e4a59-cb60-4a1a-949b-56018ea6b89c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-b4f406-6d66f86b4b-bzcnr" Oct 30 16:59:01.170653 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:01.170632 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzwnq\" (UniqueName: \"kubernetes.io/projected/246bcd4d-bddf-466e-bf5b-9f18b81fa5fd-kube-api-access-qzwnq\") pod \"payload-pre-processing-e2e-worker-w2-b4f406-786855ccd7-ns4j8\" (UID: \"246bcd4d-bddf-466e-bf5b-9f18b81fa5fd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b4f406-786855ccd7-ns4j8" Oct 30 16:59:01.170735 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:01.170717 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/246bcd4d-bddf-466e-bf5b-9f18b81fa5fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-b4f406-786855ccd7-ns4j8\" (UID: \"246bcd4d-bddf-466e-bf5b-9f18b81fa5fd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b4f406-786855ccd7-ns4j8" Oct 30 16:59:01.171331 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:01.171284 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/246bcd4d-bddf-466e-bf5b-9f18b81fa5fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-b4f406-786855ccd7-ns4j8\" (UID: \"246bcd4d-bddf-466e-bf5b-9f18b81fa5fd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b4f406-786855ccd7-ns4j8" Oct 30 16:59:01.200198 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:01.200164 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzwnq\" (UniqueName: \"kubernetes.io/projected/246bcd4d-bddf-466e-bf5b-9f18b81fa5fd-kube-api-access-qzwnq\") pod \"payload-pre-processing-e2e-worker-w2-b4f406-786855ccd7-ns4j8\" (UID: \"246bcd4d-bddf-466e-bf5b-9f18b81fa5fd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b4f406-786855ccd7-ns4j8" Oct 30 16:59:01.271731 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:01.271651 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/519e4a59-cb60-4a1a-949b-56018ea6b89c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-b4f406-6d66f86b4b-bzcnr\" (UID: \"519e4a59-cb60-4a1a-949b-56018ea6b89c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-b4f406-6d66f86b4b-bzcnr" Oct 30 16:59:01.271731 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:01.271688 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4w72\" (UniqueName: \"kubernetes.io/projected/519e4a59-cb60-4a1a-949b-56018ea6b89c-kube-api-access-h4w72\") pod \"payload-processing-e2e-worker-w2-b4f406-6d66f86b4b-bzcnr\" (UID: \"519e4a59-cb60-4a1a-949b-56018ea6b89c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-b4f406-6d66f86b4b-bzcnr" Oct 30 16:59:01.272483 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:01.272463 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/519e4a59-cb60-4a1a-949b-56018ea6b89c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-b4f406-6d66f86b4b-bzcnr\" (UID: \"519e4a59-cb60-4a1a-949b-56018ea6b89c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-b4f406-6d66f86b4b-bzcnr" Oct 30 16:59:01.288470 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:01.288443 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b4f406-786855ccd7-ns4j8" Oct 30 16:59:01.297248 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:01.297220 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4w72\" (UniqueName: \"kubernetes.io/projected/519e4a59-cb60-4a1a-949b-56018ea6b89c-kube-api-access-h4w72\") pod \"payload-processing-e2e-worker-w2-b4f406-6d66f86b4b-bzcnr\" (UID: \"519e4a59-cb60-4a1a-949b-56018ea6b89c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-b4f406-6d66f86b4b-bzcnr" Oct 30 16:59:01.368701 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:01.368663 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-b4f406-6d66f86b4b-bzcnr" Oct 30 16:59:01.601066 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:01.601039 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-b4f406-786855ccd7-ns4j8"] Oct 30 16:59:01.671552 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:01.671524 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-b4f406-6d66f86b4b-bzcnr"] Oct 30 16:59:01.672592 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:59:01.672561 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod519e4a59_cb60_4a1a_949b_56018ea6b89c.slice/crio-b58f5723477f9f02e998a1c35ff1758e1ecc49f5db5ea2aee90a88acf6a8cfd7 WatchSource:0}: Error finding container b58f5723477f9f02e998a1c35ff1758e1ecc49f5db5ea2aee90a88acf6a8cfd7: Status 404 returned error can't find the container with id b58f5723477f9f02e998a1c35ff1758e1ecc49f5db5ea2aee90a88acf6a8cfd7 Oct 30 16:59:01.988861 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:01.988824 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-d384dc-7f78968dd6-2w6cp"] Oct 30 16:59:01.993439 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:01.993409 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-d384dc-7f78968dd6-2w6cp" Oct 30 16:59:02.014334 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:02.012862 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-d384dc-7f78968dd6-2w6cp"] Oct 30 16:59:02.018085 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:02.018057 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-d384dc\"" Oct 30 16:59:02.023446 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:02.023418 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-d384dc-dockercfg-fl7bd\"" Oct 30 16:59:02.080203 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:02.080160 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6wr9\" (UniqueName: \"kubernetes.io/projected/acfdfc44-26dc-4fb0-a04a-1e15d28939ee-kube-api-access-j6wr9\") pod \"payload-processing-e2e-worker-w6-d384dc-7f78968dd6-2w6cp\" (UID: \"acfdfc44-26dc-4fb0-a04a-1e15d28939ee\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-d384dc-7f78968dd6-2w6cp" Oct 30 16:59:02.080644 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:02.080217 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acfdfc44-26dc-4fb0-a04a-1e15d28939ee-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-d384dc-7f78968dd6-2w6cp\" (UID: \"acfdfc44-26dc-4fb0-a04a-1e15d28939ee\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-d384dc-7f78968dd6-2w6cp" Oct 30 16:59:02.109247 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:02.109214 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-b4f406-6d66f86b4b-bzcnr" event={"ID":"519e4a59-cb60-4a1a-949b-56018ea6b89c","Type":"ContainerStarted","Data":"d1e6ff4af5ae577ebeb135947213d7c776786a28925de3b5ce16349c3ea38b23"} Oct 30 16:59:02.109455 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:02.109272 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-b4f406-6d66f86b4b-bzcnr" event={"ID":"519e4a59-cb60-4a1a-949b-56018ea6b89c","Type":"ContainerStarted","Data":"b58f5723477f9f02e998a1c35ff1758e1ecc49f5db5ea2aee90a88acf6a8cfd7"} Oct 30 16:59:02.109455 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:02.109300 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-b4f406-6d66f86b4b-bzcnr" Oct 30 16:59:02.110739 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:02.110710 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-096d75-67ff8648-hmszh" event={"ID":"0e716095-6ba5-49f8-a79b-5ec7a226336b","Type":"ContainerStarted","Data":"1bc673e20285e377f207342390515b137239392b2cfb111957e6a7214a22fbd8"} Oct 30 16:59:02.110841 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:02.110769 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-096d75-67ff8648-hmszh" Oct 30 16:59:02.112154 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:02.112133 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b4f406-786855ccd7-ns4j8" event={"ID":"246bcd4d-bddf-466e-bf5b-9f18b81fa5fd","Type":"ContainerStarted","Data":"b64457195612210f19529d5ade53be8ee48a9839e49dc6d0e83feb62c511511d"} Oct 30 16:59:02.112239 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:02.112169 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b4f406-786855ccd7-ns4j8" Oct 30 16:59:02.112239 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:02.112190 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b4f406-786855ccd7-ns4j8" event={"ID":"246bcd4d-bddf-466e-bf5b-9f18b81fa5fd","Type":"ContainerStarted","Data":"10f7f7cbf2ea647a9b80b9166bcde0585f253affbab8e9e741c9810b89a7c21e"} Oct 30 16:59:02.153962 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:02.153909 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-b4f406-6d66f86b4b-bzcnr" podStartSLOduration=1.153895041 podStartE2EDuration="1.153895041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:59:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:59:02.153717354 +0000 UTC m=+716.154472268" watchObservedRunningTime="2026-10-30 16:59:02.153895041 +0000 UTC m=+716.154649936" Oct 30 16:59:02.181461 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:02.181242 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6wr9\" (UniqueName: \"kubernetes.io/projected/acfdfc44-26dc-4fb0-a04a-1e15d28939ee-kube-api-access-j6wr9\") pod \"payload-processing-e2e-worker-w6-d384dc-7f78968dd6-2w6cp\" (UID: \"acfdfc44-26dc-4fb0-a04a-1e15d28939ee\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-d384dc-7f78968dd6-2w6cp" Oct 30 16:59:02.181461 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:02.181371 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acfdfc44-26dc-4fb0-a04a-1e15d28939ee-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-d384dc-7f78968dd6-2w6cp\" (UID: \"acfdfc44-26dc-4fb0-a04a-1e15d28939ee\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-d384dc-7f78968dd6-2w6cp" Oct 30 16:59:02.182193 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:02.182173 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acfdfc44-26dc-4fb0-a04a-1e15d28939ee-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-d384dc-7f78968dd6-2w6cp\" (UID: \"acfdfc44-26dc-4fb0-a04a-1e15d28939ee\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-d384dc-7f78968dd6-2w6cp" Oct 30 16:59:02.201004 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:02.200953 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-096d75-67ff8648-hmszh" podStartSLOduration=1.810608604 podStartE2EDuration="2.200937572s" totalImagesPullingTime="390.328968ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:59:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:59:02.199476313 +0000 UTC m=+716.200231239" watchObservedRunningTime="2026-10-30 16:59:02.200937572 +0000 UTC m=+716.201692474" Oct 30 16:59:02.207824 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:02.207801 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6wr9\" (UniqueName: \"kubernetes.io/projected/acfdfc44-26dc-4fb0-a04a-1e15d28939ee-kube-api-access-j6wr9\") pod \"payload-processing-e2e-worker-w6-d384dc-7f78968dd6-2w6cp\" (UID: \"acfdfc44-26dc-4fb0-a04a-1e15d28939ee\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-d384dc-7f78968dd6-2w6cp" Oct 30 16:59:02.243771 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:02.243674 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b4f406-786855ccd7-ns4j8" podStartSLOduration=2.243660046 podStartE2EDuration="2.243660046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:59:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:59:02.242722024 +0000 UTC m=+716.243476925" watchObservedRunningTime="2026-10-30 16:59:02.243660046 +0000 UTC m=+716.244414947" Oct 30 16:59:02.320193 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:02.320161 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-d384dc-7f78968dd6-2w6cp" Oct 30 16:59:02.630751 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:02.630723 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-d384dc-7f78968dd6-2w6cp"] Oct 30 16:59:02.631693 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:59:02.631663 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacfdfc44_26dc_4fb0_a04a_1e15d28939ee.slice/crio-edce3f648687fc5ef79e948f4fe6a3d9a2cd7d25d0f8ae073c432dda5f80b280 WatchSource:0}: Error finding container edce3f648687fc5ef79e948f4fe6a3d9a2cd7d25d0f8ae073c432dda5f80b280: Status 404 returned error can't find the container with id edce3f648687fc5ef79e948f4fe6a3d9a2cd7d25d0f8ae073c432dda5f80b280 Oct 30 16:59:02.858755 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:02.858723 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-04248e-6dfd587cc9-n8brs"] Oct 30 16:59:02.862260 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:02.862242 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-04248e-6dfd587cc9-n8brs" Oct 30 16:59:02.881516 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:02.881447 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-04248e\"" Oct 30 16:59:02.881667 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:02.881542 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-04248e-dockercfg-82zpn\"" Oct 30 16:59:02.883740 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:02.883718 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-04248e-6dfd587cc9-n8brs"] Oct 30 16:59:02.994839 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:02.994804 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzldc\" (UniqueName: \"kubernetes.io/projected/80f8edc4-4e9f-435e-8518-e5f53d262030-kube-api-access-hzldc\") pod \"payload-pre-processing-e2e-worker-w1-04248e-6dfd587cc9-n8brs\" (UID: \"80f8edc4-4e9f-435e-8518-e5f53d262030\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-04248e-6dfd587cc9-n8brs" Oct 30 16:59:02.995015 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:02.994867 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80f8edc4-4e9f-435e-8518-e5f53d262030-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-04248e-6dfd587cc9-n8brs\" (UID: \"80f8edc4-4e9f-435e-8518-e5f53d262030\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-04248e-6dfd587cc9-n8brs" Oct 30 16:59:03.095927 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:03.095891 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzldc\" (UniqueName: \"kubernetes.io/projected/80f8edc4-4e9f-435e-8518-e5f53d262030-kube-api-access-hzldc\") pod \"payload-pre-processing-e2e-worker-w1-04248e-6dfd587cc9-n8brs\" (UID: \"80f8edc4-4e9f-435e-8518-e5f53d262030\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-04248e-6dfd587cc9-n8brs" Oct 30 16:59:03.096433 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:03.095940 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80f8edc4-4e9f-435e-8518-e5f53d262030-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-04248e-6dfd587cc9-n8brs\" (UID: \"80f8edc4-4e9f-435e-8518-e5f53d262030\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-04248e-6dfd587cc9-n8brs" Oct 30 16:59:03.096900 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:03.096875 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80f8edc4-4e9f-435e-8518-e5f53d262030-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-04248e-6dfd587cc9-n8brs\" (UID: \"80f8edc4-4e9f-435e-8518-e5f53d262030\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-04248e-6dfd587cc9-n8brs" Oct 30 16:59:03.118644 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:03.118614 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-d384dc-7f78968dd6-2w6cp" event={"ID":"acfdfc44-26dc-4fb0-a04a-1e15d28939ee","Type":"ContainerStarted","Data":"58feeccd2dd1800d66537dfb9c04583ebddc4ba7ad9e347347fea19d27946e55"} Oct 30 16:59:03.118814 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:03.118671 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-d384dc-7f78968dd6-2w6cp" event={"ID":"acfdfc44-26dc-4fb0-a04a-1e15d28939ee","Type":"ContainerStarted","Data":"edce3f648687fc5ef79e948f4fe6a3d9a2cd7d25d0f8ae073c432dda5f80b280"} Oct 30 16:59:03.118907 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:03.118882 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-d384dc-7f78968dd6-2w6cp" Oct 30 16:59:03.121164 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:03.121144 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzldc\" (UniqueName: \"kubernetes.io/projected/80f8edc4-4e9f-435e-8518-e5f53d262030-kube-api-access-hzldc\") pod \"payload-pre-processing-e2e-worker-w1-04248e-6dfd587cc9-n8brs\" (UID: \"80f8edc4-4e9f-435e-8518-e5f53d262030\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-04248e-6dfd587cc9-n8brs" Oct 30 16:59:03.165637 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:03.165545 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-d384dc-7f78968dd6-2w6cp" podStartSLOduration=2.1655309369999998 podStartE2EDuration="2.165530937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:59:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:59:03.164495129 +0000 UTC m=+717.165250030" watchObservedRunningTime="2026-10-30 16:59:03.165530937 +0000 UTC m=+717.166285838" Oct 30 16:59:03.175569 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:03.175541 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-04248e-6dfd587cc9-n8brs" Oct 30 16:59:03.479169 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:03.479131 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-04248e-6dfd587cc9-n8brs"] Oct 30 16:59:03.481009 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:59:03.480980 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80f8edc4_4e9f_435e_8518_e5f53d262030.slice/crio-19683125e7d40ca44571c833a5144ab874ecf23ac4eeb525c1f994117c00288d WatchSource:0}: Error finding container 19683125e7d40ca44571c833a5144ab874ecf23ac4eeb525c1f994117c00288d: Status 404 returned error can't find the container with id 19683125e7d40ca44571c833a5144ab874ecf23ac4eeb525c1f994117c00288d Oct 30 16:59:03.868248 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:03.868166 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c7b9df-7498894d5f-vk5db"] Oct 30 16:59:03.871686 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:03.871668 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c7b9df-7498894d5f-vk5db" Oct 30 16:59:03.890830 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:03.890803 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-c7b9df\"" Oct 30 16:59:03.891009 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:03.890927 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-c7b9df-dockercfg-djxvn\"" Oct 30 16:59:03.892802 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:03.892778 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c7b9df-7498894d5f-vk5db"] Oct 30 16:59:03.965330 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:03.963118 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c7b9df-56764597fb-q9kdh"] Oct 30 16:59:03.970800 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:03.970766 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c7b9df-56764597fb-q9kdh"] Oct 30 16:59:03.970956 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:03.970896 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-c7b9df-56764597fb-q9kdh" Oct 30 16:59:04.006768 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:04.006731 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a9cf6b7-657b-4e3a-8f83-2946c56c43c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-c7b9df-7498894d5f-vk5db\" (UID: \"5a9cf6b7-657b-4e3a-8f83-2946c56c43c5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c7b9df-7498894d5f-vk5db" Oct 30 16:59:04.006948 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:04.006789 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv4ph\" (UniqueName: \"kubernetes.io/projected/5a9cf6b7-657b-4e3a-8f83-2946c56c43c5-kube-api-access-lv4ph\") pod \"payload-pre-processing-e2e-worker-w4-c7b9df-7498894d5f-vk5db\" (UID: \"5a9cf6b7-657b-4e3a-8f83-2946c56c43c5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c7b9df-7498894d5f-vk5db" Oct 30 16:59:04.107693 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:04.107657 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv4ph\" (UniqueName: \"kubernetes.io/projected/5a9cf6b7-657b-4e3a-8f83-2946c56c43c5-kube-api-access-lv4ph\") pod \"payload-pre-processing-e2e-worker-w4-c7b9df-7498894d5f-vk5db\" (UID: \"5a9cf6b7-657b-4e3a-8f83-2946c56c43c5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c7b9df-7498894d5f-vk5db" Oct 30 16:59:04.108168 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:04.107706 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a7fe95c-0ed3-4017-9002-20507d4c405c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-c7b9df-56764597fb-q9kdh\" (UID: \"9a7fe95c-0ed3-4017-9002-20507d4c405c\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c7b9df-56764597fb-q9kdh" Oct 30 16:59:04.108168 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:04.107800 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8xbg\" (UniqueName: \"kubernetes.io/projected/9a7fe95c-0ed3-4017-9002-20507d4c405c-kube-api-access-k8xbg\") pod \"payload-processing-e2e-worker-w4-c7b9df-56764597fb-q9kdh\" (UID: \"9a7fe95c-0ed3-4017-9002-20507d4c405c\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c7b9df-56764597fb-q9kdh" Oct 30 16:59:04.108168 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:04.107899 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a9cf6b7-657b-4e3a-8f83-2946c56c43c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-c7b9df-7498894d5f-vk5db\" (UID: \"5a9cf6b7-657b-4e3a-8f83-2946c56c43c5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c7b9df-7498894d5f-vk5db" Oct 30 16:59:04.108654 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:04.108634 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a9cf6b7-657b-4e3a-8f83-2946c56c43c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-c7b9df-7498894d5f-vk5db\" (UID: \"5a9cf6b7-657b-4e3a-8f83-2946c56c43c5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c7b9df-7498894d5f-vk5db" Oct 30 16:59:04.126438 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:04.126342 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-04248e-6dfd587cc9-n8brs" event={"ID":"80f8edc4-4e9f-435e-8518-e5f53d262030","Type":"ContainerStarted","Data":"ee44950c663dabf7409634f43dd79fe985e5ec6c00f06f95918d4042c0d9a567"} Oct 30 16:59:04.126438 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:04.126403 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-04248e-6dfd587cc9-n8brs" event={"ID":"80f8edc4-4e9f-435e-8518-e5f53d262030","Type":"ContainerStarted","Data":"19683125e7d40ca44571c833a5144ab874ecf23ac4eeb525c1f994117c00288d"} Oct 30 16:59:04.126650 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:04.126633 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-04248e-6dfd587cc9-n8brs" Oct 30 16:59:04.141405 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:04.141375 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv4ph\" (UniqueName: \"kubernetes.io/projected/5a9cf6b7-657b-4e3a-8f83-2946c56c43c5-kube-api-access-lv4ph\") pod \"payload-pre-processing-e2e-worker-w4-c7b9df-7498894d5f-vk5db\" (UID: \"5a9cf6b7-657b-4e3a-8f83-2946c56c43c5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c7b9df-7498894d5f-vk5db" Oct 30 16:59:04.171586 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:04.171517 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-04248e-6dfd587cc9-n8brs" podStartSLOduration=2.171496392 podStartE2EDuration="2.171496392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:59:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:59:04.170169004 +0000 UTC m=+718.170923926" watchObservedRunningTime="2026-10-30 16:59:04.171496392 +0000 UTC m=+718.172251294" Oct 30 16:59:04.184804 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:04.184759 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c7b9df-7498894d5f-vk5db" Oct 30 16:59:04.208640 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:04.208601 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8xbg\" (UniqueName: \"kubernetes.io/projected/9a7fe95c-0ed3-4017-9002-20507d4c405c-kube-api-access-k8xbg\") pod \"payload-processing-e2e-worker-w4-c7b9df-56764597fb-q9kdh\" (UID: \"9a7fe95c-0ed3-4017-9002-20507d4c405c\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c7b9df-56764597fb-q9kdh" Oct 30 16:59:04.208871 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:04.208845 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a7fe95c-0ed3-4017-9002-20507d4c405c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-c7b9df-56764597fb-q9kdh\" (UID: \"9a7fe95c-0ed3-4017-9002-20507d4c405c\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c7b9df-56764597fb-q9kdh" Oct 30 16:59:04.209655 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:04.209629 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a7fe95c-0ed3-4017-9002-20507d4c405c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-c7b9df-56764597fb-q9kdh\" (UID: \"9a7fe95c-0ed3-4017-9002-20507d4c405c\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c7b9df-56764597fb-q9kdh" Oct 30 16:59:04.234730 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:04.234663 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8xbg\" (UniqueName: \"kubernetes.io/projected/9a7fe95c-0ed3-4017-9002-20507d4c405c-kube-api-access-k8xbg\") pod \"payload-processing-e2e-worker-w4-c7b9df-56764597fb-q9kdh\" (UID: \"9a7fe95c-0ed3-4017-9002-20507d4c405c\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c7b9df-56764597fb-q9kdh" Oct 30 16:59:04.287721 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:04.287570 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-c7b9df-56764597fb-q9kdh" Oct 30 16:59:04.503272 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:59:04.503244 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a9cf6b7_657b_4e3a_8f83_2946c56c43c5.slice/crio-8ff0e410f9add26d3d1ea6e115f040c0ae3d5c19429b093d92c2775a039a4526 WatchSource:0}: Error finding container 8ff0e410f9add26d3d1ea6e115f040c0ae3d5c19429b093d92c2775a039a4526: Status 404 returned error can't find the container with id 8ff0e410f9add26d3d1ea6e115f040c0ae3d5c19429b093d92c2775a039a4526 Oct 30 16:59:04.503686 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:04.503665 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c7b9df-7498894d5f-vk5db"] Oct 30 16:59:04.620170 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:59:04.620139 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a7fe95c_0ed3_4017_9002_20507d4c405c.slice/crio-9953ca2d7c2082e8a9b1e3dbdba99facd0c59ca4bef34deb4f2cb26be0968da0 WatchSource:0}: Error finding container 9953ca2d7c2082e8a9b1e3dbdba99facd0c59ca4bef34deb4f2cb26be0968da0: Status 404 returned error can't find the container with id 9953ca2d7c2082e8a9b1e3dbdba99facd0c59ca4bef34deb4f2cb26be0968da0 Oct 30 16:59:04.620783 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:04.620752 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c7b9df-56764597fb-q9kdh"] Oct 30 16:59:05.135429 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:05.135390 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c7b9df-7498894d5f-vk5db" event={"ID":"5a9cf6b7-657b-4e3a-8f83-2946c56c43c5","Type":"ContainerStarted","Data":"f7d72e766210ac59bbddd541aba9c085d7f533e58310b5ddd456a3b053dfa4ca"} Oct 30 16:59:05.135922 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:05.135459 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c7b9df-7498894d5f-vk5db" event={"ID":"5a9cf6b7-657b-4e3a-8f83-2946c56c43c5","Type":"ContainerStarted","Data":"8ff0e410f9add26d3d1ea6e115f040c0ae3d5c19429b093d92c2775a039a4526"} Oct 30 16:59:05.135922 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:05.135581 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c7b9df-7498894d5f-vk5db" Oct 30 16:59:05.137238 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:05.137211 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-c7b9df-56764597fb-q9kdh" event={"ID":"9a7fe95c-0ed3-4017-9002-20507d4c405c","Type":"ContainerStarted","Data":"6d4e171435498f7a9813960b8eb2ec2cb61b1be3334916615f8c014ba1dab57a"} Oct 30 16:59:05.137388 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:05.137256 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-c7b9df-56764597fb-q9kdh" event={"ID":"9a7fe95c-0ed3-4017-9002-20507d4c405c","Type":"ContainerStarted","Data":"9953ca2d7c2082e8a9b1e3dbdba99facd0c59ca4bef34deb4f2cb26be0968da0"} Oct 30 16:59:05.179488 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:05.179439 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c7b9df-7498894d5f-vk5db" podStartSLOduration=2.179426357 podStartE2EDuration="2.179426357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:59:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:59:05.178836876 +0000 UTC m=+719.179591802" watchObservedRunningTime="2026-10-30 16:59:05.179426357 +0000 UTC m=+719.180181252" Oct 30 16:59:05.223654 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:05.223602 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-c7b9df-56764597fb-q9kdh" podStartSLOduration=2.223589025 podStartE2EDuration="2.223589025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:59:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:59:05.222378624 +0000 UTC m=+719.223133550" watchObservedRunningTime="2026-10-30 16:59:05.223589025 +0000 UTC m=+719.224343925" Oct 30 16:59:06.144640 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:06.144611 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-c7b9df-56764597fb-q9kdh" Oct 30 16:59:12.148290 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:12.148254 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-b80d83-58f8df6ff4-xt4dg" Oct 30 16:59:12.968491 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:59:12.968452 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 16:59:13.137883 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:13.137855 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-b4f406-6d66f86b4b-bzcnr" Oct 30 16:59:13.142117 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:13.141481 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-096d75-67ff8648-hmszh" Oct 30 16:59:13.143460 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:13.143435 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b4f406-786855ccd7-ns4j8" Oct 30 16:59:14.150058 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:14.150029 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-d384dc-7f78968dd6-2w6cp" Oct 30 16:59:15.125852 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:15.125818 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7b9df-59f556c645-8f7kp"] Oct 30 16:59:15.130868 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:15.130848 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7b9df-59f556c645-8f7kp" Oct 30 16:59:15.141141 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:15.141113 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-04248e-6dfd587cc9-n8brs" Oct 30 16:59:15.164777 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:15.164743 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:59:15.164777 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:15.164747 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:59:15.164777 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:15.164743 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-c7b9df-serving-cert\"" Oct 30 16:59:15.165411 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:15.164826 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cdj9d\"" Oct 30 16:59:15.165674 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:15.165655 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 16:59:15.167400 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:15.167379 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7b9df-59f556c645-8f7kp"] Oct 30 16:59:15.301128 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:15.301089 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f5a1cdd-9232-496b-8838-a21ec0cc78d3-maas-api-tls\") pod \"maas-api-e2e-worker-w4-c7b9df-59f556c645-8f7kp\" (UID: \"6f5a1cdd-9232-496b-8838-a21ec0cc78d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7b9df-59f556c645-8f7kp" Oct 30 16:59:15.301771 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:15.301745 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f5a1cdd-9232-496b-8838-a21ec0cc78d3-metrics-certs\") pod \"maas-api-e2e-worker-w4-c7b9df-59f556c645-8f7kp\" (UID: \"6f5a1cdd-9232-496b-8838-a21ec0cc78d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7b9df-59f556c645-8f7kp" Oct 30 16:59:15.308421 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:15.308381 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4xd2\" (UniqueName: \"kubernetes.io/projected/6f5a1cdd-9232-496b-8838-a21ec0cc78d3-kube-api-access-x4xd2\") pod \"maas-api-e2e-worker-w4-c7b9df-59f556c645-8f7kp\" (UID: \"6f5a1cdd-9232-496b-8838-a21ec0cc78d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7b9df-59f556c645-8f7kp" Oct 30 16:59:15.308704 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:15.308682 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f5a1cdd-9232-496b-8838-a21ec0cc78d3-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-c7b9df-59f556c645-8f7kp\" (UID: \"6f5a1cdd-9232-496b-8838-a21ec0cc78d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7b9df-59f556c645-8f7kp" Oct 30 16:59:15.415572 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:15.415487 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f5a1cdd-9232-496b-8838-a21ec0cc78d3-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-c7b9df-59f556c645-8f7kp\" (UID: \"6f5a1cdd-9232-496b-8838-a21ec0cc78d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7b9df-59f556c645-8f7kp" Oct 30 16:59:15.416451 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:15.416417 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f5a1cdd-9232-496b-8838-a21ec0cc78d3-maas-api-tls\") pod \"maas-api-e2e-worker-w4-c7b9df-59f556c645-8f7kp\" (UID: \"6f5a1cdd-9232-496b-8838-a21ec0cc78d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7b9df-59f556c645-8f7kp" Oct 30 16:59:15.416619 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:15.416599 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f5a1cdd-9232-496b-8838-a21ec0cc78d3-metrics-certs\") pod \"maas-api-e2e-worker-w4-c7b9df-59f556c645-8f7kp\" (UID: \"6f5a1cdd-9232-496b-8838-a21ec0cc78d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7b9df-59f556c645-8f7kp" Oct 30 16:59:15.416846 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:15.416758 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4xd2\" (UniqueName: \"kubernetes.io/projected/6f5a1cdd-9232-496b-8838-a21ec0cc78d3-kube-api-access-x4xd2\") pod \"maas-api-e2e-worker-w4-c7b9df-59f556c645-8f7kp\" (UID: \"6f5a1cdd-9232-496b-8838-a21ec0cc78d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7b9df-59f556c645-8f7kp" Oct 30 16:59:15.419254 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:15.419229 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f5a1cdd-9232-496b-8838-a21ec0cc78d3-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-c7b9df-59f556c645-8f7kp\" (UID: \"6f5a1cdd-9232-496b-8838-a21ec0cc78d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7b9df-59f556c645-8f7kp" Oct 30 16:59:15.423227 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:15.423204 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f5a1cdd-9232-496b-8838-a21ec0cc78d3-metrics-certs\") pod \"maas-api-e2e-worker-w4-c7b9df-59f556c645-8f7kp\" (UID: \"6f5a1cdd-9232-496b-8838-a21ec0cc78d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7b9df-59f556c645-8f7kp" Oct 30 16:59:15.425028 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:15.425003 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f5a1cdd-9232-496b-8838-a21ec0cc78d3-maas-api-tls\") pod \"maas-api-e2e-worker-w4-c7b9df-59f556c645-8f7kp\" (UID: \"6f5a1cdd-9232-496b-8838-a21ec0cc78d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7b9df-59f556c645-8f7kp" Oct 30 16:59:15.443339 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:15.443287 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4xd2\" (UniqueName: \"kubernetes.io/projected/6f5a1cdd-9232-496b-8838-a21ec0cc78d3-kube-api-access-x4xd2\") pod \"maas-api-e2e-worker-w4-c7b9df-59f556c645-8f7kp\" (UID: \"6f5a1cdd-9232-496b-8838-a21ec0cc78d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7b9df-59f556c645-8f7kp" Oct 30 16:59:15.445917 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:15.445879 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7b9df-59f556c645-8f7kp" Oct 30 16:59:15.790340 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:15.785889 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7b9df-59f556c645-8f7kp"] Oct 30 16:59:15.790340 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:59:15.786074 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f5a1cdd_9232_496b_8838_a21ec0cc78d3.slice/crio-1caf7508178b34c2925dd7f7536ba16739159f30243618b63f13d4e6d856d4dd WatchSource:0}: Error finding container 1caf7508178b34c2925dd7f7536ba16739159f30243618b63f13d4e6d856d4dd: Status 404 returned error can't find the container with id 1caf7508178b34c2925dd7f7536ba16739159f30243618b63f13d4e6d856d4dd Oct 30 16:59:15.790340 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:15.787636 4522 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 16:59:16.148285 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:16.148207 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c7b9df-7498894d5f-vk5db" Oct 30 16:59:16.321331 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:16.308828 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7b9df-59f556c645-8f7kp" event={"ID":"6f5a1cdd-9232-496b-8838-a21ec0cc78d3","Type":"ContainerStarted","Data":"1caf7508178b34c2925dd7f7536ba16739159f30243618b63f13d4e6d856d4dd"} Oct 30 16:59:17.154080 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:17.154047 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-c7b9df-56764597fb-q9kdh" Oct 30 16:59:20.351344 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:20.351286 4522 generic.go:356] "Generic (PLEG): container finished" podID="c6ffa9d7-4de9-4b98-8e7e-016db7370f08" containerID="325d2d8270cefdb0a5d94cf6e729d4271779e426379459d5f6628641cf644b40" exitCode=0 Oct 30 16:59:20.351835 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:20.351361 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7" event={"ID":"c6ffa9d7-4de9-4b98-8e7e-016db7370f08","Type":"ContainerDied","Data":"325d2d8270cefdb0a5d94cf6e729d4271779e426379459d5f6628641cf644b40"} Oct 30 16:59:20.351835 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:20.351755 4522 scope.go:122] "RemoveContainer" containerID="325d2d8270cefdb0a5d94cf6e729d4271779e426379459d5f6628641cf644b40" Oct 30 16:59:20.353278 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:20.353256 4522 generic.go:356] "Generic (PLEG): container finished" podID="93447f05-2d76-440f-b704-5e37d09eb5b9" containerID="388ee59b65c02b896fabf0d8713ff08db363affa92507f67f24e34bd21ed3bcb" exitCode=0 Oct 30 16:59:20.353416 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:20.353298 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:59:20.353416 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:20.353343 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk" event={"ID":"93447f05-2d76-440f-b704-5e37d09eb5b9","Type":"ContainerDied","Data":"388ee59b65c02b896fabf0d8713ff08db363affa92507f67f24e34bd21ed3bcb"} Oct 30 16:59:20.353416 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:20.353406 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:59:20.353596 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:20.353456 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:59:20.353707 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:20.353688 4522 scope.go:122] "RemoveContainer" containerID="388ee59b65c02b896fabf0d8713ff08db363affa92507f67f24e34bd21ed3bcb" Oct 30 16:59:20.354925 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:20.354895 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:59:20.355022 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:20.354964 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:59:20.355022 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:20.355006 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:59:20.355385 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:20.355348 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7b9df-59f556c645-8f7kp" event={"ID":"6f5a1cdd-9232-496b-8838-a21ec0cc78d3","Type":"ContainerStarted","Data":"73109e2519e55e28124cafb358ca74d3945a6bb45bbdf44481920314cf2ee263"} Oct 30 16:59:20.355574 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:20.355495 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7b9df-59f556c645-8f7kp" Oct 30 16:59:20.530384 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:20.529444 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7b9df-59f556c645-8f7kp" podStartSLOduration=1.974717174 podStartE2EDuration="5.529422187s" totalImagesPullingTime="3.554705013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:59:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:59:20.466560805 +0000 UTC m=+734.467315716" watchObservedRunningTime="2026-10-30 16:59:20.529422187 +0000 UTC m=+734.530177095" Oct 30 16:59:21.370007 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:21.369960 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7" event={"ID":"c6ffa9d7-4de9-4b98-8e7e-016db7370f08","Type":"ContainerStarted","Data":"a75994c0c66a7d2db476768b52b2221a36b163fdf6b06263a5613f123bf21b1d"} Oct 30 16:59:21.372643 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:21.372612 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk" event={"ID":"93447f05-2d76-440f-b704-5e37d09eb5b9","Type":"ContainerStarted","Data":"1058a199527188cb4606777166e1a6ebfd898bec47e834876f4e7e58ac3d6126"} Oct 30 16:59:21.649392 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:21.608637 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk" Oct 30 16:59:21.649392 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:21.609436 4522 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 16:59:21.649392 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:21.609492 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk" podUID="93447f05-2d76-440f-b704-5e37d09eb5b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 16:59:21.850328 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:21.850283 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz"] Oct 30 16:59:21.852719 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:21.852696 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz" Oct 30 16:59:21.888230 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:21.888186 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz"] Oct 30 16:59:21.956357 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:21.955057 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2010977-7978-4ee0-9b32-3449329c0b4d-istio-data\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz\" (UID: \"b2010977-7978-4ee0-9b32-3449329c0b4d\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz" Oct 30 16:59:21.956357 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:21.955121 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2010977-7978-4ee0-9b32-3449329c0b4d-workload-socket\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz\" (UID: \"b2010977-7978-4ee0-9b32-3449329c0b4d\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz" Oct 30 16:59:21.956357 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:21.955166 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2010977-7978-4ee0-9b32-3449329c0b4d-workload-certs\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz\" (UID: \"b2010977-7978-4ee0-9b32-3449329c0b4d\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz" Oct 30 16:59:21.956357 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:21.955206 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2010977-7978-4ee0-9b32-3449329c0b4d-istiod-ca-cert\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz\" (UID: \"b2010977-7978-4ee0-9b32-3449329c0b4d\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz" Oct 30 16:59:21.956357 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:21.955234 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx5x2\" (UniqueName: \"kubernetes.io/projected/b2010977-7978-4ee0-9b32-3449329c0b4d-kube-api-access-vx5x2\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz\" (UID: \"b2010977-7978-4ee0-9b32-3449329c0b4d\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz" Oct 30 16:59:21.956357 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:21.955264 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2010977-7978-4ee0-9b32-3449329c0b4d-istio-token\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz\" (UID: \"b2010977-7978-4ee0-9b32-3449329c0b4d\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz" Oct 30 16:59:21.956357 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:21.955334 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2010977-7978-4ee0-9b32-3449329c0b4d-credential-socket\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz\" (UID: \"b2010977-7978-4ee0-9b32-3449329c0b4d\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz" Oct 30 16:59:21.956357 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:21.955400 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2010977-7978-4ee0-9b32-3449329c0b4d-istio-podinfo\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz\" (UID: \"b2010977-7978-4ee0-9b32-3449329c0b4d\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz" Oct 30 16:59:21.956357 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:21.955457 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2010977-7978-4ee0-9b32-3449329c0b4d-istio-envoy\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz\" (UID: \"b2010977-7978-4ee0-9b32-3449329c0b4d\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz" Oct 30 16:59:22.034855 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.034813 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb"] Oct 30 16:59:22.046292 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.046263 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb" Oct 30 16:59:22.046889 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.046138 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77"] Oct 30 16:59:22.050083 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.050058 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77" Oct 30 16:59:22.058678 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.056821 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2010977-7978-4ee0-9b32-3449329c0b4d-istio-envoy\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz\" (UID: \"b2010977-7978-4ee0-9b32-3449329c0b4d\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz" Oct 30 16:59:22.058678 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.056887 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2010977-7978-4ee0-9b32-3449329c0b4d-istio-data\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz\" (UID: \"b2010977-7978-4ee0-9b32-3449329c0b4d\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz" Oct 30 16:59:22.058678 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.056922 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2010977-7978-4ee0-9b32-3449329c0b4d-workload-socket\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz\" (UID: \"b2010977-7978-4ee0-9b32-3449329c0b4d\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz" Oct 30 16:59:22.058678 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.057916 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2010977-7978-4ee0-9b32-3449329c0b4d-workload-certs\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz\" (UID: \"b2010977-7978-4ee0-9b32-3449329c0b4d\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz" Oct 30 16:59:22.058678 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.058338 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2010977-7978-4ee0-9b32-3449329c0b4d-istiod-ca-cert\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz\" (UID: \"b2010977-7978-4ee0-9b32-3449329c0b4d\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz" Oct 30 16:59:22.058678 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.058373 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx5x2\" (UniqueName: \"kubernetes.io/projected/b2010977-7978-4ee0-9b32-3449329c0b4d-kube-api-access-vx5x2\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz\" (UID: \"b2010977-7978-4ee0-9b32-3449329c0b4d\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz" Oct 30 16:59:22.058678 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.058398 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2010977-7978-4ee0-9b32-3449329c0b4d-istio-token\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz\" (UID: \"b2010977-7978-4ee0-9b32-3449329c0b4d\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz" Oct 30 16:59:22.058678 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.058641 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2010977-7978-4ee0-9b32-3449329c0b4d-credential-socket\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz\" (UID: \"b2010977-7978-4ee0-9b32-3449329c0b4d\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz" Oct 30 16:59:22.059164 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.058757 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2010977-7978-4ee0-9b32-3449329c0b4d-istio-podinfo\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz\" (UID: \"b2010977-7978-4ee0-9b32-3449329c0b4d\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz" Oct 30 16:59:22.059164 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.059028 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2010977-7978-4ee0-9b32-3449329c0b4d-istiod-ca-cert\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz\" (UID: \"b2010977-7978-4ee0-9b32-3449329c0b4d\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz" Oct 30 16:59:22.059164 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.058395 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2010977-7978-4ee0-9b32-3449329c0b4d-istio-data\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz\" (UID: \"b2010977-7978-4ee0-9b32-3449329c0b4d\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz" Oct 30 16:59:22.059347 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.059204 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2010977-7978-4ee0-9b32-3449329c0b4d-workload-socket\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz\" (UID: \"b2010977-7978-4ee0-9b32-3449329c0b4d\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz" Oct 30 16:59:22.059347 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.059330 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2010977-7978-4ee0-9b32-3449329c0b4d-workload-certs\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz\" (UID: \"b2010977-7978-4ee0-9b32-3449329c0b4d\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz" Oct 30 16:59:22.059680 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.059629 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2010977-7978-4ee0-9b32-3449329c0b4d-credential-socket\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz\" (UID: \"b2010977-7978-4ee0-9b32-3449329c0b4d\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz" Oct 30 16:59:22.060383 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.060363 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb"] Oct 30 16:59:22.064176 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.064143 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2010977-7978-4ee0-9b32-3449329c0b4d-istio-envoy\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz\" (UID: \"b2010977-7978-4ee0-9b32-3449329c0b4d\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz" Oct 30 16:59:22.064847 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.064781 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2010977-7978-4ee0-9b32-3449329c0b4d-istio-podinfo\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz\" (UID: \"b2010977-7978-4ee0-9b32-3449329c0b4d\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz" Oct 30 16:59:22.086807 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.086778 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-b4f406-openshift-default-dockercfg-ftgfs\"" Oct 30 16:59:22.086988 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.086965 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2010977-7978-4ee0-9b32-3449329c0b4d-istio-token\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz\" (UID: \"b2010977-7978-4ee0-9b32-3449329c0b4d\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz" Oct 30 16:59:22.087099 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.087075 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx5x2\" (UniqueName: \"kubernetes.io/projected/b2010977-7978-4ee0-9b32-3449329c0b4d-kube-api-access-vx5x2\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz\" (UID: \"b2010977-7978-4ee0-9b32-3449329c0b4d\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz" Oct 30 16:59:22.157435 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.157394 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77"] Oct 30 16:59:22.159952 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.159920 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfb55\" (UniqueName: \"kubernetes.io/projected/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46-kube-api-access-qfb55\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77\" (UID: \"bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77" Oct 30 16:59:22.160105 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.159991 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46-workload-socket\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77\" (UID: \"bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77" Oct 30 16:59:22.160105 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.160045 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46-istio-envoy\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77\" (UID: \"bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77" Oct 30 16:59:22.160105 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.160080 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46-istio-podinfo\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77\" (UID: \"bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77" Oct 30 16:59:22.160288 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.160129 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/534aeebe-4edf-425e-a16c-712cc498154b-istiod-ca-cert\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb\" (UID: \"534aeebe-4edf-425e-a16c-712cc498154b\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb" Oct 30 16:59:22.160288 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.160184 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/534aeebe-4edf-425e-a16c-712cc498154b-istio-envoy\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb\" (UID: \"534aeebe-4edf-425e-a16c-712cc498154b\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb" Oct 30 16:59:22.160288 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.160216 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/534aeebe-4edf-425e-a16c-712cc498154b-workload-socket\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb\" (UID: \"534aeebe-4edf-425e-a16c-712cc498154b\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb" Oct 30 16:59:22.160509 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.160297 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/534aeebe-4edf-425e-a16c-712cc498154b-workload-certs\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb\" (UID: \"534aeebe-4edf-425e-a16c-712cc498154b\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb" Oct 30 16:59:22.160509 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.160353 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/534aeebe-4edf-425e-a16c-712cc498154b-istio-podinfo\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb\" (UID: \"534aeebe-4edf-425e-a16c-712cc498154b\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb" Oct 30 16:59:22.160509 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.160396 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46-credential-socket\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77\" (UID: \"bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77" Oct 30 16:59:22.160509 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.160453 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46-istiod-ca-cert\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77\" (UID: \"bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77" Oct 30 16:59:22.160509 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.160484 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46-istio-data\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77\" (UID: \"bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77" Oct 30 16:59:22.160764 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.160527 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/534aeebe-4edf-425e-a16c-712cc498154b-istio-data\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb\" (UID: \"534aeebe-4edf-425e-a16c-712cc498154b\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb" Oct 30 16:59:22.160764 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.160553 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46-workload-certs\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77\" (UID: \"bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77" Oct 30 16:59:22.160764 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.160581 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46-istio-token\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77\" (UID: \"bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77" Oct 30 16:59:22.160764 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.160638 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/534aeebe-4edf-425e-a16c-712cc498154b-istio-token\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb\" (UID: \"534aeebe-4edf-425e-a16c-712cc498154b\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb" Oct 30 16:59:22.160764 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.160687 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/534aeebe-4edf-425e-a16c-712cc498154b-credential-socket\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb\" (UID: \"534aeebe-4edf-425e-a16c-712cc498154b\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb" Oct 30 16:59:22.160764 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.160734 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29tcv\" (UniqueName: \"kubernetes.io/projected/534aeebe-4edf-425e-a16c-712cc498154b-kube-api-access-29tcv\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb\" (UID: \"534aeebe-4edf-425e-a16c-712cc498154b\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb" Oct 30 16:59:22.167462 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.167437 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz" Oct 30 16:59:22.261997 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.261913 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/534aeebe-4edf-425e-a16c-712cc498154b-istio-envoy\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb\" (UID: \"534aeebe-4edf-425e-a16c-712cc498154b\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb" Oct 30 16:59:22.261997 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.261955 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/534aeebe-4edf-425e-a16c-712cc498154b-workload-socket\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb\" (UID: \"534aeebe-4edf-425e-a16c-712cc498154b\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb" Oct 30 16:59:22.261997 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.261993 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/534aeebe-4edf-425e-a16c-712cc498154b-workload-certs\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb\" (UID: \"534aeebe-4edf-425e-a16c-712cc498154b\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb" Oct 30 16:59:22.262201 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.262014 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/534aeebe-4edf-425e-a16c-712cc498154b-istio-podinfo\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb\" (UID: \"534aeebe-4edf-425e-a16c-712cc498154b\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb" Oct 30 16:59:22.262201 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.262039 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46-credential-socket\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77\" (UID: \"bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77" Oct 30 16:59:22.262201 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.262066 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46-istiod-ca-cert\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77\" (UID: \"bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77" Oct 30 16:59:22.262201 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.262085 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46-istio-data\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77\" (UID: \"bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77" Oct 30 16:59:22.262201 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.262109 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/534aeebe-4edf-425e-a16c-712cc498154b-istio-data\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb\" (UID: \"534aeebe-4edf-425e-a16c-712cc498154b\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb" Oct 30 16:59:22.262201 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.262126 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46-workload-certs\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77\" (UID: \"bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77" Oct 30 16:59:22.262201 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.262143 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46-istio-token\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77\" (UID: \"bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77" Oct 30 16:59:22.262201 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.262173 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/534aeebe-4edf-425e-a16c-712cc498154b-istio-token\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb\" (UID: \"534aeebe-4edf-425e-a16c-712cc498154b\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb" Oct 30 16:59:22.262201 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.262200 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/534aeebe-4edf-425e-a16c-712cc498154b-credential-socket\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb\" (UID: \"534aeebe-4edf-425e-a16c-712cc498154b\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb" Oct 30 16:59:22.262489 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.262227 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29tcv\" (UniqueName: \"kubernetes.io/projected/534aeebe-4edf-425e-a16c-712cc498154b-kube-api-access-29tcv\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb\" (UID: \"534aeebe-4edf-425e-a16c-712cc498154b\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb" Oct 30 16:59:22.262489 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.262243 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfb55\" (UniqueName: \"kubernetes.io/projected/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46-kube-api-access-qfb55\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77\" (UID: \"bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77" Oct 30 16:59:22.262489 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.262264 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46-workload-socket\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77\" (UID: \"bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77" Oct 30 16:59:22.262489 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.262280 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46-istio-envoy\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77\" (UID: \"bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77" Oct 30 16:59:22.262489 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.262298 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46-istio-podinfo\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77\" (UID: \"bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77" Oct 30 16:59:22.262489 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.262344 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/534aeebe-4edf-425e-a16c-712cc498154b-istiod-ca-cert\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb\" (UID: \"534aeebe-4edf-425e-a16c-712cc498154b\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb" Oct 30 16:59:22.263250 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.262977 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46-workload-certs\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77\" (UID: \"bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77" Oct 30 16:59:22.263427 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.263407 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/534aeebe-4edf-425e-a16c-712cc498154b-credential-socket\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb\" (UID: \"534aeebe-4edf-425e-a16c-712cc498154b\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb" Oct 30 16:59:22.263546 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.263518 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46-workload-socket\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77\" (UID: \"bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77" Oct 30 16:59:22.263710 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.263690 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/534aeebe-4edf-425e-a16c-712cc498154b-istiod-ca-cert\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb\" (UID: \"534aeebe-4edf-425e-a16c-712cc498154b\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb" Oct 30 16:59:22.263798 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.263779 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/534aeebe-4edf-425e-a16c-712cc498154b-istio-data\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb\" (UID: \"534aeebe-4edf-425e-a16c-712cc498154b\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb" Oct 30 16:59:22.263974 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.263954 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/534aeebe-4edf-425e-a16c-712cc498154b-workload-certs\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb\" (UID: \"534aeebe-4edf-425e-a16c-712cc498154b\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb" Oct 30 16:59:22.264099 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.264071 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46-credential-socket\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77\" (UID: \"bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77" Oct 30 16:59:22.264209 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.264171 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/534aeebe-4edf-425e-a16c-712cc498154b-workload-socket\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb\" (UID: \"534aeebe-4edf-425e-a16c-712cc498154b\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb" Oct 30 16:59:22.264423 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.264387 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46-istio-data\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77\" (UID: \"bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77" Oct 30 16:59:22.264959 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.264934 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46-istiod-ca-cert\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77\" (UID: \"bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77" Oct 30 16:59:22.265867 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.265834 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/534aeebe-4edf-425e-a16c-712cc498154b-istio-envoy\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb\" (UID: \"534aeebe-4edf-425e-a16c-712cc498154b\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb" Oct 30 16:59:22.266229 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.266213 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/534aeebe-4edf-425e-a16c-712cc498154b-istio-podinfo\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb\" (UID: \"534aeebe-4edf-425e-a16c-712cc498154b\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb" Oct 30 16:59:22.266397 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.266379 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46-istio-envoy\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77\" (UID: \"bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77" Oct 30 16:59:22.266795 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.266775 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46-istio-podinfo\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77\" (UID: \"bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77" Oct 30 16:59:22.288715 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.288641 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7" Oct 30 16:59:22.289829 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.289532 4522 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 16:59:22.289829 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.289585 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7" podUID="c6ffa9d7-4de9-4b98-8e7e-016db7370f08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 16:59:22.289997 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.289954 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/534aeebe-4edf-425e-a16c-712cc498154b-istio-token\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb\" (UID: \"534aeebe-4edf-425e-a16c-712cc498154b\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb" Oct 30 16:59:22.316448 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.296018 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfb55\" (UniqueName: \"kubernetes.io/projected/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46-kube-api-access-qfb55\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77\" (UID: \"bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77" Oct 30 16:59:22.336331 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.320045 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29tcv\" (UniqueName: \"kubernetes.io/projected/534aeebe-4edf-425e-a16c-712cc498154b-kube-api-access-29tcv\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb\" (UID: \"534aeebe-4edf-425e-a16c-712cc498154b\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb" Oct 30 16:59:22.336331 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.320392 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46-istio-token\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77\" (UID: \"bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77" Oct 30 16:59:22.368004 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.367969 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb" Oct 30 16:59:22.385266 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.385237 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77" Oct 30 16:59:22.576566 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.576534 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz"] Oct 30 16:59:22.577010 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:59:22.576977 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2010977_7978_4ee0_9b32_3449329c0b4d.slice/crio-990cda7f69c389d59b9174db2972f6f792f7f6a9bbf9bb65ed19c6b46f74a8c4 WatchSource:0}: Error finding container 990cda7f69c389d59b9174db2972f6f792f7f6a9bbf9bb65ed19c6b46f74a8c4: Status 404 returned error can't find the container with id 990cda7f69c389d59b9174db2972f6f792f7f6a9bbf9bb65ed19c6b46f74a8c4 Oct 30 16:59:22.579688 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.579661 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:59:22.579745 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.579720 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:59:22.579782 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.579748 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:59:22.647343 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.635423 4522 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 16:59:22.647343 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.635495 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk" podUID="93447f05-2d76-440f-b704-5e37d09eb5b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 16:59:22.775954 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:59:22.774616 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod534aeebe_4edf_425e_a16c_712cc498154b.slice/crio-25677683e9a4cd4b17e3ad740bc10af5d6c5e4482c96a58783f04cb5438479d9 WatchSource:0}: Error finding container 25677683e9a4cd4b17e3ad740bc10af5d6c5e4482c96a58783f04cb5438479d9: Status 404 returned error can't find the container with id 25677683e9a4cd4b17e3ad740bc10af5d6c5e4482c96a58783f04cb5438479d9 Oct 30 16:59:22.782374 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.782333 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb"] Oct 30 16:59:22.782610 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.782584 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77"] Oct 30 16:59:22.784037 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.783987 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:59:22.784160 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.784068 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:59:22.784160 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.783987 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:59:22.784160 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.784110 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:59:22.784343 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.784166 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:59:22.784343 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:22.784212 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:59:23.289011 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:23.288974 4522 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 16:59:23.289273 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:23.289244 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7" podUID="c6ffa9d7-4de9-4b98-8e7e-016db7370f08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 16:59:23.390339 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:23.389537 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb" event={"ID":"534aeebe-4edf-425e-a16c-712cc498154b","Type":"ContainerStarted","Data":"87822abdd74a876394d196ef8960d7bb1d259403aac446551fb7fc8fe3b35e9e"} Oct 30 16:59:23.390339 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:23.389643 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb" event={"ID":"534aeebe-4edf-425e-a16c-712cc498154b","Type":"ContainerStarted","Data":"25677683e9a4cd4b17e3ad740bc10af5d6c5e4482c96a58783f04cb5438479d9"} Oct 30 16:59:23.397335 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:23.395551 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77" event={"ID":"bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46","Type":"ContainerStarted","Data":"d695d95539345ee85b4ebf5ed1bb5cc069868b79a2cba30654927701839d9f5a"} Oct 30 16:59:23.397335 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:23.395614 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77" event={"ID":"bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46","Type":"ContainerStarted","Data":"2333e5e5184282dfe451f443d30fae98c23ea2967cbf44bb7ffadeb5202f39d4"} Oct 30 16:59:23.402329 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:23.398690 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz" event={"ID":"b2010977-7978-4ee0-9b32-3449329c0b4d","Type":"ContainerStarted","Data":"46a704c49374a5a30045b654592d5236cd5dfd6d0886689f0268884bd1e84237"} Oct 30 16:59:23.402329 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:23.398748 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz" event={"ID":"b2010977-7978-4ee0-9b32-3449329c0b4d","Type":"ContainerStarted","Data":"990cda7f69c389d59b9174db2972f6f792f7f6a9bbf9bb65ed19c6b46f74a8c4"} Oct 30 16:59:23.450301 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:23.450243 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb" podStartSLOduration=2.450207541 podStartE2EDuration="2.450207541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:59:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:59:23.443912151 +0000 UTC m=+737.444667078" watchObservedRunningTime="2026-10-30 16:59:23.450207541 +0000 UTC m=+737.450962443" Oct 30 16:59:23.507641 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:23.503377 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77" podStartSLOduration=1.5033575940000001 podStartE2EDuration="1.503357594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:59:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:59:23.502822331 +0000 UTC m=+737.503577233" watchObservedRunningTime="2026-10-30 16:59:23.503357594 +0000 UTC m=+737.504112497" Oct 30 16:59:23.616814 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:23.616725 4522 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 16:59:23.617051 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:23.617024 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk" podUID="93447f05-2d76-440f-b704-5e37d09eb5b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 16:59:24.062828 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:24.061970 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk" Oct 30 16:59:24.064650 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:24.063985 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7" Oct 30 16:59:24.170397 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:24.170267 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz" Oct 30 16:59:24.183370 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:24.183337 4522 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 30 16:59:24.183680 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:24.183630 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz" podUID="b2010977-7978-4ee0-9b32-3449329c0b4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 30 16:59:24.291543 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:24.291050 4522 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 16:59:24.291543 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:24.291109 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7" podUID="c6ffa9d7-4de9-4b98-8e7e-016db7370f08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 16:59:24.374261 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:24.372387 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb" Oct 30 16:59:24.374954 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:24.374591 4522 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Oct 30 16:59:24.374954 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:24.374650 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb" podUID="534aeebe-4edf-425e-a16c-712cc498154b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Oct 30 16:59:24.390610 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:24.389056 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77" Oct 30 16:59:24.391348 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:24.391215 4522 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 16:59:24.391348 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:24.391271 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77" podUID="bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 16:59:24.607270 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:24.605723 4522 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 16:59:24.607270 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:24.605785 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk" podUID="93447f05-2d76-440f-b704-5e37d09eb5b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 16:59:25.112451 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:25.111512 4522 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:59:25.112451 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:25.111580 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf" podUID="b590c8f6-9d91-49bb-8e0f-5e65eb24e631" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:59:25.168266 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:25.167839 4522 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 30 16:59:25.168266 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:25.167897 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz" podUID="b2010977-7978-4ee0-9b32-3449329c0b4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 30 16:59:25.299479 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:25.299437 4522 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 16:59:25.300172 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:25.299748 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7" podUID="c6ffa9d7-4de9-4b98-8e7e-016db7370f08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 16:59:25.373256 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:25.372745 4522 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Oct 30 16:59:25.373256 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:25.372811 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb" podUID="534aeebe-4edf-425e-a16c-712cc498154b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Oct 30 16:59:25.387822 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:25.387700 4522 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 16:59:25.387822 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:25.387760 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77" podUID="bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 16:59:25.609175 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:25.608723 4522 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 16:59:25.609175 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:25.608784 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk" podUID="93447f05-2d76-440f-b704-5e37d09eb5b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 16:59:26.177807 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:26.177774 4522 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 30 16:59:26.178080 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:26.178052 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz" podUID="b2010977-7978-4ee0-9b32-3449329c0b4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 30 16:59:26.291444 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:26.290702 4522 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 16:59:26.291444 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:26.290760 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7" podUID="c6ffa9d7-4de9-4b98-8e7e-016db7370f08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 16:59:26.382445 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:26.381940 4522 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Oct 30 16:59:26.382445 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:26.382000 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb" podUID="534aeebe-4edf-425e-a16c-712cc498154b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Oct 30 16:59:26.387483 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:26.387354 4522 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 16:59:26.387483 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:26.387410 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77" podUID="bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 16:59:26.413351 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:26.413265 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7b9df-59f556c645-8f7kp" Oct 30 16:59:26.875350 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:26.851929 4522 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 16:59:26.875350 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:26.851996 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk" podUID="93447f05-2d76-440f-b704-5e37d09eb5b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 16:59:26.886090 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:26.886027 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz" podStartSLOduration=5.886005461 podStartE2EDuration="5.886005461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:59:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:59:23.560728767 +0000 UTC m=+737.561483669" watchObservedRunningTime="2026-10-30 16:59:26.886005461 +0000 UTC m=+740.886760362" Oct 30 16:59:27.177852 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:27.177336 4522 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 30 16:59:27.177852 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:27.177400 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz" podUID="b2010977-7978-4ee0-9b32-3449329c0b4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 30 16:59:27.288859 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:27.288824 4522 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 16:59:27.289183 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:27.289134 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7" podUID="c6ffa9d7-4de9-4b98-8e7e-016db7370f08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 16:59:27.369887 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:27.369850 4522 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Oct 30 16:59:27.370196 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:27.370148 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb" podUID="534aeebe-4edf-425e-a16c-712cc498154b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Oct 30 16:59:27.387900 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:27.387864 4522 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 16:59:27.388187 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:27.388149 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77" podUID="bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 16:59:27.606934 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:27.606881 4522 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 16:59:27.607268 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:27.607218 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk" podUID="93447f05-2d76-440f-b704-5e37d09eb5b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 16:59:28.189005 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:28.178954 4522 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 30 16:59:28.189005 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:28.179016 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz" podUID="b2010977-7978-4ee0-9b32-3449329c0b4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 30 16:59:28.380329 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:28.377714 4522 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Oct 30 16:59:28.380329 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:28.377776 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb" podUID="534aeebe-4edf-425e-a16c-712cc498154b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Oct 30 16:59:28.393998 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:28.392478 4522 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 16:59:28.393998 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:28.392538 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77" podUID="bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 16:59:28.566760 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:28.564632 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7" Oct 30 16:59:28.579614 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:28.579580 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7" Oct 30 16:59:28.692066 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:28.690160 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk" Oct 30 16:59:28.699826 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:28.699575 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk" Oct 30 16:59:29.170865 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:29.170829 4522 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 30 16:59:29.171340 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:29.171265 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz" podUID="b2010977-7978-4ee0-9b32-3449329c0b4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 30 16:59:29.368784 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:29.368746 4522 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Oct 30 16:59:29.369379 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:29.369345 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb" podUID="534aeebe-4edf-425e-a16c-712cc498154b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Oct 30 16:59:29.387891 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:29.387860 4522 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 16:59:29.388122 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:29.388098 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77" podUID="bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 16:59:30.168354 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:30.168299 4522 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 30 16:59:30.168569 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:30.168393 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz" podUID="b2010977-7978-4ee0-9b32-3449329c0b4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 30 16:59:30.386895 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:30.386453 4522 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 16:59:30.386895 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:30.386514 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77" podUID="bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 16:59:30.831384 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:30.829911 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb" Oct 30 16:59:30.831384 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:30.830346 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb" Oct 30 16:59:30.832436 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:30.832394 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb" Oct 30 16:59:31.173461 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:31.173380 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz" Oct 30 16:59:31.173723 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:31.173691 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz" Oct 30 16:59:31.174705 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:31.174684 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz" Oct 30 16:59:31.491117 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:31.491086 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77" Oct 30 16:59:31.491780 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:31.491762 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77" Oct 30 16:59:31.492571 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:31.492546 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77" Oct 30 16:59:32.723278 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:32.723248 4522 generic.go:356] "Generic (PLEG): container finished" podID="b590c8f6-9d91-49bb-8e0f-5e65eb24e631" containerID="29549c3eb9f4e617a2608e54db8fa9e328f8a3843ab700753cb96f333c593ef8" exitCode=0 Oct 30 16:59:32.723712 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:32.723338 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf" event={"ID":"b590c8f6-9d91-49bb-8e0f-5e65eb24e631","Type":"ContainerDied","Data":"29549c3eb9f4e617a2608e54db8fa9e328f8a3843ab700753cb96f333c593ef8"} Oct 30 16:59:32.723898 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:32.723880 4522 scope.go:122] "RemoveContainer" containerID="29549c3eb9f4e617a2608e54db8fa9e328f8a3843ab700753cb96f333c593ef8" Oct 30 16:59:32.725055 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:32.725031 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:59:32.725179 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:32.725105 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:59:32.725179 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:32.725153 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:59:33.731877 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:33.731839 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf" event={"ID":"b590c8f6-9d91-49bb-8e0f-5e65eb24e631","Type":"ContainerStarted","Data":"7eeccabc24f491c40eb12f9abf02055c5ffc5daf41c72cf1ae7e75df2f088204"} Oct 30 16:59:33.984877 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:33.984777 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf" Oct 30 16:59:33.985060 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:33.985038 4522 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 16:59:33.985120 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:33.985100 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf" podUID="b590c8f6-9d91-49bb-8e0f-5e65eb24e631" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 16:59:34.985398 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:34.985364 4522 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 16:59:34.985794 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:34.985426 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf" podUID="b590c8f6-9d91-49bb-8e0f-5e65eb24e631" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 16:59:35.984988 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:35.984954 4522 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 16:59:35.985164 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:35.985017 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf" podUID="b590c8f6-9d91-49bb-8e0f-5e65eb24e631" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 16:59:36.985599 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:36.985561 4522 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 16:59:36.985992 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:36.985628 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf" podUID="b590c8f6-9d91-49bb-8e0f-5e65eb24e631" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 16:59:37.991741 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:37.991706 4522 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 16:59:37.992356 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:37.992322 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf" podUID="b590c8f6-9d91-49bb-8e0f-5e65eb24e631" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 16:59:38.989086 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:38.989055 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf" Oct 30 16:59:38.989482 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:38.989457 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf" Oct 30 16:59:38.990041 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:38.990021 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf" Oct 30 16:59:42.983084 ip-10-0-128-218 kubenswrapper[4522]: E1030 16:59:42.983038 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 16:59:45.374605 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:45.374572 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv"] Oct 30 16:59:45.377652 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:45.377632 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" Oct 30 16:59:45.399271 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:45.399246 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv"] Oct 30 16:59:45.409996 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:45.409961 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-d384dc\"/\"default-dockercfg-vqwcf\"" Oct 30 16:59:45.410128 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:45.410055 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-d384dc\"/\"simulator-w6-d384dc-kserve-self-signed-certs\"" Oct 30 16:59:45.410128 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:45.410075 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-d384dc\"/\"openshift-service-ca.crt\"" Oct 30 16:59:45.410128 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:45.410084 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-d384dc\"/\"kube-root-ca.crt\"" Oct 30 16:59:45.499292 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:45.499256 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3e5e732-2c67-4377-9755-e2178bf33772-tmp-dir\") pod \"simulator-w6-d384dc-kserve-546b95bbb9-vc9bv\" (UID: \"a3e5e732-2c67-4377-9755-e2178bf33772\") " pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" Oct 30 16:59:45.499292 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:45.499298 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd5pj\" (UniqueName: \"kubernetes.io/projected/a3e5e732-2c67-4377-9755-e2178bf33772-kube-api-access-qd5pj\") pod \"simulator-w6-d384dc-kserve-546b95bbb9-vc9bv\" (UID: \"a3e5e732-2c67-4377-9755-e2178bf33772\") " pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" Oct 30 16:59:45.499525 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:45.499419 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3e5e732-2c67-4377-9755-e2178bf33772-tls-certs\") pod \"simulator-w6-d384dc-kserve-546b95bbb9-vc9bv\" (UID: \"a3e5e732-2c67-4377-9755-e2178bf33772\") " pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" Oct 30 16:59:45.499525 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:45.499453 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3e5e732-2c67-4377-9755-e2178bf33772-model-cache\") pod \"simulator-w6-d384dc-kserve-546b95bbb9-vc9bv\" (UID: \"a3e5e732-2c67-4377-9755-e2178bf33772\") " pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" Oct 30 16:59:45.499525 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:45.499470 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3e5e732-2c67-4377-9755-e2178bf33772-dshm\") pod \"simulator-w6-d384dc-kserve-546b95bbb9-vc9bv\" (UID: \"a3e5e732-2c67-4377-9755-e2178bf33772\") " pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" Oct 30 16:59:45.499637 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:45.499520 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3e5e732-2c67-4377-9755-e2178bf33772-home\") pod \"simulator-w6-d384dc-kserve-546b95bbb9-vc9bv\" (UID: \"a3e5e732-2c67-4377-9755-e2178bf33772\") " pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" Oct 30 16:59:45.600947 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:45.600915 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3e5e732-2c67-4377-9755-e2178bf33772-tmp-dir\") pod \"simulator-w6-d384dc-kserve-546b95bbb9-vc9bv\" (UID: \"a3e5e732-2c67-4377-9755-e2178bf33772\") " pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" Oct 30 16:59:45.600947 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:45.600952 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd5pj\" (UniqueName: \"kubernetes.io/projected/a3e5e732-2c67-4377-9755-e2178bf33772-kube-api-access-qd5pj\") pod \"simulator-w6-d384dc-kserve-546b95bbb9-vc9bv\" (UID: \"a3e5e732-2c67-4377-9755-e2178bf33772\") " pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" Oct 30 16:59:45.601194 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:45.601018 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3e5e732-2c67-4377-9755-e2178bf33772-tls-certs\") pod \"simulator-w6-d384dc-kserve-546b95bbb9-vc9bv\" (UID: \"a3e5e732-2c67-4377-9755-e2178bf33772\") " pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" Oct 30 16:59:45.601194 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:45.601048 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3e5e732-2c67-4377-9755-e2178bf33772-model-cache\") pod \"simulator-w6-d384dc-kserve-546b95bbb9-vc9bv\" (UID: \"a3e5e732-2c67-4377-9755-e2178bf33772\") " pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" Oct 30 16:59:45.601194 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:45.601130 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3e5e732-2c67-4377-9755-e2178bf33772-dshm\") pod \"simulator-w6-d384dc-kserve-546b95bbb9-vc9bv\" (UID: \"a3e5e732-2c67-4377-9755-e2178bf33772\") " pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" Oct 30 16:59:45.601194 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:45.601171 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3e5e732-2c67-4377-9755-e2178bf33772-home\") pod \"simulator-w6-d384dc-kserve-546b95bbb9-vc9bv\" (UID: \"a3e5e732-2c67-4377-9755-e2178bf33772\") " pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" Oct 30 16:59:45.601412 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:45.601393 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3e5e732-2c67-4377-9755-e2178bf33772-tmp-dir\") pod \"simulator-w6-d384dc-kserve-546b95bbb9-vc9bv\" (UID: \"a3e5e732-2c67-4377-9755-e2178bf33772\") " pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" Oct 30 16:59:45.601503 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:45.601470 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3e5e732-2c67-4377-9755-e2178bf33772-model-cache\") pod \"simulator-w6-d384dc-kserve-546b95bbb9-vc9bv\" (UID: \"a3e5e732-2c67-4377-9755-e2178bf33772\") " pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" Oct 30 16:59:45.601636 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:45.601615 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3e5e732-2c67-4377-9755-e2178bf33772-home\") pod \"simulator-w6-d384dc-kserve-546b95bbb9-vc9bv\" (UID: \"a3e5e732-2c67-4377-9755-e2178bf33772\") " pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" Oct 30 16:59:45.603457 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:45.603434 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3e5e732-2c67-4377-9755-e2178bf33772-dshm\") pod \"simulator-w6-d384dc-kserve-546b95bbb9-vc9bv\" (UID: \"a3e5e732-2c67-4377-9755-e2178bf33772\") " pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" Oct 30 16:59:45.603619 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:45.603604 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3e5e732-2c67-4377-9755-e2178bf33772-tls-certs\") pod \"simulator-w6-d384dc-kserve-546b95bbb9-vc9bv\" (UID: \"a3e5e732-2c67-4377-9755-e2178bf33772\") " pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" Oct 30 16:59:45.630755 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:45.630646 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd5pj\" (UniqueName: \"kubernetes.io/projected/a3e5e732-2c67-4377-9755-e2178bf33772-kube-api-access-qd5pj\") pod \"simulator-w6-d384dc-kserve-546b95bbb9-vc9bv\" (UID: \"a3e5e732-2c67-4377-9755-e2178bf33772\") " pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" Oct 30 16:59:45.691630 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:45.691592 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" Oct 30 16:59:45.998963 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:59:45.998932 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda3e5e732_2c67_4377_9755_e2178bf33772.slice/crio-e174670ab51a83264656d1bfc7a95367c71748298b5e99d9073943b7075cf7cc WatchSource:0}: Error finding container e174670ab51a83264656d1bfc7a95367c71748298b5e99d9073943b7075cf7cc: Status 404 returned error can't find the container with id e174670ab51a83264656d1bfc7a95367c71748298b5e99d9073943b7075cf7cc Oct 30 16:59:46.000083 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:46.000061 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv"] Oct 30 16:59:46.806775 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:46.806740 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" event={"ID":"a3e5e732-2c67-4377-9755-e2178bf33772","Type":"ContainerStarted","Data":"4169820d4cf8e16e4fb503ca82588005c1c4d08d1195d62025bf66b21c1a6763"} Oct 30 16:59:46.807160 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:46.806787 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" event={"ID":"a3e5e732-2c67-4377-9755-e2178bf33772","Type":"ContainerStarted","Data":"e174670ab51a83264656d1bfc7a95367c71748298b5e99d9073943b7075cf7cc"} Oct 30 16:59:46.852264 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:46.852212 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" podStartSLOduration=1.852198204 podStartE2EDuration="1.852198204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:59:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:59:46.851746041 +0000 UTC m=+760.852500942" watchObservedRunningTime="2026-10-30 16:59:46.852198204 +0000 UTC m=+760.852953104" Oct 30 16:59:47.692732 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:47.692702 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" Oct 30 16:59:51.800955 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:51.800864 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5"] Oct 30 16:59:51.803922 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:51.803900 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" Oct 30 16:59:51.823746 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:51.823723 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c7b9df\"/\"openshift-service-ca.crt\"" Oct 30 16:59:51.837233 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:51.837202 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c7b9df\"/\"kube-root-ca.crt\"" Oct 30 16:59:51.837416 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:51.837232 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c7b9df\"/\"default-dockercfg-px6m6\"" Oct 30 16:59:51.837416 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:51.837284 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c7b9df\"/\"simulator-w4-c7b9df-kserve-self-signed-certs\"" Oct 30 16:59:51.841126 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:51.841100 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5"] Oct 30 16:59:51.977740 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:51.977705 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b832781-3d41-4496-aa86-e4d92792e590-home\") pod \"simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5\" (UID: \"0b832781-3d41-4496-aa86-e4d92792e590\") " pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" Oct 30 16:59:51.977740 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:51.977742 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b832781-3d41-4496-aa86-e4d92792e590-dshm\") pod \"simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5\" (UID: \"0b832781-3d41-4496-aa86-e4d92792e590\") " pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" Oct 30 16:59:51.977950 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:51.977850 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b832781-3d41-4496-aa86-e4d92792e590-model-cache\") pod \"simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5\" (UID: \"0b832781-3d41-4496-aa86-e4d92792e590\") " pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" Oct 30 16:59:51.977950 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:51.977905 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b832781-3d41-4496-aa86-e4d92792e590-tls-certs\") pod \"simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5\" (UID: \"0b832781-3d41-4496-aa86-e4d92792e590\") " pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" Oct 30 16:59:51.977950 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:51.977935 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2lgv\" (UniqueName: \"kubernetes.io/projected/0b832781-3d41-4496-aa86-e4d92792e590-kube-api-access-h2lgv\") pod \"simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5\" (UID: \"0b832781-3d41-4496-aa86-e4d92792e590\") " pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" Oct 30 16:59:51.978050 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:51.978024 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b832781-3d41-4496-aa86-e4d92792e590-tmp-dir\") pod \"simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5\" (UID: \"0b832781-3d41-4496-aa86-e4d92792e590\") " pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" Oct 30 16:59:52.081201 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:52.081058 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b832781-3d41-4496-aa86-e4d92792e590-tls-certs\") pod \"simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5\" (UID: \"0b832781-3d41-4496-aa86-e4d92792e590\") " pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" Oct 30 16:59:52.081201 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:52.081148 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2lgv\" (UniqueName: \"kubernetes.io/projected/0b832781-3d41-4496-aa86-e4d92792e590-kube-api-access-h2lgv\") pod \"simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5\" (UID: \"0b832781-3d41-4496-aa86-e4d92792e590\") " pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" Oct 30 16:59:52.081591 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:52.081277 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b832781-3d41-4496-aa86-e4d92792e590-tmp-dir\") pod \"simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5\" (UID: \"0b832781-3d41-4496-aa86-e4d92792e590\") " pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" Oct 30 16:59:52.081591 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:52.081396 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b832781-3d41-4496-aa86-e4d92792e590-home\") pod \"simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5\" (UID: \"0b832781-3d41-4496-aa86-e4d92792e590\") " pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" Oct 30 16:59:52.081591 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:52.081434 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b832781-3d41-4496-aa86-e4d92792e590-dshm\") pod \"simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5\" (UID: \"0b832781-3d41-4496-aa86-e4d92792e590\") " pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" Oct 30 16:59:52.081591 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:52.081544 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b832781-3d41-4496-aa86-e4d92792e590-model-cache\") pod \"simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5\" (UID: \"0b832781-3d41-4496-aa86-e4d92792e590\") " pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" Oct 30 16:59:52.082101 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:52.082080 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b832781-3d41-4496-aa86-e4d92792e590-model-cache\") pod \"simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5\" (UID: \"0b832781-3d41-4496-aa86-e4d92792e590\") " pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" Oct 30 16:59:52.082746 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:52.082694 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b832781-3d41-4496-aa86-e4d92792e590-tmp-dir\") pod \"simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5\" (UID: \"0b832781-3d41-4496-aa86-e4d92792e590\") " pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" Oct 30 16:59:52.082860 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:52.082749 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b832781-3d41-4496-aa86-e4d92792e590-home\") pod \"simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5\" (UID: \"0b832781-3d41-4496-aa86-e4d92792e590\") " pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" Oct 30 16:59:52.085479 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:52.085453 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b832781-3d41-4496-aa86-e4d92792e590-tls-certs\") pod \"simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5\" (UID: \"0b832781-3d41-4496-aa86-e4d92792e590\") " pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" Oct 30 16:59:52.085675 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:52.085644 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b832781-3d41-4496-aa86-e4d92792e590-dshm\") pod \"simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5\" (UID: \"0b832781-3d41-4496-aa86-e4d92792e590\") " pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" Oct 30 16:59:52.116758 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:52.116727 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2lgv\" (UniqueName: \"kubernetes.io/projected/0b832781-3d41-4496-aa86-e4d92792e590-kube-api-access-h2lgv\") pod \"simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5\" (UID: \"0b832781-3d41-4496-aa86-e4d92792e590\") " pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" Oct 30 16:59:52.416475 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:52.416390 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" Oct 30 16:59:52.723757 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:59:52.723725 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b832781_3d41_4496_aa86_e4d92792e590.slice/crio-7448356e0670fd2168213b48d3234fded53f4d2cfccdde2939741e6ac043c4d8 WatchSource:0}: Error finding container 7448356e0670fd2168213b48d3234fded53f4d2cfccdde2939741e6ac043c4d8: Status 404 returned error can't find the container with id 7448356e0670fd2168213b48d3234fded53f4d2cfccdde2939741e6ac043c4d8 Oct 30 16:59:52.724957 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:52.724930 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5"] Oct 30 16:59:52.842374 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:52.842336 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" event={"ID":"0b832781-3d41-4496-aa86-e4d92792e590","Type":"ContainerStarted","Data":"a946f686eb0efbce447453509ce825ef866102dce48f34cc92a65830b0f6ab7f"} Oct 30 16:59:52.842753 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:52.842388 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" event={"ID":"0b832781-3d41-4496-aa86-e4d92792e590","Type":"ContainerStarted","Data":"7448356e0670fd2168213b48d3234fded53f4d2cfccdde2939741e6ac043c4d8"} Oct 30 16:59:52.891039 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:52.890993 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" podStartSLOduration=1.8909785669999999 podStartE2EDuration="1.890978567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:59:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:59:52.888978823 +0000 UTC m=+766.889733734" watchObservedRunningTime="2026-10-30 16:59:52.890978567 +0000 UTC m=+766.891733545" Oct 30 16:59:53.419567 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:53.419120 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" Oct 30 16:59:55.459926 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:55.459894 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz"] Oct 30 16:59:55.464154 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:55.464129 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" Oct 30 16:59:55.494235 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:55.494207 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-b80d83\"/\"openshift-service-ca.crt\"" Oct 30 16:59:55.494235 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:55.494207 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b80d83\"/\"default-dockercfg-pnr6k\"" Oct 30 16:59:55.494483 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:55.494453 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-b80d83\"/\"kube-root-ca.crt\"" Oct 30 16:59:55.494566 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:55.494546 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b80d83\"/\"premium-w3-b80d83-kserve-self-signed-certs\"" Oct 30 16:59:55.497144 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:55.497118 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz"] Oct 30 16:59:55.521679 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:55.521653 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75321c5d-8bf8-45c6-8811-81981fe3cad4-dshm\") pod \"premium-w3-b80d83-kserve-5d5b58dc78-xw7kz\" (UID: \"75321c5d-8bf8-45c6-8811-81981fe3cad4\") " pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" Oct 30 16:59:55.521808 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:55.521696 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75321c5d-8bf8-45c6-8811-81981fe3cad4-tls-certs\") pod \"premium-w3-b80d83-kserve-5d5b58dc78-xw7kz\" (UID: \"75321c5d-8bf8-45c6-8811-81981fe3cad4\") " pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" Oct 30 16:59:55.521853 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:55.521828 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75321c5d-8bf8-45c6-8811-81981fe3cad4-home\") pod \"premium-w3-b80d83-kserve-5d5b58dc78-xw7kz\" (UID: \"75321c5d-8bf8-45c6-8811-81981fe3cad4\") " pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" Oct 30 16:59:55.521981 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:55.521953 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75321c5d-8bf8-45c6-8811-81981fe3cad4-model-cache\") pod \"premium-w3-b80d83-kserve-5d5b58dc78-xw7kz\" (UID: \"75321c5d-8bf8-45c6-8811-81981fe3cad4\") " pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" Oct 30 16:59:55.522042 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:55.522023 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcwt6\" (UniqueName: \"kubernetes.io/projected/75321c5d-8bf8-45c6-8811-81981fe3cad4-kube-api-access-bcwt6\") pod \"premium-w3-b80d83-kserve-5d5b58dc78-xw7kz\" (UID: \"75321c5d-8bf8-45c6-8811-81981fe3cad4\") " pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" Oct 30 16:59:55.522111 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:55.522097 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75321c5d-8bf8-45c6-8811-81981fe3cad4-tmp-dir\") pod \"premium-w3-b80d83-kserve-5d5b58dc78-xw7kz\" (UID: \"75321c5d-8bf8-45c6-8811-81981fe3cad4\") " pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" Oct 30 16:59:55.622807 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:55.622774 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75321c5d-8bf8-45c6-8811-81981fe3cad4-model-cache\") pod \"premium-w3-b80d83-kserve-5d5b58dc78-xw7kz\" (UID: \"75321c5d-8bf8-45c6-8811-81981fe3cad4\") " pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" Oct 30 16:59:55.622989 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:55.622824 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcwt6\" (UniqueName: \"kubernetes.io/projected/75321c5d-8bf8-45c6-8811-81981fe3cad4-kube-api-access-bcwt6\") pod \"premium-w3-b80d83-kserve-5d5b58dc78-xw7kz\" (UID: \"75321c5d-8bf8-45c6-8811-81981fe3cad4\") " pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" Oct 30 16:59:55.622989 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:55.622865 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75321c5d-8bf8-45c6-8811-81981fe3cad4-tmp-dir\") pod \"premium-w3-b80d83-kserve-5d5b58dc78-xw7kz\" (UID: \"75321c5d-8bf8-45c6-8811-81981fe3cad4\") " pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" Oct 30 16:59:55.622989 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:55.622904 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75321c5d-8bf8-45c6-8811-81981fe3cad4-dshm\") pod \"premium-w3-b80d83-kserve-5d5b58dc78-xw7kz\" (UID: \"75321c5d-8bf8-45c6-8811-81981fe3cad4\") " pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" Oct 30 16:59:55.622989 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:55.622941 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75321c5d-8bf8-45c6-8811-81981fe3cad4-tls-certs\") pod \"premium-w3-b80d83-kserve-5d5b58dc78-xw7kz\" (UID: \"75321c5d-8bf8-45c6-8811-81981fe3cad4\") " pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" Oct 30 16:59:55.622989 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:55.622973 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75321c5d-8bf8-45c6-8811-81981fe3cad4-home\") pod \"premium-w3-b80d83-kserve-5d5b58dc78-xw7kz\" (UID: \"75321c5d-8bf8-45c6-8811-81981fe3cad4\") " pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" Oct 30 16:59:55.623268 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:55.623199 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75321c5d-8bf8-45c6-8811-81981fe3cad4-model-cache\") pod \"premium-w3-b80d83-kserve-5d5b58dc78-xw7kz\" (UID: \"75321c5d-8bf8-45c6-8811-81981fe3cad4\") " pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" Oct 30 16:59:55.623354 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:55.623260 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75321c5d-8bf8-45c6-8811-81981fe3cad4-tmp-dir\") pod \"premium-w3-b80d83-kserve-5d5b58dc78-xw7kz\" (UID: \"75321c5d-8bf8-45c6-8811-81981fe3cad4\") " pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" Oct 30 16:59:55.623354 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:55.623283 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75321c5d-8bf8-45c6-8811-81981fe3cad4-home\") pod \"premium-w3-b80d83-kserve-5d5b58dc78-xw7kz\" (UID: \"75321c5d-8bf8-45c6-8811-81981fe3cad4\") " pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" Oct 30 16:59:55.625225 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:55.625203 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75321c5d-8bf8-45c6-8811-81981fe3cad4-dshm\") pod \"premium-w3-b80d83-kserve-5d5b58dc78-xw7kz\" (UID: \"75321c5d-8bf8-45c6-8811-81981fe3cad4\") " pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" Oct 30 16:59:55.625533 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:55.625516 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75321c5d-8bf8-45c6-8811-81981fe3cad4-tls-certs\") pod \"premium-w3-b80d83-kserve-5d5b58dc78-xw7kz\" (UID: \"75321c5d-8bf8-45c6-8811-81981fe3cad4\") " pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" Oct 30 16:59:55.648004 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:55.647981 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcwt6\" (UniqueName: \"kubernetes.io/projected/75321c5d-8bf8-45c6-8811-81981fe3cad4-kube-api-access-bcwt6\") pod \"premium-w3-b80d83-kserve-5d5b58dc78-xw7kz\" (UID: \"75321c5d-8bf8-45c6-8811-81981fe3cad4\") " pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" Oct 30 16:59:55.691804 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:55.691769 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" Oct 30 16:59:55.707353 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:55.707324 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" Oct 30 16:59:55.776555 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:55.776467 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" Oct 30 16:59:55.877838 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:55.877806 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" Oct 30 16:59:56.081899 ip-10-0-128-218 kubenswrapper[4522]: W1030 16:59:56.081868 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75321c5d_8bf8_45c6_8811_81981fe3cad4.slice/crio-acca2f0754083aafdf2a8ae1c88996ca9db7f96d6d5f14269eeb8277642dbc25 WatchSource:0}: Error finding container acca2f0754083aafdf2a8ae1c88996ca9db7f96d6d5f14269eeb8277642dbc25: Status 404 returned error can't find the container with id acca2f0754083aafdf2a8ae1c88996ca9db7f96d6d5f14269eeb8277642dbc25 Oct 30 16:59:56.082085 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:56.082065 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz"] Oct 30 16:59:56.866512 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:56.866475 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" event={"ID":"75321c5d-8bf8-45c6-8811-81981fe3cad4","Type":"ContainerStarted","Data":"ec5598f21e37163f7c4135de7a8a510a8fd2793f7c5e1b7e211068c818310f10"} Oct 30 16:59:56.866951 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:56.866525 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" event={"ID":"75321c5d-8bf8-45c6-8811-81981fe3cad4","Type":"ContainerStarted","Data":"acca2f0754083aafdf2a8ae1c88996ca9db7f96d6d5f14269eeb8277642dbc25"} Oct 30 16:59:56.919071 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:56.919023 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" podStartSLOduration=1.919008851 podStartE2EDuration="1.919008851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:59:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:59:56.91781887 +0000 UTC m=+770.918573789" watchObservedRunningTime="2026-10-30 16:59:56.919008851 +0000 UTC m=+770.919763751" Oct 30 16:59:57.777209 ip-10-0-128-218 kubenswrapper[4522]: I1030 16:59:57.777169 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" Oct 30 17:00:00.187135 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:00.187047 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889660-l58hw"] Oct 30 17:00:00.192503 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:00.192471 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889660-2b9gw"] Oct 30 17:00:00.192671 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:00.192584 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889660-l58hw" Oct 30 17:00:00.196742 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:00.196721 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889660-x6p67"] Oct 30 17:00:00.196871 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:00.196819 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889660-2b9gw" Oct 30 17:00:00.202007 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:00.201191 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889660-x6p67" Oct 30 17:00:00.214624 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:00.213487 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889660-l58hw"] Oct 30 17:00:00.220102 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:00.219395 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889660-2b9gw"] Oct 30 17:00:00.227782 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:00.227752 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889660-x6p67"] Oct 30 17:00:00.229495 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:00.229468 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889660-bntv4"] Oct 30 17:00:00.234264 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:00.234241 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889660-bntv4" Oct 30 17:00:00.253937 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:00.253906 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889660-bntv4"] Oct 30 17:00:00.272284 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:00.272250 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdl2x\" (UniqueName: \"kubernetes.io/projected/9004635d-a688-4763-9817-b4c39b2e8dac-kube-api-access-gdl2x\") pod \"maas-api-key-cleanup-e2e-worker-w6-d384dc-29889660-2b9gw\" (UID: \"9004635d-a688-4763-9817-b4c39b2e8dac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889660-2b9gw" Oct 30 17:00:00.272462 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:00.272438 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dx77\" (UniqueName: \"kubernetes.io/projected/f361d37b-d1f9-45ae-9874-859da12af710-kube-api-access-4dx77\") pod \"maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889660-l58hw\" (UID: \"f361d37b-d1f9-45ae-9874-859da12af710\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889660-l58hw" Oct 30 17:00:00.272521 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:00.272484 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9004635d-a688-4763-9817-b4c39b2e8dac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-d384dc-29889660-2b9gw\" (UID: \"9004635d-a688-4763-9817-b4c39b2e8dac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889660-2b9gw" Oct 30 17:00:00.272632 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:00.272611 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjcsl\" (UniqueName: \"kubernetes.io/projected/3118473a-8879-4b2c-b309-96003f326cf9-kube-api-access-sjcsl\") pod \"maas-api-key-cleanup-e2e-worker-w2-b4f406-29889660-bntv4\" (UID: \"3118473a-8879-4b2c-b309-96003f326cf9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889660-bntv4" Oct 30 17:00:00.272693 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:00.272648 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f361d37b-d1f9-45ae-9874-859da12af710-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889660-l58hw\" (UID: \"f361d37b-d1f9-45ae-9874-859da12af710\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889660-l58hw" Oct 30 17:00:00.272753 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:00.272721 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3118473a-8879-4b2c-b309-96003f326cf9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-b4f406-29889660-bntv4\" (UID: \"3118473a-8879-4b2c-b309-96003f326cf9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889660-bntv4" Oct 30 17:00:00.272807 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:00.272749 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btqhk\" (UniqueName: \"kubernetes.io/projected/aa253ff9-890d-47e8-9b2a-2a7e586502f1-kube-api-access-btqhk\") pod \"maas-api-key-cleanup-e2e-worker-w3-b80d83-29889660-x6p67\" (UID: \"aa253ff9-890d-47e8-9b2a-2a7e586502f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889660-x6p67" Oct 30 17:00:00.272906 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:00.272887 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa253ff9-890d-47e8-9b2a-2a7e586502f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b80d83-29889660-x6p67\" (UID: \"aa253ff9-890d-47e8-9b2a-2a7e586502f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889660-x6p67" Oct 30 17:00:00.309054 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:00.309029 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:00:00.374239 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:00.374193 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdl2x\" (UniqueName: \"kubernetes.io/projected/9004635d-a688-4763-9817-b4c39b2e8dac-kube-api-access-gdl2x\") pod \"maas-api-key-cleanup-e2e-worker-w6-d384dc-29889660-2b9gw\" (UID: \"9004635d-a688-4763-9817-b4c39b2e8dac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889660-2b9gw" Oct 30 17:00:00.374453 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:00.374334 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dx77\" (UniqueName: \"kubernetes.io/projected/f361d37b-d1f9-45ae-9874-859da12af710-kube-api-access-4dx77\") pod \"maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889660-l58hw\" (UID: \"f361d37b-d1f9-45ae-9874-859da12af710\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889660-l58hw" Oct 30 17:00:00.374453 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:00.374374 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9004635d-a688-4763-9817-b4c39b2e8dac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-d384dc-29889660-2b9gw\" (UID: \"9004635d-a688-4763-9817-b4c39b2e8dac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889660-2b9gw" Oct 30 17:00:00.374453 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:00.374436 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjcsl\" (UniqueName: \"kubernetes.io/projected/3118473a-8879-4b2c-b309-96003f326cf9-kube-api-access-sjcsl\") pod \"maas-api-key-cleanup-e2e-worker-w2-b4f406-29889660-bntv4\" (UID: \"3118473a-8879-4b2c-b309-96003f326cf9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889660-bntv4" Oct 30 17:00:00.374617 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:00.374459 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f361d37b-d1f9-45ae-9874-859da12af710-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889660-l58hw\" (UID: \"f361d37b-d1f9-45ae-9874-859da12af710\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889660-l58hw" Oct 30 17:00:00.375762 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:00.374527 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3118473a-8879-4b2c-b309-96003f326cf9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-b4f406-29889660-bntv4\" (UID: \"3118473a-8879-4b2c-b309-96003f326cf9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889660-bntv4" Oct 30 17:00:00.377927 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:00.375914 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9004635d-a688-4763-9817-b4c39b2e8dac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-d384dc-29889660-2b9gw\" (UID: \"9004635d-a688-4763-9817-b4c39b2e8dac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889660-2b9gw" Oct 30 17:00:00.377927 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:00.376158 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btqhk\" (UniqueName: \"kubernetes.io/projected/aa253ff9-890d-47e8-9b2a-2a7e586502f1-kube-api-access-btqhk\") pod \"maas-api-key-cleanup-e2e-worker-w3-b80d83-29889660-x6p67\" (UID: \"aa253ff9-890d-47e8-9b2a-2a7e586502f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889660-x6p67" Oct 30 17:00:00.377927 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:00.376416 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa253ff9-890d-47e8-9b2a-2a7e586502f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b80d83-29889660-x6p67\" (UID: \"aa253ff9-890d-47e8-9b2a-2a7e586502f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889660-x6p67" Oct 30 17:00:00.377927 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:00.376528 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3118473a-8879-4b2c-b309-96003f326cf9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-b4f406-29889660-bntv4\" (UID: \"3118473a-8879-4b2c-b309-96003f326cf9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889660-bntv4" Oct 30 17:00:00.377927 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:00.376974 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f361d37b-d1f9-45ae-9874-859da12af710-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889660-l58hw\" (UID: \"f361d37b-d1f9-45ae-9874-859da12af710\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889660-l58hw" Oct 30 17:00:00.381239 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:00.379269 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa253ff9-890d-47e8-9b2a-2a7e586502f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b80d83-29889660-x6p67\" (UID: \"aa253ff9-890d-47e8-9b2a-2a7e586502f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889660-x6p67" Oct 30 17:00:00.409886 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:00.409845 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btqhk\" (UniqueName: \"kubernetes.io/projected/aa253ff9-890d-47e8-9b2a-2a7e586502f1-kube-api-access-btqhk\") pod \"maas-api-key-cleanup-e2e-worker-w3-b80d83-29889660-x6p67\" (UID: \"aa253ff9-890d-47e8-9b2a-2a7e586502f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889660-x6p67" Oct 30 17:00:00.410042 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:00.409920 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdl2x\" (UniqueName: \"kubernetes.io/projected/9004635d-a688-4763-9817-b4c39b2e8dac-kube-api-access-gdl2x\") pod \"maas-api-key-cleanup-e2e-worker-w6-d384dc-29889660-2b9gw\" (UID: \"9004635d-a688-4763-9817-b4c39b2e8dac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889660-2b9gw" Oct 30 17:00:00.410367 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:00.410342 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjcsl\" (UniqueName: \"kubernetes.io/projected/3118473a-8879-4b2c-b309-96003f326cf9-kube-api-access-sjcsl\") pod \"maas-api-key-cleanup-e2e-worker-w2-b4f406-29889660-bntv4\" (UID: \"3118473a-8879-4b2c-b309-96003f326cf9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889660-bntv4" Oct 30 17:00:00.412242 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:00.412219 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dx77\" (UniqueName: \"kubernetes.io/projected/f361d37b-d1f9-45ae-9874-859da12af710-kube-api-access-4dx77\") pod \"maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889660-l58hw\" (UID: \"f361d37b-d1f9-45ae-9874-859da12af710\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889660-l58hw" Oct 30 17:00:00.510360 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:00.510287 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889660-l58hw" Oct 30 17:00:00.531928 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:00.531891 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889660-2b9gw" Oct 30 17:00:00.534731 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:00.534698 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889660-x6p67" Oct 30 17:00:00.555908 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:00.555509 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889660-bntv4" Oct 30 17:00:00.863745 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:00:00.863708 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf361d37b_d1f9_45ae_9874_859da12af710.slice/crio-cb1ea0728b785ad6f5fc531e1300c0be6a2674d6bd73524dbb57519a319f2dde WatchSource:0}: Error finding container cb1ea0728b785ad6f5fc531e1300c0be6a2674d6bd73524dbb57519a319f2dde: Status 404 returned error can't find the container with id cb1ea0728b785ad6f5fc531e1300c0be6a2674d6bd73524dbb57519a319f2dde Oct 30 17:00:00.864267 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:00.864241 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889660-l58hw"] Oct 30 17:00:00.889095 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:00.889055 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889660-l58hw" event={"ID":"f361d37b-d1f9-45ae-9874-859da12af710","Type":"ContainerStarted","Data":"cb1ea0728b785ad6f5fc531e1300c0be6a2674d6bd73524dbb57519a319f2dde"} Oct 30 17:00:00.899142 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:00:00.899114 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9004635d_a688_4763_9817_b4c39b2e8dac.slice/crio-42f0365f7179144f8bb60c5e9dea10c288fb5006afefaf9bed2b7c0dde7741ff WatchSource:0}: Error finding container 42f0365f7179144f8bb60c5e9dea10c288fb5006afefaf9bed2b7c0dde7741ff: Status 404 returned error can't find the container with id 42f0365f7179144f8bb60c5e9dea10c288fb5006afefaf9bed2b7c0dde7741ff Oct 30 17:00:00.900838 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:00.900814 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889660-2b9gw"] Oct 30 17:00:00.904388 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:00:00.904356 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa253ff9_890d_47e8_9b2a_2a7e586502f1.slice/crio-9b944c687fa278563307c42c7e201431ba9a57459c66c943deadb030138ee378 WatchSource:0}: Error finding container 9b944c687fa278563307c42c7e201431ba9a57459c66c943deadb030138ee378: Status 404 returned error can't find the container with id 9b944c687fa278563307c42c7e201431ba9a57459c66c943deadb030138ee378 Oct 30 17:00:00.905890 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:00.905793 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889660-x6p67"] Oct 30 17:00:00.919813 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:00.919787 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889660-bntv4"] Oct 30 17:00:00.920157 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:00:00.920132 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3118473a_8879_4b2c_b309_96003f326cf9.slice/crio-5607d932182315a7b6abf66c83c3d5b28034a5835ff30445796b496dd89cb206 WatchSource:0}: Error finding container 5607d932182315a7b6abf66c83c3d5b28034a5835ff30445796b496dd89cb206: Status 404 returned error can't find the container with id 5607d932182315a7b6abf66c83c3d5b28034a5835ff30445796b496dd89cb206 Oct 30 17:00:01.895630 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:01.895585 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889660-bntv4" event={"ID":"3118473a-8879-4b2c-b309-96003f326cf9","Type":"ContainerStarted","Data":"5607d932182315a7b6abf66c83c3d5b28034a5835ff30445796b496dd89cb206"} Oct 30 17:00:01.896924 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:01.896897 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889660-x6p67" event={"ID":"aa253ff9-890d-47e8-9b2a-2a7e586502f1","Type":"ContainerStarted","Data":"9b944c687fa278563307c42c7e201431ba9a57459c66c943deadb030138ee378"} Oct 30 17:00:01.898117 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:01.898096 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889660-2b9gw" event={"ID":"9004635d-a688-4763-9817-b4c39b2e8dac","Type":"ContainerStarted","Data":"42f0365f7179144f8bb60c5e9dea10c288fb5006afefaf9bed2b7c0dde7741ff"} Oct 30 17:00:02.418334 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:02.417328 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" Oct 30 17:00:02.443265 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:02.443225 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" Oct 30 17:00:02.927848 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:02.927817 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" Oct 30 17:00:05.776757 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:05.776622 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" Oct 30 17:00:05.788859 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:05.788836 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" Oct 30 17:00:05.936113 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:05.936032 4522 generic.go:356] "Generic (PLEG): container finished" podID="3118473a-8879-4b2c-b309-96003f326cf9" containerID="9e457a44771bd8d983d8f0f30134fff5d6b802a3a3627a39e896ee02c98d92d8" exitCode=60 Oct 30 17:00:05.936267 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:05.936113 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889660-bntv4" event={"ID":"3118473a-8879-4b2c-b309-96003f326cf9","Type":"ContainerDied","Data":"9e457a44771bd8d983d8f0f30134fff5d6b802a3a3627a39e896ee02c98d92d8"} Oct 30 17:00:05.936389 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:05.936375 4522 scope.go:122] "RemoveContainer" containerID="9e457a44771bd8d983d8f0f30134fff5d6b802a3a3627a39e896ee02c98d92d8" Oct 30 17:00:05.937761 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:05.937737 4522 generic.go:356] "Generic (PLEG): container finished" podID="f361d37b-d1f9-45ae-9874-859da12af710" containerID="caf31e5932404570a1718fd36a0516047242def6860a7330281aa8b2bff26d0e" exitCode=60 Oct 30 17:00:05.937861 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:05.937793 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889660-l58hw" event={"ID":"f361d37b-d1f9-45ae-9874-859da12af710","Type":"ContainerDied","Data":"caf31e5932404570a1718fd36a0516047242def6860a7330281aa8b2bff26d0e"} Oct 30 17:00:05.938038 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:05.938016 4522 scope.go:122] "RemoveContainer" containerID="caf31e5932404570a1718fd36a0516047242def6860a7330281aa8b2bff26d0e" Oct 30 17:00:05.939581 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:05.939555 4522 generic.go:356] "Generic (PLEG): container finished" podID="aa253ff9-890d-47e8-9b2a-2a7e586502f1" containerID="259e81991bc8fa2a62513c65bb28fb4c970b809e2e1417027908bf0afde60f1a" exitCode=60 Oct 30 17:00:05.939696 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:05.939651 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889660-x6p67" event={"ID":"aa253ff9-890d-47e8-9b2a-2a7e586502f1","Type":"ContainerDied","Data":"259e81991bc8fa2a62513c65bb28fb4c970b809e2e1417027908bf0afde60f1a"} Oct 30 17:00:05.939930 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:05.939803 4522 scope.go:122] "RemoveContainer" containerID="259e81991bc8fa2a62513c65bb28fb4c970b809e2e1417027908bf0afde60f1a" Oct 30 17:00:05.941527 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:05.941506 4522 generic.go:356] "Generic (PLEG): container finished" podID="9004635d-a688-4763-9817-b4c39b2e8dac" containerID="aefb3703a09b30c5eb48f74bfe5d3e09c563f48e3f17a19d7e63d14382a87df8" exitCode=60 Oct 30 17:00:05.941676 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:05.941648 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889660-2b9gw" event={"ID":"9004635d-a688-4763-9817-b4c39b2e8dac","Type":"ContainerDied","Data":"aefb3703a09b30c5eb48f74bfe5d3e09c563f48e3f17a19d7e63d14382a87df8"} Oct 30 17:00:05.941806 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:05.941755 4522 scope.go:122] "RemoveContainer" containerID="aefb3703a09b30c5eb48f74bfe5d3e09c563f48e3f17a19d7e63d14382a87df8" Oct 30 17:00:05.954809 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:05.954788 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" Oct 30 17:00:06.949159 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:06.949119 4522 generic.go:356] "Generic (PLEG): container finished" podID="aa253ff9-890d-47e8-9b2a-2a7e586502f1" containerID="c21bceabc4b5685d82fa9819fd54797eacb7bfe301205ce49e7a8b473e6ab942" exitCode=60 Oct 30 17:00:06.949637 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:06.949204 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889660-x6p67" event={"ID":"aa253ff9-890d-47e8-9b2a-2a7e586502f1","Type":"ContainerDied","Data":"c21bceabc4b5685d82fa9819fd54797eacb7bfe301205ce49e7a8b473e6ab942"} Oct 30 17:00:06.949637 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:06.949331 4522 scope.go:122] "RemoveContainer" containerID="259e81991bc8fa2a62513c65bb28fb4c970b809e2e1417027908bf0afde60f1a" Oct 30 17:00:06.949637 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:06.949563 4522 scope.go:122] "RemoveContainer" containerID="c21bceabc4b5685d82fa9819fd54797eacb7bfe301205ce49e7a8b473e6ab942" Oct 30 17:00:06.949844 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:00:06.949826 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b80d83-29889660-x6p67_odh-ai-gateway-infra(aa253ff9-890d-47e8-9b2a-2a7e586502f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889660-x6p67" podUID="aa253ff9-890d-47e8-9b2a-2a7e586502f1" Oct 30 17:00:06.951654 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:06.951630 4522 generic.go:356] "Generic (PLEG): container finished" podID="9004635d-a688-4763-9817-b4c39b2e8dac" containerID="26cf68a78a944a815417637e21a07b23e8b4f899b5bf50e6233cdec987b8de6f" exitCode=60 Oct 30 17:00:06.951767 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:06.951705 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889660-2b9gw" event={"ID":"9004635d-a688-4763-9817-b4c39b2e8dac","Type":"ContainerDied","Data":"26cf68a78a944a815417637e21a07b23e8b4f899b5bf50e6233cdec987b8de6f"} Oct 30 17:00:06.952005 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:06.951987 4522 scope.go:122] "RemoveContainer" containerID="26cf68a78a944a815417637e21a07b23e8b4f899b5bf50e6233cdec987b8de6f" Oct 30 17:00:06.952262 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:00:06.952235 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-d384dc-29889660-2b9gw_odh-ai-gateway-infra(9004635d-a688-4763-9817-b4c39b2e8dac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889660-2b9gw" podUID="9004635d-a688-4763-9817-b4c39b2e8dac" Oct 30 17:00:06.953668 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:06.953642 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889660-bntv4" event={"ID":"3118473a-8879-4b2c-b309-96003f326cf9","Type":"ContainerStarted","Data":"150ef50a4084c7539c20de5cebdbb53a4c0b3eb19566b1bf053877943c39c845"} Oct 30 17:00:06.955563 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:06.955543 4522 generic.go:356] "Generic (PLEG): container finished" podID="f361d37b-d1f9-45ae-9874-859da12af710" containerID="b0f73594cd30343282c1d17e214601dae1a6df7aa27d16747deca89ec4fa531c" exitCode=60 Oct 30 17:00:06.955661 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:06.955612 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889660-l58hw" event={"ID":"f361d37b-d1f9-45ae-9874-859da12af710","Type":"ContainerDied","Data":"b0f73594cd30343282c1d17e214601dae1a6df7aa27d16747deca89ec4fa531c"} Oct 30 17:00:06.955813 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:06.955801 4522 scope.go:122] "RemoveContainer" containerID="b0f73594cd30343282c1d17e214601dae1a6df7aa27d16747deca89ec4fa531c" Oct 30 17:00:06.955983 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:00:06.955968 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889660-l58hw_odh-ai-gateway-infra(f361d37b-d1f9-45ae-9874-859da12af710)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889660-l58hw" podUID="f361d37b-d1f9-45ae-9874-859da12af710" Oct 30 17:00:07.082674 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:07.082623 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889660-bntv4" podStartSLOduration=3.273021535 podStartE2EDuration="7.082610045s" totalImagesPullingTime="3.80958851s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:07.079616047 +0000 UTC m=+781.080370949" watchObservedRunningTime="2026-10-30 17:00:07.082610045 +0000 UTC m=+781.083364946" Oct 30 17:00:07.121547 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:07.121521 4522 scope.go:122] "RemoveContainer" containerID="aefb3703a09b30c5eb48f74bfe5d3e09c563f48e3f17a19d7e63d14382a87df8" Oct 30 17:00:07.134856 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:07.134830 4522 scope.go:122] "RemoveContainer" containerID="caf31e5932404570a1718fd36a0516047242def6860a7330281aa8b2bff26d0e" Oct 30 17:00:07.963397 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:07.963369 4522 scope.go:122] "RemoveContainer" containerID="26cf68a78a944a815417637e21a07b23e8b4f899b5bf50e6233cdec987b8de6f" Oct 30 17:00:07.963853 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:00:07.963674 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-d384dc-29889660-2b9gw_odh-ai-gateway-infra(9004635d-a688-4763-9817-b4c39b2e8dac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889660-2b9gw" podUID="9004635d-a688-4763-9817-b4c39b2e8dac" Oct 30 17:00:07.964968 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:07.964949 4522 generic.go:356] "Generic (PLEG): container finished" podID="3118473a-8879-4b2c-b309-96003f326cf9" containerID="150ef50a4084c7539c20de5cebdbb53a4c0b3eb19566b1bf053877943c39c845" exitCode=60 Oct 30 17:00:07.965054 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:07.965035 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889660-bntv4" event={"ID":"3118473a-8879-4b2c-b309-96003f326cf9","Type":"ContainerDied","Data":"150ef50a4084c7539c20de5cebdbb53a4c0b3eb19566b1bf053877943c39c845"} Oct 30 17:00:07.965124 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:07.965095 4522 scope.go:122] "RemoveContainer" containerID="9e457a44771bd8d983d8f0f30134fff5d6b802a3a3627a39e896ee02c98d92d8" Oct 30 17:00:07.965279 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:07.965261 4522 scope.go:122] "RemoveContainer" containerID="150ef50a4084c7539c20de5cebdbb53a4c0b3eb19566b1bf053877943c39c845" Oct 30 17:00:07.965572 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:00:07.965544 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-b4f406-29889660-bntv4_odh-ai-gateway-infra(3118473a-8879-4b2c-b309-96003f326cf9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889660-bntv4" podUID="3118473a-8879-4b2c-b309-96003f326cf9" Oct 30 17:00:07.967060 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:07.967045 4522 scope.go:122] "RemoveContainer" containerID="b0f73594cd30343282c1d17e214601dae1a6df7aa27d16747deca89ec4fa531c" Oct 30 17:00:07.967256 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:00:07.967239 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889660-l58hw_odh-ai-gateway-infra(f361d37b-d1f9-45ae-9874-859da12af710)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889660-l58hw" podUID="f361d37b-d1f9-45ae-9874-859da12af710" Oct 30 17:00:07.968514 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:07.968501 4522 scope.go:122] "RemoveContainer" containerID="c21bceabc4b5685d82fa9819fd54797eacb7bfe301205ce49e7a8b473e6ab942" Oct 30 17:00:07.968685 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:00:07.968669 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b80d83-29889660-x6p67_odh-ai-gateway-infra(aa253ff9-890d-47e8-9b2a-2a7e586502f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889660-x6p67" podUID="aa253ff9-890d-47e8-9b2a-2a7e586502f1" Oct 30 17:00:07.999181 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:07.999148 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p"] Oct 30 17:00:08.005262 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:08.005237 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" Oct 30 17:00:08.021752 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:08.021722 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p"] Oct 30 17:00:08.023178 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:08.023154 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-096d75\"/\"kube-root-ca.crt\"" Oct 30 17:00:08.023664 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:08.023641 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-096d75\"/\"openshift-service-ca.crt\"" Oct 30 17:00:08.023975 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:08.023959 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-096d75\"/\"default-dockercfg-m777j\"" Oct 30 17:00:08.024065 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:08.024028 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-096d75\"/\"premium-w0-096d75-kserve-self-signed-certs\"" Oct 30 17:00:08.081847 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:08.081808 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9ad71b7-ad90-4c8c-bf27-0f10b23f1593-model-cache\") pod \"premium-w0-096d75-kserve-8477779bfc-qwp2p\" (UID: \"c9ad71b7-ad90-4c8c-bf27-0f10b23f1593\") " pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" Oct 30 17:00:08.082069 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:08.082038 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9ad71b7-ad90-4c8c-bf27-0f10b23f1593-tls-certs\") pod \"premium-w0-096d75-kserve-8477779bfc-qwp2p\" (UID: \"c9ad71b7-ad90-4c8c-bf27-0f10b23f1593\") " pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" Oct 30 17:00:08.082163 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:08.082091 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9ad71b7-ad90-4c8c-bf27-0f10b23f1593-dshm\") pod \"premium-w0-096d75-kserve-8477779bfc-qwp2p\" (UID: \"c9ad71b7-ad90-4c8c-bf27-0f10b23f1593\") " pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" Oct 30 17:00:08.082163 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:08.082138 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lrmc\" (UniqueName: \"kubernetes.io/projected/c9ad71b7-ad90-4c8c-bf27-0f10b23f1593-kube-api-access-5lrmc\") pod \"premium-w0-096d75-kserve-8477779bfc-qwp2p\" (UID: \"c9ad71b7-ad90-4c8c-bf27-0f10b23f1593\") " pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" Oct 30 17:00:08.082276 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:08.082241 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9ad71b7-ad90-4c8c-bf27-0f10b23f1593-tmp-dir\") pod \"premium-w0-096d75-kserve-8477779bfc-qwp2p\" (UID: \"c9ad71b7-ad90-4c8c-bf27-0f10b23f1593\") " pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" Oct 30 17:00:08.082360 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:08.082327 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9ad71b7-ad90-4c8c-bf27-0f10b23f1593-home\") pod \"premium-w0-096d75-kserve-8477779bfc-qwp2p\" (UID: \"c9ad71b7-ad90-4c8c-bf27-0f10b23f1593\") " pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" Oct 30 17:00:08.183934 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:08.183880 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9ad71b7-ad90-4c8c-bf27-0f10b23f1593-tls-certs\") pod \"premium-w0-096d75-kserve-8477779bfc-qwp2p\" (UID: \"c9ad71b7-ad90-4c8c-bf27-0f10b23f1593\") " pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" Oct 30 17:00:08.183934 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:08.183931 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9ad71b7-ad90-4c8c-bf27-0f10b23f1593-dshm\") pod \"premium-w0-096d75-kserve-8477779bfc-qwp2p\" (UID: \"c9ad71b7-ad90-4c8c-bf27-0f10b23f1593\") " pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" Oct 30 17:00:08.184203 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:08.183975 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lrmc\" (UniqueName: \"kubernetes.io/projected/c9ad71b7-ad90-4c8c-bf27-0f10b23f1593-kube-api-access-5lrmc\") pod \"premium-w0-096d75-kserve-8477779bfc-qwp2p\" (UID: \"c9ad71b7-ad90-4c8c-bf27-0f10b23f1593\") " pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" Oct 30 17:00:08.184203 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:08.184016 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9ad71b7-ad90-4c8c-bf27-0f10b23f1593-tmp-dir\") pod \"premium-w0-096d75-kserve-8477779bfc-qwp2p\" (UID: \"c9ad71b7-ad90-4c8c-bf27-0f10b23f1593\") " pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" Oct 30 17:00:08.184203 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:08.184087 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9ad71b7-ad90-4c8c-bf27-0f10b23f1593-home\") pod \"premium-w0-096d75-kserve-8477779bfc-qwp2p\" (UID: \"c9ad71b7-ad90-4c8c-bf27-0f10b23f1593\") " pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" Oct 30 17:00:08.184203 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:08.184161 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9ad71b7-ad90-4c8c-bf27-0f10b23f1593-model-cache\") pod \"premium-w0-096d75-kserve-8477779bfc-qwp2p\" (UID: \"c9ad71b7-ad90-4c8c-bf27-0f10b23f1593\") " pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" Oct 30 17:00:08.184612 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:08.184585 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9ad71b7-ad90-4c8c-bf27-0f10b23f1593-model-cache\") pod \"premium-w0-096d75-kserve-8477779bfc-qwp2p\" (UID: \"c9ad71b7-ad90-4c8c-bf27-0f10b23f1593\") " pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" Oct 30 17:00:08.184737 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:08.184623 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9ad71b7-ad90-4c8c-bf27-0f10b23f1593-tmp-dir\") pod \"premium-w0-096d75-kserve-8477779bfc-qwp2p\" (UID: \"c9ad71b7-ad90-4c8c-bf27-0f10b23f1593\") " pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" Oct 30 17:00:08.184737 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:08.184649 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9ad71b7-ad90-4c8c-bf27-0f10b23f1593-home\") pod \"premium-w0-096d75-kserve-8477779bfc-qwp2p\" (UID: \"c9ad71b7-ad90-4c8c-bf27-0f10b23f1593\") " pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" Oct 30 17:00:08.186487 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:08.186467 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9ad71b7-ad90-4c8c-bf27-0f10b23f1593-dshm\") pod \"premium-w0-096d75-kserve-8477779bfc-qwp2p\" (UID: \"c9ad71b7-ad90-4c8c-bf27-0f10b23f1593\") " pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" Oct 30 17:00:08.186846 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:08.186824 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9ad71b7-ad90-4c8c-bf27-0f10b23f1593-tls-certs\") pod \"premium-w0-096d75-kserve-8477779bfc-qwp2p\" (UID: \"c9ad71b7-ad90-4c8c-bf27-0f10b23f1593\") " pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" Oct 30 17:00:08.221647 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:08.221563 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lrmc\" (UniqueName: \"kubernetes.io/projected/c9ad71b7-ad90-4c8c-bf27-0f10b23f1593-kube-api-access-5lrmc\") pod \"premium-w0-096d75-kserve-8477779bfc-qwp2p\" (UID: \"c9ad71b7-ad90-4c8c-bf27-0f10b23f1593\") " pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" Oct 30 17:00:08.319495 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:08.319455 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" Oct 30 17:00:08.624677 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:00:08.624641 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9ad71b7_ad90_4c8c_bf27_0f10b23f1593.slice/crio-4de15b3f22a48710206150e3f8bea0fecb80e5246799eb74c8f60dae7b3b7ee2 WatchSource:0}: Error finding container 4de15b3f22a48710206150e3f8bea0fecb80e5246799eb74c8f60dae7b3b7ee2: Status 404 returned error can't find the container with id 4de15b3f22a48710206150e3f8bea0fecb80e5246799eb74c8f60dae7b3b7ee2 Oct 30 17:00:08.625043 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:08.625023 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p"] Oct 30 17:00:08.975235 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:08.975198 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" event={"ID":"c9ad71b7-ad90-4c8c-bf27-0f10b23f1593","Type":"ContainerStarted","Data":"bf05f7463cc12ae7785f536115fbfa2a4ff775a268d2b6698e6b34725cdded74"} Oct 30 17:00:08.975723 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:08.975247 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" event={"ID":"c9ad71b7-ad90-4c8c-bf27-0f10b23f1593","Type":"ContainerStarted","Data":"4de15b3f22a48710206150e3f8bea0fecb80e5246799eb74c8f60dae7b3b7ee2"} Oct 30 17:00:08.977084 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:08.977064 4522 scope.go:122] "RemoveContainer" containerID="150ef50a4084c7539c20de5cebdbb53a4c0b3eb19566b1bf053877943c39c845" Oct 30 17:00:08.977398 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:00:08.977373 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-b4f406-29889660-bntv4_odh-ai-gateway-infra(3118473a-8879-4b2c-b309-96003f326cf9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889660-bntv4" podUID="3118473a-8879-4b2c-b309-96003f326cf9" Oct 30 17:00:09.020585 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:09.020533 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" podStartSLOduration=2.02052005 podStartE2EDuration="2.02052005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:09.019919989 +0000 UTC m=+783.020674916" watchObservedRunningTime="2026-10-30 17:00:09.02052005 +0000 UTC m=+783.021274950" Oct 30 17:00:09.320372 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:09.320279 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" Oct 30 17:00:12.950517 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:00:12.950480 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:00:18.138500 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:18.138461 4522 scope.go:122] "RemoveContainer" containerID="26cf68a78a944a815417637e21a07b23e8b4f899b5bf50e6233cdec987b8de6f" Oct 30 17:00:18.320274 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:18.320239 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" Oct 30 17:00:18.332834 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:18.332811 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" Oct 30 17:00:18.824854 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:00:18.824824 4522 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9004635d_a688_4763_9817_b4c39b2e8dac.slice/crio-7a109023f8a5e688c61ea797439b962725307a5e117e5b7c54249a8b087ac83b.scope\": RecentStats: unable to find data in memory cache]" Oct 30 17:00:19.036403 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:19.036369 4522 generic.go:356] "Generic (PLEG): container finished" podID="9004635d-a688-4763-9817-b4c39b2e8dac" containerID="7a109023f8a5e688c61ea797439b962725307a5e117e5b7c54249a8b087ac83b" exitCode=60 Oct 30 17:00:19.036579 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:19.036448 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889660-2b9gw" event={"ID":"9004635d-a688-4763-9817-b4c39b2e8dac","Type":"ContainerDied","Data":"7a109023f8a5e688c61ea797439b962725307a5e117e5b7c54249a8b087ac83b"} Oct 30 17:00:19.036579 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:19.036537 4522 scope.go:122] "RemoveContainer" containerID="26cf68a78a944a815417637e21a07b23e8b4f899b5bf50e6233cdec987b8de6f" Oct 30 17:00:19.037707 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:19.037034 4522 scope.go:122] "RemoveContainer" containerID="7a109023f8a5e688c61ea797439b962725307a5e117e5b7c54249a8b087ac83b" Oct 30 17:00:19.037707 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:00:19.037343 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-d384dc-29889660-2b9gw_odh-ai-gateway-infra(9004635d-a688-4763-9817-b4c39b2e8dac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889660-2b9gw" podUID="9004635d-a688-4763-9817-b4c39b2e8dac" Oct 30 17:00:19.049101 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:19.049051 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" Oct 30 17:00:19.137412 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:19.137333 4522 scope.go:122] "RemoveContainer" containerID="c21bceabc4b5685d82fa9819fd54797eacb7bfe301205ce49e7a8b473e6ab942" Oct 30 17:00:19.137560 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:19.137418 4522 scope.go:122] "RemoveContainer" containerID="150ef50a4084c7539c20de5cebdbb53a4c0b3eb19566b1bf053877943c39c845" Oct 30 17:00:20.044394 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:20.044357 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889660-bntv4" event={"ID":"3118473a-8879-4b2c-b309-96003f326cf9","Type":"ContainerStarted","Data":"5ede6de93b4769375f8398d89ca0027398d4c5e6fc71d0d0c922e90963058546"} Oct 30 17:00:20.046378 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:20.046349 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889660-x6p67" event={"ID":"aa253ff9-890d-47e8-9b2a-2a7e586502f1","Type":"ContainerStarted","Data":"5a4e41546fb3bd75c2ebd2618652d3904caa6070e9e8553e502b8f28ee887916"} Oct 30 17:00:20.103689 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:20.103654 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889660-2b9gw"] Oct 30 17:00:20.132116 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:20.132056 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889660-x6p67" podStartSLOduration=16.258856199 podStartE2EDuration="20.132036895s" totalImagesPullingTime="3.873180696s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:20.130763176 +0000 UTC m=+794.131518077" watchObservedRunningTime="2026-10-30 17:00:20.132036895 +0000 UTC m=+794.132791797" Oct 30 17:00:20.346153 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:20.346084 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889660-2b9gw" Oct 30 17:00:20.424145 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:20.424108 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9004635d-a688-4763-9817-b4c39b2e8dac-kube-api-access-gdl2x\" (UniqueName: \"kubernetes.io/projected/9004635d-a688-4763-9817-b4c39b2e8dac-kube-api-access-gdl2x\") pod \"9004635d-a688-4763-9817-b4c39b2e8dac\" (UID: \"9004635d-a688-4763-9817-b4c39b2e8dac\") " Oct 30 17:00:20.424399 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:20.424378 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9004635d-a688-4763-9817-b4c39b2e8dac-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9004635d-a688-4763-9817-b4c39b2e8dac-maas-api-service-ca\") pod \"9004635d-a688-4763-9817-b4c39b2e8dac\" (UID: \"9004635d-a688-4763-9817-b4c39b2e8dac\") " Oct 30 17:00:20.424700 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:20.424672 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9004635d-a688-4763-9817-b4c39b2e8dac-maas-api-service-ca" pod "9004635d-a688-4763-9817-b4c39b2e8dac" (UID: "9004635d-a688-4763-9817-b4c39b2e8dac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:20.424883 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:20.424866 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9004635d-a688-4763-9817-b4c39b2e8dac-maas-api-service-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:00:20.426433 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:20.426411 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9004635d-a688-4763-9817-b4c39b2e8dac-kube-api-access-gdl2x" pod "9004635d-a688-4763-9817-b4c39b2e8dac" (UID: "9004635d-a688-4763-9817-b4c39b2e8dac"). InnerVolumeSpecName "kube-api-access-gdl2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:20.525979 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:20.525944 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdl2x\" (UniqueName: \"kubernetes.io/projected/9004635d-a688-4763-9817-b4c39b2e8dac-kube-api-access-gdl2x\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:00:21.054781 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:21.054739 4522 generic.go:356] "Generic (PLEG): container finished" podID="aa253ff9-890d-47e8-9b2a-2a7e586502f1" containerID="5a4e41546fb3bd75c2ebd2618652d3904caa6070e9e8553e502b8f28ee887916" exitCode=60 Oct 30 17:00:21.055256 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:21.054810 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889660-x6p67" event={"ID":"aa253ff9-890d-47e8-9b2a-2a7e586502f1","Type":"ContainerDied","Data":"5a4e41546fb3bd75c2ebd2618652d3904caa6070e9e8553e502b8f28ee887916"} Oct 30 17:00:21.055256 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:21.054881 4522 scope.go:122] "RemoveContainer" containerID="c21bceabc4b5685d82fa9819fd54797eacb7bfe301205ce49e7a8b473e6ab942" Oct 30 17:00:21.055378 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:21.055277 4522 scope.go:122] "RemoveContainer" containerID="5a4e41546fb3bd75c2ebd2618652d3904caa6070e9e8553e502b8f28ee887916" Oct 30 17:00:21.055630 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:00:21.055603 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b80d83-29889660-x6p67_odh-ai-gateway-infra(aa253ff9-890d-47e8-9b2a-2a7e586502f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889660-x6p67" podUID="aa253ff9-890d-47e8-9b2a-2a7e586502f1" Oct 30 17:00:21.056416 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:21.056390 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889660-2b9gw" Oct 30 17:00:21.056533 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:21.056443 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889660-2b9gw" event={"ID":"9004635d-a688-4763-9817-b4c39b2e8dac","Type":"ContainerDied","Data":"42f0365f7179144f8bb60c5e9dea10c288fb5006afefaf9bed2b7c0dde7741ff"} Oct 30 17:00:21.058375 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:21.058348 4522 generic.go:356] "Generic (PLEG): container finished" podID="3118473a-8879-4b2c-b309-96003f326cf9" containerID="5ede6de93b4769375f8398d89ca0027398d4c5e6fc71d0d0c922e90963058546" exitCode=60 Oct 30 17:00:21.058455 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:21.058389 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889660-bntv4" event={"ID":"3118473a-8879-4b2c-b309-96003f326cf9","Type":"ContainerDied","Data":"5ede6de93b4769375f8398d89ca0027398d4c5e6fc71d0d0c922e90963058546"} Oct 30 17:00:21.058712 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:21.058697 4522 scope.go:122] "RemoveContainer" containerID="5ede6de93b4769375f8398d89ca0027398d4c5e6fc71d0d0c922e90963058546" Oct 30 17:00:21.058902 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:00:21.058886 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-b4f406-29889660-bntv4_odh-ai-gateway-infra(3118473a-8879-4b2c-b309-96003f326cf9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889660-bntv4" podUID="3118473a-8879-4b2c-b309-96003f326cf9" Oct 30 17:00:21.067407 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:21.067388 4522 scope.go:122] "RemoveContainer" containerID="7a109023f8a5e688c61ea797439b962725307a5e117e5b7c54249a8b087ac83b" Oct 30 17:00:21.075807 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:21.075788 4522 scope.go:122] "RemoveContainer" containerID="150ef50a4084c7539c20de5cebdbb53a4c0b3eb19566b1bf053877943c39c845" Oct 30 17:00:21.098181 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:21.098150 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889660-bntv4"] Oct 30 17:00:21.140476 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:21.140446 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889660-2b9gw"] Oct 30 17:00:21.142607 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:21.142575 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889660-x6p67"] Oct 30 17:00:21.148661 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:21.148637 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889660-2b9gw"] Oct 30 17:00:22.138076 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:22.137706 4522 scope.go:122] "RemoveContainer" containerID="b0f73594cd30343282c1d17e214601dae1a6df7aa27d16747deca89ec4fa531c" Oct 30 17:00:22.148684 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:22.148651 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9004635d-a688-4763-9817-b4c39b2e8dac" path="/var/lib/kubelet/pods/9004635d-a688-4763-9817-b4c39b2e8dac/volumes" Oct 30 17:00:22.243281 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:22.243255 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889660-bntv4" Oct 30 17:00:22.245830 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:22.245813 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889660-x6p67" Oct 30 17:00:22.345781 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:22.345687 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa253ff9-890d-47e8-9b2a-2a7e586502f1-kube-api-access-btqhk\" (UniqueName: \"kubernetes.io/projected/aa253ff9-890d-47e8-9b2a-2a7e586502f1-kube-api-access-btqhk\") pod \"aa253ff9-890d-47e8-9b2a-2a7e586502f1\" (UID: \"aa253ff9-890d-47e8-9b2a-2a7e586502f1\") " Oct 30 17:00:22.345781 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:22.345720 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3118473a-8879-4b2c-b309-96003f326cf9-kube-api-access-sjcsl\" (UniqueName: \"kubernetes.io/projected/3118473a-8879-4b2c-b309-96003f326cf9-kube-api-access-sjcsl\") pod \"3118473a-8879-4b2c-b309-96003f326cf9\" (UID: \"3118473a-8879-4b2c-b309-96003f326cf9\") " Oct 30 17:00:22.345781 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:22.345773 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3118473a-8879-4b2c-b309-96003f326cf9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3118473a-8879-4b2c-b309-96003f326cf9-maas-api-service-ca\") pod \"3118473a-8879-4b2c-b309-96003f326cf9\" (UID: \"3118473a-8879-4b2c-b309-96003f326cf9\") " Oct 30 17:00:22.346082 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:22.345811 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa253ff9-890d-47e8-9b2a-2a7e586502f1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa253ff9-890d-47e8-9b2a-2a7e586502f1-maas-api-service-ca\") pod \"aa253ff9-890d-47e8-9b2a-2a7e586502f1\" (UID: \"aa253ff9-890d-47e8-9b2a-2a7e586502f1\") " Oct 30 17:00:22.346192 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:22.346162 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3118473a-8879-4b2c-b309-96003f326cf9-maas-api-service-ca" pod "3118473a-8879-4b2c-b309-96003f326cf9" (UID: "3118473a-8879-4b2c-b309-96003f326cf9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:22.346299 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:22.346188 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa253ff9-890d-47e8-9b2a-2a7e586502f1-maas-api-service-ca" pod "aa253ff9-890d-47e8-9b2a-2a7e586502f1" (UID: "aa253ff9-890d-47e8-9b2a-2a7e586502f1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:22.346437 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:22.346420 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3118473a-8879-4b2c-b309-96003f326cf9-maas-api-service-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:00:22.346488 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:22.346443 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa253ff9-890d-47e8-9b2a-2a7e586502f1-maas-api-service-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:00:22.347955 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:22.347925 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3118473a-8879-4b2c-b309-96003f326cf9-kube-api-access-sjcsl" pod "3118473a-8879-4b2c-b309-96003f326cf9" (UID: "3118473a-8879-4b2c-b309-96003f326cf9"). InnerVolumeSpecName "kube-api-access-sjcsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:22.348060 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:22.348028 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa253ff9-890d-47e8-9b2a-2a7e586502f1-kube-api-access-btqhk" pod "aa253ff9-890d-47e8-9b2a-2a7e586502f1" (UID: "aa253ff9-890d-47e8-9b2a-2a7e586502f1"). InnerVolumeSpecName "kube-api-access-btqhk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:22.447253 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:22.447227 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btqhk\" (UniqueName: \"kubernetes.io/projected/aa253ff9-890d-47e8-9b2a-2a7e586502f1-kube-api-access-btqhk\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:00:22.447253 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:22.447254 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjcsl\" (UniqueName: \"kubernetes.io/projected/3118473a-8879-4b2c-b309-96003f326cf9-kube-api-access-sjcsl\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:00:23.074711 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:23.074681 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889660-x6p67" Oct 30 17:00:23.074880 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:23.074722 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889660-x6p67" event={"ID":"aa253ff9-890d-47e8-9b2a-2a7e586502f1","Type":"ContainerDied","Data":"9b944c687fa278563307c42c7e201431ba9a57459c66c943deadb030138ee378"} Oct 30 17:00:23.074880 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:23.074791 4522 scope.go:122] "RemoveContainer" containerID="5a4e41546fb3bd75c2ebd2618652d3904caa6070e9e8553e502b8f28ee887916" Oct 30 17:00:23.076184 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:23.076168 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889660-bntv4" Oct 30 17:00:23.076272 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:23.076219 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889660-bntv4" event={"ID":"3118473a-8879-4b2c-b309-96003f326cf9","Type":"ContainerDied","Data":"5607d932182315a7b6abf66c83c3d5b28034a5835ff30445796b496dd89cb206"} Oct 30 17:00:23.078461 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:23.078437 4522 generic.go:356] "Generic (PLEG): container finished" podID="f361d37b-d1f9-45ae-9874-859da12af710" containerID="0627fbd85b2934fdbfe1980963706e6793a5e57c759a8cb7365f5d452525e4fd" exitCode=60 Oct 30 17:00:23.078579 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:23.078512 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889660-l58hw" event={"ID":"f361d37b-d1f9-45ae-9874-859da12af710","Type":"ContainerDied","Data":"0627fbd85b2934fdbfe1980963706e6793a5e57c759a8cb7365f5d452525e4fd"} Oct 30 17:00:23.078957 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:23.078936 4522 scope.go:122] "RemoveContainer" containerID="0627fbd85b2934fdbfe1980963706e6793a5e57c759a8cb7365f5d452525e4fd" Oct 30 17:00:23.079236 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:00:23.079206 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889660-l58hw_odh-ai-gateway-infra(f361d37b-d1f9-45ae-9874-859da12af710)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889660-l58hw" podUID="f361d37b-d1f9-45ae-9874-859da12af710" Oct 30 17:00:23.087215 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:23.087093 4522 scope.go:122] "RemoveContainer" containerID="5ede6de93b4769375f8398d89ca0027398d4c5e6fc71d0d0c922e90963058546" Oct 30 17:00:23.096530 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:23.096510 4522 scope.go:122] "RemoveContainer" containerID="b0f73594cd30343282c1d17e214601dae1a6df7aa27d16747deca89ec4fa531c" Oct 30 17:00:23.130112 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:23.130086 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889660-x6p67"] Oct 30 17:00:23.135816 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:23.135792 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889660-x6p67"] Oct 30 17:00:23.175834 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:23.175806 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889660-bntv4"] Oct 30 17:00:23.186082 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:23.186055 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889660-bntv4"] Oct 30 17:00:24.144375 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:24.144344 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3118473a-8879-4b2c-b309-96003f326cf9" path="/var/lib/kubelet/pods/3118473a-8879-4b2c-b309-96003f326cf9/volumes" Oct 30 17:00:24.144789 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:24.144775 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa253ff9-890d-47e8-9b2a-2a7e586502f1" path="/var/lib/kubelet/pods/aa253ff9-890d-47e8-9b2a-2a7e586502f1/volumes" Oct 30 17:00:24.282790 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:24.282741 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889660-l58hw"] Oct 30 17:00:24.569806 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:24.569773 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889660-l58hw" Oct 30 17:00:24.687743 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:24.687711 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f361d37b-d1f9-45ae-9874-859da12af710-kube-api-access-4dx77\" (UniqueName: \"kubernetes.io/projected/f361d37b-d1f9-45ae-9874-859da12af710-kube-api-access-4dx77\") pod \"f361d37b-d1f9-45ae-9874-859da12af710\" (UID: \"f361d37b-d1f9-45ae-9874-859da12af710\") " Oct 30 17:00:24.688005 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:24.687985 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f361d37b-d1f9-45ae-9874-859da12af710-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f361d37b-d1f9-45ae-9874-859da12af710-maas-api-service-ca\") pod \"f361d37b-d1f9-45ae-9874-859da12af710\" (UID: \"f361d37b-d1f9-45ae-9874-859da12af710\") " Oct 30 17:00:24.688590 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:24.688558 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f361d37b-d1f9-45ae-9874-859da12af710-maas-api-service-ca" pod "f361d37b-d1f9-45ae-9874-859da12af710" (UID: "f361d37b-d1f9-45ae-9874-859da12af710"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:24.690059 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:24.690027 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f361d37b-d1f9-45ae-9874-859da12af710-kube-api-access-4dx77" pod "f361d37b-d1f9-45ae-9874-859da12af710" (UID: "f361d37b-d1f9-45ae-9874-859da12af710"). InnerVolumeSpecName "kube-api-access-4dx77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:24.789799 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:24.789714 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f361d37b-d1f9-45ae-9874-859da12af710-maas-api-service-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:00:24.789799 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:24.789743 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dx77\" (UniqueName: \"kubernetes.io/projected/f361d37b-d1f9-45ae-9874-859da12af710-kube-api-access-4dx77\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:00:25.096756 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:25.096676 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889660-l58hw" Oct 30 17:00:25.096756 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:25.096684 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889660-l58hw" event={"ID":"f361d37b-d1f9-45ae-9874-859da12af710","Type":"ContainerDied","Data":"cb1ea0728b785ad6f5fc531e1300c0be6a2674d6bd73524dbb57519a319f2dde"} Oct 30 17:00:25.096954 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:25.096793 4522 scope.go:122] "RemoveContainer" containerID="0627fbd85b2934fdbfe1980963706e6793a5e57c759a8cb7365f5d452525e4fd" Oct 30 17:00:25.179375 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:25.179345 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889660-l58hw"] Oct 30 17:00:25.184067 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:25.184034 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889660-l58hw"] Oct 30 17:00:26.143895 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:26.143867 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f361d37b-d1f9-45ae-9874-859da12af710" path="/var/lib/kubelet/pods/f361d37b-d1f9-45ae-9874-859da12af710/volumes" Oct 30 17:00:28.527897 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:28.527856 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f"] Oct 30 17:00:28.528439 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:28.528413 4522 cpu_manager.go:414] "containerMap: removing container" podUID="f361d37b-d1f9-45ae-9874-859da12af710" containerName="cleanup" Oct 30 17:00:28.528439 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:28.528429 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f361d37b-d1f9-45ae-9874-859da12af710" containerName="cleanup" Oct 30 17:00:28.528439 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:28.528441 4522 cpu_manager.go:414] "containerMap: removing container" podUID="aa253ff9-890d-47e8-9b2a-2a7e586502f1" containerName="cleanup" Oct 30 17:00:28.528607 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:28.528447 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa253ff9-890d-47e8-9b2a-2a7e586502f1" containerName="cleanup" Oct 30 17:00:28.528607 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:28.528460 4522 cpu_manager.go:414] "containerMap: removing container" podUID="aa253ff9-890d-47e8-9b2a-2a7e586502f1" containerName="cleanup" Oct 30 17:00:28.528607 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:28.528465 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa253ff9-890d-47e8-9b2a-2a7e586502f1" containerName="cleanup" Oct 30 17:00:28.528607 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:28.528473 4522 cpu_manager.go:414] "containerMap: removing container" podUID="9004635d-a688-4763-9817-b4c39b2e8dac" containerName="cleanup" Oct 30 17:00:28.528607 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:28.528478 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9004635d-a688-4763-9817-b4c39b2e8dac" containerName="cleanup" Oct 30 17:00:28.528607 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:28.528486 4522 cpu_manager.go:414] "containerMap: removing container" podUID="3118473a-8879-4b2c-b309-96003f326cf9" containerName="cleanup" Oct 30 17:00:28.528607 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:28.528491 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3118473a-8879-4b2c-b309-96003f326cf9" containerName="cleanup" Oct 30 17:00:28.528607 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:28.528500 4522 cpu_manager.go:414] "containerMap: removing container" podUID="aa253ff9-890d-47e8-9b2a-2a7e586502f1" containerName="cleanup" Oct 30 17:00:28.528607 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:28.528504 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa253ff9-890d-47e8-9b2a-2a7e586502f1" containerName="cleanup" Oct 30 17:00:28.528607 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:28.528511 4522 cpu_manager.go:414] "containerMap: removing container" podUID="f361d37b-d1f9-45ae-9874-859da12af710" containerName="cleanup" Oct 30 17:00:28.528607 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:28.528515 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f361d37b-d1f9-45ae-9874-859da12af710" containerName="cleanup" Oct 30 17:00:28.528607 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:28.528522 4522 cpu_manager.go:414] "containerMap: removing container" podUID="3118473a-8879-4b2c-b309-96003f326cf9" containerName="cleanup" Oct 30 17:00:28.528607 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:28.528526 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3118473a-8879-4b2c-b309-96003f326cf9" containerName="cleanup" Oct 30 17:00:28.528607 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:28.528534 4522 cpu_manager.go:414] "containerMap: removing container" podUID="f361d37b-d1f9-45ae-9874-859da12af710" containerName="cleanup" Oct 30 17:00:28.528607 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:28.528538 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f361d37b-d1f9-45ae-9874-859da12af710" containerName="cleanup" Oct 30 17:00:28.528607 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:28.528548 4522 cpu_manager.go:414] "containerMap: removing container" podUID="9004635d-a688-4763-9817-b4c39b2e8dac" containerName="cleanup" Oct 30 17:00:28.528607 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:28.528552 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9004635d-a688-4763-9817-b4c39b2e8dac" containerName="cleanup" Oct 30 17:00:28.528607 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:28.528571 4522 cpu_manager.go:414] "containerMap: removing container" podUID="9004635d-a688-4763-9817-b4c39b2e8dac" containerName="cleanup" Oct 30 17:00:28.528607 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:28.528575 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9004635d-a688-4763-9817-b4c39b2e8dac" containerName="cleanup" Oct 30 17:00:28.529531 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:28.528645 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa253ff9-890d-47e8-9b2a-2a7e586502f1" containerName="cleanup" Oct 30 17:00:28.529531 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:28.528652 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="f361d37b-d1f9-45ae-9874-859da12af710" containerName="cleanup" Oct 30 17:00:28.529531 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:28.528657 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="3118473a-8879-4b2c-b309-96003f326cf9" containerName="cleanup" Oct 30 17:00:28.529531 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:28.528662 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="f361d37b-d1f9-45ae-9874-859da12af710" containerName="cleanup" Oct 30 17:00:28.529531 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:28.528668 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="9004635d-a688-4763-9817-b4c39b2e8dac" containerName="cleanup" Oct 30 17:00:28.529531 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:28.528673 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="3118473a-8879-4b2c-b309-96003f326cf9" containerName="cleanup" Oct 30 17:00:28.529531 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:28.528679 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="9004635d-a688-4763-9817-b4c39b2e8dac" containerName="cleanup" Oct 30 17:00:28.529531 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:28.528684 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="f361d37b-d1f9-45ae-9874-859da12af710" containerName="cleanup" Oct 30 17:00:28.529531 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:28.528689 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa253ff9-890d-47e8-9b2a-2a7e586502f1" containerName="cleanup" Oct 30 17:00:28.529531 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:28.528694 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="9004635d-a688-4763-9817-b4c39b2e8dac" containerName="cleanup" Oct 30 17:00:28.529531 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:28.528699 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa253ff9-890d-47e8-9b2a-2a7e586502f1" containerName="cleanup" Oct 30 17:00:28.538235 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:28.538205 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" Oct 30 17:00:28.551796 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:28.551766 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f"] Oct 30 17:00:28.562757 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:28.562729 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-d384dc\"/\"premium-w6-d384dc-kserve-self-signed-certs\"" Oct 30 17:00:28.633072 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:28.633039 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a6a85ad-c944-450f-b0bb-e3b64b0f56f0-tmp-dir\") pod \"premium-w6-d384dc-kserve-7fcb98cb96-8956f\" (UID: \"8a6a85ad-c944-450f-b0bb-e3b64b0f56f0\") " pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" Oct 30 17:00:28.633263 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:28.633101 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a6a85ad-c944-450f-b0bb-e3b64b0f56f0-home\") pod \"premium-w6-d384dc-kserve-7fcb98cb96-8956f\" (UID: \"8a6a85ad-c944-450f-b0bb-e3b64b0f56f0\") " pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" Oct 30 17:00:28.633263 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:28.633212 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvprb\" (UniqueName: \"kubernetes.io/projected/8a6a85ad-c944-450f-b0bb-e3b64b0f56f0-kube-api-access-hvprb\") pod \"premium-w6-d384dc-kserve-7fcb98cb96-8956f\" (UID: \"8a6a85ad-c944-450f-b0bb-e3b64b0f56f0\") " pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" Oct 30 17:00:28.633406 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:28.633340 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a6a85ad-c944-450f-b0bb-e3b64b0f56f0-model-cache\") pod \"premium-w6-d384dc-kserve-7fcb98cb96-8956f\" (UID: \"8a6a85ad-c944-450f-b0bb-e3b64b0f56f0\") " pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" Oct 30 17:00:28.633406 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:28.633380 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a6a85ad-c944-450f-b0bb-e3b64b0f56f0-dshm\") pod \"premium-w6-d384dc-kserve-7fcb98cb96-8956f\" (UID: \"8a6a85ad-c944-450f-b0bb-e3b64b0f56f0\") " pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" Oct 30 17:00:28.633504 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:28.633409 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a6a85ad-c944-450f-b0bb-e3b64b0f56f0-tls-certs\") pod \"premium-w6-d384dc-kserve-7fcb98cb96-8956f\" (UID: \"8a6a85ad-c944-450f-b0bb-e3b64b0f56f0\") " pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" Oct 30 17:00:28.734714 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:28.734678 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a6a85ad-c944-450f-b0bb-e3b64b0f56f0-model-cache\") pod \"premium-w6-d384dc-kserve-7fcb98cb96-8956f\" (UID: \"8a6a85ad-c944-450f-b0bb-e3b64b0f56f0\") " pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" Oct 30 17:00:28.734714 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:28.734720 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a6a85ad-c944-450f-b0bb-e3b64b0f56f0-dshm\") pod \"premium-w6-d384dc-kserve-7fcb98cb96-8956f\" (UID: \"8a6a85ad-c944-450f-b0bb-e3b64b0f56f0\") " pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" Oct 30 17:00:28.734969 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:28.734739 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a6a85ad-c944-450f-b0bb-e3b64b0f56f0-tls-certs\") pod \"premium-w6-d384dc-kserve-7fcb98cb96-8956f\" (UID: \"8a6a85ad-c944-450f-b0bb-e3b64b0f56f0\") " pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" Oct 30 17:00:28.734969 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:28.734848 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a6a85ad-c944-450f-b0bb-e3b64b0f56f0-tmp-dir\") pod \"premium-w6-d384dc-kserve-7fcb98cb96-8956f\" (UID: \"8a6a85ad-c944-450f-b0bb-e3b64b0f56f0\") " pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" Oct 30 17:00:28.734969 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:28.734919 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a6a85ad-c944-450f-b0bb-e3b64b0f56f0-home\") pod \"premium-w6-d384dc-kserve-7fcb98cb96-8956f\" (UID: \"8a6a85ad-c944-450f-b0bb-e3b64b0f56f0\") " pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" Oct 30 17:00:28.734969 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:28.734945 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvprb\" (UniqueName: \"kubernetes.io/projected/8a6a85ad-c944-450f-b0bb-e3b64b0f56f0-kube-api-access-hvprb\") pod \"premium-w6-d384dc-kserve-7fcb98cb96-8956f\" (UID: \"8a6a85ad-c944-450f-b0bb-e3b64b0f56f0\") " pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" Oct 30 17:00:28.735175 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:28.735135 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a6a85ad-c944-450f-b0bb-e3b64b0f56f0-model-cache\") pod \"premium-w6-d384dc-kserve-7fcb98cb96-8956f\" (UID: \"8a6a85ad-c944-450f-b0bb-e3b64b0f56f0\") " pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" Oct 30 17:00:28.735241 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:28.735217 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a6a85ad-c944-450f-b0bb-e3b64b0f56f0-tmp-dir\") pod \"premium-w6-d384dc-kserve-7fcb98cb96-8956f\" (UID: \"8a6a85ad-c944-450f-b0bb-e3b64b0f56f0\") " pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" Oct 30 17:00:28.735386 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:28.735363 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a6a85ad-c944-450f-b0bb-e3b64b0f56f0-home\") pod \"premium-w6-d384dc-kserve-7fcb98cb96-8956f\" (UID: \"8a6a85ad-c944-450f-b0bb-e3b64b0f56f0\") " pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" Oct 30 17:00:28.737701 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:28.737680 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a6a85ad-c944-450f-b0bb-e3b64b0f56f0-tls-certs\") pod \"premium-w6-d384dc-kserve-7fcb98cb96-8956f\" (UID: \"8a6a85ad-c944-450f-b0bb-e3b64b0f56f0\") " pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" Oct 30 17:00:28.740017 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:28.739994 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a6a85ad-c944-450f-b0bb-e3b64b0f56f0-dshm\") pod \"premium-w6-d384dc-kserve-7fcb98cb96-8956f\" (UID: \"8a6a85ad-c944-450f-b0bb-e3b64b0f56f0\") " pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" Oct 30 17:00:28.763740 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:28.762374 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvprb\" (UniqueName: \"kubernetes.io/projected/8a6a85ad-c944-450f-b0bb-e3b64b0f56f0-kube-api-access-hvprb\") pod \"premium-w6-d384dc-kserve-7fcb98cb96-8956f\" (UID: \"8a6a85ad-c944-450f-b0bb-e3b64b0f56f0\") " pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" Oct 30 17:00:28.850606 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:28.850515 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" Oct 30 17:00:29.156122 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:29.156072 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f"] Oct 30 17:00:29.157088 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:00:29.157056 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a6a85ad_c944_450f_b0bb_e3b64b0f56f0.slice/crio-5cf12ca50dcd0c769d232c8a09e7a0bc54790beed990a25aff34a2647edcf351 WatchSource:0}: Error finding container 5cf12ca50dcd0c769d232c8a09e7a0bc54790beed990a25aff34a2647edcf351: Status 404 returned error can't find the container with id 5cf12ca50dcd0c769d232c8a09e7a0bc54790beed990a25aff34a2647edcf351 Oct 30 17:00:30.126571 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:30.126534 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" event={"ID":"8a6a85ad-c944-450f-b0bb-e3b64b0f56f0","Type":"ContainerStarted","Data":"a2b7cbb389bdee3d842c9043f5abd81df99eac77a4b68b502e2fb699c8fbe56b"} Oct 30 17:00:30.126978 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:30.126582 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" event={"ID":"8a6a85ad-c944-450f-b0bb-e3b64b0f56f0","Type":"ContainerStarted","Data":"5cf12ca50dcd0c769d232c8a09e7a0bc54790beed990a25aff34a2647edcf351"} Oct 30 17:00:30.171026 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:30.170976 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" podStartSLOduration=2.170962446 podStartE2EDuration="2.170962446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:30.170189731 +0000 UTC m=+804.170944633" watchObservedRunningTime="2026-10-30 17:00:30.170962446 +0000 UTC m=+804.171717347" Oct 30 17:00:30.851575 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:30.851548 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" Oct 30 17:00:34.175337 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:34.172785 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw"] Oct 30 17:00:34.181326 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:34.181158 4522 cpu_manager.go:414] "containerMap: removing container" podUID="3118473a-8879-4b2c-b309-96003f326cf9" containerName="cleanup" Oct 30 17:00:34.181326 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:34.181187 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3118473a-8879-4b2c-b309-96003f326cf9" containerName="cleanup" Oct 30 17:00:34.185438 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:34.185412 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="3118473a-8879-4b2c-b309-96003f326cf9" containerName="cleanup" Oct 30 17:00:34.198974 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:34.198941 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw"] Oct 30 17:00:34.199140 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:34.199073 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" Oct 30 17:00:34.229417 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:34.229392 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c7b9df\"/\"premium-w4-c7b9df-kserve-self-signed-certs\"" Oct 30 17:00:34.300248 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:34.300214 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43ca764d-ae0e-4858-9841-5bbd0ffdbb85-home\") pod \"premium-w4-c7b9df-kserve-7cd7bb565-gjrvw\" (UID: \"43ca764d-ae0e-4858-9841-5bbd0ffdbb85\") " pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" Oct 30 17:00:34.300248 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:34.300250 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43ca764d-ae0e-4858-9841-5bbd0ffdbb85-dshm\") pod \"premium-w4-c7b9df-kserve-7cd7bb565-gjrvw\" (UID: \"43ca764d-ae0e-4858-9841-5bbd0ffdbb85\") " pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" Oct 30 17:00:34.300498 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:34.300281 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43ca764d-ae0e-4858-9841-5bbd0ffdbb85-model-cache\") pod \"premium-w4-c7b9df-kserve-7cd7bb565-gjrvw\" (UID: \"43ca764d-ae0e-4858-9841-5bbd0ffdbb85\") " pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" Oct 30 17:00:34.300498 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:34.300426 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43ca764d-ae0e-4858-9841-5bbd0ffdbb85-tls-certs\") pod \"premium-w4-c7b9df-kserve-7cd7bb565-gjrvw\" (UID: \"43ca764d-ae0e-4858-9841-5bbd0ffdbb85\") " pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" Oct 30 17:00:34.300498 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:34.300475 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjmdc\" (UniqueName: \"kubernetes.io/projected/43ca764d-ae0e-4858-9841-5bbd0ffdbb85-kube-api-access-cjmdc\") pod \"premium-w4-c7b9df-kserve-7cd7bb565-gjrvw\" (UID: \"43ca764d-ae0e-4858-9841-5bbd0ffdbb85\") " pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" Oct 30 17:00:34.300597 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:34.300533 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43ca764d-ae0e-4858-9841-5bbd0ffdbb85-tmp-dir\") pod \"premium-w4-c7b9df-kserve-7cd7bb565-gjrvw\" (UID: \"43ca764d-ae0e-4858-9841-5bbd0ffdbb85\") " pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" Oct 30 17:00:34.402633 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:34.402243 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43ca764d-ae0e-4858-9841-5bbd0ffdbb85-tls-certs\") pod \"premium-w4-c7b9df-kserve-7cd7bb565-gjrvw\" (UID: \"43ca764d-ae0e-4858-9841-5bbd0ffdbb85\") " pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" Oct 30 17:00:34.402633 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:34.402351 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjmdc\" (UniqueName: \"kubernetes.io/projected/43ca764d-ae0e-4858-9841-5bbd0ffdbb85-kube-api-access-cjmdc\") pod \"premium-w4-c7b9df-kserve-7cd7bb565-gjrvw\" (UID: \"43ca764d-ae0e-4858-9841-5bbd0ffdbb85\") " pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" Oct 30 17:00:34.402633 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:34.402402 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43ca764d-ae0e-4858-9841-5bbd0ffdbb85-tmp-dir\") pod \"premium-w4-c7b9df-kserve-7cd7bb565-gjrvw\" (UID: \"43ca764d-ae0e-4858-9841-5bbd0ffdbb85\") " pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" Oct 30 17:00:34.402633 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:34.402495 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43ca764d-ae0e-4858-9841-5bbd0ffdbb85-home\") pod \"premium-w4-c7b9df-kserve-7cd7bb565-gjrvw\" (UID: \"43ca764d-ae0e-4858-9841-5bbd0ffdbb85\") " pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" Oct 30 17:00:34.402633 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:34.402522 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43ca764d-ae0e-4858-9841-5bbd0ffdbb85-dshm\") pod \"premium-w4-c7b9df-kserve-7cd7bb565-gjrvw\" (UID: \"43ca764d-ae0e-4858-9841-5bbd0ffdbb85\") " pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" Oct 30 17:00:34.402633 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:34.402551 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43ca764d-ae0e-4858-9841-5bbd0ffdbb85-model-cache\") pod \"premium-w4-c7b9df-kserve-7cd7bb565-gjrvw\" (UID: \"43ca764d-ae0e-4858-9841-5bbd0ffdbb85\") " pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" Oct 30 17:00:34.403105 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:34.403021 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43ca764d-ae0e-4858-9841-5bbd0ffdbb85-model-cache\") pod \"premium-w4-c7b9df-kserve-7cd7bb565-gjrvw\" (UID: \"43ca764d-ae0e-4858-9841-5bbd0ffdbb85\") " pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" Oct 30 17:00:34.404110 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:34.404077 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43ca764d-ae0e-4858-9841-5bbd0ffdbb85-tmp-dir\") pod \"premium-w4-c7b9df-kserve-7cd7bb565-gjrvw\" (UID: \"43ca764d-ae0e-4858-9841-5bbd0ffdbb85\") " pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" Oct 30 17:00:34.408702 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:34.404836 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43ca764d-ae0e-4858-9841-5bbd0ffdbb85-home\") pod \"premium-w4-c7b9df-kserve-7cd7bb565-gjrvw\" (UID: \"43ca764d-ae0e-4858-9841-5bbd0ffdbb85\") " pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" Oct 30 17:00:34.415649 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:34.415619 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43ca764d-ae0e-4858-9841-5bbd0ffdbb85-tls-certs\") pod \"premium-w4-c7b9df-kserve-7cd7bb565-gjrvw\" (UID: \"43ca764d-ae0e-4858-9841-5bbd0ffdbb85\") " pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" Oct 30 17:00:34.431773 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:34.431703 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43ca764d-ae0e-4858-9841-5bbd0ffdbb85-dshm\") pod \"premium-w4-c7b9df-kserve-7cd7bb565-gjrvw\" (UID: \"43ca764d-ae0e-4858-9841-5bbd0ffdbb85\") " pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" Oct 30 17:00:34.433493 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:34.433466 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjmdc\" (UniqueName: \"kubernetes.io/projected/43ca764d-ae0e-4858-9841-5bbd0ffdbb85-kube-api-access-cjmdc\") pod \"premium-w4-c7b9df-kserve-7cd7bb565-gjrvw\" (UID: \"43ca764d-ae0e-4858-9841-5bbd0ffdbb85\") " pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" Oct 30 17:00:34.521473 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:34.521435 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" Oct 30 17:00:34.908120 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:34.902240 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw"] Oct 30 17:00:35.163015 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:35.162981 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" event={"ID":"43ca764d-ae0e-4858-9841-5bbd0ffdbb85","Type":"ContainerStarted","Data":"81eb5c139949d65f6ec60db66498efa287a1b2a434ce8f91d626772bdec48053"} Oct 30 17:00:36.178782 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:36.178738 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" event={"ID":"43ca764d-ae0e-4858-9841-5bbd0ffdbb85","Type":"ContainerStarted","Data":"45e88413ae21492a881ddb38f38e1ccb7482d6fd2927208f40ef2f93b1ee8928"} Oct 30 17:00:36.230167 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:36.230099 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" podStartSLOduration=2.230078785 podStartE2EDuration="2.230078785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:36.227264857 +0000 UTC m=+810.228019761" watchObservedRunningTime="2026-10-30 17:00:36.230078785 +0000 UTC m=+810.230833687" Oct 30 17:00:36.522866 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:36.522827 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" Oct 30 17:00:38.851192 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:38.851158 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" Oct 30 17:00:38.868233 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:38.868198 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" Oct 30 17:00:39.205869 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:39.205840 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" Oct 30 17:00:42.959330 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:00:42.958379 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:00:44.522838 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:44.522796 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" Oct 30 17:00:44.538262 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:44.538234 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" Oct 30 17:00:45.327723 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:45.327681 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" Oct 30 17:00:49.101165 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:49.101129 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d03a028-7cmmw"] Oct 30 17:00:49.107344 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:49.107301 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d03a028-7cmmw" Oct 30 17:00:49.140950 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:49.140830 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:00:49.145609 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:49.145571 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d03a028-7cmmw"] Oct 30 17:00:49.145779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:49.145724 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:00:49.187575 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:49.187538 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/338f53d4-163f-49cf-a772-bc10a691ef68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d03a028-7cmmw\" (UID: \"338f53d4-163f-49cf-a772-bc10a691ef68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d03a028-7cmmw" Oct 30 17:00:49.187974 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:49.187951 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxhx6\" (UniqueName: \"kubernetes.io/projected/338f53d4-163f-49cf-a772-bc10a691ef68-kube-api-access-bxhx6\") pod \"maas-api-revoke-sub-6d03a028-7cmmw\" (UID: \"338f53d4-163f-49cf-a772-bc10a691ef68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d03a028-7cmmw" Oct 30 17:00:49.188228 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:49.188209 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/338f53d4-163f-49cf-a772-bc10a691ef68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d03a028-7cmmw\" (UID: \"338f53d4-163f-49cf-a772-bc10a691ef68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d03a028-7cmmw" Oct 30 17:00:49.289059 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:49.289025 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/338f53d4-163f-49cf-a772-bc10a691ef68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d03a028-7cmmw\" (UID: \"338f53d4-163f-49cf-a772-bc10a691ef68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d03a028-7cmmw" Oct 30 17:00:49.289518 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:49.289482 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxhx6\" (UniqueName: \"kubernetes.io/projected/338f53d4-163f-49cf-a772-bc10a691ef68-kube-api-access-bxhx6\") pod \"maas-api-revoke-sub-6d03a028-7cmmw\" (UID: \"338f53d4-163f-49cf-a772-bc10a691ef68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d03a028-7cmmw" Oct 30 17:00:49.289643 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:49.289541 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/338f53d4-163f-49cf-a772-bc10a691ef68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d03a028-7cmmw\" (UID: \"338f53d4-163f-49cf-a772-bc10a691ef68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d03a028-7cmmw" Oct 30 17:00:49.290329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:49.290281 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/338f53d4-163f-49cf-a772-bc10a691ef68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d03a028-7cmmw\" (UID: \"338f53d4-163f-49cf-a772-bc10a691ef68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d03a028-7cmmw" Oct 30 17:00:49.292727 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:49.292700 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/338f53d4-163f-49cf-a772-bc10a691ef68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d03a028-7cmmw\" (UID: \"338f53d4-163f-49cf-a772-bc10a691ef68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d03a028-7cmmw" Oct 30 17:00:49.314826 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:49.314795 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxhx6\" (UniqueName: \"kubernetes.io/projected/338f53d4-163f-49cf-a772-bc10a691ef68-kube-api-access-bxhx6\") pod \"maas-api-revoke-sub-6d03a028-7cmmw\" (UID: \"338f53d4-163f-49cf-a772-bc10a691ef68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d03a028-7cmmw" Oct 30 17:00:49.421326 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:49.421230 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d03a028-7cmmw" Oct 30 17:00:49.741212 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:49.741175 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d03a028-7cmmw"] Oct 30 17:00:49.746352 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:00:49.746294 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod338f53d4_163f_49cf_a772_bc10a691ef68.slice/crio-c51000fee2a49e3413425de63d59b6a324c323e1d406bce037726a34684c0c37 WatchSource:0}: Error finding container c51000fee2a49e3413425de63d59b6a324c323e1d406bce037726a34684c0c37: Status 404 returned error can't find the container with id c51000fee2a49e3413425de63d59b6a324c323e1d406bce037726a34684c0c37 Oct 30 17:00:50.300392 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:50.300059 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d03a028-7cmmw" event={"ID":"338f53d4-163f-49cf-a772-bc10a691ef68","Type":"ContainerStarted","Data":"ecce0806f2ee29c940e91b1ee9b06b3c328972b799f85b1737e38b3dc21645d1"} Oct 30 17:00:50.300853 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:50.300434 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d03a028-7cmmw" event={"ID":"338f53d4-163f-49cf-a772-bc10a691ef68","Type":"ContainerStarted","Data":"c51000fee2a49e3413425de63d59b6a324c323e1d406bce037726a34684c0c37"} Oct 30 17:00:50.358623 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:50.358505 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d03a028-7cmmw" podStartSLOduration=1.358489974 podStartE2EDuration="1.358489974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:50.357297248 +0000 UTC m=+824.358052169" watchObservedRunningTime="2026-10-30 17:00:50.358489974 +0000 UTC m=+824.359244875" Oct 30 17:00:51.306782 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:51.306746 4522 generic.go:356] "Generic (PLEG): container finished" podID="338f53d4-163f-49cf-a772-bc10a691ef68" containerID="ecce0806f2ee29c940e91b1ee9b06b3c328972b799f85b1737e38b3dc21645d1" exitCode=0 Oct 30 17:00:51.307167 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:51.306835 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d03a028-7cmmw" event={"ID":"338f53d4-163f-49cf-a772-bc10a691ef68","Type":"ContainerDied","Data":"ecce0806f2ee29c940e91b1ee9b06b3c328972b799f85b1737e38b3dc21645d1"} Oct 30 17:00:51.802911 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:51.802866 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs"] Oct 30 17:00:51.807220 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:51.807195 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs" Oct 30 17:00:51.818828 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:51.818801 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ecae9c7-ada3-4612-909f-1c1493369daf-workload-certs\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs\" (UID: \"2ecae9c7-ada3-4612-909f-1c1493369daf\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs" Oct 30 17:00:51.818936 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:51.818837 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ecae9c7-ada3-4612-909f-1c1493369daf-istio-token\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs\" (UID: \"2ecae9c7-ada3-4612-909f-1c1493369daf\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs" Oct 30 17:00:51.818936 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:51.818868 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecae9c7-ada3-4612-909f-1c1493369daf-credential-socket\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs\" (UID: \"2ecae9c7-ada3-4612-909f-1c1493369daf\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs" Oct 30 17:00:51.819025 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:51.819004 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ecae9c7-ada3-4612-909f-1c1493369daf-istiod-ca-cert\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs\" (UID: \"2ecae9c7-ada3-4612-909f-1c1493369daf\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs" Oct 30 17:00:51.819066 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:51.819049 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ecae9c7-ada3-4612-909f-1c1493369daf-istio-podinfo\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs\" (UID: \"2ecae9c7-ada3-4612-909f-1c1493369daf\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs" Oct 30 17:00:51.819147 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:51.819125 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecae9c7-ada3-4612-909f-1c1493369daf-workload-socket\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs\" (UID: \"2ecae9c7-ada3-4612-909f-1c1493369daf\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs" Oct 30 17:00:51.819254 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:51.819236 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ecae9c7-ada3-4612-909f-1c1493369daf-istio-data\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs\" (UID: \"2ecae9c7-ada3-4612-909f-1c1493369daf\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs" Oct 30 17:00:51.819292 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:51.819267 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ecae9c7-ada3-4612-909f-1c1493369daf-istio-envoy\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs\" (UID: \"2ecae9c7-ada3-4612-909f-1c1493369daf\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs" Oct 30 17:00:51.819406 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:51.819393 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp7rj\" (UniqueName: \"kubernetes.io/projected/2ecae9c7-ada3-4612-909f-1c1493369daf-kube-api-access-rp7rj\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs\" (UID: \"2ecae9c7-ada3-4612-909f-1c1493369daf\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs" Oct 30 17:00:51.837110 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:51.837087 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs"] Oct 30 17:00:51.847713 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:51.847692 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-b80d83-openshift-default-dockercfg-6zl9l\"" Oct 30 17:00:51.921469 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:51.921434 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ecae9c7-ada3-4612-909f-1c1493369daf-istiod-ca-cert\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs\" (UID: \"2ecae9c7-ada3-4612-909f-1c1493369daf\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs" Oct 30 17:00:51.921620 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:51.921484 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ecae9c7-ada3-4612-909f-1c1493369daf-istio-podinfo\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs\" (UID: \"2ecae9c7-ada3-4612-909f-1c1493369daf\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs" Oct 30 17:00:51.921620 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:51.921518 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecae9c7-ada3-4612-909f-1c1493369daf-workload-socket\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs\" (UID: \"2ecae9c7-ada3-4612-909f-1c1493369daf\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs" Oct 30 17:00:51.921620 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:51.921558 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ecae9c7-ada3-4612-909f-1c1493369daf-istio-data\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs\" (UID: \"2ecae9c7-ada3-4612-909f-1c1493369daf\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs" Oct 30 17:00:51.921620 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:51.921586 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ecae9c7-ada3-4612-909f-1c1493369daf-istio-envoy\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs\" (UID: \"2ecae9c7-ada3-4612-909f-1c1493369daf\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs" Oct 30 17:00:51.921839 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:51.921716 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp7rj\" (UniqueName: \"kubernetes.io/projected/2ecae9c7-ada3-4612-909f-1c1493369daf-kube-api-access-rp7rj\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs\" (UID: \"2ecae9c7-ada3-4612-909f-1c1493369daf\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs" Oct 30 17:00:51.921839 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:51.921759 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ecae9c7-ada3-4612-909f-1c1493369daf-workload-certs\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs\" (UID: \"2ecae9c7-ada3-4612-909f-1c1493369daf\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs" Oct 30 17:00:51.921839 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:51.921784 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ecae9c7-ada3-4612-909f-1c1493369daf-istio-token\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs\" (UID: \"2ecae9c7-ada3-4612-909f-1c1493369daf\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs" Oct 30 17:00:51.921839 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:51.921816 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecae9c7-ada3-4612-909f-1c1493369daf-credential-socket\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs\" (UID: \"2ecae9c7-ada3-4612-909f-1c1493369daf\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs" Oct 30 17:00:51.922042 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:51.922003 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecae9c7-ada3-4612-909f-1c1493369daf-workload-socket\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs\" (UID: \"2ecae9c7-ada3-4612-909f-1c1493369daf\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs" Oct 30 17:00:51.922099 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:51.922054 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ecae9c7-ada3-4612-909f-1c1493369daf-istio-data\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs\" (UID: \"2ecae9c7-ada3-4612-909f-1c1493369daf\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs" Oct 30 17:00:51.922220 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:51.922200 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecae9c7-ada3-4612-909f-1c1493369daf-credential-socket\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs\" (UID: \"2ecae9c7-ada3-4612-909f-1c1493369daf\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs" Oct 30 17:00:51.922285 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:51.922253 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ecae9c7-ada3-4612-909f-1c1493369daf-workload-certs\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs\" (UID: \"2ecae9c7-ada3-4612-909f-1c1493369daf\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs" Oct 30 17:00:51.922362 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:51.922279 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ecae9c7-ada3-4612-909f-1c1493369daf-istiod-ca-cert\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs\" (UID: \"2ecae9c7-ada3-4612-909f-1c1493369daf\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs" Oct 30 17:00:51.923887 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:51.923867 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ecae9c7-ada3-4612-909f-1c1493369daf-istio-envoy\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs\" (UID: \"2ecae9c7-ada3-4612-909f-1c1493369daf\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs" Oct 30 17:00:51.924051 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:51.924028 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ecae9c7-ada3-4612-909f-1c1493369daf-istio-podinfo\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs\" (UID: \"2ecae9c7-ada3-4612-909f-1c1493369daf\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs" Oct 30 17:00:51.949044 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:51.949014 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp7rj\" (UniqueName: \"kubernetes.io/projected/2ecae9c7-ada3-4612-909f-1c1493369daf-kube-api-access-rp7rj\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs\" (UID: \"2ecae9c7-ada3-4612-909f-1c1493369daf\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs" Oct 30 17:00:51.949177 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:51.949077 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ecae9c7-ada3-4612-909f-1c1493369daf-istio-token\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs\" (UID: \"2ecae9c7-ada3-4612-909f-1c1493369daf\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs" Oct 30 17:00:52.120740 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:52.120660 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs" Oct 30 17:00:52.430067 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:52.429981 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m"] Oct 30 17:00:52.451626 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:52.451589 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m" Oct 30 17:00:52.459293 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:52.459260 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m"] Oct 30 17:00:52.470914 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:52.470887 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d03a028-7cmmw" Oct 30 17:00:52.476813 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:00:52.476787 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ecae9c7_ada3_4612_909f_1c1493369daf.slice/crio-5bf13af2d1d8183b5ee4cc9007f301ebabd4a0bfa12b329c0dc1a675201e50d3 WatchSource:0}: Error finding container 5bf13af2d1d8183b5ee4cc9007f301ebabd4a0bfa12b329c0dc1a675201e50d3: Status 404 returned error can't find the container with id 5bf13af2d1d8183b5ee4cc9007f301ebabd4a0bfa12b329c0dc1a675201e50d3 Oct 30 17:00:52.478134 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:52.478069 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs"] Oct 30 17:00:52.479580 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:52.479545 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:00:52.479673 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:52.479627 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:00:52.479714 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:52.479674 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:00:52.531892 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:52.531866 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/338f53d4-163f-49cf-a772-bc10a691ef68-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/338f53d4-163f-49cf-a772-bc10a691ef68-maas-api-openshift-ca\") pod \"338f53d4-163f-49cf-a772-bc10a691ef68\" (UID: \"338f53d4-163f-49cf-a772-bc10a691ef68\") " Oct 30 17:00:52.532037 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:52.531919 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/338f53d4-163f-49cf-a772-bc10a691ef68-kube-api-access-bxhx6\" (UniqueName: \"kubernetes.io/projected/338f53d4-163f-49cf-a772-bc10a691ef68-kube-api-access-bxhx6\") pod \"338f53d4-163f-49cf-a772-bc10a691ef68\" (UID: \"338f53d4-163f-49cf-a772-bc10a691ef68\") " Oct 30 17:00:52.532037 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:52.531943 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/338f53d4-163f-49cf-a772-bc10a691ef68-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/338f53d4-163f-49cf-a772-bc10a691ef68-maas-api-xks-ca\") pod \"338f53d4-163f-49cf-a772-bc10a691ef68\" (UID: \"338f53d4-163f-49cf-a772-bc10a691ef68\") " Oct 30 17:00:52.532214 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:52.532186 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/338f53d4-163f-49cf-a772-bc10a691ef68-maas-api-openshift-ca" pod "338f53d4-163f-49cf-a772-bc10a691ef68" (UID: "338f53d4-163f-49cf-a772-bc10a691ef68"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:52.532444 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:52.532424 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/232a5af4-ecfb-4e38-a937-fc555a9b946b-workload-certs\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m\" (UID: \"232a5af4-ecfb-4e38-a937-fc555a9b946b\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m" Oct 30 17:00:52.532522 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:52.532461 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/232a5af4-ecfb-4e38-a937-fc555a9b946b-istio-envoy\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m\" (UID: \"232a5af4-ecfb-4e38-a937-fc555a9b946b\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m" Oct 30 17:00:52.532522 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:52.532480 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/232a5af4-ecfb-4e38-a937-fc555a9b946b-istio-token\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m\" (UID: \"232a5af4-ecfb-4e38-a937-fc555a9b946b\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m" Oct 30 17:00:52.532655 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:52.532630 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6z5q\" (UniqueName: \"kubernetes.io/projected/232a5af4-ecfb-4e38-a937-fc555a9b946b-kube-api-access-v6z5q\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m\" (UID: \"232a5af4-ecfb-4e38-a937-fc555a9b946b\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m" Oct 30 17:00:52.532709 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:52.532699 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/232a5af4-ecfb-4e38-a937-fc555a9b946b-credential-socket\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m\" (UID: \"232a5af4-ecfb-4e38-a937-fc555a9b946b\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m" Oct 30 17:00:52.532766 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:52.532745 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/232a5af4-ecfb-4e38-a937-fc555a9b946b-istiod-ca-cert\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m\" (UID: \"232a5af4-ecfb-4e38-a937-fc555a9b946b\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m" Oct 30 17:00:52.532905 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:52.532867 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/232a5af4-ecfb-4e38-a937-fc555a9b946b-workload-socket\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m\" (UID: \"232a5af4-ecfb-4e38-a937-fc555a9b946b\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m" Oct 30 17:00:52.533007 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:52.532990 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/232a5af4-ecfb-4e38-a937-fc555a9b946b-istio-podinfo\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m\" (UID: \"232a5af4-ecfb-4e38-a937-fc555a9b946b\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m" Oct 30 17:00:52.533057 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:52.533024 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/232a5af4-ecfb-4e38-a937-fc555a9b946b-istio-data\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m\" (UID: \"232a5af4-ecfb-4e38-a937-fc555a9b946b\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m" Oct 30 17:00:52.533149 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:52.533130 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/338f53d4-163f-49cf-a772-bc10a691ef68-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:00:52.533976 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:52.533955 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/338f53d4-163f-49cf-a772-bc10a691ef68-kube-api-access-bxhx6" pod "338f53d4-163f-49cf-a772-bc10a691ef68" (UID: "338f53d4-163f-49cf-a772-bc10a691ef68"). InnerVolumeSpecName "kube-api-access-bxhx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:52.534962 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:52.534942 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/338f53d4-163f-49cf-a772-bc10a691ef68-maas-api-xks-ca" pod "338f53d4-163f-49cf-a772-bc10a691ef68" (UID: "338f53d4-163f-49cf-a772-bc10a691ef68"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:00:52.634418 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:52.634374 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/232a5af4-ecfb-4e38-a937-fc555a9b946b-istio-podinfo\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m\" (UID: \"232a5af4-ecfb-4e38-a937-fc555a9b946b\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m" Oct 30 17:00:52.634418 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:52.634412 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/232a5af4-ecfb-4e38-a937-fc555a9b946b-istio-data\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m\" (UID: \"232a5af4-ecfb-4e38-a937-fc555a9b946b\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m" Oct 30 17:00:52.634674 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:52.634494 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/232a5af4-ecfb-4e38-a937-fc555a9b946b-workload-certs\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m\" (UID: \"232a5af4-ecfb-4e38-a937-fc555a9b946b\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m" Oct 30 17:00:52.634674 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:52.634528 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/232a5af4-ecfb-4e38-a937-fc555a9b946b-istio-envoy\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m\" (UID: \"232a5af4-ecfb-4e38-a937-fc555a9b946b\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m" Oct 30 17:00:52.634674 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:52.634652 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/232a5af4-ecfb-4e38-a937-fc555a9b946b-istio-token\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m\" (UID: \"232a5af4-ecfb-4e38-a937-fc555a9b946b\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m" Oct 30 17:00:52.634851 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:52.634745 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6z5q\" (UniqueName: \"kubernetes.io/projected/232a5af4-ecfb-4e38-a937-fc555a9b946b-kube-api-access-v6z5q\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m\" (UID: \"232a5af4-ecfb-4e38-a937-fc555a9b946b\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m" Oct 30 17:00:52.634851 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:52.634806 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/232a5af4-ecfb-4e38-a937-fc555a9b946b-credential-socket\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m\" (UID: \"232a5af4-ecfb-4e38-a937-fc555a9b946b\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m" Oct 30 17:00:52.634958 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:52.634851 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/232a5af4-ecfb-4e38-a937-fc555a9b946b-istiod-ca-cert\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m\" (UID: \"232a5af4-ecfb-4e38-a937-fc555a9b946b\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m" Oct 30 17:00:52.634958 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:52.634863 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/232a5af4-ecfb-4e38-a937-fc555a9b946b-workload-certs\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m\" (UID: \"232a5af4-ecfb-4e38-a937-fc555a9b946b\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m" Oct 30 17:00:52.634958 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:52.634926 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/232a5af4-ecfb-4e38-a937-fc555a9b946b-istio-data\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m\" (UID: \"232a5af4-ecfb-4e38-a937-fc555a9b946b\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m" Oct 30 17:00:52.635122 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:52.635007 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/232a5af4-ecfb-4e38-a937-fc555a9b946b-workload-socket\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m\" (UID: \"232a5af4-ecfb-4e38-a937-fc555a9b946b\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m" Oct 30 17:00:52.635189 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:52.635128 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxhx6\" (UniqueName: \"kubernetes.io/projected/338f53d4-163f-49cf-a772-bc10a691ef68-kube-api-access-bxhx6\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:00:52.635189 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:52.635148 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/338f53d4-163f-49cf-a772-bc10a691ef68-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:00:52.635189 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:52.635151 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/232a5af4-ecfb-4e38-a937-fc555a9b946b-credential-socket\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m\" (UID: \"232a5af4-ecfb-4e38-a937-fc555a9b946b\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m" Oct 30 17:00:52.635412 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:52.635388 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/232a5af4-ecfb-4e38-a937-fc555a9b946b-workload-socket\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m\" (UID: \"232a5af4-ecfb-4e38-a937-fc555a9b946b\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m" Oct 30 17:00:52.635515 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:52.635492 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/232a5af4-ecfb-4e38-a937-fc555a9b946b-istiod-ca-cert\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m\" (UID: \"232a5af4-ecfb-4e38-a937-fc555a9b946b\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m" Oct 30 17:00:52.637333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:52.637297 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/232a5af4-ecfb-4e38-a937-fc555a9b946b-istio-envoy\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m\" (UID: \"232a5af4-ecfb-4e38-a937-fc555a9b946b\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m" Oct 30 17:00:52.637673 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:52.637649 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/232a5af4-ecfb-4e38-a937-fc555a9b946b-istio-podinfo\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m\" (UID: \"232a5af4-ecfb-4e38-a937-fc555a9b946b\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m" Oct 30 17:00:52.674007 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:52.673970 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6z5q\" (UniqueName: \"kubernetes.io/projected/232a5af4-ecfb-4e38-a937-fc555a9b946b-kube-api-access-v6z5q\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m\" (UID: \"232a5af4-ecfb-4e38-a937-fc555a9b946b\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m" Oct 30 17:00:52.676033 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:52.676009 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/232a5af4-ecfb-4e38-a937-fc555a9b946b-istio-token\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m\" (UID: \"232a5af4-ecfb-4e38-a937-fc555a9b946b\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m" Oct 30 17:00:52.775331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:52.774533 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m" Oct 30 17:00:53.122536 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:00:53.122499 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod232a5af4_ecfb_4e38_a937_fc555a9b946b.slice/crio-660681750b12d50ea7a28ce5662f28db03ae9464219b5aea90d98e54a6a9d594 WatchSource:0}: Error finding container 660681750b12d50ea7a28ce5662f28db03ae9464219b5aea90d98e54a6a9d594: Status 404 returned error can't find the container with id 660681750b12d50ea7a28ce5662f28db03ae9464219b5aea90d98e54a6a9d594 Oct 30 17:00:53.122738 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:53.122716 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m"] Oct 30 17:00:53.124991 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:53.124958 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:00:53.125082 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:53.125023 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:00:53.125082 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:53.125051 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:00:53.327326 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:53.326766 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs" event={"ID":"2ecae9c7-ada3-4612-909f-1c1493369daf","Type":"ContainerStarted","Data":"8be21e369bb85f51030c572ee03a89ee10e1ca6c400efcf5ad5e38d1941d3d51"} Oct 30 17:00:53.327326 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:53.326822 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs" event={"ID":"2ecae9c7-ada3-4612-909f-1c1493369daf","Type":"ContainerStarted","Data":"5bf13af2d1d8183b5ee4cc9007f301ebabd4a0bfa12b329c0dc1a675201e50d3"} Oct 30 17:00:53.329530 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:53.329502 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d03a028-7cmmw" Oct 30 17:00:53.329664 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:53.329579 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d03a028-7cmmw" event={"ID":"338f53d4-163f-49cf-a772-bc10a691ef68","Type":"ContainerDied","Data":"c51000fee2a49e3413425de63d59b6a324c323e1d406bce037726a34684c0c37"} Oct 30 17:00:53.329664 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:53.329617 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c51000fee2a49e3413425de63d59b6a324c323e1d406bce037726a34684c0c37" Oct 30 17:00:53.332388 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:53.332360 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m" event={"ID":"232a5af4-ecfb-4e38-a937-fc555a9b946b","Type":"ContainerStarted","Data":"907d3419675b1ee04fb79572c2238986099b43909a692f3d98dcc358efc70ce2"} Oct 30 17:00:53.332473 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:53.332420 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m" event={"ID":"232a5af4-ecfb-4e38-a937-fc555a9b946b","Type":"ContainerStarted","Data":"660681750b12d50ea7a28ce5662f28db03ae9464219b5aea90d98e54a6a9d594"} Oct 30 17:00:53.379936 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:53.379828 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs" podStartSLOduration=2.3798125580000002 podStartE2EDuration="2.379812558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:53.375352982 +0000 UTC m=+827.376107883" watchObservedRunningTime="2026-10-30 17:00:53.379812558 +0000 UTC m=+827.380567459" Oct 30 17:00:53.449975 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:53.449915 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m" podStartSLOduration=1.449895486 podStartE2EDuration="1.449895486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:53.446782966 +0000 UTC m=+827.447537898" watchObservedRunningTime="2026-10-30 17:00:53.449895486 +0000 UTC m=+827.450650387" Oct 30 17:00:53.775428 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:53.775395 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m" Oct 30 17:00:54.122590 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:54.122470 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs" Oct 30 17:00:54.131361 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:54.131300 4522 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 17:00:54.131682 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:54.131634 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs" podUID="2ecae9c7-ada3-4612-909f-1c1493369daf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 17:00:54.689415 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:54.689383 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c27b1e00-5kx5t"] Oct 30 17:00:54.689911 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:54.689898 4522 cpu_manager.go:414] "containerMap: removing container" podUID="338f53d4-163f-49cf-a772-bc10a691ef68" containerName="revoke-keys" Oct 30 17:00:54.689969 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:54.689912 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="338f53d4-163f-49cf-a772-bc10a691ef68" containerName="revoke-keys" Oct 30 17:00:54.690002 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:54.689995 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="338f53d4-163f-49cf-a772-bc10a691ef68" containerName="revoke-keys" Oct 30 17:00:54.704328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:54.702612 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27b1e00-5kx5t" Oct 30 17:00:54.726341 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:54.723725 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:00:54.752361 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:54.742014 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:00:54.753844 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:54.753346 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c27b1e00-5kx5t"] Oct 30 17:00:54.763719 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:54.763692 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm56s\" (UniqueName: \"kubernetes.io/projected/111a5d49-933a-4434-87a1-d6714c0a49ed-kube-api-access-gm56s\") pod \"maas-api-revoke-sub-c27b1e00-5kx5t\" (UID: \"111a5d49-933a-4434-87a1-d6714c0a49ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27b1e00-5kx5t" Oct 30 17:00:54.763833 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:54.763762 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/111a5d49-933a-4434-87a1-d6714c0a49ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c27b1e00-5kx5t\" (UID: \"111a5d49-933a-4434-87a1-d6714c0a49ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27b1e00-5kx5t" Oct 30 17:00:54.763899 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:54.763857 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/111a5d49-933a-4434-87a1-d6714c0a49ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c27b1e00-5kx5t\" (UID: \"111a5d49-933a-4434-87a1-d6714c0a49ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27b1e00-5kx5t" Oct 30 17:00:54.776591 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:54.776142 4522 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 17:00:54.776591 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:54.776181 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m" podUID="232a5af4-ecfb-4e38-a937-fc555a9b946b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 17:00:54.865233 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:54.865198 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm56s\" (UniqueName: \"kubernetes.io/projected/111a5d49-933a-4434-87a1-d6714c0a49ed-kube-api-access-gm56s\") pod \"maas-api-revoke-sub-c27b1e00-5kx5t\" (UID: \"111a5d49-933a-4434-87a1-d6714c0a49ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27b1e00-5kx5t" Oct 30 17:00:54.865474 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:54.865250 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/111a5d49-933a-4434-87a1-d6714c0a49ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c27b1e00-5kx5t\" (UID: \"111a5d49-933a-4434-87a1-d6714c0a49ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27b1e00-5kx5t" Oct 30 17:00:54.865474 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:54.865273 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/111a5d49-933a-4434-87a1-d6714c0a49ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c27b1e00-5kx5t\" (UID: \"111a5d49-933a-4434-87a1-d6714c0a49ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27b1e00-5kx5t" Oct 30 17:00:54.865954 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:54.865935 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/111a5d49-933a-4434-87a1-d6714c0a49ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c27b1e00-5kx5t\" (UID: \"111a5d49-933a-4434-87a1-d6714c0a49ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27b1e00-5kx5t" Oct 30 17:00:54.867869 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:54.867842 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/111a5d49-933a-4434-87a1-d6714c0a49ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c27b1e00-5kx5t\" (UID: \"111a5d49-933a-4434-87a1-d6714c0a49ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27b1e00-5kx5t" Oct 30 17:00:54.899821 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:54.899792 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm56s\" (UniqueName: \"kubernetes.io/projected/111a5d49-933a-4434-87a1-d6714c0a49ed-kube-api-access-gm56s\") pod \"maas-api-revoke-sub-c27b1e00-5kx5t\" (UID: \"111a5d49-933a-4434-87a1-d6714c0a49ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27b1e00-5kx5t" Oct 30 17:00:55.063514 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:55.063478 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27b1e00-5kx5t" Oct 30 17:00:55.143540 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:55.143354 4522 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 17:00:55.143540 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:55.143409 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs" podUID="2ecae9c7-ada3-4612-909f-1c1493369daf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 17:00:55.446112 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:55.446068 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c27b1e00-5kx5t"] Oct 30 17:00:55.446717 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:00:55.446670 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod111a5d49_933a_4434_87a1_d6714c0a49ed.slice/crio-b87db8d25c4befec6a50fd7eb490b132bc890b7276381c4502da3fa7afc90962 WatchSource:0}: Error finding container b87db8d25c4befec6a50fd7eb490b132bc890b7276381c4502da3fa7afc90962: Status 404 returned error can't find the container with id b87db8d25c4befec6a50fd7eb490b132bc890b7276381c4502da3fa7afc90962 Oct 30 17:00:55.775949 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:55.775913 4522 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 17:00:55.776299 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:55.775969 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m" podUID="232a5af4-ecfb-4e38-a937-fc555a9b946b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 17:00:56.122388 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:56.121967 4522 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 17:00:56.122388 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:56.122027 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs" podUID="2ecae9c7-ada3-4612-909f-1c1493369daf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 17:00:56.359592 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:56.359550 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27b1e00-5kx5t" event={"ID":"111a5d49-933a-4434-87a1-d6714c0a49ed","Type":"ContainerStarted","Data":"2d91be15d4729f16224665ad75cf7dae38bafc99fc2a46f5729dcfa2963f490c"} Oct 30 17:00:56.359760 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:56.359605 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27b1e00-5kx5t" event={"ID":"111a5d49-933a-4434-87a1-d6714c0a49ed","Type":"ContainerStarted","Data":"b87db8d25c4befec6a50fd7eb490b132bc890b7276381c4502da3fa7afc90962"} Oct 30 17:00:56.776091 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:56.776052 4522 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 17:00:56.776518 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:56.776099 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m" podUID="232a5af4-ecfb-4e38-a937-fc555a9b946b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 17:00:57.124149 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:57.123713 4522 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 17:00:57.124149 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:57.123777 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs" podUID="2ecae9c7-ada3-4612-909f-1c1493369daf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 17:00:57.366249 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:57.366218 4522 generic.go:356] "Generic (PLEG): container finished" podID="111a5d49-933a-4434-87a1-d6714c0a49ed" containerID="2d91be15d4729f16224665ad75cf7dae38bafc99fc2a46f5729dcfa2963f490c" exitCode=0 Oct 30 17:00:57.366462 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:57.366281 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27b1e00-5kx5t" event={"ID":"111a5d49-933a-4434-87a1-d6714c0a49ed","Type":"ContainerDied","Data":"2d91be15d4729f16224665ad75cf7dae38bafc99fc2a46f5729dcfa2963f490c"} Oct 30 17:00:57.775174 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:57.775142 4522 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 17:00:57.775358 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:57.775194 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m" podUID="232a5af4-ecfb-4e38-a937-fc555a9b946b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 17:00:58.122390 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:58.121779 4522 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 17:00:58.122390 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:58.121843 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs" podUID="2ecae9c7-ada3-4612-909f-1c1493369daf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 17:00:58.659116 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:58.659094 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27b1e00-5kx5t" Oct 30 17:00:58.762447 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:58.761962 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/111a5d49-933a-4434-87a1-d6714c0a49ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/111a5d49-933a-4434-87a1-d6714c0a49ed-maas-api-openshift-ca\") pod \"111a5d49-933a-4434-87a1-d6714c0a49ed\" (UID: \"111a5d49-933a-4434-87a1-d6714c0a49ed\") " Oct 30 17:00:58.762447 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:58.762203 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/111a5d49-933a-4434-87a1-d6714c0a49ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/111a5d49-933a-4434-87a1-d6714c0a49ed-maas-api-xks-ca\") pod \"111a5d49-933a-4434-87a1-d6714c0a49ed\" (UID: \"111a5d49-933a-4434-87a1-d6714c0a49ed\") " Oct 30 17:00:58.762447 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:58.762277 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/111a5d49-933a-4434-87a1-d6714c0a49ed-kube-api-access-gm56s\" (UniqueName: \"kubernetes.io/projected/111a5d49-933a-4434-87a1-d6714c0a49ed-kube-api-access-gm56s\") pod \"111a5d49-933a-4434-87a1-d6714c0a49ed\" (UID: \"111a5d49-933a-4434-87a1-d6714c0a49ed\") " Oct 30 17:00:58.769089 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:58.769057 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/111a5d49-933a-4434-87a1-d6714c0a49ed-maas-api-openshift-ca" pod "111a5d49-933a-4434-87a1-d6714c0a49ed" (UID: "111a5d49-933a-4434-87a1-d6714c0a49ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:58.773198 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:58.773148 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/111a5d49-933a-4434-87a1-d6714c0a49ed-maas-api-xks-ca" pod "111a5d49-933a-4434-87a1-d6714c0a49ed" (UID: "111a5d49-933a-4434-87a1-d6714c0a49ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:00:58.773510 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:58.773488 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/111a5d49-933a-4434-87a1-d6714c0a49ed-kube-api-access-gm56s" pod "111a5d49-933a-4434-87a1-d6714c0a49ed" (UID: "111a5d49-933a-4434-87a1-d6714c0a49ed"). InnerVolumeSpecName "kube-api-access-gm56s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:58.784824 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:58.784800 4522 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 17:00:58.785005 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:58.784981 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m" podUID="232a5af4-ecfb-4e38-a937-fc555a9b946b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 17:00:58.863605 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:58.863568 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/111a5d49-933a-4434-87a1-d6714c0a49ed-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:00:58.863605 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:58.863599 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gm56s\" (UniqueName: \"kubernetes.io/projected/111a5d49-933a-4434-87a1-d6714c0a49ed-kube-api-access-gm56s\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:00:58.863605 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:58.863610 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/111a5d49-933a-4434-87a1-d6714c0a49ed-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:00:59.121664 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:59.121568 4522 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 17:00:59.121912 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:59.121878 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs" podUID="2ecae9c7-ada3-4612-909f-1c1493369daf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 17:00:59.382462 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:59.382357 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27b1e00-5kx5t" event={"ID":"111a5d49-933a-4434-87a1-d6714c0a49ed","Type":"ContainerDied","Data":"b87db8d25c4befec6a50fd7eb490b132bc890b7276381c4502da3fa7afc90962"} Oct 30 17:00:59.382462 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:59.382389 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27b1e00-5kx5t" Oct 30 17:00:59.382462 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:59.382426 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b87db8d25c4befec6a50fd7eb490b132bc890b7276381c4502da3fa7afc90962" Oct 30 17:00:59.408727 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:59.408656 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7df5a782-xmptt"] Oct 30 17:00:59.409249 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:59.409228 4522 cpu_manager.go:414] "containerMap: removing container" podUID="111a5d49-933a-4434-87a1-d6714c0a49ed" containerName="revoke-keys" Oct 30 17:00:59.409249 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:59.409249 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="111a5d49-933a-4434-87a1-d6714c0a49ed" containerName="revoke-keys" Oct 30 17:00:59.409425 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:59.409410 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="111a5d49-933a-4434-87a1-d6714c0a49ed" containerName="revoke-keys" Oct 30 17:00:59.414656 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:59.414637 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df5a782-xmptt" Oct 30 17:00:59.434262 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:59.434234 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:00:59.434525 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:59.434502 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:00:59.444635 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:59.442824 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7df5a782-xmptt"] Oct 30 17:00:59.573659 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:59.573613 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffmt6\" (UniqueName: \"kubernetes.io/projected/f87925c2-329d-49b5-a1fd-4b74f2859a2a-kube-api-access-ffmt6\") pod \"maas-api-revoke-sub-7df5a782-xmptt\" (UID: \"f87925c2-329d-49b5-a1fd-4b74f2859a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df5a782-xmptt" Oct 30 17:00:59.573885 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:59.573828 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f87925c2-329d-49b5-a1fd-4b74f2859a2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7df5a782-xmptt\" (UID: \"f87925c2-329d-49b5-a1fd-4b74f2859a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df5a782-xmptt" Oct 30 17:00:59.573958 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:59.573899 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f87925c2-329d-49b5-a1fd-4b74f2859a2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7df5a782-xmptt\" (UID: \"f87925c2-329d-49b5-a1fd-4b74f2859a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df5a782-xmptt" Oct 30 17:00:59.675140 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:59.675035 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f87925c2-329d-49b5-a1fd-4b74f2859a2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7df5a782-xmptt\" (UID: \"f87925c2-329d-49b5-a1fd-4b74f2859a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df5a782-xmptt" Oct 30 17:00:59.675140 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:59.675101 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f87925c2-329d-49b5-a1fd-4b74f2859a2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7df5a782-xmptt\" (UID: \"f87925c2-329d-49b5-a1fd-4b74f2859a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df5a782-xmptt" Oct 30 17:00:59.675415 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:59.675152 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffmt6\" (UniqueName: \"kubernetes.io/projected/f87925c2-329d-49b5-a1fd-4b74f2859a2a-kube-api-access-ffmt6\") pod \"maas-api-revoke-sub-7df5a782-xmptt\" (UID: \"f87925c2-329d-49b5-a1fd-4b74f2859a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df5a782-xmptt" Oct 30 17:00:59.675761 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:59.675738 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f87925c2-329d-49b5-a1fd-4b74f2859a2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7df5a782-xmptt\" (UID: \"f87925c2-329d-49b5-a1fd-4b74f2859a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df5a782-xmptt" Oct 30 17:00:59.677441 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:59.677417 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f87925c2-329d-49b5-a1fd-4b74f2859a2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7df5a782-xmptt\" (UID: \"f87925c2-329d-49b5-a1fd-4b74f2859a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df5a782-xmptt" Oct 30 17:00:59.702900 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:59.702862 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffmt6\" (UniqueName: \"kubernetes.io/projected/f87925c2-329d-49b5-a1fd-4b74f2859a2a-kube-api-access-ffmt6\") pod \"maas-api-revoke-sub-7df5a782-xmptt\" (UID: \"f87925c2-329d-49b5-a1fd-4b74f2859a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df5a782-xmptt" Oct 30 17:00:59.735033 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:59.735002 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df5a782-xmptt" Oct 30 17:00:59.775689 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:59.775653 4522 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 17:00:59.775869 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:00:59.775718 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m" podUID="232a5af4-ecfb-4e38-a937-fc555a9b946b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 17:01:00.050670 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:00.043815 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7df5a782-xmptt"] Oct 30 17:01:00.051133 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:01:00.051108 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf87925c2_329d_49b5_a1fd_4b74f2859a2a.slice/crio-8c0d3d920d442e9aaf3b9d076f4adb2e866bc93aa1ff48c5e4a0f90ffc19b245 WatchSource:0}: Error finding container 8c0d3d920d442e9aaf3b9d076f4adb2e866bc93aa1ff48c5e4a0f90ffc19b245: Status 404 returned error can't find the container with id 8c0d3d920d442e9aaf3b9d076f4adb2e866bc93aa1ff48c5e4a0f90ffc19b245 Oct 30 17:01:00.127546 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:00.127517 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs" Oct 30 17:01:00.128013 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:00.127988 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs" Oct 30 17:01:00.128803 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:00.128782 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs" Oct 30 17:01:00.389902 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:00.389864 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df5a782-xmptt" event={"ID":"f87925c2-329d-49b5-a1fd-4b74f2859a2a","Type":"ContainerStarted","Data":"88474834a289b245a637490c9e041f5fb68c42ab96eb7c5b99b1cea2a873057c"} Oct 30 17:01:00.390380 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:00.389926 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df5a782-xmptt" event={"ID":"f87925c2-329d-49b5-a1fd-4b74f2859a2a","Type":"ContainerStarted","Data":"8c0d3d920d442e9aaf3b9d076f4adb2e866bc93aa1ff48c5e4a0f90ffc19b245"} Oct 30 17:01:00.780348 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:00.780299 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m" Oct 30 17:01:00.780643 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:00.780622 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m" Oct 30 17:01:00.781458 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:00.781438 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m" Oct 30 17:01:00.833020 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:00.832972 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df5a782-xmptt" podStartSLOduration=1.832956574 podStartE2EDuration="1.832956574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:01:00.436378541 +0000 UTC m=+834.437133441" watchObservedRunningTime="2026-10-30 17:01:00.832956574 +0000 UTC m=+834.833711475" Oct 30 17:01:01.396975 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:01.396942 4522 generic.go:356] "Generic (PLEG): container finished" podID="f87925c2-329d-49b5-a1fd-4b74f2859a2a" containerID="88474834a289b245a637490c9e041f5fb68c42ab96eb7c5b99b1cea2a873057c" exitCode=0 Oct 30 17:01:01.397435 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:01.396999 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df5a782-xmptt" event={"ID":"f87925c2-329d-49b5-a1fd-4b74f2859a2a","Type":"ContainerDied","Data":"88474834a289b245a637490c9e041f5fb68c42ab96eb7c5b99b1cea2a873057c"} Oct 30 17:01:02.558988 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:02.558964 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df5a782-xmptt" Oct 30 17:01:02.713262 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:02.713231 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f87925c2-329d-49b5-a1fd-4b74f2859a2a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f87925c2-329d-49b5-a1fd-4b74f2859a2a-maas-api-openshift-ca\") pod \"f87925c2-329d-49b5-a1fd-4b74f2859a2a\" (UID: \"f87925c2-329d-49b5-a1fd-4b74f2859a2a\") " Oct 30 17:01:02.713493 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:02.713400 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f87925c2-329d-49b5-a1fd-4b74f2859a2a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f87925c2-329d-49b5-a1fd-4b74f2859a2a-maas-api-xks-ca\") pod \"f87925c2-329d-49b5-a1fd-4b74f2859a2a\" (UID: \"f87925c2-329d-49b5-a1fd-4b74f2859a2a\") " Oct 30 17:01:02.713558 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:02.713503 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f87925c2-329d-49b5-a1fd-4b74f2859a2a-kube-api-access-ffmt6\" (UniqueName: \"kubernetes.io/projected/f87925c2-329d-49b5-a1fd-4b74f2859a2a-kube-api-access-ffmt6\") pod \"f87925c2-329d-49b5-a1fd-4b74f2859a2a\" (UID: \"f87925c2-329d-49b5-a1fd-4b74f2859a2a\") " Oct 30 17:01:02.713608 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:02.713577 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f87925c2-329d-49b5-a1fd-4b74f2859a2a-maas-api-openshift-ca" pod "f87925c2-329d-49b5-a1fd-4b74f2859a2a" (UID: "f87925c2-329d-49b5-a1fd-4b74f2859a2a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:01:02.713789 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:02.713774 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f87925c2-329d-49b5-a1fd-4b74f2859a2a-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:01:02.715369 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:02.715342 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f87925c2-329d-49b5-a1fd-4b74f2859a2a-maas-api-xks-ca" pod "f87925c2-329d-49b5-a1fd-4b74f2859a2a" (UID: "f87925c2-329d-49b5-a1fd-4b74f2859a2a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:01:02.715564 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:02.715544 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f87925c2-329d-49b5-a1fd-4b74f2859a2a-kube-api-access-ffmt6" pod "f87925c2-329d-49b5-a1fd-4b74f2859a2a" (UID: "f87925c2-329d-49b5-a1fd-4b74f2859a2a"). InnerVolumeSpecName "kube-api-access-ffmt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:01:02.815067 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:02.815034 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f87925c2-329d-49b5-a1fd-4b74f2859a2a-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:01:02.815067 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:02.815067 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffmt6\" (UniqueName: \"kubernetes.io/projected/f87925c2-329d-49b5-a1fd-4b74f2859a2a-kube-api-access-ffmt6\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:01:03.411636 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:03.411599 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df5a782-xmptt" event={"ID":"f87925c2-329d-49b5-a1fd-4b74f2859a2a","Type":"ContainerDied","Data":"8c0d3d920d442e9aaf3b9d076f4adb2e866bc93aa1ff48c5e4a0f90ffc19b245"} Oct 30 17:01:03.411636 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:03.411647 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c0d3d920d442e9aaf3b9d076f4adb2e866bc93aa1ff48c5e4a0f90ffc19b245" Oct 30 17:01:03.411849 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:03.411647 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df5a782-xmptt" Oct 30 17:01:03.659509 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:03.659468 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1be7fd0-x6sgq"] Oct 30 17:01:03.660032 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:03.660012 4522 cpu_manager.go:414] "containerMap: removing container" podUID="f87925c2-329d-49b5-a1fd-4b74f2859a2a" containerName="revoke-keys" Oct 30 17:01:03.660032 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:03.660032 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f87925c2-329d-49b5-a1fd-4b74f2859a2a" containerName="revoke-keys" Oct 30 17:01:03.660150 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:03.660129 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="f87925c2-329d-49b5-a1fd-4b74f2859a2a" containerName="revoke-keys" Oct 30 17:01:03.663577 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:03.663522 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1be7fd0-x6sgq" Oct 30 17:01:03.684757 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:03.684728 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:01:03.687339 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:03.687288 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1be7fd0-x6sgq"] Oct 30 17:01:03.695759 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:03.695735 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:01:03.826134 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:03.826098 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35684f3d-77e1-42cf-8c59-633053012794-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1be7fd0-x6sgq\" (UID: \"35684f3d-77e1-42cf-8c59-633053012794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1be7fd0-x6sgq" Oct 30 17:01:03.826337 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:03.826160 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phbvr\" (UniqueName: \"kubernetes.io/projected/35684f3d-77e1-42cf-8c59-633053012794-kube-api-access-phbvr\") pod \"maas-api-revoke-sub-d1be7fd0-x6sgq\" (UID: \"35684f3d-77e1-42cf-8c59-633053012794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1be7fd0-x6sgq" Oct 30 17:01:03.826388 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:03.826365 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35684f3d-77e1-42cf-8c59-633053012794-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1be7fd0-x6sgq\" (UID: \"35684f3d-77e1-42cf-8c59-633053012794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1be7fd0-x6sgq" Oct 30 17:01:03.927543 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:03.927460 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35684f3d-77e1-42cf-8c59-633053012794-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1be7fd0-x6sgq\" (UID: \"35684f3d-77e1-42cf-8c59-633053012794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1be7fd0-x6sgq" Oct 30 17:01:03.927543 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:03.927532 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35684f3d-77e1-42cf-8c59-633053012794-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1be7fd0-x6sgq\" (UID: \"35684f3d-77e1-42cf-8c59-633053012794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1be7fd0-x6sgq" Oct 30 17:01:03.927742 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:03.927593 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phbvr\" (UniqueName: \"kubernetes.io/projected/35684f3d-77e1-42cf-8c59-633053012794-kube-api-access-phbvr\") pod \"maas-api-revoke-sub-d1be7fd0-x6sgq\" (UID: \"35684f3d-77e1-42cf-8c59-633053012794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1be7fd0-x6sgq" Oct 30 17:01:03.928150 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:03.928131 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35684f3d-77e1-42cf-8c59-633053012794-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1be7fd0-x6sgq\" (UID: \"35684f3d-77e1-42cf-8c59-633053012794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1be7fd0-x6sgq" Oct 30 17:01:03.929886 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:03.929868 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35684f3d-77e1-42cf-8c59-633053012794-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1be7fd0-x6sgq\" (UID: \"35684f3d-77e1-42cf-8c59-633053012794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1be7fd0-x6sgq" Oct 30 17:01:03.953231 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:03.953197 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phbvr\" (UniqueName: \"kubernetes.io/projected/35684f3d-77e1-42cf-8c59-633053012794-kube-api-access-phbvr\") pod \"maas-api-revoke-sub-d1be7fd0-x6sgq\" (UID: \"35684f3d-77e1-42cf-8c59-633053012794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1be7fd0-x6sgq" Oct 30 17:01:03.979351 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:03.979326 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1be7fd0-x6sgq" Oct 30 17:01:04.305501 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:01:04.305472 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35684f3d_77e1_42cf_8c59_633053012794.slice/crio-435392a9110a99dd2db78c46a2c44e72fd2d40351a81c20054b85d9fea904589 WatchSource:0}: Error finding container 435392a9110a99dd2db78c46a2c44e72fd2d40351a81c20054b85d9fea904589: Status 404 returned error can't find the container with id 435392a9110a99dd2db78c46a2c44e72fd2d40351a81c20054b85d9fea904589 Oct 30 17:01:04.306627 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:04.306606 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1be7fd0-x6sgq"] Oct 30 17:01:04.418487 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:04.418454 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1be7fd0-x6sgq" event={"ID":"35684f3d-77e1-42cf-8c59-633053012794","Type":"ContainerStarted","Data":"435392a9110a99dd2db78c46a2c44e72fd2d40351a81c20054b85d9fea904589"} Oct 30 17:01:05.440330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:05.438615 4522 generic.go:356] "Generic (PLEG): container finished" podID="35684f3d-77e1-42cf-8c59-633053012794" containerID="4b2f11f60141d2b33b2301113c9b0db2c18bb1ddec1ce60fa4983407751fee99" exitCode=0 Oct 30 17:01:05.440330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:05.438752 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1be7fd0-x6sgq" event={"ID":"35684f3d-77e1-42cf-8c59-633053012794","Type":"ContainerDied","Data":"4b2f11f60141d2b33b2301113c9b0db2c18bb1ddec1ce60fa4983407751fee99"} Oct 30 17:01:06.860289 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:06.860247 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1be7fd0-x6sgq" Oct 30 17:01:06.891053 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:06.891022 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35684f3d-77e1-42cf-8c59-633053012794-kube-api-access-phbvr\" (UniqueName: \"kubernetes.io/projected/35684f3d-77e1-42cf-8c59-633053012794-kube-api-access-phbvr\") pod \"35684f3d-77e1-42cf-8c59-633053012794\" (UID: \"35684f3d-77e1-42cf-8c59-633053012794\") " Oct 30 17:01:06.891271 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:06.891255 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35684f3d-77e1-42cf-8c59-633053012794-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35684f3d-77e1-42cf-8c59-633053012794-maas-api-openshift-ca\") pod \"35684f3d-77e1-42cf-8c59-633053012794\" (UID: \"35684f3d-77e1-42cf-8c59-633053012794\") " Oct 30 17:01:06.891381 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:06.891291 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35684f3d-77e1-42cf-8c59-633053012794-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35684f3d-77e1-42cf-8c59-633053012794-maas-api-xks-ca\") pod \"35684f3d-77e1-42cf-8c59-633053012794\" (UID: \"35684f3d-77e1-42cf-8c59-633053012794\") " Oct 30 17:01:06.892336 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:06.892287 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35684f3d-77e1-42cf-8c59-633053012794-maas-api-openshift-ca" pod "35684f3d-77e1-42cf-8c59-633053012794" (UID: "35684f3d-77e1-42cf-8c59-633053012794"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:01:06.893901 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:06.893872 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35684f3d-77e1-42cf-8c59-633053012794-maas-api-xks-ca" pod "35684f3d-77e1-42cf-8c59-633053012794" (UID: "35684f3d-77e1-42cf-8c59-633053012794"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:01:06.893995 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:06.893872 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35684f3d-77e1-42cf-8c59-633053012794-kube-api-access-phbvr" pod "35684f3d-77e1-42cf-8c59-633053012794" (UID: "35684f3d-77e1-42cf-8c59-633053012794"). InnerVolumeSpecName "kube-api-access-phbvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:01:06.993001 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:06.992960 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35684f3d-77e1-42cf-8c59-633053012794-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:01:06.993001 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:06.992994 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35684f3d-77e1-42cf-8c59-633053012794-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:01:06.993001 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:06.993007 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phbvr\" (UniqueName: \"kubernetes.io/projected/35684f3d-77e1-42cf-8c59-633053012794-kube-api-access-phbvr\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:01:07.491296 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:07.491266 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1be7fd0-x6sgq" Oct 30 17:01:07.491501 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:07.491327 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1be7fd0-x6sgq" event={"ID":"35684f3d-77e1-42cf-8c59-633053012794","Type":"ContainerDied","Data":"435392a9110a99dd2db78c46a2c44e72fd2d40351a81c20054b85d9fea904589"} Oct 30 17:01:07.491501 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:07.491373 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="435392a9110a99dd2db78c46a2c44e72fd2d40351a81c20054b85d9fea904589" Oct 30 17:01:07.645293 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:07.645255 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq"] Oct 30 17:01:07.645862 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:07.645843 4522 cpu_manager.go:414] "containerMap: removing container" podUID="35684f3d-77e1-42cf-8c59-633053012794" containerName="revoke-keys" Oct 30 17:01:07.645977 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:07.645863 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35684f3d-77e1-42cf-8c59-633053012794" containerName="revoke-keys" Oct 30 17:01:07.646034 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:07.645988 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="35684f3d-77e1-42cf-8c59-633053012794" containerName="revoke-keys" Oct 30 17:01:07.649505 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:07.649485 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq" Oct 30 17:01:07.667281 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:07.667261 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-04248e-openshift-default-dockercfg-bbxjd\"" Oct 30 17:01:07.683205 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:07.683177 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq"] Oct 30 17:01:07.701190 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:07.701159 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46027d41-98ad-409d-a66f-4bc050bc2648-workload-socket\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq\" (UID: \"46027d41-98ad-409d-a66f-4bc050bc2648\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq" Oct 30 17:01:07.701482 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:07.701457 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46027d41-98ad-409d-a66f-4bc050bc2648-istiod-ca-cert\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq\" (UID: \"46027d41-98ad-409d-a66f-4bc050bc2648\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq" Oct 30 17:01:07.701643 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:07.701612 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nnf2\" (UniqueName: \"kubernetes.io/projected/46027d41-98ad-409d-a66f-4bc050bc2648-kube-api-access-6nnf2\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq\" (UID: \"46027d41-98ad-409d-a66f-4bc050bc2648\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq" Oct 30 17:01:07.701753 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:07.701683 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46027d41-98ad-409d-a66f-4bc050bc2648-credential-socket\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq\" (UID: \"46027d41-98ad-409d-a66f-4bc050bc2648\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq" Oct 30 17:01:07.701753 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:07.701733 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46027d41-98ad-409d-a66f-4bc050bc2648-workload-certs\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq\" (UID: \"46027d41-98ad-409d-a66f-4bc050bc2648\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq" Oct 30 17:01:07.701899 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:07.701866 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46027d41-98ad-409d-a66f-4bc050bc2648-istio-envoy\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq\" (UID: \"46027d41-98ad-409d-a66f-4bc050bc2648\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq" Oct 30 17:01:07.701968 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:07.701914 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46027d41-98ad-409d-a66f-4bc050bc2648-istio-token\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq\" (UID: \"46027d41-98ad-409d-a66f-4bc050bc2648\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq" Oct 30 17:01:07.701968 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:07.701952 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46027d41-98ad-409d-a66f-4bc050bc2648-istio-podinfo\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq\" (UID: \"46027d41-98ad-409d-a66f-4bc050bc2648\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq" Oct 30 17:01:07.702073 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:07.702054 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46027d41-98ad-409d-a66f-4bc050bc2648-istio-data\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq\" (UID: \"46027d41-98ad-409d-a66f-4bc050bc2648\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq" Oct 30 17:01:07.802941 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:07.802853 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46027d41-98ad-409d-a66f-4bc050bc2648-istio-data\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq\" (UID: \"46027d41-98ad-409d-a66f-4bc050bc2648\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq" Oct 30 17:01:07.803107 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:07.802974 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46027d41-98ad-409d-a66f-4bc050bc2648-workload-socket\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq\" (UID: \"46027d41-98ad-409d-a66f-4bc050bc2648\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq" Oct 30 17:01:07.803107 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:07.803037 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46027d41-98ad-409d-a66f-4bc050bc2648-istiod-ca-cert\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq\" (UID: \"46027d41-98ad-409d-a66f-4bc050bc2648\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq" Oct 30 17:01:07.803107 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:07.803069 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nnf2\" (UniqueName: \"kubernetes.io/projected/46027d41-98ad-409d-a66f-4bc050bc2648-kube-api-access-6nnf2\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq\" (UID: \"46027d41-98ad-409d-a66f-4bc050bc2648\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq" Oct 30 17:01:07.803280 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:07.803143 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46027d41-98ad-409d-a66f-4bc050bc2648-credential-socket\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq\" (UID: \"46027d41-98ad-409d-a66f-4bc050bc2648\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq" Oct 30 17:01:07.803280 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:07.803169 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46027d41-98ad-409d-a66f-4bc050bc2648-workload-certs\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq\" (UID: \"46027d41-98ad-409d-a66f-4bc050bc2648\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq" Oct 30 17:01:07.803280 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:07.803228 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46027d41-98ad-409d-a66f-4bc050bc2648-istio-envoy\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq\" (UID: \"46027d41-98ad-409d-a66f-4bc050bc2648\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq" Oct 30 17:01:07.803280 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:07.803257 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46027d41-98ad-409d-a66f-4bc050bc2648-istio-token\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq\" (UID: \"46027d41-98ad-409d-a66f-4bc050bc2648\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq" Oct 30 17:01:07.803516 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:07.803288 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46027d41-98ad-409d-a66f-4bc050bc2648-istio-data\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq\" (UID: \"46027d41-98ad-409d-a66f-4bc050bc2648\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq" Oct 30 17:01:07.803516 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:07.803294 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46027d41-98ad-409d-a66f-4bc050bc2648-istio-podinfo\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq\" (UID: \"46027d41-98ad-409d-a66f-4bc050bc2648\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq" Oct 30 17:01:07.803516 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:07.803406 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46027d41-98ad-409d-a66f-4bc050bc2648-workload-socket\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq\" (UID: \"46027d41-98ad-409d-a66f-4bc050bc2648\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq" Oct 30 17:01:07.803731 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:07.803713 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46027d41-98ad-409d-a66f-4bc050bc2648-credential-socket\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq\" (UID: \"46027d41-98ad-409d-a66f-4bc050bc2648\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq" Oct 30 17:01:07.803846 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:07.803821 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46027d41-98ad-409d-a66f-4bc050bc2648-istiod-ca-cert\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq\" (UID: \"46027d41-98ad-409d-a66f-4bc050bc2648\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq" Oct 30 17:01:07.804015 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:07.803992 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46027d41-98ad-409d-a66f-4bc050bc2648-workload-certs\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq\" (UID: \"46027d41-98ad-409d-a66f-4bc050bc2648\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq" Oct 30 17:01:07.807336 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:07.805894 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46027d41-98ad-409d-a66f-4bc050bc2648-istio-envoy\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq\" (UID: \"46027d41-98ad-409d-a66f-4bc050bc2648\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq" Oct 30 17:01:07.807336 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:07.806376 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46027d41-98ad-409d-a66f-4bc050bc2648-istio-podinfo\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq\" (UID: \"46027d41-98ad-409d-a66f-4bc050bc2648\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq" Oct 30 17:01:07.829344 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:07.829294 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46027d41-98ad-409d-a66f-4bc050bc2648-istio-token\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq\" (UID: \"46027d41-98ad-409d-a66f-4bc050bc2648\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq" Oct 30 17:01:07.829476 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:07.829458 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nnf2\" (UniqueName: \"kubernetes.io/projected/46027d41-98ad-409d-a66f-4bc050bc2648-kube-api-access-6nnf2\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq\" (UID: \"46027d41-98ad-409d-a66f-4bc050bc2648\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq" Oct 30 17:01:07.964032 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:07.963997 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq" Oct 30 17:01:08.300586 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:08.300550 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq"] Oct 30 17:01:08.303821 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:01:08.303794 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46027d41_98ad_409d_a66f_4bc050bc2648.slice/crio-2a7121ed0a6a5f33732d3d22cccfe3924f233937acd8da6258605605060aad6b WatchSource:0}: Error finding container 2a7121ed0a6a5f33732d3d22cccfe3924f233937acd8da6258605605060aad6b: Status 404 returned error can't find the container with id 2a7121ed0a6a5f33732d3d22cccfe3924f233937acd8da6258605605060aad6b Oct 30 17:01:08.306222 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:08.306189 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:01:08.306289 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:08.306257 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:01:08.306289 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:08.306287 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:01:08.503334 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:08.502602 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq" event={"ID":"46027d41-98ad-409d-a66f-4bc050bc2648","Type":"ContainerStarted","Data":"c09766a2cb5828462c8d758dc680c4b58133409e57b861a19da7f333f2604862"} Oct 30 17:01:08.503334 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:08.502674 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq" event={"ID":"46027d41-98ad-409d-a66f-4bc050bc2648","Type":"ContainerStarted","Data":"2a7121ed0a6a5f33732d3d22cccfe3924f233937acd8da6258605605060aad6b"} Oct 30 17:01:08.560767 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:08.560662 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq" podStartSLOduration=1.560646975 podStartE2EDuration="1.560646975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:01:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:01:08.558197922 +0000 UTC m=+842.558952842" watchObservedRunningTime="2026-10-30 17:01:08.560646975 +0000 UTC m=+842.561401938" Oct 30 17:01:08.964644 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:08.964610 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq" Oct 30 17:01:09.968427 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:09.968379 4522 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 17:01:09.968927 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:09.968445 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq" podUID="46027d41-98ad-409d-a66f-4bc050bc2648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 17:01:10.964676 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:10.964641 4522 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 17:01:10.964856 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:10.964691 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq" podUID="46027d41-98ad-409d-a66f-4bc050bc2648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 17:01:11.966173 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:11.964785 4522 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 17:01:11.966173 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:11.964849 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq" podUID="46027d41-98ad-409d-a66f-4bc050bc2648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 17:01:12.955328 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:01:12.955242 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:01:12.965183 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:12.965151 4522 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 17:01:12.965386 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:12.965207 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq" podUID="46027d41-98ad-409d-a66f-4bc050bc2648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 17:01:13.965392 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:13.965354 4522 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 17:01:13.965800 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:13.965432 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq" podUID="46027d41-98ad-409d-a66f-4bc050bc2648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 17:01:14.655448 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:14.655407 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdbb1f67-t7bps"] Oct 30 17:01:14.660592 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:14.660568 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbb1f67-t7bps" Oct 30 17:01:14.693549 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:14.693515 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:01:14.697907 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:14.697858 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:01:14.698826 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:14.698714 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdbb1f67-t7bps"] Oct 30 17:01:14.755328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:14.755123 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcfdcc85-e3e7-4c06-94c8-c9d22a3862b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdbb1f67-t7bps\" (UID: \"fcfdcc85-e3e7-4c06-94c8-c9d22a3862b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbb1f67-t7bps" Oct 30 17:01:14.755328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:14.755218 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcfdcc85-e3e7-4c06-94c8-c9d22a3862b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdbb1f67-t7bps\" (UID: \"fcfdcc85-e3e7-4c06-94c8-c9d22a3862b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbb1f67-t7bps" Oct 30 17:01:14.755563 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:14.755394 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c5h2\" (UniqueName: \"kubernetes.io/projected/fcfdcc85-e3e7-4c06-94c8-c9d22a3862b7-kube-api-access-4c5h2\") pod \"maas-api-revoke-sub-cdbb1f67-t7bps\" (UID: \"fcfdcc85-e3e7-4c06-94c8-c9d22a3862b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbb1f67-t7bps" Oct 30 17:01:14.860330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:14.860213 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c5h2\" (UniqueName: \"kubernetes.io/projected/fcfdcc85-e3e7-4c06-94c8-c9d22a3862b7-kube-api-access-4c5h2\") pod \"maas-api-revoke-sub-cdbb1f67-t7bps\" (UID: \"fcfdcc85-e3e7-4c06-94c8-c9d22a3862b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbb1f67-t7bps" Oct 30 17:01:14.860524 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:14.860379 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcfdcc85-e3e7-4c06-94c8-c9d22a3862b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdbb1f67-t7bps\" (UID: \"fcfdcc85-e3e7-4c06-94c8-c9d22a3862b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbb1f67-t7bps" Oct 30 17:01:14.860524 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:14.860461 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcfdcc85-e3e7-4c06-94c8-c9d22a3862b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdbb1f67-t7bps\" (UID: \"fcfdcc85-e3e7-4c06-94c8-c9d22a3862b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbb1f67-t7bps" Oct 30 17:01:14.863938 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:14.861555 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcfdcc85-e3e7-4c06-94c8-c9d22a3862b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdbb1f67-t7bps\" (UID: \"fcfdcc85-e3e7-4c06-94c8-c9d22a3862b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbb1f67-t7bps" Oct 30 17:01:14.866322 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:14.865718 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcfdcc85-e3e7-4c06-94c8-c9d22a3862b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdbb1f67-t7bps\" (UID: \"fcfdcc85-e3e7-4c06-94c8-c9d22a3862b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbb1f67-t7bps" Oct 30 17:01:14.889071 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:14.889000 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c5h2\" (UniqueName: \"kubernetes.io/projected/fcfdcc85-e3e7-4c06-94c8-c9d22a3862b7-kube-api-access-4c5h2\") pod \"maas-api-revoke-sub-cdbb1f67-t7bps\" (UID: \"fcfdcc85-e3e7-4c06-94c8-c9d22a3862b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbb1f67-t7bps" Oct 30 17:01:14.964868 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:14.964831 4522 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 17:01:14.965050 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:14.964897 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq" podUID="46027d41-98ad-409d-a66f-4bc050bc2648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 17:01:15.013261 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:15.013226 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbb1f67-t7bps" Oct 30 17:01:15.344959 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:15.344917 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdbb1f67-t7bps"] Oct 30 17:01:15.345934 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:01:15.345905 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcfdcc85_e3e7_4c06_94c8_c9d22a3862b7.slice/crio-b10f7e21e4d5aa225e712210a465e061e3ac304d590829a22aaca8fc17a99a6c WatchSource:0}: Error finding container b10f7e21e4d5aa225e712210a465e061e3ac304d590829a22aaca8fc17a99a6c: Status 404 returned error can't find the container with id b10f7e21e4d5aa225e712210a465e061e3ac304d590829a22aaca8fc17a99a6c Oct 30 17:01:15.577953 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:15.577921 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbb1f67-t7bps" event={"ID":"fcfdcc85-e3e7-4c06-94c8-c9d22a3862b7","Type":"ContainerStarted","Data":"b10f7e21e4d5aa225e712210a465e061e3ac304d590829a22aaca8fc17a99a6c"} Oct 30 17:01:15.968783 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:15.968744 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq" Oct 30 17:01:15.969220 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:15.969194 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq" Oct 30 17:01:15.970058 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:15.970036 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq" Oct 30 17:01:16.587195 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:16.587106 4522 generic.go:356] "Generic (PLEG): container finished" podID="fcfdcc85-e3e7-4c06-94c8-c9d22a3862b7" containerID="810c545e1a8dbf8222b4aa8dfcce30e268c8e9da13283002f80ecb2025afe5b3" exitCode=0 Oct 30 17:01:16.587616 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:16.587191 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbb1f67-t7bps" event={"ID":"fcfdcc85-e3e7-4c06-94c8-c9d22a3862b7","Type":"ContainerDied","Data":"810c545e1a8dbf8222b4aa8dfcce30e268c8e9da13283002f80ecb2025afe5b3"} Oct 30 17:01:17.747547 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:17.747524 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbb1f67-t7bps" Oct 30 17:01:17.906227 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:17.906126 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcfdcc85-e3e7-4c06-94c8-c9d22a3862b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcfdcc85-e3e7-4c06-94c8-c9d22a3862b7-maas-api-openshift-ca\") pod \"fcfdcc85-e3e7-4c06-94c8-c9d22a3862b7\" (UID: \"fcfdcc85-e3e7-4c06-94c8-c9d22a3862b7\") " Oct 30 17:01:17.906447 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:17.906430 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcfdcc85-e3e7-4c06-94c8-c9d22a3862b7-kube-api-access-4c5h2\" (UniqueName: \"kubernetes.io/projected/fcfdcc85-e3e7-4c06-94c8-c9d22a3862b7-kube-api-access-4c5h2\") pod \"fcfdcc85-e3e7-4c06-94c8-c9d22a3862b7\" (UID: \"fcfdcc85-e3e7-4c06-94c8-c9d22a3862b7\") " Oct 30 17:01:17.906520 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:17.906480 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcfdcc85-e3e7-4c06-94c8-c9d22a3862b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcfdcc85-e3e7-4c06-94c8-c9d22a3862b7-maas-api-xks-ca\") pod \"fcfdcc85-e3e7-4c06-94c8-c9d22a3862b7\" (UID: \"fcfdcc85-e3e7-4c06-94c8-c9d22a3862b7\") " Oct 30 17:01:17.906589 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:17.906513 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcfdcc85-e3e7-4c06-94c8-c9d22a3862b7-maas-api-openshift-ca" pod "fcfdcc85-e3e7-4c06-94c8-c9d22a3862b7" (UID: "fcfdcc85-e3e7-4c06-94c8-c9d22a3862b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:01:17.906839 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:17.906822 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcfdcc85-e3e7-4c06-94c8-c9d22a3862b7-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:01:17.908496 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:17.908469 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcfdcc85-e3e7-4c06-94c8-c9d22a3862b7-kube-api-access-4c5h2" pod "fcfdcc85-e3e7-4c06-94c8-c9d22a3862b7" (UID: "fcfdcc85-e3e7-4c06-94c8-c9d22a3862b7"). InnerVolumeSpecName "kube-api-access-4c5h2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:01:17.908496 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:17.908480 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcfdcc85-e3e7-4c06-94c8-c9d22a3862b7-maas-api-xks-ca" pod "fcfdcc85-e3e7-4c06-94c8-c9d22a3862b7" (UID: "fcfdcc85-e3e7-4c06-94c8-c9d22a3862b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:01:18.008328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:18.008265 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c5h2\" (UniqueName: \"kubernetes.io/projected/fcfdcc85-e3e7-4c06-94c8-c9d22a3862b7-kube-api-access-4c5h2\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:01:18.008328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:18.008297 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcfdcc85-e3e7-4c06-94c8-c9d22a3862b7-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:01:18.599984 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:18.599950 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbb1f67-t7bps" Oct 30 17:01:18.600163 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:18.599975 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbb1f67-t7bps" event={"ID":"fcfdcc85-e3e7-4c06-94c8-c9d22a3862b7","Type":"ContainerDied","Data":"b10f7e21e4d5aa225e712210a465e061e3ac304d590829a22aaca8fc17a99a6c"} Oct 30 17:01:18.600163 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:18.600029 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b10f7e21e4d5aa225e712210a465e061e3ac304d590829a22aaca8fc17a99a6c" Oct 30 17:01:19.486299 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:19.486265 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02dc2587-j2jds"] Oct 30 17:01:19.487045 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:19.487007 4522 cpu_manager.go:414] "containerMap: removing container" podUID="fcfdcc85-e3e7-4c06-94c8-c9d22a3862b7" containerName="revoke-keys" Oct 30 17:01:19.487135 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:19.487046 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcfdcc85-e3e7-4c06-94c8-c9d22a3862b7" containerName="revoke-keys" Oct 30 17:01:19.487199 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:19.487171 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcfdcc85-e3e7-4c06-94c8-c9d22a3862b7" containerName="revoke-keys" Oct 30 17:01:19.490529 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:19.490509 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02dc2587-j2jds" Oct 30 17:01:19.509466 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:19.509442 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02dc2587-j2jds"] Oct 30 17:01:19.519505 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:19.519480 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:01:19.519656 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:19.519548 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:01:19.625919 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:19.625881 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df27828a-9121-4255-88a5-1ea5f5066337-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02dc2587-j2jds\" (UID: \"df27828a-9121-4255-88a5-1ea5f5066337\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02dc2587-j2jds" Oct 30 17:01:19.626152 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:19.626010 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df27828a-9121-4255-88a5-1ea5f5066337-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02dc2587-j2jds\" (UID: \"df27828a-9121-4255-88a5-1ea5f5066337\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02dc2587-j2jds" Oct 30 17:01:19.626231 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:19.626159 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jch7q\" (UniqueName: \"kubernetes.io/projected/df27828a-9121-4255-88a5-1ea5f5066337-kube-api-access-jch7q\") pod \"maas-api-revoke-sub-02dc2587-j2jds\" (UID: \"df27828a-9121-4255-88a5-1ea5f5066337\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02dc2587-j2jds" Oct 30 17:01:19.727259 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:19.727220 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df27828a-9121-4255-88a5-1ea5f5066337-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02dc2587-j2jds\" (UID: \"df27828a-9121-4255-88a5-1ea5f5066337\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02dc2587-j2jds" Oct 30 17:01:19.727447 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:19.727340 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df27828a-9121-4255-88a5-1ea5f5066337-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02dc2587-j2jds\" (UID: \"df27828a-9121-4255-88a5-1ea5f5066337\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02dc2587-j2jds" Oct 30 17:01:19.727507 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:19.727493 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jch7q\" (UniqueName: \"kubernetes.io/projected/df27828a-9121-4255-88a5-1ea5f5066337-kube-api-access-jch7q\") pod \"maas-api-revoke-sub-02dc2587-j2jds\" (UID: \"df27828a-9121-4255-88a5-1ea5f5066337\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02dc2587-j2jds" Oct 30 17:01:19.727872 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:19.727846 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df27828a-9121-4255-88a5-1ea5f5066337-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02dc2587-j2jds\" (UID: \"df27828a-9121-4255-88a5-1ea5f5066337\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02dc2587-j2jds" Oct 30 17:01:19.729821 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:19.729796 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df27828a-9121-4255-88a5-1ea5f5066337-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02dc2587-j2jds\" (UID: \"df27828a-9121-4255-88a5-1ea5f5066337\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02dc2587-j2jds" Oct 30 17:01:19.757823 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:19.753898 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jch7q\" (UniqueName: \"kubernetes.io/projected/df27828a-9121-4255-88a5-1ea5f5066337-kube-api-access-jch7q\") pod \"maas-api-revoke-sub-02dc2587-j2jds\" (UID: \"df27828a-9121-4255-88a5-1ea5f5066337\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02dc2587-j2jds" Oct 30 17:01:19.808238 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:19.808192 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02dc2587-j2jds" Oct 30 17:01:20.134895 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:20.134730 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02dc2587-j2jds"] Oct 30 17:01:20.633984 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:20.633930 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02dc2587-j2jds" event={"ID":"df27828a-9121-4255-88a5-1ea5f5066337","Type":"ContainerStarted","Data":"3738f941f6a7d993244e8310aed350aa1b56bad3c22c48e1c269c1ad173331d4"} Oct 30 17:01:20.634557 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:20.634014 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02dc2587-j2jds" event={"ID":"df27828a-9121-4255-88a5-1ea5f5066337","Type":"ContainerStarted","Data":"c0187c4b461bc97968ceaa3d202dfcdd058907975c0ecd2a625fc2ce1151fadd"} Oct 30 17:01:20.684427 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:20.684275 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02dc2587-j2jds" podStartSLOduration=1.68425549 podStartE2EDuration="1.68425549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:01:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:01:20.682186001 +0000 UTC m=+854.682940931" watchObservedRunningTime="2026-10-30 17:01:20.68425549 +0000 UTC m=+854.685010393" Oct 30 17:01:21.645904 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:21.645287 4522 generic.go:356] "Generic (PLEG): container finished" podID="df27828a-9121-4255-88a5-1ea5f5066337" containerID="3738f941f6a7d993244e8310aed350aa1b56bad3c22c48e1c269c1ad173331d4" exitCode=0 Oct 30 17:01:21.645904 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:21.645502 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02dc2587-j2jds" event={"ID":"df27828a-9121-4255-88a5-1ea5f5066337","Type":"ContainerDied","Data":"3738f941f6a7d993244e8310aed350aa1b56bad3c22c48e1c269c1ad173331d4"} Oct 30 17:01:22.905503 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:22.905472 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02dc2587-j2jds" Oct 30 17:01:22.984573 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:22.984545 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df27828a-9121-4255-88a5-1ea5f5066337-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df27828a-9121-4255-88a5-1ea5f5066337-maas-api-openshift-ca\") pod \"df27828a-9121-4255-88a5-1ea5f5066337\" (UID: \"df27828a-9121-4255-88a5-1ea5f5066337\") " Oct 30 17:01:22.984726 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:22.984592 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df27828a-9121-4255-88a5-1ea5f5066337-kube-api-access-jch7q\" (UniqueName: \"kubernetes.io/projected/df27828a-9121-4255-88a5-1ea5f5066337-kube-api-access-jch7q\") pod \"df27828a-9121-4255-88a5-1ea5f5066337\" (UID: \"df27828a-9121-4255-88a5-1ea5f5066337\") " Oct 30 17:01:22.984782 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:22.984747 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df27828a-9121-4255-88a5-1ea5f5066337-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df27828a-9121-4255-88a5-1ea5f5066337-maas-api-xks-ca\") pod \"df27828a-9121-4255-88a5-1ea5f5066337\" (UID: \"df27828a-9121-4255-88a5-1ea5f5066337\") " Oct 30 17:01:22.984862 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:22.984839 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df27828a-9121-4255-88a5-1ea5f5066337-maas-api-openshift-ca" pod "df27828a-9121-4255-88a5-1ea5f5066337" (UID: "df27828a-9121-4255-88a5-1ea5f5066337"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:01:22.985231 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:22.985208 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df27828a-9121-4255-88a5-1ea5f5066337-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:01:22.986823 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:22.986795 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df27828a-9121-4255-88a5-1ea5f5066337-maas-api-xks-ca" pod "df27828a-9121-4255-88a5-1ea5f5066337" (UID: "df27828a-9121-4255-88a5-1ea5f5066337"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:01:22.987432 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:22.987409 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df27828a-9121-4255-88a5-1ea5f5066337-kube-api-access-jch7q" pod "df27828a-9121-4255-88a5-1ea5f5066337" (UID: "df27828a-9121-4255-88a5-1ea5f5066337"). InnerVolumeSpecName "kube-api-access-jch7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:01:23.086502 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:23.086469 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jch7q\" (UniqueName: \"kubernetes.io/projected/df27828a-9121-4255-88a5-1ea5f5066337-kube-api-access-jch7q\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:01:23.086502 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:23.086498 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df27828a-9121-4255-88a5-1ea5f5066337-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:01:23.398385 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:23.398286 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6434aa3d-556sw"] Oct 30 17:01:23.398819 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:23.398805 4522 cpu_manager.go:414] "containerMap: removing container" podUID="df27828a-9121-4255-88a5-1ea5f5066337" containerName="revoke-keys" Oct 30 17:01:23.398865 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:23.398819 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df27828a-9121-4255-88a5-1ea5f5066337" containerName="revoke-keys" Oct 30 17:01:23.398948 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:23.398938 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="df27828a-9121-4255-88a5-1ea5f5066337" containerName="revoke-keys" Oct 30 17:01:23.402216 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:23.402199 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6434aa3d-556sw" Oct 30 17:01:23.429509 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:23.429471 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6434aa3d-556sw"] Oct 30 17:01:23.491735 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:23.491697 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa8d133a-3261-41e5-88f4-18fcaeb3ec46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6434aa3d-556sw\" (UID: \"fa8d133a-3261-41e5-88f4-18fcaeb3ec46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6434aa3d-556sw" Oct 30 17:01:23.491910 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:23.491841 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa8d133a-3261-41e5-88f4-18fcaeb3ec46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6434aa3d-556sw\" (UID: \"fa8d133a-3261-41e5-88f4-18fcaeb3ec46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6434aa3d-556sw" Oct 30 17:01:23.491994 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:23.491980 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srxfv\" (UniqueName: \"kubernetes.io/projected/fa8d133a-3261-41e5-88f4-18fcaeb3ec46-kube-api-access-srxfv\") pod \"maas-api-revoke-sub-6434aa3d-556sw\" (UID: \"fa8d133a-3261-41e5-88f4-18fcaeb3ec46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6434aa3d-556sw" Oct 30 17:01:23.592686 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:23.592647 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa8d133a-3261-41e5-88f4-18fcaeb3ec46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6434aa3d-556sw\" (UID: \"fa8d133a-3261-41e5-88f4-18fcaeb3ec46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6434aa3d-556sw" Oct 30 17:01:23.592894 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:23.592749 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa8d133a-3261-41e5-88f4-18fcaeb3ec46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6434aa3d-556sw\" (UID: \"fa8d133a-3261-41e5-88f4-18fcaeb3ec46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6434aa3d-556sw" Oct 30 17:01:23.592894 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:23.592856 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srxfv\" (UniqueName: \"kubernetes.io/projected/fa8d133a-3261-41e5-88f4-18fcaeb3ec46-kube-api-access-srxfv\") pod \"maas-api-revoke-sub-6434aa3d-556sw\" (UID: \"fa8d133a-3261-41e5-88f4-18fcaeb3ec46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6434aa3d-556sw" Oct 30 17:01:23.593423 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:23.593398 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa8d133a-3261-41e5-88f4-18fcaeb3ec46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6434aa3d-556sw\" (UID: \"fa8d133a-3261-41e5-88f4-18fcaeb3ec46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6434aa3d-556sw" Oct 30 17:01:23.595087 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:23.595066 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa8d133a-3261-41e5-88f4-18fcaeb3ec46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6434aa3d-556sw\" (UID: \"fa8d133a-3261-41e5-88f4-18fcaeb3ec46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6434aa3d-556sw" Oct 30 17:01:23.617561 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:23.617535 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srxfv\" (UniqueName: \"kubernetes.io/projected/fa8d133a-3261-41e5-88f4-18fcaeb3ec46-kube-api-access-srxfv\") pod \"maas-api-revoke-sub-6434aa3d-556sw\" (UID: \"fa8d133a-3261-41e5-88f4-18fcaeb3ec46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6434aa3d-556sw" Oct 30 17:01:23.668373 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:23.668284 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02dc2587-j2jds" Oct 30 17:01:23.668373 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:23.668319 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02dc2587-j2jds" event={"ID":"df27828a-9121-4255-88a5-1ea5f5066337","Type":"ContainerDied","Data":"c0187c4b461bc97968ceaa3d202dfcdd058907975c0ecd2a625fc2ce1151fadd"} Oct 30 17:01:23.668373 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:23.668368 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0187c4b461bc97968ceaa3d202dfcdd058907975c0ecd2a625fc2ce1151fadd" Oct 30 17:01:23.716487 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:23.716380 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6434aa3d-556sw" Oct 30 17:01:24.019381 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:01:24.019350 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa8d133a_3261_41e5_88f4_18fcaeb3ec46.slice/crio-ef9c54df5d4da46bd477008aa71ffddef3f0136f7143022ecdb48fac83d55a01 WatchSource:0}: Error finding container ef9c54df5d4da46bd477008aa71ffddef3f0136f7143022ecdb48fac83d55a01: Status 404 returned error can't find the container with id ef9c54df5d4da46bd477008aa71ffddef3f0136f7143022ecdb48fac83d55a01 Oct 30 17:01:24.021193 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:24.021170 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6434aa3d-556sw"] Oct 30 17:01:24.686415 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:24.686283 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6434aa3d-556sw" event={"ID":"fa8d133a-3261-41e5-88f4-18fcaeb3ec46","Type":"ContainerStarted","Data":"78a8ea388c8d5d0ea1da0c59f040b9c5c2d0a006cb3c78620391a7db2d691a40"} Oct 30 17:01:24.686415 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:24.686355 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6434aa3d-556sw" event={"ID":"fa8d133a-3261-41e5-88f4-18fcaeb3ec46","Type":"ContainerStarted","Data":"ef9c54df5d4da46bd477008aa71ffddef3f0136f7143022ecdb48fac83d55a01"} Oct 30 17:01:24.733641 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:24.733588 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6434aa3d-556sw" podStartSLOduration=1.73357461 podStartE2EDuration="1.73357461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:01:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:01:24.730889528 +0000 UTC m=+858.731644428" watchObservedRunningTime="2026-10-30 17:01:24.73357461 +0000 UTC m=+858.734329510" Oct 30 17:01:25.694957 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:25.694920 4522 generic.go:356] "Generic (PLEG): container finished" podID="fa8d133a-3261-41e5-88f4-18fcaeb3ec46" containerID="78a8ea388c8d5d0ea1da0c59f040b9c5c2d0a006cb3c78620391a7db2d691a40" exitCode=0 Oct 30 17:01:25.695400 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:25.695002 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6434aa3d-556sw" event={"ID":"fa8d133a-3261-41e5-88f4-18fcaeb3ec46","Type":"ContainerDied","Data":"78a8ea388c8d5d0ea1da0c59f040b9c5c2d0a006cb3c78620391a7db2d691a40"} Oct 30 17:01:26.865003 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:26.864975 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6434aa3d-556sw" Oct 30 17:01:26.929746 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:26.929708 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa8d133a-3261-41e5-88f4-18fcaeb3ec46-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa8d133a-3261-41e5-88f4-18fcaeb3ec46-maas-api-xks-ca\") pod \"fa8d133a-3261-41e5-88f4-18fcaeb3ec46\" (UID: \"fa8d133a-3261-41e5-88f4-18fcaeb3ec46\") " Oct 30 17:01:26.929884 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:26.929841 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa8d133a-3261-41e5-88f4-18fcaeb3ec46-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa8d133a-3261-41e5-88f4-18fcaeb3ec46-maas-api-openshift-ca\") pod \"fa8d133a-3261-41e5-88f4-18fcaeb3ec46\" (UID: \"fa8d133a-3261-41e5-88f4-18fcaeb3ec46\") " Oct 30 17:01:26.929943 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:26.929899 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa8d133a-3261-41e5-88f4-18fcaeb3ec46-kube-api-access-srxfv\" (UniqueName: \"kubernetes.io/projected/fa8d133a-3261-41e5-88f4-18fcaeb3ec46-kube-api-access-srxfv\") pod \"fa8d133a-3261-41e5-88f4-18fcaeb3ec46\" (UID: \"fa8d133a-3261-41e5-88f4-18fcaeb3ec46\") " Oct 30 17:01:26.930189 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:26.930166 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa8d133a-3261-41e5-88f4-18fcaeb3ec46-maas-api-openshift-ca" pod "fa8d133a-3261-41e5-88f4-18fcaeb3ec46" (UID: "fa8d133a-3261-41e5-88f4-18fcaeb3ec46"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:01:26.930462 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:26.930440 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa8d133a-3261-41e5-88f4-18fcaeb3ec46-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:01:26.931956 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:26.931933 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa8d133a-3261-41e5-88f4-18fcaeb3ec46-maas-api-xks-ca" pod "fa8d133a-3261-41e5-88f4-18fcaeb3ec46" (UID: "fa8d133a-3261-41e5-88f4-18fcaeb3ec46"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:01:26.932450 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:26.932426 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa8d133a-3261-41e5-88f4-18fcaeb3ec46-kube-api-access-srxfv" pod "fa8d133a-3261-41e5-88f4-18fcaeb3ec46" (UID: "fa8d133a-3261-41e5-88f4-18fcaeb3ec46"). InnerVolumeSpecName "kube-api-access-srxfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:01:27.031880 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:27.031797 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa8d133a-3261-41e5-88f4-18fcaeb3ec46-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:01:27.031880 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:27.031830 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srxfv\" (UniqueName: \"kubernetes.io/projected/fa8d133a-3261-41e5-88f4-18fcaeb3ec46-kube-api-access-srxfv\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:01:27.712109 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:27.712082 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6434aa3d-556sw" Oct 30 17:01:27.712109 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:27.712096 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6434aa3d-556sw" event={"ID":"fa8d133a-3261-41e5-88f4-18fcaeb3ec46","Type":"ContainerDied","Data":"ef9c54df5d4da46bd477008aa71ffddef3f0136f7143022ecdb48fac83d55a01"} Oct 30 17:01:27.712400 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:27.712150 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef9c54df5d4da46bd477008aa71ffddef3f0136f7143022ecdb48fac83d55a01" Oct 30 17:01:29.079252 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:29.079215 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eea1ffaf-l78l7"] Oct 30 17:01:29.079760 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:29.079746 4522 cpu_manager.go:414] "containerMap: removing container" podUID="fa8d133a-3261-41e5-88f4-18fcaeb3ec46" containerName="revoke-keys" Oct 30 17:01:29.079804 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:29.079761 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa8d133a-3261-41e5-88f4-18fcaeb3ec46" containerName="revoke-keys" Oct 30 17:01:29.079855 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:29.079846 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa8d133a-3261-41e5-88f4-18fcaeb3ec46" containerName="revoke-keys" Oct 30 17:01:29.083279 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:29.083262 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea1ffaf-l78l7" Oct 30 17:01:29.114784 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:29.114750 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:01:29.114999 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:29.114882 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:01:29.118447 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:29.118420 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eea1ffaf-l78l7"] Oct 30 17:01:29.157670 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:29.157623 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b04afd1f-0a81-4e8f-bb65-1fb57a6df53d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eea1ffaf-l78l7\" (UID: \"b04afd1f-0a81-4e8f-bb65-1fb57a6df53d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea1ffaf-l78l7" Oct 30 17:01:29.157869 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:29.157792 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns9bg\" (UniqueName: \"kubernetes.io/projected/b04afd1f-0a81-4e8f-bb65-1fb57a6df53d-kube-api-access-ns9bg\") pod \"maas-api-revoke-sub-eea1ffaf-l78l7\" (UID: \"b04afd1f-0a81-4e8f-bb65-1fb57a6df53d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea1ffaf-l78l7" Oct 30 17:01:29.157942 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:29.157905 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b04afd1f-0a81-4e8f-bb65-1fb57a6df53d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eea1ffaf-l78l7\" (UID: \"b04afd1f-0a81-4e8f-bb65-1fb57a6df53d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea1ffaf-l78l7" Oct 30 17:01:29.258747 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:29.258709 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns9bg\" (UniqueName: \"kubernetes.io/projected/b04afd1f-0a81-4e8f-bb65-1fb57a6df53d-kube-api-access-ns9bg\") pod \"maas-api-revoke-sub-eea1ffaf-l78l7\" (UID: \"b04afd1f-0a81-4e8f-bb65-1fb57a6df53d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea1ffaf-l78l7" Oct 30 17:01:29.258929 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:29.258786 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b04afd1f-0a81-4e8f-bb65-1fb57a6df53d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eea1ffaf-l78l7\" (UID: \"b04afd1f-0a81-4e8f-bb65-1fb57a6df53d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea1ffaf-l78l7" Oct 30 17:01:29.258929 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:29.258848 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b04afd1f-0a81-4e8f-bb65-1fb57a6df53d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eea1ffaf-l78l7\" (UID: \"b04afd1f-0a81-4e8f-bb65-1fb57a6df53d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea1ffaf-l78l7" Oct 30 17:01:29.259427 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:29.259402 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b04afd1f-0a81-4e8f-bb65-1fb57a6df53d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eea1ffaf-l78l7\" (UID: \"b04afd1f-0a81-4e8f-bb65-1fb57a6df53d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea1ffaf-l78l7" Oct 30 17:01:29.261147 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:29.261127 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b04afd1f-0a81-4e8f-bb65-1fb57a6df53d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eea1ffaf-l78l7\" (UID: \"b04afd1f-0a81-4e8f-bb65-1fb57a6df53d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea1ffaf-l78l7" Oct 30 17:01:29.284979 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:29.284953 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns9bg\" (UniqueName: \"kubernetes.io/projected/b04afd1f-0a81-4e8f-bb65-1fb57a6df53d-kube-api-access-ns9bg\") pod \"maas-api-revoke-sub-eea1ffaf-l78l7\" (UID: \"b04afd1f-0a81-4e8f-bb65-1fb57a6df53d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea1ffaf-l78l7" Oct 30 17:01:29.396751 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:29.396672 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea1ffaf-l78l7" Oct 30 17:01:29.726175 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:29.726143 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eea1ffaf-l78l7"] Oct 30 17:01:29.732345 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:01:29.732279 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb04afd1f_0a81_4e8f_bb65_1fb57a6df53d.slice/crio-c2918a35af00a0e1ad47b818e550e61ec11e47c2ea856711c41b4ae20189d227 WatchSource:0}: Error finding container c2918a35af00a0e1ad47b818e550e61ec11e47c2ea856711c41b4ae20189d227: Status 404 returned error can't find the container with id c2918a35af00a0e1ad47b818e550e61ec11e47c2ea856711c41b4ae20189d227 Oct 30 17:01:30.734470 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:30.734438 4522 generic.go:356] "Generic (PLEG): container finished" podID="b04afd1f-0a81-4e8f-bb65-1fb57a6df53d" containerID="1ece4e9e30d17b7b59483e7da57ee98c6ea168764ed07c547f5bd107513d4096" exitCode=0 Oct 30 17:01:30.734848 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:30.734491 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea1ffaf-l78l7" event={"ID":"b04afd1f-0a81-4e8f-bb65-1fb57a6df53d","Type":"ContainerDied","Data":"1ece4e9e30d17b7b59483e7da57ee98c6ea168764ed07c547f5bd107513d4096"} Oct 30 17:01:30.734848 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:30.734547 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea1ffaf-l78l7" event={"ID":"b04afd1f-0a81-4e8f-bb65-1fb57a6df53d","Type":"ContainerStarted","Data":"c2918a35af00a0e1ad47b818e550e61ec11e47c2ea856711c41b4ae20189d227"} Oct 30 17:01:31.885802 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:31.885770 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea1ffaf-l78l7" Oct 30 17:01:31.991779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:31.991749 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b04afd1f-0a81-4e8f-bb65-1fb57a6df53d-kube-api-access-ns9bg\" (UniqueName: \"kubernetes.io/projected/b04afd1f-0a81-4e8f-bb65-1fb57a6df53d-kube-api-access-ns9bg\") pod \"b04afd1f-0a81-4e8f-bb65-1fb57a6df53d\" (UID: \"b04afd1f-0a81-4e8f-bb65-1fb57a6df53d\") " Oct 30 17:01:31.991958 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:31.991791 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b04afd1f-0a81-4e8f-bb65-1fb57a6df53d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b04afd1f-0a81-4e8f-bb65-1fb57a6df53d-maas-api-xks-ca\") pod \"b04afd1f-0a81-4e8f-bb65-1fb57a6df53d\" (UID: \"b04afd1f-0a81-4e8f-bb65-1fb57a6df53d\") " Oct 30 17:01:31.991958 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:31.991849 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b04afd1f-0a81-4e8f-bb65-1fb57a6df53d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b04afd1f-0a81-4e8f-bb65-1fb57a6df53d-maas-api-openshift-ca\") pod \"b04afd1f-0a81-4e8f-bb65-1fb57a6df53d\" (UID: \"b04afd1f-0a81-4e8f-bb65-1fb57a6df53d\") " Oct 30 17:01:31.992269 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:31.992244 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b04afd1f-0a81-4e8f-bb65-1fb57a6df53d-maas-api-openshift-ca" pod "b04afd1f-0a81-4e8f-bb65-1fb57a6df53d" (UID: "b04afd1f-0a81-4e8f-bb65-1fb57a6df53d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:01:31.993873 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:31.993846 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b04afd1f-0a81-4e8f-bb65-1fb57a6df53d-maas-api-xks-ca" pod "b04afd1f-0a81-4e8f-bb65-1fb57a6df53d" (UID: "b04afd1f-0a81-4e8f-bb65-1fb57a6df53d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:01:31.993990 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:31.993975 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b04afd1f-0a81-4e8f-bb65-1fb57a6df53d-kube-api-access-ns9bg" pod "b04afd1f-0a81-4e8f-bb65-1fb57a6df53d" (UID: "b04afd1f-0a81-4e8f-bb65-1fb57a6df53d"). InnerVolumeSpecName "kube-api-access-ns9bg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:01:32.093244 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:32.093207 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ns9bg\" (UniqueName: \"kubernetes.io/projected/b04afd1f-0a81-4e8f-bb65-1fb57a6df53d-kube-api-access-ns9bg\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:01:32.093244 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:32.093239 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b04afd1f-0a81-4e8f-bb65-1fb57a6df53d-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:01:32.093244 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:32.093249 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b04afd1f-0a81-4e8f-bb65-1fb57a6df53d-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:01:32.752221 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:32.752191 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea1ffaf-l78l7" Oct 30 17:01:32.752439 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:32.752224 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea1ffaf-l78l7" event={"ID":"b04afd1f-0a81-4e8f-bb65-1fb57a6df53d","Type":"ContainerDied","Data":"c2918a35af00a0e1ad47b818e550e61ec11e47c2ea856711c41b4ae20189d227"} Oct 30 17:01:32.752439 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:32.752271 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2918a35af00a0e1ad47b818e550e61ec11e47c2ea856711c41b4ae20189d227" Oct 30 17:01:34.094530 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:34.094491 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15a438c9-99cvt"] Oct 30 17:01:34.095047 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:34.095034 4522 cpu_manager.go:414] "containerMap: removing container" podUID="b04afd1f-0a81-4e8f-bb65-1fb57a6df53d" containerName="revoke-keys" Oct 30 17:01:34.095087 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:34.095048 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b04afd1f-0a81-4e8f-bb65-1fb57a6df53d" containerName="revoke-keys" Oct 30 17:01:34.095146 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:34.095137 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="b04afd1f-0a81-4e8f-bb65-1fb57a6df53d" containerName="revoke-keys" Oct 30 17:01:34.098651 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:34.098632 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a438c9-99cvt" Oct 30 17:01:34.121591 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:34.121556 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15a438c9-99cvt"] Oct 30 17:01:34.129948 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:34.129921 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:01:34.129948 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:34.129942 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:01:34.220051 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:34.220020 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9b46f5d-5403-4847-b498-bfae1b880d6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15a438c9-99cvt\" (UID: \"a9b46f5d-5403-4847-b498-bfae1b880d6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a438c9-99cvt" Oct 30 17:01:34.220214 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:34.220192 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9b46f5d-5403-4847-b498-bfae1b880d6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15a438c9-99cvt\" (UID: \"a9b46f5d-5403-4847-b498-bfae1b880d6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a438c9-99cvt" Oct 30 17:01:34.220288 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:34.220253 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn2bv\" (UniqueName: \"kubernetes.io/projected/a9b46f5d-5403-4847-b498-bfae1b880d6a-kube-api-access-wn2bv\") pod \"maas-api-revoke-sub-15a438c9-99cvt\" (UID: \"a9b46f5d-5403-4847-b498-bfae1b880d6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a438c9-99cvt" Oct 30 17:01:34.321149 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:34.321115 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9b46f5d-5403-4847-b498-bfae1b880d6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15a438c9-99cvt\" (UID: \"a9b46f5d-5403-4847-b498-bfae1b880d6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a438c9-99cvt" Oct 30 17:01:34.321341 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:34.321172 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn2bv\" (UniqueName: \"kubernetes.io/projected/a9b46f5d-5403-4847-b498-bfae1b880d6a-kube-api-access-wn2bv\") pod \"maas-api-revoke-sub-15a438c9-99cvt\" (UID: \"a9b46f5d-5403-4847-b498-bfae1b880d6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a438c9-99cvt" Oct 30 17:01:34.321341 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:34.321247 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9b46f5d-5403-4847-b498-bfae1b880d6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15a438c9-99cvt\" (UID: \"a9b46f5d-5403-4847-b498-bfae1b880d6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a438c9-99cvt" Oct 30 17:01:34.321967 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:34.321944 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9b46f5d-5403-4847-b498-bfae1b880d6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15a438c9-99cvt\" (UID: \"a9b46f5d-5403-4847-b498-bfae1b880d6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a438c9-99cvt" Oct 30 17:01:34.323574 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:34.323555 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9b46f5d-5403-4847-b498-bfae1b880d6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15a438c9-99cvt\" (UID: \"a9b46f5d-5403-4847-b498-bfae1b880d6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a438c9-99cvt" Oct 30 17:01:34.346479 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:34.346404 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn2bv\" (UniqueName: \"kubernetes.io/projected/a9b46f5d-5403-4847-b498-bfae1b880d6a-kube-api-access-wn2bv\") pod \"maas-api-revoke-sub-15a438c9-99cvt\" (UID: \"a9b46f5d-5403-4847-b498-bfae1b880d6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a438c9-99cvt" Oct 30 17:01:34.411800 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:34.411767 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a438c9-99cvt" Oct 30 17:01:34.716717 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:01:34.716687 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9b46f5d_5403_4847_b498_bfae1b880d6a.slice/crio-45fd263b001c22dcb9f36d0ce2f740481b1f31a67110916761c55d9743a6b29b WatchSource:0}: Error finding container 45fd263b001c22dcb9f36d0ce2f740481b1f31a67110916761c55d9743a6b29b: Status 404 returned error can't find the container with id 45fd263b001c22dcb9f36d0ce2f740481b1f31a67110916761c55d9743a6b29b Oct 30 17:01:34.718883 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:34.718798 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15a438c9-99cvt"] Oct 30 17:01:34.769551 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:34.769520 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a438c9-99cvt" event={"ID":"a9b46f5d-5403-4847-b498-bfae1b880d6a","Type":"ContainerStarted","Data":"45fd263b001c22dcb9f36d0ce2f740481b1f31a67110916761c55d9743a6b29b"} Oct 30 17:01:35.777559 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:35.777477 4522 generic.go:356] "Generic (PLEG): container finished" podID="a9b46f5d-5403-4847-b498-bfae1b880d6a" containerID="cb73fc8da8a19694d9f513b2a7bbb9f9db30548966647dccf31ee17387d58747" exitCode=0 Oct 30 17:01:35.777559 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:35.777532 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a438c9-99cvt" event={"ID":"a9b46f5d-5403-4847-b498-bfae1b880d6a","Type":"ContainerDied","Data":"cb73fc8da8a19694d9f513b2a7bbb9f9db30548966647dccf31ee17387d58747"} Oct 30 17:01:36.969908 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:36.969872 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a438c9-99cvt" Oct 30 17:01:37.058260 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:37.058169 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9b46f5d-5403-4847-b498-bfae1b880d6a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9b46f5d-5403-4847-b498-bfae1b880d6a-maas-api-openshift-ca\") pod \"a9b46f5d-5403-4847-b498-bfae1b880d6a\" (UID: \"a9b46f5d-5403-4847-b498-bfae1b880d6a\") " Oct 30 17:01:37.058453 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:37.058382 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9b46f5d-5403-4847-b498-bfae1b880d6a-kube-api-access-wn2bv\" (UniqueName: \"kubernetes.io/projected/a9b46f5d-5403-4847-b498-bfae1b880d6a-kube-api-access-wn2bv\") pod \"a9b46f5d-5403-4847-b498-bfae1b880d6a\" (UID: \"a9b46f5d-5403-4847-b498-bfae1b880d6a\") " Oct 30 17:01:37.058453 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:37.058433 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9b46f5d-5403-4847-b498-bfae1b880d6a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9b46f5d-5403-4847-b498-bfae1b880d6a-maas-api-xks-ca\") pod \"a9b46f5d-5403-4847-b498-bfae1b880d6a\" (UID: \"a9b46f5d-5403-4847-b498-bfae1b880d6a\") " Oct 30 17:01:37.059184 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:37.059149 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9b46f5d-5403-4847-b498-bfae1b880d6a-maas-api-openshift-ca" pod "a9b46f5d-5403-4847-b498-bfae1b880d6a" (UID: "a9b46f5d-5403-4847-b498-bfae1b880d6a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:01:37.067819 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:37.067784 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9b46f5d-5403-4847-b498-bfae1b880d6a-maas-api-xks-ca" pod "a9b46f5d-5403-4847-b498-bfae1b880d6a" (UID: "a9b46f5d-5403-4847-b498-bfae1b880d6a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:01:37.068422 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:37.068392 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9b46f5d-5403-4847-b498-bfae1b880d6a-kube-api-access-wn2bv" pod "a9b46f5d-5403-4847-b498-bfae1b880d6a" (UID: "a9b46f5d-5403-4847-b498-bfae1b880d6a"). InnerVolumeSpecName "kube-api-access-wn2bv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:01:37.160433 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:37.160394 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9b46f5d-5403-4847-b498-bfae1b880d6a-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:01:37.160683 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:37.160661 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wn2bv\" (UniqueName: \"kubernetes.io/projected/a9b46f5d-5403-4847-b498-bfae1b880d6a-kube-api-access-wn2bv\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:01:37.160792 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:37.160779 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9b46f5d-5403-4847-b498-bfae1b880d6a-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:01:37.797381 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:37.797341 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a438c9-99cvt" event={"ID":"a9b46f5d-5403-4847-b498-bfae1b880d6a","Type":"ContainerDied","Data":"45fd263b001c22dcb9f36d0ce2f740481b1f31a67110916761c55d9743a6b29b"} Oct 30 17:01:37.797684 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:37.797665 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45fd263b001c22dcb9f36d0ce2f740481b1f31a67110916761c55d9743a6b29b" Oct 30 17:01:37.797858 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:37.797844 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a438c9-99cvt" Oct 30 17:01:39.977461 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:39.977423 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-637f442f-h8ckw"] Oct 30 17:01:39.978148 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:39.978127 4522 cpu_manager.go:414] "containerMap: removing container" podUID="a9b46f5d-5403-4847-b498-bfae1b880d6a" containerName="revoke-keys" Oct 30 17:01:39.978243 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:39.978150 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9b46f5d-5403-4847-b498-bfae1b880d6a" containerName="revoke-keys" Oct 30 17:01:39.978353 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:39.978339 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9b46f5d-5403-4847-b498-bfae1b880d6a" containerName="revoke-keys" Oct 30 17:01:39.982526 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:39.982506 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-637f442f-h8ckw" Oct 30 17:01:40.001078 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:40.001047 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-637f442f-h8ckw"] Oct 30 17:01:40.010803 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:40.010775 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:01:40.010923 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:40.010815 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:01:40.119547 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:40.119510 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c753be2-3052-46c0-844c-f7eebc0cbdcf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-637f442f-h8ckw\" (UID: \"6c753be2-3052-46c0-844c-f7eebc0cbdcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-637f442f-h8ckw" Oct 30 17:01:40.119726 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:40.119689 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c753be2-3052-46c0-844c-f7eebc0cbdcf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-637f442f-h8ckw\" (UID: \"6c753be2-3052-46c0-844c-f7eebc0cbdcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-637f442f-h8ckw" Oct 30 17:01:40.119726 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:40.119720 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpzk8\" (UniqueName: \"kubernetes.io/projected/6c753be2-3052-46c0-844c-f7eebc0cbdcf-kube-api-access-lpzk8\") pod \"maas-api-revoke-sub-637f442f-h8ckw\" (UID: \"6c753be2-3052-46c0-844c-f7eebc0cbdcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-637f442f-h8ckw" Oct 30 17:01:40.220588 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:40.220551 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c753be2-3052-46c0-844c-f7eebc0cbdcf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-637f442f-h8ckw\" (UID: \"6c753be2-3052-46c0-844c-f7eebc0cbdcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-637f442f-h8ckw" Oct 30 17:01:40.220774 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:40.220600 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpzk8\" (UniqueName: \"kubernetes.io/projected/6c753be2-3052-46c0-844c-f7eebc0cbdcf-kube-api-access-lpzk8\") pod \"maas-api-revoke-sub-637f442f-h8ckw\" (UID: \"6c753be2-3052-46c0-844c-f7eebc0cbdcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-637f442f-h8ckw" Oct 30 17:01:40.220774 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:40.220744 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c753be2-3052-46c0-844c-f7eebc0cbdcf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-637f442f-h8ckw\" (UID: \"6c753be2-3052-46c0-844c-f7eebc0cbdcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-637f442f-h8ckw" Oct 30 17:01:40.221209 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:40.221187 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c753be2-3052-46c0-844c-f7eebc0cbdcf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-637f442f-h8ckw\" (UID: \"6c753be2-3052-46c0-844c-f7eebc0cbdcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-637f442f-h8ckw" Oct 30 17:01:40.223094 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:40.223070 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c753be2-3052-46c0-844c-f7eebc0cbdcf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-637f442f-h8ckw\" (UID: \"6c753be2-3052-46c0-844c-f7eebc0cbdcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-637f442f-h8ckw" Oct 30 17:01:40.245785 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:40.245720 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpzk8\" (UniqueName: \"kubernetes.io/projected/6c753be2-3052-46c0-844c-f7eebc0cbdcf-kube-api-access-lpzk8\") pod \"maas-api-revoke-sub-637f442f-h8ckw\" (UID: \"6c753be2-3052-46c0-844c-f7eebc0cbdcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-637f442f-h8ckw" Oct 30 17:01:40.295933 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:40.295883 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-637f442f-h8ckw" Oct 30 17:01:40.607224 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:01:40.607190 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c753be2_3052_46c0_844c_f7eebc0cbdcf.slice/crio-67889fa5cd8e813161c877d81c4341a06a4aa98dea141a86014a52ec97ba0acd WatchSource:0}: Error finding container 67889fa5cd8e813161c877d81c4341a06a4aa98dea141a86014a52ec97ba0acd: Status 404 returned error can't find the container with id 67889fa5cd8e813161c877d81c4341a06a4aa98dea141a86014a52ec97ba0acd Oct 30 17:01:40.607895 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:40.607866 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-637f442f-h8ckw"] Oct 30 17:01:40.829218 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:40.829187 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-637f442f-h8ckw" event={"ID":"6c753be2-3052-46c0-844c-f7eebc0cbdcf","Type":"ContainerStarted","Data":"67889fa5cd8e813161c877d81c4341a06a4aa98dea141a86014a52ec97ba0acd"} Oct 30 17:01:41.843966 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:41.843877 4522 generic.go:356] "Generic (PLEG): container finished" podID="6c753be2-3052-46c0-844c-f7eebc0cbdcf" containerID="0acead728e9f4a6068a06c6ab55a84934c877d74ac67897c4af0a6d9ebfa362f" exitCode=0 Oct 30 17:01:41.843966 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:41.843919 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-637f442f-h8ckw" event={"ID":"6c753be2-3052-46c0-844c-f7eebc0cbdcf","Type":"ContainerDied","Data":"0acead728e9f4a6068a06c6ab55a84934c877d74ac67897c4af0a6d9ebfa362f"} Oct 30 17:01:42.964338 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:01:42.964272 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:01:43.027259 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:43.027232 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-637f442f-h8ckw" Oct 30 17:01:43.054195 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:43.054165 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c753be2-3052-46c0-844c-f7eebc0cbdcf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c753be2-3052-46c0-844c-f7eebc0cbdcf-maas-api-xks-ca\") pod \"6c753be2-3052-46c0-844c-f7eebc0cbdcf\" (UID: \"6c753be2-3052-46c0-844c-f7eebc0cbdcf\") " Oct 30 17:01:43.054418 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:43.054225 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c753be2-3052-46c0-844c-f7eebc0cbdcf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c753be2-3052-46c0-844c-f7eebc0cbdcf-maas-api-openshift-ca\") pod \"6c753be2-3052-46c0-844c-f7eebc0cbdcf\" (UID: \"6c753be2-3052-46c0-844c-f7eebc0cbdcf\") " Oct 30 17:01:43.054418 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:43.054263 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c753be2-3052-46c0-844c-f7eebc0cbdcf-kube-api-access-lpzk8\" (UniqueName: \"kubernetes.io/projected/6c753be2-3052-46c0-844c-f7eebc0cbdcf-kube-api-access-lpzk8\") pod \"6c753be2-3052-46c0-844c-f7eebc0cbdcf\" (UID: \"6c753be2-3052-46c0-844c-f7eebc0cbdcf\") " Oct 30 17:01:43.054676 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:43.054628 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c753be2-3052-46c0-844c-f7eebc0cbdcf-maas-api-openshift-ca" pod "6c753be2-3052-46c0-844c-f7eebc0cbdcf" (UID: "6c753be2-3052-46c0-844c-f7eebc0cbdcf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:01:43.054870 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:43.054850 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c753be2-3052-46c0-844c-f7eebc0cbdcf-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:01:43.056381 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:43.056352 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c753be2-3052-46c0-844c-f7eebc0cbdcf-maas-api-xks-ca" pod "6c753be2-3052-46c0-844c-f7eebc0cbdcf" (UID: "6c753be2-3052-46c0-844c-f7eebc0cbdcf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:01:43.056501 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:43.056392 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c753be2-3052-46c0-844c-f7eebc0cbdcf-kube-api-access-lpzk8" pod "6c753be2-3052-46c0-844c-f7eebc0cbdcf" (UID: "6c753be2-3052-46c0-844c-f7eebc0cbdcf"). InnerVolumeSpecName "kube-api-access-lpzk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:01:43.156524 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:43.156393 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c753be2-3052-46c0-844c-f7eebc0cbdcf-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:01:43.156524 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:43.156439 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpzk8\" (UniqueName: \"kubernetes.io/projected/6c753be2-3052-46c0-844c-f7eebc0cbdcf-kube-api-access-lpzk8\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:01:43.859920 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:43.859888 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-637f442f-h8ckw" Oct 30 17:01:43.860121 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:43.859925 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-637f442f-h8ckw" event={"ID":"6c753be2-3052-46c0-844c-f7eebc0cbdcf","Type":"ContainerDied","Data":"67889fa5cd8e813161c877d81c4341a06a4aa98dea141a86014a52ec97ba0acd"} Oct 30 17:01:43.860121 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:43.859971 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67889fa5cd8e813161c877d81c4341a06a4aa98dea141a86014a52ec97ba0acd" Oct 30 17:01:48.863319 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:48.863271 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7mdgd"] Oct 30 17:01:48.863859 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:48.863844 4522 cpu_manager.go:414] "containerMap: removing container" podUID="6c753be2-3052-46c0-844c-f7eebc0cbdcf" containerName="revoke-keys" Oct 30 17:01:48.863902 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:48.863860 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c753be2-3052-46c0-844c-f7eebc0cbdcf" containerName="revoke-keys" Oct 30 17:01:48.864008 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:48.863998 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c753be2-3052-46c0-844c-f7eebc0cbdcf" containerName="revoke-keys" Oct 30 17:01:48.867657 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:48.867636 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7mdgd" Oct 30 17:01:48.903906 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:48.903871 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7mdgd"] Oct 30 17:01:48.927483 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:48.927447 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c0131b-dd1d-426b-ae9d-124a6e45e010-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7mdgd\" (UID: \"86c0131b-dd1d-426b-ae9d-124a6e45e010\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7mdgd" Oct 30 17:01:48.927483 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:48.927487 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c0131b-dd1d-426b-ae9d-124a6e45e010-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7mdgd\" (UID: \"86c0131b-dd1d-426b-ae9d-124a6e45e010\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7mdgd" Oct 30 17:01:48.927705 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:48.927611 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86c0131b-dd1d-426b-ae9d-124a6e45e010-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7mdgd\" (UID: \"86c0131b-dd1d-426b-ae9d-124a6e45e010\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7mdgd" Oct 30 17:01:48.927705 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:48.927633 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86c0131b-dd1d-426b-ae9d-124a6e45e010-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7mdgd\" (UID: \"86c0131b-dd1d-426b-ae9d-124a6e45e010\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7mdgd" Oct 30 17:01:48.927778 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:48.927714 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86c0131b-dd1d-426b-ae9d-124a6e45e010-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7mdgd\" (UID: \"86c0131b-dd1d-426b-ae9d-124a6e45e010\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7mdgd" Oct 30 17:01:48.927778 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:48.927742 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86c0131b-dd1d-426b-ae9d-124a6e45e010-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7mdgd\" (UID: \"86c0131b-dd1d-426b-ae9d-124a6e45e010\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7mdgd" Oct 30 17:01:48.927889 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:48.927861 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86c0131b-dd1d-426b-ae9d-124a6e45e010-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7mdgd\" (UID: \"86c0131b-dd1d-426b-ae9d-124a6e45e010\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7mdgd" Oct 30 17:01:48.927927 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:48.927911 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86c0131b-dd1d-426b-ae9d-124a6e45e010-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7mdgd\" (UID: \"86c0131b-dd1d-426b-ae9d-124a6e45e010\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7mdgd" Oct 30 17:01:48.927984 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:48.927966 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm7wz\" (UniqueName: \"kubernetes.io/projected/86c0131b-dd1d-426b-ae9d-124a6e45e010-kube-api-access-hm7wz\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7mdgd\" (UID: \"86c0131b-dd1d-426b-ae9d-124a6e45e010\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7mdgd" Oct 30 17:01:49.029120 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:49.029087 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86c0131b-dd1d-426b-ae9d-124a6e45e010-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7mdgd\" (UID: \"86c0131b-dd1d-426b-ae9d-124a6e45e010\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7mdgd" Oct 30 17:01:49.029362 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:49.029131 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86c0131b-dd1d-426b-ae9d-124a6e45e010-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7mdgd\" (UID: \"86c0131b-dd1d-426b-ae9d-124a6e45e010\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7mdgd" Oct 30 17:01:49.029362 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:49.029165 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86c0131b-dd1d-426b-ae9d-124a6e45e010-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7mdgd\" (UID: \"86c0131b-dd1d-426b-ae9d-124a6e45e010\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7mdgd" Oct 30 17:01:49.029362 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:49.029277 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86c0131b-dd1d-426b-ae9d-124a6e45e010-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7mdgd\" (UID: \"86c0131b-dd1d-426b-ae9d-124a6e45e010\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7mdgd" Oct 30 17:01:49.029362 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:49.029363 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm7wz\" (UniqueName: \"kubernetes.io/projected/86c0131b-dd1d-426b-ae9d-124a6e45e010-kube-api-access-hm7wz\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7mdgd\" (UID: \"86c0131b-dd1d-426b-ae9d-124a6e45e010\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7mdgd" Oct 30 17:01:49.029588 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:49.029408 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c0131b-dd1d-426b-ae9d-124a6e45e010-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7mdgd\" (UID: \"86c0131b-dd1d-426b-ae9d-124a6e45e010\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7mdgd" Oct 30 17:01:49.029588 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:49.029427 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c0131b-dd1d-426b-ae9d-124a6e45e010-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7mdgd\" (UID: \"86c0131b-dd1d-426b-ae9d-124a6e45e010\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7mdgd" Oct 30 17:01:49.029588 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:49.029515 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86c0131b-dd1d-426b-ae9d-124a6e45e010-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7mdgd\" (UID: \"86c0131b-dd1d-426b-ae9d-124a6e45e010\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7mdgd" Oct 30 17:01:49.029588 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:49.029539 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86c0131b-dd1d-426b-ae9d-124a6e45e010-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7mdgd\" (UID: \"86c0131b-dd1d-426b-ae9d-124a6e45e010\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7mdgd" Oct 30 17:01:49.029793 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:49.029694 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86c0131b-dd1d-426b-ae9d-124a6e45e010-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7mdgd\" (UID: \"86c0131b-dd1d-426b-ae9d-124a6e45e010\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7mdgd" Oct 30 17:01:49.029793 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:49.029724 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c0131b-dd1d-426b-ae9d-124a6e45e010-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7mdgd\" (UID: \"86c0131b-dd1d-426b-ae9d-124a6e45e010\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7mdgd" Oct 30 17:01:49.029902 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:49.029811 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c0131b-dd1d-426b-ae9d-124a6e45e010-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7mdgd\" (UID: \"86c0131b-dd1d-426b-ae9d-124a6e45e010\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7mdgd" Oct 30 17:01:49.029902 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:49.029877 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86c0131b-dd1d-426b-ae9d-124a6e45e010-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7mdgd\" (UID: \"86c0131b-dd1d-426b-ae9d-124a6e45e010\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7mdgd" Oct 30 17:01:49.030209 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:49.030186 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86c0131b-dd1d-426b-ae9d-124a6e45e010-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7mdgd\" (UID: \"86c0131b-dd1d-426b-ae9d-124a6e45e010\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7mdgd" Oct 30 17:01:49.031571 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:49.031549 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86c0131b-dd1d-426b-ae9d-124a6e45e010-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7mdgd\" (UID: \"86c0131b-dd1d-426b-ae9d-124a6e45e010\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7mdgd" Oct 30 17:01:49.032097 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:49.032081 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86c0131b-dd1d-426b-ae9d-124a6e45e010-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7mdgd\" (UID: \"86c0131b-dd1d-426b-ae9d-124a6e45e010\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7mdgd" Oct 30 17:01:49.054004 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:49.053977 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86c0131b-dd1d-426b-ae9d-124a6e45e010-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7mdgd\" (UID: \"86c0131b-dd1d-426b-ae9d-124a6e45e010\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7mdgd" Oct 30 17:01:49.054119 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:49.054049 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm7wz\" (UniqueName: \"kubernetes.io/projected/86c0131b-dd1d-426b-ae9d-124a6e45e010-kube-api-access-hm7wz\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7mdgd\" (UID: \"86c0131b-dd1d-426b-ae9d-124a6e45e010\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7mdgd" Oct 30 17:01:49.183678 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:49.183604 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7mdgd" Oct 30 17:01:49.512870 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:01:49.512841 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86c0131b_dd1d_426b_ae9d_124a6e45e010.slice/crio-2b3d6725d5c6303ea289868259a41300b5c4fd41e0d60776b9fc638883bacd7a WatchSource:0}: Error finding container 2b3d6725d5c6303ea289868259a41300b5c4fd41e0d60776b9fc638883bacd7a: Status 404 returned error can't find the container with id 2b3d6725d5c6303ea289868259a41300b5c4fd41e0d60776b9fc638883bacd7a Oct 30 17:01:49.514027 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:49.514004 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7mdgd"] Oct 30 17:01:49.515246 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:49.515220 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:01:49.515293 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:49.515277 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:01:49.515354 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:49.515319 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:01:49.900167 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:49.900078 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7mdgd" event={"ID":"86c0131b-dd1d-426b-ae9d-124a6e45e010","Type":"ContainerStarted","Data":"7266d81c21073083f051b3596b54023d612e854ca4a6055bee056fc9cfe6bed1"} Oct 30 17:01:49.900167 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:49.900160 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7mdgd" event={"ID":"86c0131b-dd1d-426b-ae9d-124a6e45e010","Type":"ContainerStarted","Data":"2b3d6725d5c6303ea289868259a41300b5c4fd41e0d60776b9fc638883bacd7a"} Oct 30 17:01:49.950796 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:49.950744 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7mdgd" podStartSLOduration=1.950731148 podStartE2EDuration="1.950731148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:01:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:01:49.949136678 +0000 UTC m=+883.949891579" watchObservedRunningTime="2026-10-30 17:01:49.950731148 +0000 UTC m=+883.951486117" Oct 30 17:01:50.184160 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:50.184075 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7mdgd" Oct 30 17:01:50.185723 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:50.185698 4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-7mdgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 17:01:50.185845 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:50.185743 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7mdgd" podUID="86c0131b-dd1d-426b-ae9d-124a6e45e010" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 17:01:51.184194 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:51.184162 4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-7mdgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 17:01:51.184622 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:51.184221 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7mdgd" podUID="86c0131b-dd1d-426b-ae9d-124a6e45e010" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 17:01:51.829947 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:51.829914 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl"] Oct 30 17:01:51.834116 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:51.834092 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" Oct 30 17:01:51.855537 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:51.855512 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl"] Oct 30 17:01:51.864912 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:51.864884 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-096d75\"/\"distinct-w0-096d75-kserve-self-signed-certs\"" Oct 30 17:01:51.968670 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:51.968638 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f1e3bbb-ab1c-455d-a05d-d62f72c5d009-model-cache\") pod \"distinct-w0-096d75-kserve-567644df56-8fnkl\" (UID: \"2f1e3bbb-ab1c-455d-a05d-d62f72c5d009\") " pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" Oct 30 17:01:51.968852 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:51.968798 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f1e3bbb-ab1c-455d-a05d-d62f72c5d009-dshm\") pod \"distinct-w0-096d75-kserve-567644df56-8fnkl\" (UID: \"2f1e3bbb-ab1c-455d-a05d-d62f72c5d009\") " pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" Oct 30 17:01:51.969031 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:51.969000 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f1e3bbb-ab1c-455d-a05d-d62f72c5d009-home\") pod \"distinct-w0-096d75-kserve-567644df56-8fnkl\" (UID: \"2f1e3bbb-ab1c-455d-a05d-d62f72c5d009\") " pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" Oct 30 17:01:51.969110 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:51.969086 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f1e3bbb-ab1c-455d-a05d-d62f72c5d009-tls-certs\") pod \"distinct-w0-096d75-kserve-567644df56-8fnkl\" (UID: \"2f1e3bbb-ab1c-455d-a05d-d62f72c5d009\") " pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" Oct 30 17:01:51.969175 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:51.969158 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p47sj\" (UniqueName: \"kubernetes.io/projected/2f1e3bbb-ab1c-455d-a05d-d62f72c5d009-kube-api-access-p47sj\") pod \"distinct-w0-096d75-kserve-567644df56-8fnkl\" (UID: \"2f1e3bbb-ab1c-455d-a05d-d62f72c5d009\") " pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" Oct 30 17:01:51.969293 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:51.969273 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f1e3bbb-ab1c-455d-a05d-d62f72c5d009-tmp-dir\") pod \"distinct-w0-096d75-kserve-567644df56-8fnkl\" (UID: \"2f1e3bbb-ab1c-455d-a05d-d62f72c5d009\") " pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" Oct 30 17:01:52.070214 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.070169 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f1e3bbb-ab1c-455d-a05d-d62f72c5d009-home\") pod \"distinct-w0-096d75-kserve-567644df56-8fnkl\" (UID: \"2f1e3bbb-ab1c-455d-a05d-d62f72c5d009\") " pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" Oct 30 17:01:52.071999 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.071763 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f1e3bbb-ab1c-455d-a05d-d62f72c5d009-home\") pod \"distinct-w0-096d75-kserve-567644df56-8fnkl\" (UID: \"2f1e3bbb-ab1c-455d-a05d-d62f72c5d009\") " pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" Oct 30 17:01:52.072324 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.072285 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f1e3bbb-ab1c-455d-a05d-d62f72c5d009-tls-certs\") pod \"distinct-w0-096d75-kserve-567644df56-8fnkl\" (UID: \"2f1e3bbb-ab1c-455d-a05d-d62f72c5d009\") " pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" Oct 30 17:01:52.072524 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.072506 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p47sj\" (UniqueName: \"kubernetes.io/projected/2f1e3bbb-ab1c-455d-a05d-d62f72c5d009-kube-api-access-p47sj\") pod \"distinct-w0-096d75-kserve-567644df56-8fnkl\" (UID: \"2f1e3bbb-ab1c-455d-a05d-d62f72c5d009\") " pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" Oct 30 17:01:52.072744 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.072730 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f1e3bbb-ab1c-455d-a05d-d62f72c5d009-tmp-dir\") pod \"distinct-w0-096d75-kserve-567644df56-8fnkl\" (UID: \"2f1e3bbb-ab1c-455d-a05d-d62f72c5d009\") " pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" Oct 30 17:01:52.073226 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.073208 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f1e3bbb-ab1c-455d-a05d-d62f72c5d009-model-cache\") pod \"distinct-w0-096d75-kserve-567644df56-8fnkl\" (UID: \"2f1e3bbb-ab1c-455d-a05d-d62f72c5d009\") " pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" Oct 30 17:01:52.073715 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.073699 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f1e3bbb-ab1c-455d-a05d-d62f72c5d009-dshm\") pod \"distinct-w0-096d75-kserve-567644df56-8fnkl\" (UID: \"2f1e3bbb-ab1c-455d-a05d-d62f72c5d009\") " pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" Oct 30 17:01:52.074215 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.073110 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f1e3bbb-ab1c-455d-a05d-d62f72c5d009-tmp-dir\") pod \"distinct-w0-096d75-kserve-567644df56-8fnkl\" (UID: \"2f1e3bbb-ab1c-455d-a05d-d62f72c5d009\") " pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" Oct 30 17:01:52.074419 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.073593 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f1e3bbb-ab1c-455d-a05d-d62f72c5d009-model-cache\") pod \"distinct-w0-096d75-kserve-567644df56-8fnkl\" (UID: \"2f1e3bbb-ab1c-455d-a05d-d62f72c5d009\") " pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" Oct 30 17:01:52.078579 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.078557 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f1e3bbb-ab1c-455d-a05d-d62f72c5d009-tls-certs\") pod \"distinct-w0-096d75-kserve-567644df56-8fnkl\" (UID: \"2f1e3bbb-ab1c-455d-a05d-d62f72c5d009\") " pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" Oct 30 17:01:52.084517 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.084343 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f1e3bbb-ab1c-455d-a05d-d62f72c5d009-dshm\") pod \"distinct-w0-096d75-kserve-567644df56-8fnkl\" (UID: \"2f1e3bbb-ab1c-455d-a05d-d62f72c5d009\") " pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" Oct 30 17:01:52.131658 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.130179 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p47sj\" (UniqueName: \"kubernetes.io/projected/2f1e3bbb-ab1c-455d-a05d-d62f72c5d009-kube-api-access-p47sj\") pod \"distinct-w0-096d75-kserve-567644df56-8fnkl\" (UID: \"2f1e3bbb-ab1c-455d-a05d-d62f72c5d009\") " pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" Oct 30 17:01:52.147725 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.147690 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" Oct 30 17:01:52.184737 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.184699 4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-7mdgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 17:01:52.185169 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.184751 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7mdgd" podUID="86c0131b-dd1d-426b-ae9d-124a6e45e010" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 17:01:52.306075 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.291297 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp"] Oct 30 17:01:52.322974 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.322727 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp" Oct 30 17:01:52.324494 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.324429 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp"] Oct 30 17:01:52.327920 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.327898 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn"] Oct 30 17:01:52.332197 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.332178 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn" Oct 30 17:01:52.370406 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.370040 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn"] Oct 30 17:01:52.376576 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.376547 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/015c4531-2766-4522-acf0-0427f1ba0a3d-istio-token\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn\" (UID: \"015c4531-2766-4522-acf0-0427f1ba0a3d\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn" Oct 30 17:01:52.376735 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.376596 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/015c4531-2766-4522-acf0-0427f1ba0a3d-credential-socket\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn\" (UID: \"015c4531-2766-4522-acf0-0427f1ba0a3d\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn" Oct 30 17:01:52.376735 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.376613 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2-credential-socket\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp\" (UID: \"9eb030f0-e0a2-446e-b35f-4c497d9a3aa2\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp" Oct 30 17:01:52.376735 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.376667 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2-istio-token\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp\" (UID: \"9eb030f0-e0a2-446e-b35f-4c497d9a3aa2\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp" Oct 30 17:01:52.376909 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.376804 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2-workload-socket\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp\" (UID: \"9eb030f0-e0a2-446e-b35f-4c497d9a3aa2\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp" Oct 30 17:01:52.376909 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.376871 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/015c4531-2766-4522-acf0-0427f1ba0a3d-istio-podinfo\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn\" (UID: \"015c4531-2766-4522-acf0-0427f1ba0a3d\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn" Oct 30 17:01:52.377025 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.376914 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/015c4531-2766-4522-acf0-0427f1ba0a3d-istio-envoy\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn\" (UID: \"015c4531-2766-4522-acf0-0427f1ba0a3d\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn" Oct 30 17:01:52.377025 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.377013 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2-istio-data\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp\" (UID: \"9eb030f0-e0a2-446e-b35f-4c497d9a3aa2\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp" Oct 30 17:01:52.377126 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.377069 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs4wp\" (UniqueName: \"kubernetes.io/projected/015c4531-2766-4522-acf0-0427f1ba0a3d-kube-api-access-zs4wp\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn\" (UID: \"015c4531-2766-4522-acf0-0427f1ba0a3d\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn" Oct 30 17:01:52.377126 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.377094 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2-workload-certs\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp\" (UID: \"9eb030f0-e0a2-446e-b35f-4c497d9a3aa2\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp" Oct 30 17:01:52.377203 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.377176 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2-istio-podinfo\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp\" (UID: \"9eb030f0-e0a2-446e-b35f-4c497d9a3aa2\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp" Oct 30 17:01:52.377247 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.377220 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fdz6\" (UniqueName: \"kubernetes.io/projected/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2-kube-api-access-4fdz6\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp\" (UID: \"9eb030f0-e0a2-446e-b35f-4c497d9a3aa2\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp" Oct 30 17:01:52.377328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.377296 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/015c4531-2766-4522-acf0-0427f1ba0a3d-workload-certs\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn\" (UID: \"015c4531-2766-4522-acf0-0427f1ba0a3d\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn" Oct 30 17:01:52.377364 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.377354 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/015c4531-2766-4522-acf0-0427f1ba0a3d-workload-socket\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn\" (UID: \"015c4531-2766-4522-acf0-0427f1ba0a3d\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn" Oct 30 17:01:52.377410 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.377380 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2-istiod-ca-cert\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp\" (UID: \"9eb030f0-e0a2-446e-b35f-4c497d9a3aa2\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp" Oct 30 17:01:52.377500 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.377484 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/015c4531-2766-4522-acf0-0427f1ba0a3d-istiod-ca-cert\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn\" (UID: \"015c4531-2766-4522-acf0-0427f1ba0a3d\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn" Oct 30 17:01:52.377559 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.377514 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/015c4531-2766-4522-acf0-0427f1ba0a3d-istio-data\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn\" (UID: \"015c4531-2766-4522-acf0-0427f1ba0a3d\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn" Oct 30 17:01:52.377627 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.377611 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2-istio-envoy\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp\" (UID: \"9eb030f0-e0a2-446e-b35f-4c497d9a3aa2\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp" Oct 30 17:01:52.476112 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:01:52.476087 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f1e3bbb_ab1c_455d_a05d_d62f72c5d009.slice/crio-eceb20fe407065110de59829235140d5b4eb9dca5d825c7c97f18254f2824c03 WatchSource:0}: Error finding container eceb20fe407065110de59829235140d5b4eb9dca5d825c7c97f18254f2824c03: Status 404 returned error can't find the container with id eceb20fe407065110de59829235140d5b4eb9dca5d825c7c97f18254f2824c03 Oct 30 17:01:52.477874 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.477847 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl"] Oct 30 17:01:52.478416 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.478392 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2-istio-envoy\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp\" (UID: \"9eb030f0-e0a2-446e-b35f-4c497d9a3aa2\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp" Oct 30 17:01:52.478517 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.478427 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/015c4531-2766-4522-acf0-0427f1ba0a3d-istio-token\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn\" (UID: \"015c4531-2766-4522-acf0-0427f1ba0a3d\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn" Oct 30 17:01:52.478517 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.478471 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/015c4531-2766-4522-acf0-0427f1ba0a3d-credential-socket\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn\" (UID: \"015c4531-2766-4522-acf0-0427f1ba0a3d\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn" Oct 30 17:01:52.478517 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.478495 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2-credential-socket\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp\" (UID: \"9eb030f0-e0a2-446e-b35f-4c497d9a3aa2\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp" Oct 30 17:01:52.478709 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.478522 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2-istio-token\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp\" (UID: \"9eb030f0-e0a2-446e-b35f-4c497d9a3aa2\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp" Oct 30 17:01:52.478709 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.478560 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2-workload-socket\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp\" (UID: \"9eb030f0-e0a2-446e-b35f-4c497d9a3aa2\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp" Oct 30 17:01:52.478709 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.478617 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/015c4531-2766-4522-acf0-0427f1ba0a3d-istio-podinfo\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn\" (UID: \"015c4531-2766-4522-acf0-0427f1ba0a3d\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn" Oct 30 17:01:52.478709 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.478661 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/015c4531-2766-4522-acf0-0427f1ba0a3d-istio-envoy\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn\" (UID: \"015c4531-2766-4522-acf0-0427f1ba0a3d\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn" Oct 30 17:01:52.479003 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.478722 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2-istio-data\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp\" (UID: \"9eb030f0-e0a2-446e-b35f-4c497d9a3aa2\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp" Oct 30 17:01:52.479003 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.478779 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs4wp\" (UniqueName: \"kubernetes.io/projected/015c4531-2766-4522-acf0-0427f1ba0a3d-kube-api-access-zs4wp\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn\" (UID: \"015c4531-2766-4522-acf0-0427f1ba0a3d\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn" Oct 30 17:01:52.479003 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.478801 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2-workload-certs\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp\" (UID: \"9eb030f0-e0a2-446e-b35f-4c497d9a3aa2\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp" Oct 30 17:01:52.479003 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.478830 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2-istio-podinfo\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp\" (UID: \"9eb030f0-e0a2-446e-b35f-4c497d9a3aa2\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp" Oct 30 17:01:52.479003 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.478849 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fdz6\" (UniqueName: \"kubernetes.io/projected/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2-kube-api-access-4fdz6\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp\" (UID: \"9eb030f0-e0a2-446e-b35f-4c497d9a3aa2\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp" Oct 30 17:01:52.479003 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.478873 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/015c4531-2766-4522-acf0-0427f1ba0a3d-workload-certs\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn\" (UID: \"015c4531-2766-4522-acf0-0427f1ba0a3d\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn" Oct 30 17:01:52.479003 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.478896 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/015c4531-2766-4522-acf0-0427f1ba0a3d-workload-socket\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn\" (UID: \"015c4531-2766-4522-acf0-0427f1ba0a3d\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn" Oct 30 17:01:52.479003 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.478910 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2-istiod-ca-cert\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp\" (UID: \"9eb030f0-e0a2-446e-b35f-4c497d9a3aa2\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp" Oct 30 17:01:52.479003 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.478911 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2-credential-socket\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp\" (UID: \"9eb030f0-e0a2-446e-b35f-4c497d9a3aa2\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp" Oct 30 17:01:52.479003 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.478965 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/015c4531-2766-4522-acf0-0427f1ba0a3d-istiod-ca-cert\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn\" (UID: \"015c4531-2766-4522-acf0-0427f1ba0a3d\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn" Oct 30 17:01:52.479003 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.478976 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2-workload-socket\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp\" (UID: \"9eb030f0-e0a2-446e-b35f-4c497d9a3aa2\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp" Oct 30 17:01:52.479491 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.479196 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/015c4531-2766-4522-acf0-0427f1ba0a3d-credential-socket\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn\" (UID: \"015c4531-2766-4522-acf0-0427f1ba0a3d\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn" Oct 30 17:01:52.479491 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.479264 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/015c4531-2766-4522-acf0-0427f1ba0a3d-istio-data\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn\" (UID: \"015c4531-2766-4522-acf0-0427f1ba0a3d\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn" Oct 30 17:01:52.479491 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.478983 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/015c4531-2766-4522-acf0-0427f1ba0a3d-istio-data\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn\" (UID: \"015c4531-2766-4522-acf0-0427f1ba0a3d\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn" Oct 30 17:01:52.479787 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.479765 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2-workload-certs\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp\" (UID: \"9eb030f0-e0a2-446e-b35f-4c497d9a3aa2\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp" Oct 30 17:01:52.479968 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.479947 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2-istio-data\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp\" (UID: \"9eb030f0-e0a2-446e-b35f-4c497d9a3aa2\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp" Oct 30 17:01:52.480226 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.480202 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/015c4531-2766-4522-acf0-0427f1ba0a3d-workload-socket\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn\" (UID: \"015c4531-2766-4522-acf0-0427f1ba0a3d\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn" Oct 30 17:01:52.480514 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.480491 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/015c4531-2766-4522-acf0-0427f1ba0a3d-workload-certs\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn\" (UID: \"015c4531-2766-4522-acf0-0427f1ba0a3d\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn" Oct 30 17:01:52.482469 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.481126 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2-istiod-ca-cert\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp\" (UID: \"9eb030f0-e0a2-446e-b35f-4c497d9a3aa2\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp" Oct 30 17:01:52.482469 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.481346 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/015c4531-2766-4522-acf0-0427f1ba0a3d-istiod-ca-cert\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn\" (UID: \"015c4531-2766-4522-acf0-0427f1ba0a3d\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn" Oct 30 17:01:52.482469 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.481548 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2-istio-envoy\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp\" (UID: \"9eb030f0-e0a2-446e-b35f-4c497d9a3aa2\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp" Oct 30 17:01:52.482469 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.481552 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/015c4531-2766-4522-acf0-0427f1ba0a3d-istio-podinfo\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn\" (UID: \"015c4531-2766-4522-acf0-0427f1ba0a3d\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn" Oct 30 17:01:52.482469 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.481552 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/015c4531-2766-4522-acf0-0427f1ba0a3d-istio-envoy\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn\" (UID: \"015c4531-2766-4522-acf0-0427f1ba0a3d\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn" Oct 30 17:01:52.482469 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.482061 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2-istio-podinfo\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp\" (UID: \"9eb030f0-e0a2-446e-b35f-4c497d9a3aa2\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp" Oct 30 17:01:52.519576 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.519548 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fdz6\" (UniqueName: \"kubernetes.io/projected/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2-kube-api-access-4fdz6\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp\" (UID: \"9eb030f0-e0a2-446e-b35f-4c497d9a3aa2\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp" Oct 30 17:01:52.519760 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.519730 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2-istio-token\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp\" (UID: \"9eb030f0-e0a2-446e-b35f-4c497d9a3aa2\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp" Oct 30 17:01:52.519853 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.519667 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs4wp\" (UniqueName: \"kubernetes.io/projected/015c4531-2766-4522-acf0-0427f1ba0a3d-kube-api-access-zs4wp\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn\" (UID: \"015c4531-2766-4522-acf0-0427f1ba0a3d\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn" Oct 30 17:01:52.520654 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.520619 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/015c4531-2766-4522-acf0-0427f1ba0a3d-istio-token\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn\" (UID: \"015c4531-2766-4522-acf0-0427f1ba0a3d\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn" Oct 30 17:01:52.646196 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.646105 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp" Oct 30 17:01:52.667376 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.667344 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn" Oct 30 17:01:52.688045 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.688012 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx"] Oct 30 17:01:52.717061 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.700599 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx" Oct 30 17:01:52.736565 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.736534 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx"] Oct 30 17:01:52.786004 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.783223 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8208e132-462b-4b94-9246-e0ecc27a13a9-workload-certs\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx\" (UID: \"8208e132-462b-4b94-9246-e0ecc27a13a9\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx" Oct 30 17:01:52.786004 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.783352 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74jbd\" (UniqueName: \"kubernetes.io/projected/8208e132-462b-4b94-9246-e0ecc27a13a9-kube-api-access-74jbd\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx\" (UID: \"8208e132-462b-4b94-9246-e0ecc27a13a9\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx" Oct 30 17:01:52.786004 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.783446 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8208e132-462b-4b94-9246-e0ecc27a13a9-istio-token\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx\" (UID: \"8208e132-462b-4b94-9246-e0ecc27a13a9\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx" Oct 30 17:01:52.786004 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.783513 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8208e132-462b-4b94-9246-e0ecc27a13a9-credential-socket\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx\" (UID: \"8208e132-462b-4b94-9246-e0ecc27a13a9\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx" Oct 30 17:01:52.786004 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.783561 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8208e132-462b-4b94-9246-e0ecc27a13a9-istio-data\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx\" (UID: \"8208e132-462b-4b94-9246-e0ecc27a13a9\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx" Oct 30 17:01:52.786004 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.783604 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8208e132-462b-4b94-9246-e0ecc27a13a9-workload-socket\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx\" (UID: \"8208e132-462b-4b94-9246-e0ecc27a13a9\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx" Oct 30 17:01:52.786004 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.783626 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8208e132-462b-4b94-9246-e0ecc27a13a9-istio-envoy\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx\" (UID: \"8208e132-462b-4b94-9246-e0ecc27a13a9\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx" Oct 30 17:01:52.786004 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.783688 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8208e132-462b-4b94-9246-e0ecc27a13a9-istio-podinfo\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx\" (UID: \"8208e132-462b-4b94-9246-e0ecc27a13a9\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx" Oct 30 17:01:52.786004 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.783726 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8208e132-462b-4b94-9246-e0ecc27a13a9-istiod-ca-cert\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx\" (UID: \"8208e132-462b-4b94-9246-e0ecc27a13a9\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx" Oct 30 17:01:52.884987 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.884944 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8208e132-462b-4b94-9246-e0ecc27a13a9-credential-socket\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx\" (UID: \"8208e132-462b-4b94-9246-e0ecc27a13a9\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx" Oct 30 17:01:52.885145 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.885001 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8208e132-462b-4b94-9246-e0ecc27a13a9-istio-data\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx\" (UID: \"8208e132-462b-4b94-9246-e0ecc27a13a9\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx" Oct 30 17:01:52.885145 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.885028 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8208e132-462b-4b94-9246-e0ecc27a13a9-workload-socket\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx\" (UID: \"8208e132-462b-4b94-9246-e0ecc27a13a9\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx" Oct 30 17:01:52.885145 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.885062 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8208e132-462b-4b94-9246-e0ecc27a13a9-istio-envoy\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx\" (UID: \"8208e132-462b-4b94-9246-e0ecc27a13a9\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx" Oct 30 17:01:52.885145 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.885122 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8208e132-462b-4b94-9246-e0ecc27a13a9-istio-podinfo\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx\" (UID: \"8208e132-462b-4b94-9246-e0ecc27a13a9\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx" Oct 30 17:01:52.885399 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.885156 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8208e132-462b-4b94-9246-e0ecc27a13a9-istiod-ca-cert\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx\" (UID: \"8208e132-462b-4b94-9246-e0ecc27a13a9\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx" Oct 30 17:01:52.885399 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.885223 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8208e132-462b-4b94-9246-e0ecc27a13a9-workload-certs\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx\" (UID: \"8208e132-462b-4b94-9246-e0ecc27a13a9\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx" Oct 30 17:01:52.885399 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.885299 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74jbd\" (UniqueName: \"kubernetes.io/projected/8208e132-462b-4b94-9246-e0ecc27a13a9-kube-api-access-74jbd\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx\" (UID: \"8208e132-462b-4b94-9246-e0ecc27a13a9\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx" Oct 30 17:01:52.885619 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.885401 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8208e132-462b-4b94-9246-e0ecc27a13a9-istio-token\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx\" (UID: \"8208e132-462b-4b94-9246-e0ecc27a13a9\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx" Oct 30 17:01:52.885619 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.885436 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8208e132-462b-4b94-9246-e0ecc27a13a9-credential-socket\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx\" (UID: \"8208e132-462b-4b94-9246-e0ecc27a13a9\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx" Oct 30 17:01:52.885619 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.885500 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8208e132-462b-4b94-9246-e0ecc27a13a9-istio-data\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx\" (UID: \"8208e132-462b-4b94-9246-e0ecc27a13a9\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx" Oct 30 17:01:52.885619 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.885519 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8208e132-462b-4b94-9246-e0ecc27a13a9-workload-socket\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx\" (UID: \"8208e132-462b-4b94-9246-e0ecc27a13a9\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx" Oct 30 17:01:52.885834 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.885803 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8208e132-462b-4b94-9246-e0ecc27a13a9-workload-certs\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx\" (UID: \"8208e132-462b-4b94-9246-e0ecc27a13a9\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx" Oct 30 17:01:52.886131 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.886101 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8208e132-462b-4b94-9246-e0ecc27a13a9-istiod-ca-cert\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx\" (UID: \"8208e132-462b-4b94-9246-e0ecc27a13a9\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx" Oct 30 17:01:52.888924 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.888900 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8208e132-462b-4b94-9246-e0ecc27a13a9-istio-envoy\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx\" (UID: \"8208e132-462b-4b94-9246-e0ecc27a13a9\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx" Oct 30 17:01:52.889285 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.889263 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8208e132-462b-4b94-9246-e0ecc27a13a9-istio-podinfo\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx\" (UID: \"8208e132-462b-4b94-9246-e0ecc27a13a9\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx" Oct 30 17:01:52.919091 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.919017 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8208e132-462b-4b94-9246-e0ecc27a13a9-istio-token\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx\" (UID: \"8208e132-462b-4b94-9246-e0ecc27a13a9\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx" Oct 30 17:01:52.919221 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.919157 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74jbd\" (UniqueName: \"kubernetes.io/projected/8208e132-462b-4b94-9246-e0ecc27a13a9-kube-api-access-74jbd\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx\" (UID: \"8208e132-462b-4b94-9246-e0ecc27a13a9\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx" Oct 30 17:01:52.936513 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.936477 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" event={"ID":"2f1e3bbb-ab1c-455d-a05d-d62f72c5d009","Type":"ContainerStarted","Data":"56fda00ed5e16c409cc69d65fb9063109ae58c0bd6c429b41977240d42a559c8"} Oct 30 17:01:52.936668 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.936525 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" event={"ID":"2f1e3bbb-ab1c-455d-a05d-d62f72c5d009","Type":"ContainerStarted","Data":"eceb20fe407065110de59829235140d5b4eb9dca5d825c7c97f18254f2824c03"} Oct 30 17:01:52.983586 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.983538 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" podStartSLOduration=1.983523849 podStartE2EDuration="1.983523849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:01:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:01:52.98165975 +0000 UTC m=+886.982414651" watchObservedRunningTime="2026-10-30 17:01:52.983523849 +0000 UTC m=+886.984278750" Oct 30 17:01:52.988497 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:01:52.988467 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eb030f0_e0a2_446e_b35f_4c497d9a3aa2.slice/crio-c50002850adb34f86d769e15c0dd24a169908ad128be77f2b2ad7fa6708cbc62 WatchSource:0}: Error finding container c50002850adb34f86d769e15c0dd24a169908ad128be77f2b2ad7fa6708cbc62: Status 404 returned error can't find the container with id c50002850adb34f86d769e15c0dd24a169908ad128be77f2b2ad7fa6708cbc62 Oct 30 17:01:52.990210 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.990185 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp"] Oct 30 17:01:52.990906 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.990877 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:01:52.990962 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.990936 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:01:52.990997 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:52.990964 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:01:53.036629 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:01:53.036596 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod015c4531_2766_4522_acf0_0427f1ba0a3d.slice/crio-860191f8246c8d4324f1b4adcbb2ca96b58444dd8335b21940ba49e89fa09f9b WatchSource:0}: Error finding container 860191f8246c8d4324f1b4adcbb2ca96b58444dd8335b21940ba49e89fa09f9b: Status 404 returned error can't find the container with id 860191f8246c8d4324f1b4adcbb2ca96b58444dd8335b21940ba49e89fa09f9b Oct 30 17:01:53.038871 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:53.038848 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn"] Oct 30 17:01:53.039254 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:53.039223 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:01:53.039369 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:53.039299 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:01:53.039431 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:53.039408 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:01:53.043519 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:53.043497 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx" Oct 30 17:01:53.150790 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:53.150743 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" Oct 30 17:01:53.210463 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:53.184990 4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-7mdgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 17:01:53.210463 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:53.185050 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7mdgd" podUID="86c0131b-dd1d-426b-ae9d-124a6e45e010" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 17:01:53.430527 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:53.429180 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx"] Oct 30 17:01:53.430527 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:01:53.429440 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8208e132_462b_4b94_9246_e0ecc27a13a9.slice/crio-0375c114a7ae20f68d88688d6d90f550a0cbe15dbcb56f9cd6e6c0f6f7b94bda WatchSource:0}: Error finding container 0375c114a7ae20f68d88688d6d90f550a0cbe15dbcb56f9cd6e6c0f6f7b94bda: Status 404 returned error can't find the container with id 0375c114a7ae20f68d88688d6d90f550a0cbe15dbcb56f9cd6e6c0f6f7b94bda Oct 30 17:01:53.432797 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:53.432755 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:01:53.432899 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:53.432848 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:01:53.432899 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:53.432892 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:01:53.972439 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:53.972395 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp" event={"ID":"9eb030f0-e0a2-446e-b35f-4c497d9a3aa2","Type":"ContainerStarted","Data":"a87317b6fc68451491a9fbed0fc18be64d8efac699dc6db5620732df065e2be6"} Oct 30 17:01:53.976389 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:53.972708 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp" event={"ID":"9eb030f0-e0a2-446e-b35f-4c497d9a3aa2","Type":"ContainerStarted","Data":"c50002850adb34f86d769e15c0dd24a169908ad128be77f2b2ad7fa6708cbc62"} Oct 30 17:01:53.980295 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:53.980265 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx" event={"ID":"8208e132-462b-4b94-9246-e0ecc27a13a9","Type":"ContainerStarted","Data":"0df7b4963c6c0b0fcb666d87adb6ba20cecd890acf9d8e99d35f023b4f7c449b"} Oct 30 17:01:53.980527 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:53.980499 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx" event={"ID":"8208e132-462b-4b94-9246-e0ecc27a13a9","Type":"ContainerStarted","Data":"0375c114a7ae20f68d88688d6d90f550a0cbe15dbcb56f9cd6e6c0f6f7b94bda"} Oct 30 17:01:53.983222 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:53.983195 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn" event={"ID":"015c4531-2766-4522-acf0-0427f1ba0a3d","Type":"ContainerStarted","Data":"ec16c40a6b42782747d2a94269022619f5fb1a13bafcac50046ba80bf2766aa9"} Oct 30 17:01:53.983370 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:53.983257 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn" event={"ID":"015c4531-2766-4522-acf0-0427f1ba0a3d","Type":"ContainerStarted","Data":"860191f8246c8d4324f1b4adcbb2ca96b58444dd8335b21940ba49e89fa09f9b"} Oct 30 17:01:54.042337 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:54.040907 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp" podStartSLOduration=2.040885131 podStartE2EDuration="2.040885131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:01:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:01:54.021789546 +0000 UTC m=+888.022544451" watchObservedRunningTime="2026-10-30 17:01:54.040885131 +0000 UTC m=+888.041640034" Oct 30 17:01:54.061330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:54.058380 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx" Oct 30 17:01:54.061330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:54.058529 4522 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 17:01:54.061330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:54.058576 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx" podUID="8208e132-462b-4b94-9246-e0ecc27a13a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 17:01:54.113329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:54.113079 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx" podStartSLOduration=2.113059785 podStartE2EDuration="2.113059785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:01:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:01:54.106803463 +0000 UTC m=+888.107558387" watchObservedRunningTime="2026-10-30 17:01:54.113059785 +0000 UTC m=+888.113814688" Oct 30 17:01:54.184807 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:54.184770 4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-7mdgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 17:01:54.185296 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:54.185260 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7mdgd" podUID="86c0131b-dd1d-426b-ae9d-124a6e45e010" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 17:01:54.212596 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:54.212519 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn" podStartSLOduration=2.212496954 podStartE2EDuration="2.212496954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:01:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:01:54.209945741 +0000 UTC m=+888.210700643" watchObservedRunningTime="2026-10-30 17:01:54.212496954 +0000 UTC m=+888.213251856" Oct 30 17:01:54.648612 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:54.648583 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp" Oct 30 17:01:54.648990 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:54.648968 4522 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 17:01:54.649139 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:54.649114 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp" podUID="9eb030f0-e0a2-446e-b35f-4c497d9a3aa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 17:01:54.671419 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:54.671390 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn" Oct 30 17:01:54.672534 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:54.671678 4522 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 17:01:54.672739 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:54.672715 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn" podUID="015c4531-2766-4522-acf0-0427f1ba0a3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 17:01:55.102331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:55.056441 4522 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 17:01:55.102331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:55.056527 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx" podUID="8208e132-462b-4b94-9246-e0ecc27a13a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 17:01:55.193331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:55.185855 4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-7mdgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 17:01:55.193331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:55.185961 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7mdgd" podUID="86c0131b-dd1d-426b-ae9d-124a6e45e010" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 17:01:55.293533 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:55.293472 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" podUID="43ca764d-ae0e-4858-9841-5bbd0ffdbb85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:01:55.526462 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:55.526414 4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" podUID="43ca764d-ae0e-4858-9841-5bbd0ffdbb85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:01:55.646790 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:55.646751 4522 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 17:01:55.647068 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:55.647037 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp" podUID="9eb030f0-e0a2-446e-b35f-4c497d9a3aa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 17:01:55.669122 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:55.669090 4522 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 17:01:55.669418 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:55.669391 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn" podUID="015c4531-2766-4522-acf0-0427f1ba0a3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 17:01:55.719331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:55.718679 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" podUID="4360e9dc-c001-4102-8af6-b55b692fcf2c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:01:55.881605 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:55.881469 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" podUID="a3e5e732-2c67-4377-9755-e2178bf33772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:01:55.927510 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:55.927454 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" podUID="0b832781-3d41-4496-aa86-e4d92792e590" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:01:55.946519 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:55.946459 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" podUID="75321c5d-8bf8-45c6-8811-81981fe3cad4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:01:56.039455 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:56.039402 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" podUID="c9ad71b7-ad90-4c8c-bf27-0f10b23f1593" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:01:56.044718 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:56.044681 4522 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 17:01:56.044979 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:56.044951 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx" podUID="8208e132-462b-4b94-9246-e0ecc27a13a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 17:01:56.201390 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:56.190066 4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-7mdgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 17:01:56.201671 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:56.201634 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7mdgd" podUID="86c0131b-dd1d-426b-ae9d-124a6e45e010" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 17:01:56.212270 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:56.211408 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" podUID="8a6a85ad-c944-450f-b0bb-e3b64b0f56f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:01:56.297478 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:56.297430 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" podUID="43ca764d-ae0e-4858-9841-5bbd0ffdbb85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 17:01:56.298950 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:56.298913 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" Oct 30 17:01:56.715406 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:56.664960 4522 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 17:01:56.715406 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:56.665017 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp" podUID="9eb030f0-e0a2-446e-b35f-4c497d9a3aa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 17:01:56.715406 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:56.690001 4522 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" podUID="4360e9dc-c001-4102-8af6-b55b692fcf2c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:01:56.715406 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:56.690821 4522 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 17:01:56.715406 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:56.690866 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn" podUID="015c4531-2766-4522-acf0-0427f1ba0a3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 17:01:56.752678 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:56.747450 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" podUID="4360e9dc-c001-4102-8af6-b55b692fcf2c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:01:56.752678 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:56.747852 4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" podUID="a3e5e732-2c67-4377-9755-e2178bf33772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:01:56.752678 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:56.747968 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" Oct 30 17:01:56.879924 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:56.845412 4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" podUID="75321c5d-8bf8-45c6-8811-81981fe3cad4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": context deadline exceeded" Oct 30 17:01:56.890756 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:56.888397 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" podUID="a3e5e732-2c67-4377-9755-e2178bf33772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:01:56.890756 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:56.888829 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" Oct 30 17:01:56.991391 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:56.978383 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" podUID="0b832781-3d41-4496-aa86-e4d92792e590" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:01:56.991391 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:56.979039 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" Oct 30 17:01:57.025330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:57.024332 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" podUID="75321c5d-8bf8-45c6-8811-81981fe3cad4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:01:57.041332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:57.040394 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" podUID="c9ad71b7-ad90-4c8c-bf27-0f10b23f1593" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 17:01:57.041332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:57.040894 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" Oct 30 17:01:57.069712 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:57.062744 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" Oct 30 17:01:57.096330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:57.094615 4522 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 17:01:57.096330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:57.094678 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx" podUID="8208e132-462b-4b94-9246-e0ecc27a13a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 17:01:57.271720 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:57.271632 4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-7mdgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 17:01:57.271945 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:57.271918 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7mdgd" podUID="86c0131b-dd1d-426b-ae9d-124a6e45e010" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 17:01:57.274369 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:57.274346 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" Oct 30 17:01:57.275685 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:57.275654 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" podUID="8a6a85ad-c944-450f-b0bb-e3b64b0f56f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 17:01:57.279353 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:57.275993 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" Oct 30 17:01:57.283827 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:57.283802 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" Oct 30 17:01:57.301448 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:57.301398 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" podUID="43ca764d-ae0e-4858-9841-5bbd0ffdbb85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:01:57.692387 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:57.688754 4522 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 17:01:57.692387 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:57.688834 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp" podUID="9eb030f0-e0a2-446e-b35f-4c497d9a3aa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 17:01:57.822330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:57.820955 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" podUID="4360e9dc-c001-4102-8af6-b55b692fcf2c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:01:57.845339 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:57.843262 4522 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 17:01:57.845339 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:57.843342 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn" podUID="015c4531-2766-4522-acf0-0427f1ba0a3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 17:01:57.987325 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:57.979623 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" podUID="0b832781-3d41-4496-aa86-e4d92792e590" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 17:01:58.053632 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:58.052386 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" podUID="c9ad71b7-ad90-4c8c-bf27-0f10b23f1593" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:01:58.094342 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:58.073449 4522 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 17:01:58.094342 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:58.073506 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx" podUID="8208e132-462b-4b94-9246-e0ecc27a13a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 17:01:58.216341 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:58.216133 4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-7mdgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 17:01:58.216341 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:58.216187 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7mdgd" podUID="86c0131b-dd1d-426b-ae9d-124a6e45e010" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 17:01:58.284470 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:58.283631 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" podUID="8a6a85ad-c944-450f-b0bb-e3b64b0f56f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:01:58.668938 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:58.668839 4522 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 17:01:58.673634 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:58.673592 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp" podUID="9eb030f0-e0a2-446e-b35f-4c497d9a3aa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 17:01:58.735344 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:58.724690 4522 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 17:01:58.735344 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:58.724762 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn" podUID="015c4531-2766-4522-acf0-0427f1ba0a3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 17:01:58.782823 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:58.782782 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" Oct 30 17:01:58.864330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:58.837415 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" podUID="4360e9dc-c001-4102-8af6-b55b692fcf2c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Oct 30 17:01:59.046126 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:59.024755 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" podUID="0b832781-3d41-4496-aa86-e4d92792e590" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:01:59.073079 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:59.073038 4522 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 17:01:59.073434 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:59.073374 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx" podUID="8208e132-462b-4b94-9246-e0ecc27a13a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 17:01:59.074256 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:59.074235 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" Oct 30 17:01:59.274341 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:59.272601 4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-7mdgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 17:01:59.274341 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:59.272725 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7mdgd" podUID="86c0131b-dd1d-426b-ae9d-124a6e45e010" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 17:01:59.295439 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:59.295393 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s"] Oct 30 17:01:59.449396 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:59.422762 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" Oct 30 17:01:59.450790 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:59.450425 4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" podUID="c9ad71b7-ad90-4c8c-bf27-0f10b23f1593" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:01:59.511333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:59.489988 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bfbc6d3-91fd-41a7-9585-40a8f252f743-tmp-dir\") pod \"distinct-w3-b80d83-kserve-8697d79dd5-rjk8s\" (UID: \"9bfbc6d3-91fd-41a7-9585-40a8f252f743\") " pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" Oct 30 17:01:59.511333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:59.490047 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bfbc6d3-91fd-41a7-9585-40a8f252f743-dshm\") pod \"distinct-w3-b80d83-kserve-8697d79dd5-rjk8s\" (UID: \"9bfbc6d3-91fd-41a7-9585-40a8f252f743\") " pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" Oct 30 17:01:59.511333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:59.490090 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bfbc6d3-91fd-41a7-9585-40a8f252f743-home\") pod \"distinct-w3-b80d83-kserve-8697d79dd5-rjk8s\" (UID: \"9bfbc6d3-91fd-41a7-9585-40a8f252f743\") " pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" Oct 30 17:01:59.511333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:59.490164 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bfbc6d3-91fd-41a7-9585-40a8f252f743-model-cache\") pod \"distinct-w3-b80d83-kserve-8697d79dd5-rjk8s\" (UID: \"9bfbc6d3-91fd-41a7-9585-40a8f252f743\") " pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" Oct 30 17:01:59.511333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:59.490209 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bfbc6d3-91fd-41a7-9585-40a8f252f743-tls-certs\") pod \"distinct-w3-b80d83-kserve-8697d79dd5-rjk8s\" (UID: \"9bfbc6d3-91fd-41a7-9585-40a8f252f743\") " pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" Oct 30 17:01:59.511333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:59.490256 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2nxn\" (UniqueName: \"kubernetes.io/projected/9bfbc6d3-91fd-41a7-9585-40a8f252f743-kube-api-access-g2nxn\") pod \"distinct-w3-b80d83-kserve-8697d79dd5-rjk8s\" (UID: \"9bfbc6d3-91fd-41a7-9585-40a8f252f743\") " pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" Oct 30 17:01:59.511333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:59.491099 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b80d83\"/\"distinct-w3-b80d83-kserve-self-signed-certs\"" Oct 30 17:01:59.539331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:59.533987 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s"] Oct 30 17:01:59.588344 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:59.587599 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" podUID="8a6a85ad-c944-450f-b0bb-e3b64b0f56f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:01:59.619248 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:59.617184 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bfbc6d3-91fd-41a7-9585-40a8f252f743-tmp-dir\") pod \"distinct-w3-b80d83-kserve-8697d79dd5-rjk8s\" (UID: \"9bfbc6d3-91fd-41a7-9585-40a8f252f743\") " pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" Oct 30 17:01:59.619248 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:59.617245 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bfbc6d3-91fd-41a7-9585-40a8f252f743-dshm\") pod \"distinct-w3-b80d83-kserve-8697d79dd5-rjk8s\" (UID: \"9bfbc6d3-91fd-41a7-9585-40a8f252f743\") " pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" Oct 30 17:01:59.619248 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:59.617292 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bfbc6d3-91fd-41a7-9585-40a8f252f743-home\") pod \"distinct-w3-b80d83-kserve-8697d79dd5-rjk8s\" (UID: \"9bfbc6d3-91fd-41a7-9585-40a8f252f743\") " pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" Oct 30 17:01:59.619248 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:59.617395 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bfbc6d3-91fd-41a7-9585-40a8f252f743-model-cache\") pod \"distinct-w3-b80d83-kserve-8697d79dd5-rjk8s\" (UID: \"9bfbc6d3-91fd-41a7-9585-40a8f252f743\") " pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" Oct 30 17:01:59.619248 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:59.617443 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bfbc6d3-91fd-41a7-9585-40a8f252f743-tls-certs\") pod \"distinct-w3-b80d83-kserve-8697d79dd5-rjk8s\" (UID: \"9bfbc6d3-91fd-41a7-9585-40a8f252f743\") " pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" Oct 30 17:01:59.619248 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:59.617496 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2nxn\" (UniqueName: \"kubernetes.io/projected/9bfbc6d3-91fd-41a7-9585-40a8f252f743-kube-api-access-g2nxn\") pod \"distinct-w3-b80d83-kserve-8697d79dd5-rjk8s\" (UID: \"9bfbc6d3-91fd-41a7-9585-40a8f252f743\") " pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" Oct 30 17:01:59.619248 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:59.618925 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bfbc6d3-91fd-41a7-9585-40a8f252f743-tmp-dir\") pod \"distinct-w3-b80d83-kserve-8697d79dd5-rjk8s\" (UID: \"9bfbc6d3-91fd-41a7-9585-40a8f252f743\") " pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" Oct 30 17:01:59.630048 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:59.622191 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" Oct 30 17:01:59.630048 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:59.629653 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bfbc6d3-91fd-41a7-9585-40a8f252f743-home\") pod \"distinct-w3-b80d83-kserve-8697d79dd5-rjk8s\" (UID: \"9bfbc6d3-91fd-41a7-9585-40a8f252f743\") " pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" Oct 30 17:01:59.647413 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:59.647367 4522 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 17:01:59.647702 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:59.647664 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp" podUID="9eb030f0-e0a2-446e-b35f-4c497d9a3aa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 17:01:59.675333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:59.671639 4522 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 17:01:59.675333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:59.671700 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn" podUID="015c4531-2766-4522-acf0-0427f1ba0a3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 17:01:59.675333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:59.673393 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bfbc6d3-91fd-41a7-9585-40a8f252f743-model-cache\") pod \"distinct-w3-b80d83-kserve-8697d79dd5-rjk8s\" (UID: \"9bfbc6d3-91fd-41a7-9585-40a8f252f743\") " pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" Oct 30 17:01:59.691326 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:59.683773 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" Oct 30 17:01:59.723391 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:59.695155 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2nxn\" (UniqueName: \"kubernetes.io/projected/9bfbc6d3-91fd-41a7-9585-40a8f252f743-kube-api-access-g2nxn\") pod \"distinct-w3-b80d83-kserve-8697d79dd5-rjk8s\" (UID: \"9bfbc6d3-91fd-41a7-9585-40a8f252f743\") " pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" Oct 30 17:01:59.932330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:01:59.902927 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bfbc6d3-91fd-41a7-9585-40a8f252f743-tls-certs\") pod \"distinct-w3-b80d83-kserve-8697d79dd5-rjk8s\" (UID: \"9bfbc6d3-91fd-41a7-9585-40a8f252f743\") " pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" Oct 30 17:02:00.050390 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:00.049392 4522 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 17:02:00.050390 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:00.049456 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx" podUID="8208e132-462b-4b94-9246-e0ecc27a13a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 17:02:00.077980 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:00.076666 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" podUID="c9ad71b7-ad90-4c8c-bf27-0f10b23f1593" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:02:00.296330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:00.287049 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" podUID="75321c5d-8bf8-45c6-8811-81981fe3cad4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:02:00.332391 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:00.332264 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" podUID="43ca764d-ae0e-4858-9841-5bbd0ffdbb85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:02:00.593529 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:00.593403 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" podUID="8a6a85ad-c944-450f-b0bb-e3b64b0f56f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:02:00.649575 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:00.649537 4522 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 17:02:00.649815 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:00.649788 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp" podUID="9eb030f0-e0a2-446e-b35f-4c497d9a3aa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 17:02:00.650716 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:00.650652 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bfbc6d3-91fd-41a7-9585-40a8f252f743-dshm\") pod \"distinct-w3-b80d83-kserve-8697d79dd5-rjk8s\" (UID: \"9bfbc6d3-91fd-41a7-9585-40a8f252f743\") " pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" Oct 30 17:02:00.671686 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:00.671651 4522 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 17:02:00.671933 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:00.671905 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn" podUID="015c4531-2766-4522-acf0-0427f1ba0a3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 17:02:00.805829 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:00.805560 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" Oct 30 17:02:00.858745 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:00.857528 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" podUID="4360e9dc-c001-4102-8af6-b55b692fcf2c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Oct 30 17:02:01.047746 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:01.047705 4522 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 17:02:01.048016 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:01.047982 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx" podUID="8208e132-462b-4b94-9246-e0ecc27a13a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 17:02:01.079053 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:01.079013 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" podUID="c9ad71b7-ad90-4c8c-bf27-0f10b23f1593" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 17:02:01.079725 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:01.079700 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" Oct 30 17:02:01.349098 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:01.348983 4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-7mdgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:02:01.349372 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:01.349340 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7mdgd" podUID="86c0131b-dd1d-426b-ae9d-124a6e45e010" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:02:01.371331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:01.369614 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" Oct 30 17:02:01.389118 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:01.389089 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" Oct 30 17:02:01.391033 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:01.390978 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" podUID="75321c5d-8bf8-45c6-8811-81981fe3cad4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:02:01.391232 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:01.391212 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" Oct 30 17:02:01.465488 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:01.465413 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" Oct 30 17:02:01.551700 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:01.551619 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7mdgd" Oct 30 17:02:01.553505 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:01.553483 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7mdgd" Oct 30 17:02:01.555944 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:01.555926 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7mdgd" Oct 30 17:02:01.646946 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:01.646899 4522 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 17:02:01.647199 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:01.647163 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp" podUID="9eb030f0-e0a2-446e-b35f-4c497d9a3aa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 17:02:01.669614 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:01.669573 4522 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 17:02:01.669868 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:01.669838 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn" podUID="015c4531-2766-4522-acf0-0427f1ba0a3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 17:02:01.779007 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:01.778098 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s"] Oct 30 17:02:02.043406 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:02.043354 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" event={"ID":"9bfbc6d3-91fd-41a7-9585-40a8f252f743","Type":"ContainerStarted","Data":"dd8ed889c56d104044f6b5565e22bb2e08a8caf2d9715bbf964f46275c2eac13"} Oct 30 17:02:02.043912 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:02.043413 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" event={"ID":"9bfbc6d3-91fd-41a7-9585-40a8f252f743","Type":"ContainerStarted","Data":"d4d16356dd88e628a12267bdb7d9ee580a99c6041ec3b662361299ab9c044575"} Oct 30 17:02:02.044191 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:02.044163 4522 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 17:02:02.044279 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:02.044215 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx" podUID="8208e132-462b-4b94-9246-e0ecc27a13a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 17:02:02.148302 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:02.148216 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" Oct 30 17:02:02.161658 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:02.161626 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" Oct 30 17:02:02.216329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:02.215300 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" podStartSLOduration=4.215279256 podStartE2EDuration="4.215279256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:01:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:02:02.107619523 +0000 UTC m=+896.108374455" watchObservedRunningTime="2026-10-30 17:02:02.215279256 +0000 UTC m=+896.216034158" Oct 30 17:02:02.729331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:02.729173 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn" Oct 30 17:02:02.732344 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:02.730689 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn" Oct 30 17:02:02.732344 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:02.731557 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn" Oct 30 17:02:02.807126 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:02.807091 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" Oct 30 17:02:02.907252 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:02.907218 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp" Oct 30 17:02:02.907629 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:02.907603 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp" Oct 30 17:02:02.908501 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:02.908479 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp" Oct 30 17:02:03.068450 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:03.068367 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" Oct 30 17:02:03.961291 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:03.961254 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx" Oct 30 17:02:03.961752 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:03.961725 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx" Oct 30 17:02:03.962606 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:03.962589 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx" Oct 30 17:02:07.202469 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:07.202392 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9"] Oct 30 17:02:07.386390 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:07.386356 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9"] Oct 30 17:02:07.386700 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:07.386682 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9" Oct 30 17:02:07.406609 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:07.406264 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-c7b9df-openshift-default-dockercfg-qhctr\"" Oct 30 17:02:07.515324 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:07.515212 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw55g\" (UniqueName: \"kubernetes.io/projected/79ea6704-625a-4af9-83c6-7f5214a6b8b4-kube-api-access-dw55g\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9\" (UID: \"79ea6704-625a-4af9-83c6-7f5214a6b8b4\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9" Oct 30 17:02:07.516289 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:07.516262 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ea6704-625a-4af9-83c6-7f5214a6b8b4-workload-socket\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9\" (UID: \"79ea6704-625a-4af9-83c6-7f5214a6b8b4\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9" Oct 30 17:02:07.516494 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:07.516478 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79ea6704-625a-4af9-83c6-7f5214a6b8b4-istio-data\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9\" (UID: \"79ea6704-625a-4af9-83c6-7f5214a6b8b4\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9" Oct 30 17:02:07.516638 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:07.516623 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79ea6704-625a-4af9-83c6-7f5214a6b8b4-istiod-ca-cert\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9\" (UID: \"79ea6704-625a-4af9-83c6-7f5214a6b8b4\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9" Oct 30 17:02:07.516733 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:07.516719 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ea6704-625a-4af9-83c6-7f5214a6b8b4-credential-socket\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9\" (UID: \"79ea6704-625a-4af9-83c6-7f5214a6b8b4\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9" Oct 30 17:02:07.516830 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:07.516818 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79ea6704-625a-4af9-83c6-7f5214a6b8b4-istio-podinfo\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9\" (UID: \"79ea6704-625a-4af9-83c6-7f5214a6b8b4\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9" Oct 30 17:02:07.516932 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:07.516919 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79ea6704-625a-4af9-83c6-7f5214a6b8b4-istio-token\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9\" (UID: \"79ea6704-625a-4af9-83c6-7f5214a6b8b4\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9" Oct 30 17:02:07.517028 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:07.517015 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79ea6704-625a-4af9-83c6-7f5214a6b8b4-workload-certs\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9\" (UID: \"79ea6704-625a-4af9-83c6-7f5214a6b8b4\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9" Oct 30 17:02:07.517118 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:07.517106 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79ea6704-625a-4af9-83c6-7f5214a6b8b4-istio-envoy\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9\" (UID: \"79ea6704-625a-4af9-83c6-7f5214a6b8b4\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9" Oct 30 17:02:07.618829 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:07.618789 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ea6704-625a-4af9-83c6-7f5214a6b8b4-workload-socket\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9\" (UID: \"79ea6704-625a-4af9-83c6-7f5214a6b8b4\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9" Oct 30 17:02:07.619736 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:07.619713 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79ea6704-625a-4af9-83c6-7f5214a6b8b4-istio-data\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9\" (UID: \"79ea6704-625a-4af9-83c6-7f5214a6b8b4\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9" Oct 30 17:02:07.620232 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:07.620211 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79ea6704-625a-4af9-83c6-7f5214a6b8b4-istiod-ca-cert\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9\" (UID: \"79ea6704-625a-4af9-83c6-7f5214a6b8b4\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9" Oct 30 17:02:07.622428 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:07.622409 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ea6704-625a-4af9-83c6-7f5214a6b8b4-credential-socket\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9\" (UID: \"79ea6704-625a-4af9-83c6-7f5214a6b8b4\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9" Oct 30 17:02:07.622930 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:07.622351 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79ea6704-625a-4af9-83c6-7f5214a6b8b4-istiod-ca-cert\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9\" (UID: \"79ea6704-625a-4af9-83c6-7f5214a6b8b4\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9" Oct 30 17:02:07.623075 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:07.620117 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79ea6704-625a-4af9-83c6-7f5214a6b8b4-istio-data\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9\" (UID: \"79ea6704-625a-4af9-83c6-7f5214a6b8b4\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9" Oct 30 17:02:07.623273 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:07.619636 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ea6704-625a-4af9-83c6-7f5214a6b8b4-workload-socket\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9\" (UID: \"79ea6704-625a-4af9-83c6-7f5214a6b8b4\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9" Oct 30 17:02:07.623402 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:07.622788 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ea6704-625a-4af9-83c6-7f5214a6b8b4-credential-socket\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9\" (UID: \"79ea6704-625a-4af9-83c6-7f5214a6b8b4\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9" Oct 30 17:02:07.623511 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:07.622899 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79ea6704-625a-4af9-83c6-7f5214a6b8b4-istio-podinfo\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9\" (UID: \"79ea6704-625a-4af9-83c6-7f5214a6b8b4\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9" Oct 30 17:02:07.624067 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:07.624043 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79ea6704-625a-4af9-83c6-7f5214a6b8b4-istio-token\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9\" (UID: \"79ea6704-625a-4af9-83c6-7f5214a6b8b4\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9" Oct 30 17:02:07.624706 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:07.624686 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79ea6704-625a-4af9-83c6-7f5214a6b8b4-workload-certs\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9\" (UID: \"79ea6704-625a-4af9-83c6-7f5214a6b8b4\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9" Oct 30 17:02:07.625136 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:07.625117 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79ea6704-625a-4af9-83c6-7f5214a6b8b4-istio-envoy\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9\" (UID: \"79ea6704-625a-4af9-83c6-7f5214a6b8b4\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9" Oct 30 17:02:07.626273 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:07.626249 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79ea6704-625a-4af9-83c6-7f5214a6b8b4-istio-podinfo\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9\" (UID: \"79ea6704-625a-4af9-83c6-7f5214a6b8b4\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9" Oct 30 17:02:07.626437 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:07.625074 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79ea6704-625a-4af9-83c6-7f5214a6b8b4-workload-certs\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9\" (UID: \"79ea6704-625a-4af9-83c6-7f5214a6b8b4\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9" Oct 30 17:02:07.626651 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:07.626634 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw55g\" (UniqueName: \"kubernetes.io/projected/79ea6704-625a-4af9-83c6-7f5214a6b8b4-kube-api-access-dw55g\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9\" (UID: \"79ea6704-625a-4af9-83c6-7f5214a6b8b4\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9" Oct 30 17:02:07.631332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:07.631283 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79ea6704-625a-4af9-83c6-7f5214a6b8b4-istio-envoy\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9\" (UID: \"79ea6704-625a-4af9-83c6-7f5214a6b8b4\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9" Oct 30 17:02:07.649474 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:07.649450 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79ea6704-625a-4af9-83c6-7f5214a6b8b4-istio-token\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9\" (UID: \"79ea6704-625a-4af9-83c6-7f5214a6b8b4\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9" Oct 30 17:02:07.655293 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:07.655231 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw55g\" (UniqueName: \"kubernetes.io/projected/79ea6704-625a-4af9-83c6-7f5214a6b8b4-kube-api-access-dw55g\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9\" (UID: \"79ea6704-625a-4af9-83c6-7f5214a6b8b4\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9" Oct 30 17:02:07.785664 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:07.785577 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9" Oct 30 17:02:08.449738 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:08.447971 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9"] Oct 30 17:02:08.466580 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:08.466538 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:02:08.467958 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:08.467930 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:02:08.468300 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:08.468275 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:02:09.162487 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:09.162378 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9" event={"ID":"79ea6704-625a-4af9-83c6-7f5214a6b8b4","Type":"ContainerStarted","Data":"b30e42bcea4c35136a893f35f69ba397516426654b5cf251fedc902b3388ac0c"} Oct 30 17:02:09.162725 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:09.162487 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9" event={"ID":"79ea6704-625a-4af9-83c6-7f5214a6b8b4","Type":"ContainerStarted","Data":"027ae8786d72b07ed8a2aa8d93e3ee842cd51c9c5fbc5e0b0ea994c56fe57090"} Oct 30 17:02:09.226930 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:09.226861 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9" podStartSLOduration=2.22684153 podStartE2EDuration="2.22684153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:02:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:02:09.224575564 +0000 UTC m=+903.225330483" watchObservedRunningTime="2026-10-30 17:02:09.22684153 +0000 UTC m=+903.227596434" Oct 30 17:02:09.786773 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:09.786739 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9" Oct 30 17:02:09.788115 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:09.788092 4522 patch_prober.go:29] interesting pod/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 17:02:09.788203 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:09.788149 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9" podUID="79ea6704-625a-4af9-83c6-7f5214a6b8b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 17:02:10.787024 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:10.786988 4522 patch_prober.go:29] interesting pod/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 17:02:10.787466 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:10.787049 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9" podUID="79ea6704-625a-4af9-83c6-7f5214a6b8b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 17:02:10.816459 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:10.816418 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" Oct 30 17:02:10.832248 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:10.832218 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" Oct 30 17:02:11.190173 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:11.190093 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" Oct 30 17:02:11.786716 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:11.786675 4522 patch_prober.go:29] interesting pod/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 17:02:11.786901 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:11.786762 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9" podUID="79ea6704-625a-4af9-83c6-7f5214a6b8b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 17:02:12.787199 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:12.787159 4522 patch_prober.go:29] interesting pod/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 17:02:12.787622 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:12.787230 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9" podUID="79ea6704-625a-4af9-83c6-7f5214a6b8b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 17:02:12.973534 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:02:12.973492 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:02:13.786678 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:13.786641 4522 patch_prober.go:29] interesting pod/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 17:02:13.786863 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:13.786697 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9" podUID="79ea6704-625a-4af9-83c6-7f5214a6b8b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 17:02:14.786618 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:14.786584 4522 patch_prober.go:29] interesting pod/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 17:02:14.787068 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:14.786647 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9" podUID="79ea6704-625a-4af9-83c6-7f5214a6b8b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 17:02:15.795667 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:15.795633 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9" Oct 30 17:02:15.796278 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:15.796054 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9" Oct 30 17:02:15.797644 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:15.797622 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9" Oct 30 17:02:17.143889 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:17.143857 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43815402-p2dnc"] Oct 30 17:02:17.148392 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:17.148372 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43815402-p2dnc" Oct 30 17:02:17.169048 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:17.169015 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43815402-p2dnc"] Oct 30 17:02:17.178511 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:17.178474 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:02:17.178653 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:17.178533 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:02:17.194832 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:17.194799 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb92d902-69e6-4f48-946e-035115b4559a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43815402-p2dnc\" (UID: \"bb92d902-69e6-4f48-946e-035115b4559a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43815402-p2dnc" Oct 30 17:02:17.195026 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:17.195005 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xstnr\" (UniqueName: \"kubernetes.io/projected/bb92d902-69e6-4f48-946e-035115b4559a-kube-api-access-xstnr\") pod \"maas-api-revoke-sub-43815402-p2dnc\" (UID: \"bb92d902-69e6-4f48-946e-035115b4559a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43815402-p2dnc" Oct 30 17:02:17.195206 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:17.195182 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb92d902-69e6-4f48-946e-035115b4559a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43815402-p2dnc\" (UID: \"bb92d902-69e6-4f48-946e-035115b4559a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43815402-p2dnc" Oct 30 17:02:17.296369 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:17.296331 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xstnr\" (UniqueName: \"kubernetes.io/projected/bb92d902-69e6-4f48-946e-035115b4559a-kube-api-access-xstnr\") pod \"maas-api-revoke-sub-43815402-p2dnc\" (UID: \"bb92d902-69e6-4f48-946e-035115b4559a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43815402-p2dnc" Oct 30 17:02:17.296553 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:17.296485 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb92d902-69e6-4f48-946e-035115b4559a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43815402-p2dnc\" (UID: \"bb92d902-69e6-4f48-946e-035115b4559a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43815402-p2dnc" Oct 30 17:02:17.296604 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:17.296574 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb92d902-69e6-4f48-946e-035115b4559a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43815402-p2dnc\" (UID: \"bb92d902-69e6-4f48-946e-035115b4559a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43815402-p2dnc" Oct 30 17:02:17.297266 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:17.297244 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb92d902-69e6-4f48-946e-035115b4559a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43815402-p2dnc\" (UID: \"bb92d902-69e6-4f48-946e-035115b4559a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43815402-p2dnc" Oct 30 17:02:17.298850 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:17.298825 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb92d902-69e6-4f48-946e-035115b4559a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43815402-p2dnc\" (UID: \"bb92d902-69e6-4f48-946e-035115b4559a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43815402-p2dnc" Oct 30 17:02:17.320956 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:17.320926 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xstnr\" (UniqueName: \"kubernetes.io/projected/bb92d902-69e6-4f48-946e-035115b4559a-kube-api-access-xstnr\") pod \"maas-api-revoke-sub-43815402-p2dnc\" (UID: \"bb92d902-69e6-4f48-946e-035115b4559a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43815402-p2dnc" Oct 30 17:02:17.464023 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:17.463988 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43815402-p2dnc" Oct 30 17:02:17.768820 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:02:17.768785 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb92d902_69e6_4f48_946e_035115b4559a.slice/crio-36112ec42054d5d6f32c7ef7468e83df6a313f275b10222c20b4756a9a7040bd WatchSource:0}: Error finding container 36112ec42054d5d6f32c7ef7468e83df6a313f275b10222c20b4756a9a7040bd: Status 404 returned error can't find the container with id 36112ec42054d5d6f32c7ef7468e83df6a313f275b10222c20b4756a9a7040bd Oct 30 17:02:17.770489 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:17.770470 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43815402-p2dnc"] Oct 30 17:02:18.268558 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:18.268511 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43815402-p2dnc" event={"ID":"bb92d902-69e6-4f48-946e-035115b4559a","Type":"ContainerStarted","Data":"63b79ea2c7d60912e2d1dcd3f716cf20cbcd064422ce781e10b35319d01257eb"} Oct 30 17:02:18.269056 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:18.268622 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43815402-p2dnc" event={"ID":"bb92d902-69e6-4f48-946e-035115b4559a","Type":"ContainerStarted","Data":"36112ec42054d5d6f32c7ef7468e83df6a313f275b10222c20b4756a9a7040bd"} Oct 30 17:02:18.313994 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:18.313943 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43815402-p2dnc" podStartSLOduration=1.313931081 podStartE2EDuration="1.313931081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:02:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:02:18.313367307 +0000 UTC m=+912.314122209" watchObservedRunningTime="2026-10-30 17:02:18.313931081 +0000 UTC m=+912.314686012" Oct 30 17:02:19.276260 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:19.276223 4522 generic.go:356] "Generic (PLEG): container finished" podID="bb92d902-69e6-4f48-946e-035115b4559a" containerID="63b79ea2c7d60912e2d1dcd3f716cf20cbcd064422ce781e10b35319d01257eb" exitCode=0 Oct 30 17:02:19.276761 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:19.276336 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43815402-p2dnc" event={"ID":"bb92d902-69e6-4f48-946e-035115b4559a","Type":"ContainerDied","Data":"63b79ea2c7d60912e2d1dcd3f716cf20cbcd064422ce781e10b35319d01257eb"} Oct 30 17:02:20.449843 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:20.449815 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43815402-p2dnc" Oct 30 17:02:20.538136 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:20.538053 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb92d902-69e6-4f48-946e-035115b4559a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb92d902-69e6-4f48-946e-035115b4559a-maas-api-openshift-ca\") pod \"bb92d902-69e6-4f48-946e-035115b4559a\" (UID: \"bb92d902-69e6-4f48-946e-035115b4559a\") " Oct 30 17:02:20.538317 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:20.538148 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb92d902-69e6-4f48-946e-035115b4559a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb92d902-69e6-4f48-946e-035115b4559a-maas-api-xks-ca\") pod \"bb92d902-69e6-4f48-946e-035115b4559a\" (UID: \"bb92d902-69e6-4f48-946e-035115b4559a\") " Oct 30 17:02:20.538393 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:20.538363 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb92d902-69e6-4f48-946e-035115b4559a-kube-api-access-xstnr\" (UniqueName: \"kubernetes.io/projected/bb92d902-69e6-4f48-946e-035115b4559a-kube-api-access-xstnr\") pod \"bb92d902-69e6-4f48-946e-035115b4559a\" (UID: \"bb92d902-69e6-4f48-946e-035115b4559a\") " Oct 30 17:02:20.538445 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:20.538427 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb92d902-69e6-4f48-946e-035115b4559a-maas-api-openshift-ca" pod "bb92d902-69e6-4f48-946e-035115b4559a" (UID: "bb92d902-69e6-4f48-946e-035115b4559a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:02:20.538990 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:20.538965 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb92d902-69e6-4f48-946e-035115b4559a-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:02:20.540496 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:20.540469 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb92d902-69e6-4f48-946e-035115b4559a-maas-api-xks-ca" pod "bb92d902-69e6-4f48-946e-035115b4559a" (UID: "bb92d902-69e6-4f48-946e-035115b4559a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:02:20.540621 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:20.540562 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb92d902-69e6-4f48-946e-035115b4559a-kube-api-access-xstnr" pod "bb92d902-69e6-4f48-946e-035115b4559a" (UID: "bb92d902-69e6-4f48-946e-035115b4559a"). InnerVolumeSpecName "kube-api-access-xstnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:02:20.640283 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:20.640250 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xstnr\" (UniqueName: \"kubernetes.io/projected/bb92d902-69e6-4f48-946e-035115b4559a-kube-api-access-xstnr\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:02:20.640283 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:20.640279 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb92d902-69e6-4f48-946e-035115b4559a-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:02:21.303936 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:21.303900 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43815402-p2dnc" Oct 30 17:02:21.304130 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:21.303907 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43815402-p2dnc" event={"ID":"bb92d902-69e6-4f48-946e-035115b4559a","Type":"ContainerDied","Data":"36112ec42054d5d6f32c7ef7468e83df6a313f275b10222c20b4756a9a7040bd"} Oct 30 17:02:21.304130 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:21.304043 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36112ec42054d5d6f32c7ef7468e83df6a313f275b10222c20b4756a9a7040bd" Oct 30 17:02:22.107951 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:22.107906 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw"] Oct 30 17:02:22.108809 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:22.108786 4522 cpu_manager.go:414] "containerMap: removing container" podUID="bb92d902-69e6-4f48-946e-035115b4559a" containerName="revoke-keys" Oct 30 17:02:22.108941 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:22.108924 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb92d902-69e6-4f48-946e-035115b4559a" containerName="revoke-keys" Oct 30 17:02:22.109160 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:22.109143 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb92d902-69e6-4f48-946e-035115b4559a" containerName="revoke-keys" Oct 30 17:02:22.114263 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:22.114234 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw" Oct 30 17:02:22.237329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:22.222811 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e123c50-767e-478c-a7e5-0d2476ccbc3b-istio-data\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw\" (UID: \"0e123c50-767e-478c-a7e5-0d2476ccbc3b\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw" Oct 30 17:02:22.237329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:22.222958 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e123c50-767e-478c-a7e5-0d2476ccbc3b-credential-socket\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw\" (UID: \"0e123c50-767e-478c-a7e5-0d2476ccbc3b\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw" Oct 30 17:02:22.237329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:22.223004 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e123c50-767e-478c-a7e5-0d2476ccbc3b-istio-token\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw\" (UID: \"0e123c50-767e-478c-a7e5-0d2476ccbc3b\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw" Oct 30 17:02:22.237329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:22.223040 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk7b4\" (UniqueName: \"kubernetes.io/projected/0e123c50-767e-478c-a7e5-0d2476ccbc3b-kube-api-access-bk7b4\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw\" (UID: \"0e123c50-767e-478c-a7e5-0d2476ccbc3b\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw" Oct 30 17:02:22.237329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:22.223071 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e123c50-767e-478c-a7e5-0d2476ccbc3b-workload-socket\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw\" (UID: \"0e123c50-767e-478c-a7e5-0d2476ccbc3b\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw" Oct 30 17:02:22.237329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:22.223099 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e123c50-767e-478c-a7e5-0d2476ccbc3b-istiod-ca-cert\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw\" (UID: \"0e123c50-767e-478c-a7e5-0d2476ccbc3b\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw" Oct 30 17:02:22.237329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:22.223166 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e123c50-767e-478c-a7e5-0d2476ccbc3b-istio-envoy\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw\" (UID: \"0e123c50-767e-478c-a7e5-0d2476ccbc3b\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw" Oct 30 17:02:22.237329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:22.223253 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e123c50-767e-478c-a7e5-0d2476ccbc3b-workload-certs\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw\" (UID: \"0e123c50-767e-478c-a7e5-0d2476ccbc3b\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw" Oct 30 17:02:22.237329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:22.223291 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e123c50-767e-478c-a7e5-0d2476ccbc3b-istio-podinfo\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw\" (UID: \"0e123c50-767e-478c-a7e5-0d2476ccbc3b\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw" Oct 30 17:02:22.328327 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:22.328191 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e123c50-767e-478c-a7e5-0d2476ccbc3b-credential-socket\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw\" (UID: \"0e123c50-767e-478c-a7e5-0d2476ccbc3b\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw" Oct 30 17:02:22.328327 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:22.328254 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e123c50-767e-478c-a7e5-0d2476ccbc3b-istio-token\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw\" (UID: \"0e123c50-767e-478c-a7e5-0d2476ccbc3b\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw" Oct 30 17:02:22.328327 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:22.328285 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk7b4\" (UniqueName: \"kubernetes.io/projected/0e123c50-767e-478c-a7e5-0d2476ccbc3b-kube-api-access-bk7b4\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw\" (UID: \"0e123c50-767e-478c-a7e5-0d2476ccbc3b\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw" Oct 30 17:02:22.328629 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:22.328336 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e123c50-767e-478c-a7e5-0d2476ccbc3b-workload-socket\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw\" (UID: \"0e123c50-767e-478c-a7e5-0d2476ccbc3b\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw" Oct 30 17:02:22.328629 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:22.328359 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e123c50-767e-478c-a7e5-0d2476ccbc3b-istiod-ca-cert\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw\" (UID: \"0e123c50-767e-478c-a7e5-0d2476ccbc3b\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw" Oct 30 17:02:22.328629 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:22.328421 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e123c50-767e-478c-a7e5-0d2476ccbc3b-istio-envoy\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw\" (UID: \"0e123c50-767e-478c-a7e5-0d2476ccbc3b\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw" Oct 30 17:02:22.328629 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:22.328484 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e123c50-767e-478c-a7e5-0d2476ccbc3b-workload-certs\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw\" (UID: \"0e123c50-767e-478c-a7e5-0d2476ccbc3b\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw" Oct 30 17:02:22.328629 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:22.328514 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e123c50-767e-478c-a7e5-0d2476ccbc3b-istio-podinfo\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw\" (UID: \"0e123c50-767e-478c-a7e5-0d2476ccbc3b\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw" Oct 30 17:02:22.328889 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:22.328669 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e123c50-767e-478c-a7e5-0d2476ccbc3b-istio-data\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw\" (UID: \"0e123c50-767e-478c-a7e5-0d2476ccbc3b\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw" Oct 30 17:02:22.365386 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:22.329141 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e123c50-767e-478c-a7e5-0d2476ccbc3b-istio-data\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw\" (UID: \"0e123c50-767e-478c-a7e5-0d2476ccbc3b\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw" Oct 30 17:02:22.365386 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:22.341577 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e123c50-767e-478c-a7e5-0d2476ccbc3b-credential-socket\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw\" (UID: \"0e123c50-767e-478c-a7e5-0d2476ccbc3b\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw" Oct 30 17:02:22.365386 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:22.342438 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e123c50-767e-478c-a7e5-0d2476ccbc3b-workload-socket\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw\" (UID: \"0e123c50-767e-478c-a7e5-0d2476ccbc3b\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw" Oct 30 17:02:22.365386 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:22.343120 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e123c50-767e-478c-a7e5-0d2476ccbc3b-istiod-ca-cert\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw\" (UID: \"0e123c50-767e-478c-a7e5-0d2476ccbc3b\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw" Oct 30 17:02:22.365386 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:22.343929 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e123c50-767e-478c-a7e5-0d2476ccbc3b-workload-certs\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw\" (UID: \"0e123c50-767e-478c-a7e5-0d2476ccbc3b\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw" Oct 30 17:02:22.388825 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:22.382613 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw"] Oct 30 17:02:22.441873 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:22.441803 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk7b4\" (UniqueName: \"kubernetes.io/projected/0e123c50-767e-478c-a7e5-0d2476ccbc3b-kube-api-access-bk7b4\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw\" (UID: \"0e123c50-767e-478c-a7e5-0d2476ccbc3b\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw" Oct 30 17:02:22.456389 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:22.454878 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e123c50-767e-478c-a7e5-0d2476ccbc3b-istio-token\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw\" (UID: \"0e123c50-767e-478c-a7e5-0d2476ccbc3b\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw" Oct 30 17:02:22.471086 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:22.471036 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e123c50-767e-478c-a7e5-0d2476ccbc3b-istio-podinfo\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw\" (UID: \"0e123c50-767e-478c-a7e5-0d2476ccbc3b\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw" Oct 30 17:02:22.501260 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:22.501227 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e123c50-767e-478c-a7e5-0d2476ccbc3b-istio-envoy\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw\" (UID: \"0e123c50-767e-478c-a7e5-0d2476ccbc3b\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw" Oct 30 17:02:22.509059 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:22.509027 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw" Oct 30 17:02:22.546844 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:22.546802 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh"] Oct 30 17:02:22.560748 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:22.560717 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh"] Oct 30 17:02:22.561055 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:22.561037 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh" Oct 30 17:02:22.654249 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:22.654155 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa811185-1eb8-451b-afb7-9580783a30f2-workload-socket\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh\" (UID: \"aa811185-1eb8-451b-afb7-9580783a30f2\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh" Oct 30 17:02:22.654249 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:22.654209 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa811185-1eb8-451b-afb7-9580783a30f2-workload-certs\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh\" (UID: \"aa811185-1eb8-451b-afb7-9580783a30f2\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh" Oct 30 17:02:22.654504 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:22.654249 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa811185-1eb8-451b-afb7-9580783a30f2-credential-socket\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh\" (UID: \"aa811185-1eb8-451b-afb7-9580783a30f2\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh" Oct 30 17:02:22.654504 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:22.654341 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa811185-1eb8-451b-afb7-9580783a30f2-istio-data\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh\" (UID: \"aa811185-1eb8-451b-afb7-9580783a30f2\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh" Oct 30 17:02:22.654504 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:22.654478 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa811185-1eb8-451b-afb7-9580783a30f2-istio-token\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh\" (UID: \"aa811185-1eb8-451b-afb7-9580783a30f2\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh" Oct 30 17:02:22.654684 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:22.654519 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa811185-1eb8-451b-afb7-9580783a30f2-istio-podinfo\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh\" (UID: \"aa811185-1eb8-451b-afb7-9580783a30f2\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh" Oct 30 17:02:22.654684 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:22.654571 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf247\" (UniqueName: \"kubernetes.io/projected/aa811185-1eb8-451b-afb7-9580783a30f2-kube-api-access-pf247\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh\" (UID: \"aa811185-1eb8-451b-afb7-9580783a30f2\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh" Oct 30 17:02:22.654684 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:22.654617 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa811185-1eb8-451b-afb7-9580783a30f2-istio-envoy\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh\" (UID: \"aa811185-1eb8-451b-afb7-9580783a30f2\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh" Oct 30 17:02:22.654684 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:22.654660 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa811185-1eb8-451b-afb7-9580783a30f2-istiod-ca-cert\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh\" (UID: \"aa811185-1eb8-451b-afb7-9580783a30f2\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh" Oct 30 17:02:22.762452 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:22.762393 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa811185-1eb8-451b-afb7-9580783a30f2-workload-socket\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh\" (UID: \"aa811185-1eb8-451b-afb7-9580783a30f2\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh" Oct 30 17:02:22.762452 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:22.762439 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa811185-1eb8-451b-afb7-9580783a30f2-workload-certs\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh\" (UID: \"aa811185-1eb8-451b-afb7-9580783a30f2\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh" Oct 30 17:02:22.762677 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:22.762476 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa811185-1eb8-451b-afb7-9580783a30f2-credential-socket\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh\" (UID: \"aa811185-1eb8-451b-afb7-9580783a30f2\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh" Oct 30 17:02:22.762677 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:22.762549 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa811185-1eb8-451b-afb7-9580783a30f2-istio-data\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh\" (UID: \"aa811185-1eb8-451b-afb7-9580783a30f2\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh" Oct 30 17:02:22.762789 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:22.762688 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa811185-1eb8-451b-afb7-9580783a30f2-istio-token\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh\" (UID: \"aa811185-1eb8-451b-afb7-9580783a30f2\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh" Oct 30 17:02:22.762789 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:22.762733 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa811185-1eb8-451b-afb7-9580783a30f2-istio-podinfo\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh\" (UID: \"aa811185-1eb8-451b-afb7-9580783a30f2\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh" Oct 30 17:02:22.762897 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:22.762791 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf247\" (UniqueName: \"kubernetes.io/projected/aa811185-1eb8-451b-afb7-9580783a30f2-kube-api-access-pf247\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh\" (UID: \"aa811185-1eb8-451b-afb7-9580783a30f2\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh" Oct 30 17:02:22.762897 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:22.762840 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa811185-1eb8-451b-afb7-9580783a30f2-istio-envoy\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh\" (UID: \"aa811185-1eb8-451b-afb7-9580783a30f2\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh" Oct 30 17:02:22.762897 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:22.762882 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa811185-1eb8-451b-afb7-9580783a30f2-istiod-ca-cert\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh\" (UID: \"aa811185-1eb8-451b-afb7-9580783a30f2\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh" Oct 30 17:02:22.787086 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:22.763746 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa811185-1eb8-451b-afb7-9580783a30f2-istiod-ca-cert\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh\" (UID: \"aa811185-1eb8-451b-afb7-9580783a30f2\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh" Oct 30 17:02:22.787086 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:22.764054 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa811185-1eb8-451b-afb7-9580783a30f2-workload-socket\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh\" (UID: \"aa811185-1eb8-451b-afb7-9580783a30f2\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh" Oct 30 17:02:22.787086 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:22.764300 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa811185-1eb8-451b-afb7-9580783a30f2-workload-certs\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh\" (UID: \"aa811185-1eb8-451b-afb7-9580783a30f2\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh" Oct 30 17:02:22.787086 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:22.764580 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa811185-1eb8-451b-afb7-9580783a30f2-credential-socket\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh\" (UID: \"aa811185-1eb8-451b-afb7-9580783a30f2\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh" Oct 30 17:02:22.787086 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:22.764816 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa811185-1eb8-451b-afb7-9580783a30f2-istio-data\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh\" (UID: \"aa811185-1eb8-451b-afb7-9580783a30f2\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh" Oct 30 17:02:22.804754 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:22.804106 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa811185-1eb8-451b-afb7-9580783a30f2-istio-envoy\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh\" (UID: \"aa811185-1eb8-451b-afb7-9580783a30f2\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh" Oct 30 17:02:23.175931 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:23.148768 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc"] Oct 30 17:02:23.233777 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:23.233745 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa811185-1eb8-451b-afb7-9580783a30f2-istio-token\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh\" (UID: \"aa811185-1eb8-451b-afb7-9580783a30f2\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh" Oct 30 17:02:23.265371 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:23.258725 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc" Oct 30 17:02:23.317483 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:23.317391 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-v527k"] Oct 30 17:02:23.412588 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:23.379679 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/570bc5c9-330c-402e-b262-16b3be6340e7-istio-token\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc\" (UID: \"570bc5c9-330c-402e-b262-16b3be6340e7\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc" Oct 30 17:02:23.412588 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:23.379776 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/570bc5c9-330c-402e-b262-16b3be6340e7-istio-envoy\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc\" (UID: \"570bc5c9-330c-402e-b262-16b3be6340e7\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc" Oct 30 17:02:23.412588 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:23.379867 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/570bc5c9-330c-402e-b262-16b3be6340e7-istio-data\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc\" (UID: \"570bc5c9-330c-402e-b262-16b3be6340e7\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc" Oct 30 17:02:23.412588 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:23.379921 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/570bc5c9-330c-402e-b262-16b3be6340e7-istio-podinfo\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc\" (UID: \"570bc5c9-330c-402e-b262-16b3be6340e7\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc" Oct 30 17:02:23.412588 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:23.379953 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/570bc5c9-330c-402e-b262-16b3be6340e7-workload-socket\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc\" (UID: \"570bc5c9-330c-402e-b262-16b3be6340e7\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc" Oct 30 17:02:23.412588 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:23.380037 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/570bc5c9-330c-402e-b262-16b3be6340e7-istiod-ca-cert\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc\" (UID: \"570bc5c9-330c-402e-b262-16b3be6340e7\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc" Oct 30 17:02:23.412588 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:23.380166 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz2bx\" (UniqueName: \"kubernetes.io/projected/570bc5c9-330c-402e-b262-16b3be6340e7-kube-api-access-bz2bx\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc\" (UID: \"570bc5c9-330c-402e-b262-16b3be6340e7\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc" Oct 30 17:02:23.412588 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:23.380198 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/570bc5c9-330c-402e-b262-16b3be6340e7-credential-socket\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc\" (UID: \"570bc5c9-330c-402e-b262-16b3be6340e7\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc" Oct 30 17:02:23.412588 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:23.380233 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/570bc5c9-330c-402e-b262-16b3be6340e7-workload-certs\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc\" (UID: \"570bc5c9-330c-402e-b262-16b3be6340e7\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc" Oct 30 17:02:23.584498 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:23.580974 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf247\" (UniqueName: \"kubernetes.io/projected/aa811185-1eb8-451b-afb7-9580783a30f2-kube-api-access-pf247\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh\" (UID: \"aa811185-1eb8-451b-afb7-9580783a30f2\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh" Oct 30 17:02:23.584498 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:23.581557 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa811185-1eb8-451b-afb7-9580783a30f2-istio-podinfo\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh\" (UID: \"aa811185-1eb8-451b-afb7-9580783a30f2\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh" Oct 30 17:02:23.584498 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:23.581821 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-v527k" Oct 30 17:02:23.610716 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:23.608731 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh" Oct 30 17:02:23.644038 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:23.643943 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz2bx\" (UniqueName: \"kubernetes.io/projected/570bc5c9-330c-402e-b262-16b3be6340e7-kube-api-access-bz2bx\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc\" (UID: \"570bc5c9-330c-402e-b262-16b3be6340e7\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc" Oct 30 17:02:23.649578 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:23.647713 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/570bc5c9-330c-402e-b262-16b3be6340e7-credential-socket\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc\" (UID: \"570bc5c9-330c-402e-b262-16b3be6340e7\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc" Oct 30 17:02:23.649578 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:23.647778 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/570bc5c9-330c-402e-b262-16b3be6340e7-workload-certs\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc\" (UID: \"570bc5c9-330c-402e-b262-16b3be6340e7\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc" Oct 30 17:02:23.649578 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:23.647831 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/570bc5c9-330c-402e-b262-16b3be6340e7-istio-token\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc\" (UID: \"570bc5c9-330c-402e-b262-16b3be6340e7\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc" Oct 30 17:02:23.649578 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:23.647928 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/570bc5c9-330c-402e-b262-16b3be6340e7-istio-envoy\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc\" (UID: \"570bc5c9-330c-402e-b262-16b3be6340e7\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc" Oct 30 17:02:23.649578 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:23.648022 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/570bc5c9-330c-402e-b262-16b3be6340e7-istio-data\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc\" (UID: \"570bc5c9-330c-402e-b262-16b3be6340e7\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc" Oct 30 17:02:23.649578 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:23.648084 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/570bc5c9-330c-402e-b262-16b3be6340e7-istio-podinfo\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc\" (UID: \"570bc5c9-330c-402e-b262-16b3be6340e7\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc" Oct 30 17:02:23.649578 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:23.648114 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/570bc5c9-330c-402e-b262-16b3be6340e7-workload-socket\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc\" (UID: \"570bc5c9-330c-402e-b262-16b3be6340e7\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc" Oct 30 17:02:23.649578 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:23.648197 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/570bc5c9-330c-402e-b262-16b3be6340e7-istiod-ca-cert\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc\" (UID: \"570bc5c9-330c-402e-b262-16b3be6340e7\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc" Oct 30 17:02:23.649578 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:23.649276 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/570bc5c9-330c-402e-b262-16b3be6340e7-istiod-ca-cert\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc\" (UID: \"570bc5c9-330c-402e-b262-16b3be6340e7\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc" Oct 30 17:02:23.671585 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:23.654690 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/570bc5c9-330c-402e-b262-16b3be6340e7-credential-socket\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc\" (UID: \"570bc5c9-330c-402e-b262-16b3be6340e7\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc" Oct 30 17:02:23.671585 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:23.654977 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/570bc5c9-330c-402e-b262-16b3be6340e7-workload-certs\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc\" (UID: \"570bc5c9-330c-402e-b262-16b3be6340e7\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc" Oct 30 17:02:23.768327 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:23.743233 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/570bc5c9-330c-402e-b262-16b3be6340e7-istio-data\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc\" (UID: \"570bc5c9-330c-402e-b262-16b3be6340e7\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc" Oct 30 17:02:23.778336 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:23.746739 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc"] Oct 30 17:02:23.804418 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:23.757478 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:02:23.804418 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:23.785576 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:02:23.804418 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:23.785730 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:02:23.804418 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:23.765950 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/570bc5c9-330c-402e-b262-16b3be6340e7-workload-socket\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc\" (UID: \"570bc5c9-330c-402e-b262-16b3be6340e7\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc" Oct 30 17:02:23.805702 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:23.805441 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/570bc5c9-330c-402e-b262-16b3be6340e7-istio-envoy\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc\" (UID: \"570bc5c9-330c-402e-b262-16b3be6340e7\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc" Oct 30 17:02:23.806075 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:23.806043 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/570bc5c9-330c-402e-b262-16b3be6340e7-istio-podinfo\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc\" (UID: \"570bc5c9-330c-402e-b262-16b3be6340e7\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc" Oct 30 17:02:23.844381 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:23.817916 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/570bc5c9-330c-402e-b262-16b3be6340e7-istio-token\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc\" (UID: \"570bc5c9-330c-402e-b262-16b3be6340e7\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc" Oct 30 17:02:23.844381 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:23.821725 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz2bx\" (UniqueName: \"kubernetes.io/projected/570bc5c9-330c-402e-b262-16b3be6340e7-kube-api-access-bz2bx\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc\" (UID: \"570bc5c9-330c-402e-b262-16b3be6340e7\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc" Oct 30 17:02:23.844381 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:23.835116 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" podUID="4360e9dc-c001-4102-8af6-b55b692fcf2c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:02:23.870077 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:23.852870 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvsq7\" (UniqueName: \"kubernetes.io/projected/8b6f1a27-3c5a-447a-9043-72c9f70fbe70-kube-api-access-nvsq7\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-v527k\" (UID: \"8b6f1a27-3c5a-447a-9043-72c9f70fbe70\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-v527k" Oct 30 17:02:23.870077 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:23.864481 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b6f1a27-3c5a-447a-9043-72c9f70fbe70-istio-token\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-v527k\" (UID: \"8b6f1a27-3c5a-447a-9043-72c9f70fbe70\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-v527k" Oct 30 17:02:23.870077 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:23.864605 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b6f1a27-3c5a-447a-9043-72c9f70fbe70-istiod-ca-cert\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-v527k\" (UID: \"8b6f1a27-3c5a-447a-9043-72c9f70fbe70\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-v527k" Oct 30 17:02:23.870077 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:23.864787 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b6f1a27-3c5a-447a-9043-72c9f70fbe70-workload-certs\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-v527k\" (UID: \"8b6f1a27-3c5a-447a-9043-72c9f70fbe70\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-v527k" Oct 30 17:02:23.891334 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:23.889360 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc" Oct 30 17:02:23.891334 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:23.890893 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-v527k"] Oct 30 17:02:23.938545 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:23.938377 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6f1a27-3c5a-447a-9043-72c9f70fbe70-credential-socket\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-v527k\" (UID: \"8b6f1a27-3c5a-447a-9043-72c9f70fbe70\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-v527k" Oct 30 17:02:23.959103 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:23.958948 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b6f1a27-3c5a-447a-9043-72c9f70fbe70-istio-data\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-v527k\" (UID: \"8b6f1a27-3c5a-447a-9043-72c9f70fbe70\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-v527k" Oct 30 17:02:24.013435 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:24.013395 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6f1a27-3c5a-447a-9043-72c9f70fbe70-workload-socket\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-v527k\" (UID: \"8b6f1a27-3c5a-447a-9043-72c9f70fbe70\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-v527k" Oct 30 17:02:24.062269 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:24.062226 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b6f1a27-3c5a-447a-9043-72c9f70fbe70-istio-envoy\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-v527k\" (UID: \"8b6f1a27-3c5a-447a-9043-72c9f70fbe70\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-v527k" Oct 30 17:02:24.062561 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:24.062538 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b6f1a27-3c5a-447a-9043-72c9f70fbe70-istio-podinfo\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-v527k\" (UID: \"8b6f1a27-3c5a-447a-9043-72c9f70fbe70\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-v527k" Oct 30 17:02:24.089362 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:24.075127 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" podUID="2f1e3bbb-ab1c-455d-a05d-d62f72c5d009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:02:24.308384 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:24.292104 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw"] Oct 30 17:02:24.308384 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:24.294626 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" podUID="9bfbc6d3-91fd-41a7-9585-40a8f252f743" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 30 17:02:24.308384 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:24.297855 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b6f1a27-3c5a-447a-9043-72c9f70fbe70-istio-token\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-v527k\" (UID: \"8b6f1a27-3c5a-447a-9043-72c9f70fbe70\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-v527k" Oct 30 17:02:24.308384 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:24.297895 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b6f1a27-3c5a-447a-9043-72c9f70fbe70-istiod-ca-cert\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-v527k\" (UID: \"8b6f1a27-3c5a-447a-9043-72c9f70fbe70\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-v527k" Oct 30 17:02:24.308384 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:24.297984 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b6f1a27-3c5a-447a-9043-72c9f70fbe70-workload-certs\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-v527k\" (UID: \"8b6f1a27-3c5a-447a-9043-72c9f70fbe70\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-v527k" Oct 30 17:02:24.308384 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:24.298023 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6f1a27-3c5a-447a-9043-72c9f70fbe70-credential-socket\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-v527k\" (UID: \"8b6f1a27-3c5a-447a-9043-72c9f70fbe70\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-v527k" Oct 30 17:02:24.308384 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:24.298139 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b6f1a27-3c5a-447a-9043-72c9f70fbe70-istio-data\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-v527k\" (UID: \"8b6f1a27-3c5a-447a-9043-72c9f70fbe70\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-v527k" Oct 30 17:02:24.308384 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:24.298172 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6f1a27-3c5a-447a-9043-72c9f70fbe70-workload-socket\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-v527k\" (UID: \"8b6f1a27-3c5a-447a-9043-72c9f70fbe70\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-v527k" Oct 30 17:02:24.308384 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:24.298196 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b6f1a27-3c5a-447a-9043-72c9f70fbe70-istio-envoy\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-v527k\" (UID: \"8b6f1a27-3c5a-447a-9043-72c9f70fbe70\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-v527k" Oct 30 17:02:24.308384 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:24.298222 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b6f1a27-3c5a-447a-9043-72c9f70fbe70-istio-podinfo\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-v527k\" (UID: \"8b6f1a27-3c5a-447a-9043-72c9f70fbe70\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-v527k" Oct 30 17:02:24.308384 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:24.298323 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvsq7\" (UniqueName: \"kubernetes.io/projected/8b6f1a27-3c5a-447a-9043-72c9f70fbe70-kube-api-access-nvsq7\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-v527k\" (UID: \"8b6f1a27-3c5a-447a-9043-72c9f70fbe70\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-v527k" Oct 30 17:02:24.308384 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:24.300182 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b6f1a27-3c5a-447a-9043-72c9f70fbe70-istiod-ca-cert\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-v527k\" (UID: \"8b6f1a27-3c5a-447a-9043-72c9f70fbe70\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-v527k" Oct 30 17:02:24.308384 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:24.300691 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b6f1a27-3c5a-447a-9043-72c9f70fbe70-workload-certs\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-v527k\" (UID: \"8b6f1a27-3c5a-447a-9043-72c9f70fbe70\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-v527k" Oct 30 17:02:24.308384 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:24.300924 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6f1a27-3c5a-447a-9043-72c9f70fbe70-credential-socket\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-v527k\" (UID: \"8b6f1a27-3c5a-447a-9043-72c9f70fbe70\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-v527k" Oct 30 17:02:24.308384 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:24.301870 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b6f1a27-3c5a-447a-9043-72c9f70fbe70-istio-data\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-v527k\" (UID: \"8b6f1a27-3c5a-447a-9043-72c9f70fbe70\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-v527k" Oct 30 17:02:24.308384 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:24.302100 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6f1a27-3c5a-447a-9043-72c9f70fbe70-workload-socket\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-v527k\" (UID: \"8b6f1a27-3c5a-447a-9043-72c9f70fbe70\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-v527k" Oct 30 17:02:24.308384 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:24.306255 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" podUID="a3e5e732-2c67-4377-9755-e2178bf33772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:02:24.358131 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:24.326203 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" podUID="43ca764d-ae0e-4858-9841-5bbd0ffdbb85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 17:02:24.358131 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:24.350848 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvsq7\" (UniqueName: \"kubernetes.io/projected/8b6f1a27-3c5a-447a-9043-72c9f70fbe70-kube-api-access-nvsq7\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-v527k\" (UID: \"8b6f1a27-3c5a-447a-9043-72c9f70fbe70\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-v527k" Oct 30 17:02:24.453345 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:24.452381 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" podUID="c9ad71b7-ad90-4c8c-bf27-0f10b23f1593" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:02:24.494155 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:24.467999 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b6f1a27-3c5a-447a-9043-72c9f70fbe70-istio-envoy\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-v527k\" (UID: \"8b6f1a27-3c5a-447a-9043-72c9f70fbe70\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-v527k" Oct 30 17:02:24.494155 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:24.470447 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b6f1a27-3c5a-447a-9043-72c9f70fbe70-istio-token\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-v527k\" (UID: \"8b6f1a27-3c5a-447a-9043-72c9f70fbe70\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-v527k" Oct 30 17:02:24.494155 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:24.475991 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b6f1a27-3c5a-447a-9043-72c9f70fbe70-istio-podinfo\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-v527k\" (UID: \"8b6f1a27-3c5a-447a-9043-72c9f70fbe70\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-v527k" Oct 30 17:02:24.624194 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:24.624156 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw" event={"ID":"0e123c50-767e-478c-a7e5-0d2476ccbc3b","Type":"ContainerStarted","Data":"5a55745e4b054f0015ed3df749d3c576744a49e28d2dc31ba121e8a6bfa4674c"} Oct 30 17:02:24.624396 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:24.624219 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw" event={"ID":"0e123c50-767e-478c-a7e5-0d2476ccbc3b","Type":"ContainerStarted","Data":"86b8c2afe64a24c7e9a3b0c6bf524a8241f24195f92936d8e8f436fa2a503d19"} Oct 30 17:02:24.676929 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:24.676819 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw" podStartSLOduration=2.676801482 podStartE2EDuration="2.676801482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:02:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:02:24.674948728 +0000 UTC m=+918.675703631" watchObservedRunningTime="2026-10-30 17:02:24.676801482 +0000 UTC m=+918.677556384" Oct 30 17:02:24.686905 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:24.686758 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh"] Oct 30 17:02:24.694847 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:24.694805 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:02:24.694944 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:24.694916 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:02:24.695735 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:24.694960 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:02:24.757924 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:24.757895 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-v527k" Oct 30 17:02:24.809664 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:24.809027 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc"] Oct 30 17:02:24.810662 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:02:24.810628 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod570bc5c9_330c_402e_b262_16b3be6340e7.slice/crio-168d1157b2d86cad74e65d178fff3547e4ff9b1ca26c626bfe1c9b466d73b3e0 WatchSource:0}: Error finding container 168d1157b2d86cad74e65d178fff3547e4ff9b1ca26c626bfe1c9b466d73b3e0: Status 404 returned error can't find the container with id 168d1157b2d86cad74e65d178fff3547e4ff9b1ca26c626bfe1c9b466d73b3e0 Oct 30 17:02:24.814634 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:24.814597 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:02:24.814743 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:24.814719 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:02:24.814808 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:24.814787 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:02:25.085020 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:02:25.084980 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b6f1a27_3c5a_447a_9043_72c9f70fbe70.slice/crio-eaaa1e3e0323c282727186ef650ae428c3909c2b50c50cb49a464d38b969e57b WatchSource:0}: Error finding container eaaa1e3e0323c282727186ef650ae428c3909c2b50c50cb49a464d38b969e57b: Status 404 returned error can't find the container with id eaaa1e3e0323c282727186ef650ae428c3909c2b50c50cb49a464d38b969e57b Oct 30 17:02:25.086349 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:25.086322 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-v527k"] Oct 30 17:02:25.087699 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:25.087661 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:02:25.087804 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:25.087726 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:02:25.087804 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:25.087761 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:02:25.521329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:25.520712 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw" Oct 30 17:02:25.530339 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:25.529596 4522 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 17:02:25.530339 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:25.529656 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw" podUID="0e123c50-767e-478c-a7e5-0d2476ccbc3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 17:02:25.678221 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:25.674721 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc" event={"ID":"570bc5c9-330c-402e-b262-16b3be6340e7","Type":"ContainerStarted","Data":"c4291b8c23114bb9edea91f8b1eaf0eda4f75862b2ce722eafd724879aabf378"} Oct 30 17:02:25.678221 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:25.674832 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc" event={"ID":"570bc5c9-330c-402e-b262-16b3be6340e7","Type":"ContainerStarted","Data":"168d1157b2d86cad74e65d178fff3547e4ff9b1ca26c626bfe1c9b466d73b3e0"} Oct 30 17:02:25.678221 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:25.678204 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-v527k" event={"ID":"8b6f1a27-3c5a-447a-9043-72c9f70fbe70","Type":"ContainerStarted","Data":"a5862a44e6344ff74cf1e8f204bff6a2ee606520ce08168979bf651011e21cde"} Oct 30 17:02:25.678476 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:25.678245 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-v527k" event={"ID":"8b6f1a27-3c5a-447a-9043-72c9f70fbe70","Type":"ContainerStarted","Data":"eaaa1e3e0323c282727186ef650ae428c3909c2b50c50cb49a464d38b969e57b"} Oct 30 17:02:25.681282 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:25.681255 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh" event={"ID":"aa811185-1eb8-451b-afb7-9580783a30f2","Type":"ContainerStarted","Data":"ee7a7bb2cd70fb9312ee4948bd61f60505bd923fdced7d9ce6d77d059886eb2e"} Oct 30 17:02:25.681350 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:25.681333 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh" event={"ID":"aa811185-1eb8-451b-afb7-9580783a30f2","Type":"ContainerStarted","Data":"ce618febfc7802b12ff2c3798c002ad6c97214188e8cd4ca4a37ba622b2b20c8"} Oct 30 17:02:25.769349 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:25.769316 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-v527k" Oct 30 17:02:25.777384 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:25.775740 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh" Oct 30 17:02:25.777384 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:25.775863 4522 patch_prober.go:29] interesting pod/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 17:02:25.777384 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:25.775911 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh" podUID="aa811185-1eb8-451b-afb7-9580783a30f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 17:02:25.895329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:25.894492 4522 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 17:02:25.895329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:25.894572 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc" podUID="570bc5c9-330c-402e-b262-16b3be6340e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 17:02:25.907330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:25.903664 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc" Oct 30 17:02:25.918344 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:25.912358 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh" podStartSLOduration=3.9123368149999997 podStartE2EDuration="3.912336815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:02:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:02:25.817613331 +0000 UTC m=+919.818368232" watchObservedRunningTime="2026-10-30 17:02:25.912336815 +0000 UTC m=+919.913091717" Oct 30 17:02:25.918344 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:25.912810 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc" podStartSLOduration=3.9128019099999998 podStartE2EDuration="3.91280191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:02:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:02:25.769044002 +0000 UTC m=+919.769798903" watchObservedRunningTime="2026-10-30 17:02:25.91280191 +0000 UTC m=+919.913556814" Oct 30 17:02:25.961332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:25.934842 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-v527k" podStartSLOduration=3.934821088 podStartE2EDuration="3.934821088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:02:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:02:25.865837012 +0000 UTC m=+919.866591941" watchObservedRunningTime="2026-10-30 17:02:25.934821088 +0000 UTC m=+919.935575990" Oct 30 17:02:26.521330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:26.520734 4522 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 17:02:26.521330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:26.520808 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw" podUID="0e123c50-767e-478c-a7e5-0d2476ccbc3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 17:02:26.779391 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:26.767478 4522 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-v527k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 17:02:26.779391 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:26.767539 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-v527k" podUID="8b6f1a27-3c5a-447a-9043-72c9f70fbe70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 17:02:26.799333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:26.785580 4522 patch_prober.go:29] interesting pod/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 17:02:26.799333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:26.785636 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh" podUID="aa811185-1eb8-451b-afb7-9580783a30f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 17:02:26.908330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:26.908046 4522 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 17:02:26.908330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:26.908134 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc" podUID="570bc5c9-330c-402e-b262-16b3be6340e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 17:02:27.077386 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:27.076641 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" podUID="2f1e3bbb-ab1c-455d-a05d-d62f72c5d009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:02:27.181332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:27.177391 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" podUID="9bfbc6d3-91fd-41a7-9585-40a8f252f743" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:02:27.339402 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:27.323433 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" podUID="a3e5e732-2c67-4377-9755-e2178bf33772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:02:27.339402 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:27.324239 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" podUID="43ca764d-ae0e-4858-9841-5bbd0ffdbb85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:02:27.370024 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:27.369982 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" podUID="c9ad71b7-ad90-4c8c-bf27-0f10b23f1593" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 17:02:27.520914 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:27.520878 4522 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 17:02:27.521222 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:27.521180 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw" podUID="0e123c50-767e-478c-a7e5-0d2476ccbc3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 17:02:27.592568 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:27.592457 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" podUID="8a6a85ad-c944-450f-b0bb-e3b64b0f56f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:02:27.759044 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:27.759011 4522 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-v527k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 17:02:27.759363 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:27.759300 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-v527k" podUID="8b6f1a27-3c5a-447a-9043-72c9f70fbe70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 17:02:27.769825 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:27.769708 4522 patch_prober.go:29] interesting pod/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 17:02:27.769825 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:27.769767 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh" podUID="aa811185-1eb8-451b-afb7-9580783a30f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 17:02:27.830222 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:27.830179 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" podUID="4360e9dc-c001-4102-8af6-b55b692fcf2c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:02:27.892389 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:27.891761 4522 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 17:02:27.892389 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:27.891822 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc" podUID="570bc5c9-330c-402e-b262-16b3be6340e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 17:02:28.190391 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:28.184497 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" podUID="9bfbc6d3-91fd-41a7-9585-40a8f252f743" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 30 17:02:28.190391 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:28.186097 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" Oct 30 17:02:28.326334 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:28.325428 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" podUID="43ca764d-ae0e-4858-9841-5bbd0ffdbb85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:02:28.352333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:28.327276 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" podUID="a3e5e732-2c67-4377-9755-e2178bf33772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 17:02:28.352333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:28.327961 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" Oct 30 17:02:28.352333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:28.328021 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" Oct 30 17:02:28.394332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:28.392825 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" podUID="c9ad71b7-ad90-4c8c-bf27-0f10b23f1593" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 17:02:28.394332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:28.393376 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" Oct 30 17:02:28.513549 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:28.513498 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" podUID="75321c5d-8bf8-45c6-8811-81981fe3cad4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:02:28.582332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:28.561600 4522 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 17:02:28.582332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:28.561661 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw" podUID="0e123c50-767e-478c-a7e5-0d2476ccbc3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 17:02:28.597930 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:28.594151 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" podUID="8a6a85ad-c944-450f-b0bb-e3b64b0f56f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:02:28.597930 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:28.594643 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" Oct 30 17:02:28.720237 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:28.717500 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" podUID="0b832781-3d41-4496-aa86-e4d92792e590" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 17:02:28.823387 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:28.819554 4522 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-v527k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 17:02:28.823387 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:28.819609 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-v527k" podUID="8b6f1a27-3c5a-447a-9043-72c9f70fbe70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 17:02:28.841342 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:28.835777 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" podUID="4360e9dc-c001-4102-8af6-b55b692fcf2c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Oct 30 17:02:28.870501 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:28.843023 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" Oct 30 17:02:28.870501 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:28.843265 4522 patch_prober.go:29] interesting pod/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 17:02:28.870501 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:28.843332 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh" podUID="aa811185-1eb8-451b-afb7-9580783a30f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 17:02:28.899127 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:28.899062 4522 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 17:02:28.899418 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:28.899360 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc" podUID="570bc5c9-330c-402e-b262-16b3be6340e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 17:02:29.067337 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:29.066449 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" podUID="2f1e3bbb-ab1c-455d-a05d-d62f72c5d009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:02:29.187688 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:29.186399 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" podUID="9bfbc6d3-91fd-41a7-9585-40a8f252f743" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:02:29.327450 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:29.327401 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" podUID="43ca764d-ae0e-4858-9841-5bbd0ffdbb85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 17:02:29.329382 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:29.327705 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" podUID="a3e5e732-2c67-4377-9755-e2178bf33772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 17:02:29.405641 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:29.405596 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" podUID="c9ad71b7-ad90-4c8c-bf27-0f10b23f1593" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:02:29.407079 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:29.405888 4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" podUID="c9ad71b7-ad90-4c8c-bf27-0f10b23f1593" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:02:29.534442 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:29.534383 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" podUID="75321c5d-8bf8-45c6-8811-81981fe3cad4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:02:29.535642 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:29.535549 4522 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 17:02:29.535642 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:29.535616 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw" podUID="0e123c50-767e-478c-a7e5-0d2476ccbc3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 17:02:29.539334 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:29.536150 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" Oct 30 17:02:29.665488 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:29.653845 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" podUID="8a6a85ad-c944-450f-b0bb-e3b64b0f56f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 17:02:29.799383 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:29.796782 4522 patch_prober.go:29] interesting pod/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 17:02:29.799383 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:29.796853 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh" podUID="aa811185-1eb8-451b-afb7-9580783a30f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 17:02:29.799383 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:29.797377 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" podUID="0b832781-3d41-4496-aa86-e4d92792e590" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 17:02:29.799383 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:29.797665 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" Oct 30 17:02:29.799383 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:29.799195 4522 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-v527k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 17:02:29.799383 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:29.799231 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-v527k" podUID="8b6f1a27-3c5a-447a-9043-72c9f70fbe70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 17:02:29.897328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:29.872036 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" podUID="4360e9dc-c001-4102-8af6-b55b692fcf2c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:02:29.900471 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:29.900424 4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" podUID="8a6a85ad-c944-450f-b0bb-e3b64b0f56f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": context deadline exceeded" Oct 30 17:02:29.929468 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:29.929432 4522 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 17:02:29.929756 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:29.929722 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc" podUID="570bc5c9-330c-402e-b262-16b3be6340e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 17:02:30.172508 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:30.172396 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" podUID="2f1e3bbb-ab1c-455d-a05d-d62f72c5d009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:02:30.200336 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:30.196797 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" podUID="9bfbc6d3-91fd-41a7-9585-40a8f252f743" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:02:30.338446 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:30.338398 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" podUID="43ca764d-ae0e-4858-9841-5bbd0ffdbb85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 17:02:30.361331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:30.357994 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" podUID="a3e5e732-2c67-4377-9755-e2178bf33772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:02:30.476332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:30.400884 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" Oct 30 17:02:30.476332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:30.411402 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" podUID="c9ad71b7-ad90-4c8c-bf27-0f10b23f1593" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 17:02:30.524205 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:30.524080 4522 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 17:02:30.524511 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:30.524485 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw" podUID="0e123c50-767e-478c-a7e5-0d2476ccbc3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 17:02:30.537559 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:30.537410 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" podUID="75321c5d-8bf8-45c6-8811-81981fe3cad4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:02:30.654545 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:30.654478 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" podUID="8a6a85ad-c944-450f-b0bb-e3b64b0f56f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:02:30.762245 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:30.762151 4522 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-v527k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 17:02:30.762828 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:30.762798 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-v527k" podUID="8b6f1a27-3c5a-447a-9043-72c9f70fbe70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 17:02:30.838328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:30.837486 4522 patch_prober.go:29] interesting pod/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 17:02:30.838328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:30.837576 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh" podUID="aa811185-1eb8-451b-afb7-9580783a30f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 17:02:30.860447 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:30.857744 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" podUID="0b832781-3d41-4496-aa86-e4d92792e590" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 17:02:30.902140 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:30.900891 4522 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 17:02:30.902140 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:30.900959 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc" podUID="570bc5c9-330c-402e-b262-16b3be6340e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 17:02:30.910517 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:30.906159 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" podUID="4360e9dc-c001-4102-8af6-b55b692fcf2c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:02:31.207395 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:31.174421 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" podUID="2f1e3bbb-ab1c-455d-a05d-d62f72c5d009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 17:02:31.207395 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:31.207278 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" podUID="9bfbc6d3-91fd-41a7-9585-40a8f252f743" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:02:31.340385 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:31.338818 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" podUID="43ca764d-ae0e-4858-9841-5bbd0ffdbb85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:02:31.450525 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:31.448394 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" podUID="a3e5e732-2c67-4377-9755-e2178bf33772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 17:02:31.475522 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:31.475410 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" podUID="c9ad71b7-ad90-4c8c-bf27-0f10b23f1593" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:02:31.539333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:31.532211 4522 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 17:02:31.539333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:31.532269 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw" podUID="0e123c50-767e-478c-a7e5-0d2476ccbc3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 17:02:31.540218 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:31.539664 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" podUID="75321c5d-8bf8-45c6-8811-81981fe3cad4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:02:31.660179 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:31.659541 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" podUID="8a6a85ad-c944-450f-b0bb-e3b64b0f56f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 17:02:31.764696 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:31.764124 4522 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-v527k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 17:02:31.764696 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:31.764186 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-v527k" podUID="8b6f1a27-3c5a-447a-9043-72c9f70fbe70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 17:02:31.779902 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:31.779868 4522 patch_prober.go:29] interesting pod/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 17:02:31.780227 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:31.780165 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh" podUID="aa811185-1eb8-451b-afb7-9580783a30f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 17:02:31.844525 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:31.844414 4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" podUID="9bfbc6d3-91fd-41a7-9585-40a8f252f743" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:02:31.872449 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:31.872398 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" podUID="0b832781-3d41-4496-aa86-e4d92792e590" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:02:31.900561 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:31.900527 4522 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 17:02:31.900811 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:31.900785 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc" podUID="570bc5c9-330c-402e-b262-16b3be6340e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 17:02:31.928149 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:31.928107 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" podUID="4360e9dc-c001-4102-8af6-b55b692fcf2c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Oct 30 17:02:32.175384 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:32.173913 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" Oct 30 17:02:32.175384 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:32.175101 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" podUID="2f1e3bbb-ab1c-455d-a05d-d62f72c5d009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:02:32.187488 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:32.187461 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" Oct 30 17:02:32.208860 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:32.208462 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" podUID="9bfbc6d3-91fd-41a7-9585-40a8f252f743" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 30 17:02:32.240546 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:32.239630 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" Oct 30 17:02:32.343552 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:32.343521 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" Oct 30 17:02:32.344970 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:32.344913 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" Oct 30 17:02:32.347391 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:32.347020 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" Oct 30 17:02:32.359030 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:32.358635 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" Oct 30 17:02:32.359393 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:32.359370 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" Oct 30 17:02:32.378328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:32.376999 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" Oct 30 17:02:32.522327 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:32.521403 4522 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 17:02:32.522327 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:32.521484 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw" podUID="0e123c50-767e-478c-a7e5-0d2476ccbc3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 17:02:32.761047 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:32.760026 4522 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-v527k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 17:02:32.761047 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:32.760093 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-v527k" podUID="8b6f1a27-3c5a-447a-9043-72c9f70fbe70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 17:02:32.769812 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:32.769400 4522 patch_prober.go:29] interesting pod/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 17:02:32.769812 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:32.769457 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh" podUID="aa811185-1eb8-451b-afb7-9580783a30f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 17:02:32.894930 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:32.894829 4522 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 17:02:32.895217 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:32.895187 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc" podUID="570bc5c9-330c-402e-b262-16b3be6340e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 17:02:33.547330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:33.545592 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw" Oct 30 17:02:33.547330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:33.547111 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw" Oct 30 17:02:33.553571 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:33.553540 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw" Oct 30 17:02:33.778067 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:33.778037 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh" Oct 30 17:02:33.778997 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:33.778967 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh" Oct 30 17:02:33.779811 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:33.779791 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh" Oct 30 17:02:33.898698 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:33.898614 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-v527k" Oct 30 17:02:33.899204 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:33.899143 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-v527k" Oct 30 17:02:33.900780 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:33.900760 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-v527k" Oct 30 17:02:33.908237 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:33.908214 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc" Oct 30 17:02:33.908726 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:33.908707 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc" Oct 30 17:02:33.909759 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:33.909738 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc" Oct 30 17:02:42.950532 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:02:42.950488 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:02:53.221138 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:53.221062 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9810290f-2lnpx"] Oct 30 17:02:53.225273 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:53.225255 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9810290f-2lnpx" Oct 30 17:02:53.245734 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:53.245703 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9810290f-2lnpx"] Oct 30 17:02:53.253952 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:53.253926 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:02:53.254091 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:53.253955 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:02:53.346321 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:53.346283 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16d26c33-9ea4-406c-b1d5-0cb773380404-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9810290f-2lnpx\" (UID: \"16d26c33-9ea4-406c-b1d5-0cb773380404\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9810290f-2lnpx" Oct 30 17:02:53.346511 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:53.346492 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg672\" (UniqueName: \"kubernetes.io/projected/16d26c33-9ea4-406c-b1d5-0cb773380404-kube-api-access-hg672\") pod \"maas-api-revoke-sub-9810290f-2lnpx\" (UID: \"16d26c33-9ea4-406c-b1d5-0cb773380404\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9810290f-2lnpx" Oct 30 17:02:53.346642 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:53.346628 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16d26c33-9ea4-406c-b1d5-0cb773380404-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9810290f-2lnpx\" (UID: \"16d26c33-9ea4-406c-b1d5-0cb773380404\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9810290f-2lnpx" Oct 30 17:02:53.447758 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:53.447720 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16d26c33-9ea4-406c-b1d5-0cb773380404-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9810290f-2lnpx\" (UID: \"16d26c33-9ea4-406c-b1d5-0cb773380404\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9810290f-2lnpx" Oct 30 17:02:53.448031 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:53.448004 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg672\" (UniqueName: \"kubernetes.io/projected/16d26c33-9ea4-406c-b1d5-0cb773380404-kube-api-access-hg672\") pod \"maas-api-revoke-sub-9810290f-2lnpx\" (UID: \"16d26c33-9ea4-406c-b1d5-0cb773380404\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9810290f-2lnpx" Oct 30 17:02:53.448188 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:53.448171 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16d26c33-9ea4-406c-b1d5-0cb773380404-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9810290f-2lnpx\" (UID: \"16d26c33-9ea4-406c-b1d5-0cb773380404\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9810290f-2lnpx" Oct 30 17:02:53.449112 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:53.449084 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16d26c33-9ea4-406c-b1d5-0cb773380404-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9810290f-2lnpx\" (UID: \"16d26c33-9ea4-406c-b1d5-0cb773380404\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9810290f-2lnpx" Oct 30 17:02:53.450656 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:53.450633 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16d26c33-9ea4-406c-b1d5-0cb773380404-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9810290f-2lnpx\" (UID: \"16d26c33-9ea4-406c-b1d5-0cb773380404\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9810290f-2lnpx" Oct 30 17:02:53.474677 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:53.474596 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg672\" (UniqueName: \"kubernetes.io/projected/16d26c33-9ea4-406c-b1d5-0cb773380404-kube-api-access-hg672\") pod \"maas-api-revoke-sub-9810290f-2lnpx\" (UID: \"16d26c33-9ea4-406c-b1d5-0cb773380404\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9810290f-2lnpx" Oct 30 17:02:53.539475 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:53.539447 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9810290f-2lnpx" Oct 30 17:02:53.852546 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:02:53.852510 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16d26c33_9ea4_406c_b1d5_0cb773380404.slice/crio-d8707bb0d47e9d363c3f41d22164dfa0373b8b7ac38541b84aeab020cb85888b WatchSource:0}: Error finding container d8707bb0d47e9d363c3f41d22164dfa0373b8b7ac38541b84aeab020cb85888b: Status 404 returned error can't find the container with id d8707bb0d47e9d363c3f41d22164dfa0373b8b7ac38541b84aeab020cb85888b Oct 30 17:02:53.854087 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:53.854054 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9810290f-2lnpx"] Oct 30 17:02:54.771355 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:54.771318 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9810290f-2lnpx" event={"ID":"16d26c33-9ea4-406c-b1d5-0cb773380404","Type":"ContainerStarted","Data":"7b6554b0b238d9af1218131b038b154e6a0ae4ac4675e31cea18c539e0c1431f"} Oct 30 17:02:54.771853 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:54.771380 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9810290f-2lnpx" event={"ID":"16d26c33-9ea4-406c-b1d5-0cb773380404","Type":"ContainerStarted","Data":"d8707bb0d47e9d363c3f41d22164dfa0373b8b7ac38541b84aeab020cb85888b"} Oct 30 17:02:54.817399 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:54.817353 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9810290f-2lnpx" podStartSLOduration=1.8173383749999998 podStartE2EDuration="1.817338375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:02:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:02:54.816297006 +0000 UTC m=+948.817051906" watchObservedRunningTime="2026-10-30 17:02:54.817338375 +0000 UTC m=+948.818093267" Oct 30 17:02:55.315371 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:55.315324 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9"] Oct 30 17:02:55.325810 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:55.320538 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" Oct 30 17:02:55.349442 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:55.349411 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9"] Oct 30 17:02:55.354821 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:55.354790 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-86f50a\"/\"distinct-w5-86f50a-kserve-self-signed-certs\"" Oct 30 17:02:55.354821 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:55.354791 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-86f50a\"/\"kube-root-ca.crt\"" Oct 30 17:02:55.355020 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:55.354830 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-86f50a\"/\"openshift-service-ca.crt\"" Oct 30 17:02:55.355120 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:55.355104 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-86f50a\"/\"default-dockercfg-tw9rv\"" Oct 30 17:02:55.481942 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:55.481908 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b4ad752-e053-4cea-a3bc-fe2be6f92021-tmp-dir\") pod \"distinct-w5-86f50a-kserve-994c9b44-vssh9\" (UID: \"4b4ad752-e053-4cea-a3bc-fe2be6f92021\") " pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" Oct 30 17:02:55.481942 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:55.481950 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sxx8\" (UniqueName: \"kubernetes.io/projected/4b4ad752-e053-4cea-a3bc-fe2be6f92021-kube-api-access-9sxx8\") pod \"distinct-w5-86f50a-kserve-994c9b44-vssh9\" (UID: \"4b4ad752-e053-4cea-a3bc-fe2be6f92021\") " pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" Oct 30 17:02:55.482202 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:55.482155 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b4ad752-e053-4cea-a3bc-fe2be6f92021-model-cache\") pod \"distinct-w5-86f50a-kserve-994c9b44-vssh9\" (UID: \"4b4ad752-e053-4cea-a3bc-fe2be6f92021\") " pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" Oct 30 17:02:55.482202 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:55.482187 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b4ad752-e053-4cea-a3bc-fe2be6f92021-dshm\") pod \"distinct-w5-86f50a-kserve-994c9b44-vssh9\" (UID: \"4b4ad752-e053-4cea-a3bc-fe2be6f92021\") " pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" Oct 30 17:02:55.482276 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:55.482222 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b4ad752-e053-4cea-a3bc-fe2be6f92021-tls-certs\") pod \"distinct-w5-86f50a-kserve-994c9b44-vssh9\" (UID: \"4b4ad752-e053-4cea-a3bc-fe2be6f92021\") " pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" Oct 30 17:02:55.482386 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:55.482360 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b4ad752-e053-4cea-a3bc-fe2be6f92021-home\") pod \"distinct-w5-86f50a-kserve-994c9b44-vssh9\" (UID: \"4b4ad752-e053-4cea-a3bc-fe2be6f92021\") " pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" Oct 30 17:02:55.585726 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:55.585619 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sxx8\" (UniqueName: \"kubernetes.io/projected/4b4ad752-e053-4cea-a3bc-fe2be6f92021-kube-api-access-9sxx8\") pod \"distinct-w5-86f50a-kserve-994c9b44-vssh9\" (UID: \"4b4ad752-e053-4cea-a3bc-fe2be6f92021\") " pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" Oct 30 17:02:55.585904 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:55.585887 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b4ad752-e053-4cea-a3bc-fe2be6f92021-model-cache\") pod \"distinct-w5-86f50a-kserve-994c9b44-vssh9\" (UID: \"4b4ad752-e053-4cea-a3bc-fe2be6f92021\") " pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" Oct 30 17:02:55.585968 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:55.585938 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b4ad752-e053-4cea-a3bc-fe2be6f92021-dshm\") pod \"distinct-w5-86f50a-kserve-994c9b44-vssh9\" (UID: \"4b4ad752-e053-4cea-a3bc-fe2be6f92021\") " pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" Oct 30 17:02:55.586025 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:55.585982 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b4ad752-e053-4cea-a3bc-fe2be6f92021-tls-certs\") pod \"distinct-w5-86f50a-kserve-994c9b44-vssh9\" (UID: \"4b4ad752-e053-4cea-a3bc-fe2be6f92021\") " pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" Oct 30 17:02:55.586093 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:55.586026 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b4ad752-e053-4cea-a3bc-fe2be6f92021-home\") pod \"distinct-w5-86f50a-kserve-994c9b44-vssh9\" (UID: \"4b4ad752-e053-4cea-a3bc-fe2be6f92021\") " pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" Oct 30 17:02:55.586151 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:55.586104 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b4ad752-e053-4cea-a3bc-fe2be6f92021-tmp-dir\") pod \"distinct-w5-86f50a-kserve-994c9b44-vssh9\" (UID: \"4b4ad752-e053-4cea-a3bc-fe2be6f92021\") " pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" Oct 30 17:02:55.586650 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:55.586619 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b4ad752-e053-4cea-a3bc-fe2be6f92021-tmp-dir\") pod \"distinct-w5-86f50a-kserve-994c9b44-vssh9\" (UID: \"4b4ad752-e053-4cea-a3bc-fe2be6f92021\") " pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" Oct 30 17:02:55.587243 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:55.587218 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b4ad752-e053-4cea-a3bc-fe2be6f92021-model-cache\") pod \"distinct-w5-86f50a-kserve-994c9b44-vssh9\" (UID: \"4b4ad752-e053-4cea-a3bc-fe2be6f92021\") " pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" Oct 30 17:02:55.593347 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:55.593260 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b4ad752-e053-4cea-a3bc-fe2be6f92021-dshm\") pod \"distinct-w5-86f50a-kserve-994c9b44-vssh9\" (UID: \"4b4ad752-e053-4cea-a3bc-fe2be6f92021\") " pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" Oct 30 17:02:55.593774 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:55.593744 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b4ad752-e053-4cea-a3bc-fe2be6f92021-tls-certs\") pod \"distinct-w5-86f50a-kserve-994c9b44-vssh9\" (UID: \"4b4ad752-e053-4cea-a3bc-fe2be6f92021\") " pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" Oct 30 17:02:55.597535 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:55.597497 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b4ad752-e053-4cea-a3bc-fe2be6f92021-home\") pod \"distinct-w5-86f50a-kserve-994c9b44-vssh9\" (UID: \"4b4ad752-e053-4cea-a3bc-fe2be6f92021\") " pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" Oct 30 17:02:55.631329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:55.617958 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sxx8\" (UniqueName: \"kubernetes.io/projected/4b4ad752-e053-4cea-a3bc-fe2be6f92021-kube-api-access-9sxx8\") pod \"distinct-w5-86f50a-kserve-994c9b44-vssh9\" (UID: \"4b4ad752-e053-4cea-a3bc-fe2be6f92021\") " pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" Oct 30 17:02:55.665379 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:55.663523 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" Oct 30 17:02:55.823142 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:55.811993 4522 generic.go:356] "Generic (PLEG): container finished" podID="16d26c33-9ea4-406c-b1d5-0cb773380404" containerID="7b6554b0b238d9af1218131b038b154e6a0ae4ac4675e31cea18c539e0c1431f" exitCode=0 Oct 30 17:02:55.823142 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:55.812246 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9810290f-2lnpx" event={"ID":"16d26c33-9ea4-406c-b1d5-0cb773380404","Type":"ContainerDied","Data":"7b6554b0b238d9af1218131b038b154e6a0ae4ac4675e31cea18c539e0c1431f"} Oct 30 17:02:56.031328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:56.030935 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9"] Oct 30 17:02:56.820532 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:56.820485 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" event={"ID":"4b4ad752-e053-4cea-a3bc-fe2be6f92021","Type":"ContainerStarted","Data":"4f08eb9d781721239b79b3f6e885d7b55d3347c64721b985ed55c8e50914f1c2"} Oct 30 17:02:56.820715 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:56.820547 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" event={"ID":"4b4ad752-e053-4cea-a3bc-fe2be6f92021","Type":"ContainerStarted","Data":"87f8ffa1118f3b10a9102de0bd5d730d3e2b5ca9ba004b9cf081e9fd7a64b3b8"} Oct 30 17:02:56.866499 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:56.866443 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" podStartSLOduration=1.866429089 podStartE2EDuration="1.866429089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:02:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:02:56.865735905 +0000 UTC m=+950.866490806" watchObservedRunningTime="2026-10-30 17:02:56.866429089 +0000 UTC m=+950.867183990" Oct 30 17:02:56.978410 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:56.978382 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9810290f-2lnpx" Oct 30 17:02:57.128434 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:57.128338 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16d26c33-9ea4-406c-b1d5-0cb773380404-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16d26c33-9ea4-406c-b1d5-0cb773380404-maas-api-openshift-ca\") pod \"16d26c33-9ea4-406c-b1d5-0cb773380404\" (UID: \"16d26c33-9ea4-406c-b1d5-0cb773380404\") " Oct 30 17:02:57.128434 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:57.128422 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16d26c33-9ea4-406c-b1d5-0cb773380404-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16d26c33-9ea4-406c-b1d5-0cb773380404-maas-api-xks-ca\") pod \"16d26c33-9ea4-406c-b1d5-0cb773380404\" (UID: \"16d26c33-9ea4-406c-b1d5-0cb773380404\") " Oct 30 17:02:57.128672 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:57.128657 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16d26c33-9ea4-406c-b1d5-0cb773380404-kube-api-access-hg672\" (UniqueName: \"kubernetes.io/projected/16d26c33-9ea4-406c-b1d5-0cb773380404-kube-api-access-hg672\") pod \"16d26c33-9ea4-406c-b1d5-0cb773380404\" (UID: \"16d26c33-9ea4-406c-b1d5-0cb773380404\") " Oct 30 17:02:57.128725 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:57.128702 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16d26c33-9ea4-406c-b1d5-0cb773380404-maas-api-openshift-ca" pod "16d26c33-9ea4-406c-b1d5-0cb773380404" (UID: "16d26c33-9ea4-406c-b1d5-0cb773380404"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:02:57.129175 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:57.129153 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16d26c33-9ea4-406c-b1d5-0cb773380404-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:02:57.130516 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:57.130490 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16d26c33-9ea4-406c-b1d5-0cb773380404-maas-api-xks-ca" pod "16d26c33-9ea4-406c-b1d5-0cb773380404" (UID: "16d26c33-9ea4-406c-b1d5-0cb773380404"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:02:57.130775 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:57.130757 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16d26c33-9ea4-406c-b1d5-0cb773380404-kube-api-access-hg672" pod "16d26c33-9ea4-406c-b1d5-0cb773380404" (UID: "16d26c33-9ea4-406c-b1d5-0cb773380404"). InnerVolumeSpecName "kube-api-access-hg672". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:02:57.230529 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:57.230497 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16d26c33-9ea4-406c-b1d5-0cb773380404-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:02:57.230529 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:57.230525 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg672\" (UniqueName: \"kubernetes.io/projected/16d26c33-9ea4-406c-b1d5-0cb773380404-kube-api-access-hg672\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:02:57.665018 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:57.664984 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" Oct 30 17:02:57.828541 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:57.828505 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9810290f-2lnpx" event={"ID":"16d26c33-9ea4-406c-b1d5-0cb773380404","Type":"ContainerDied","Data":"d8707bb0d47e9d363c3f41d22164dfa0373b8b7ac38541b84aeab020cb85888b"} Oct 30 17:02:57.828739 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:57.828556 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8707bb0d47e9d363c3f41d22164dfa0373b8b7ac38541b84aeab020cb85888b" Oct 30 17:02:57.828739 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:02:57.828651 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9810290f-2lnpx" Oct 30 17:03:05.303550 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:05.303509 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65f9f450-lxclp"] Oct 30 17:03:05.304977 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:05.304950 4522 cpu_manager.go:414] "containerMap: removing container" podUID="16d26c33-9ea4-406c-b1d5-0cb773380404" containerName="revoke-keys" Oct 30 17:03:05.304977 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:05.304976 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16d26c33-9ea4-406c-b1d5-0cb773380404" containerName="revoke-keys" Oct 30 17:03:05.305340 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:05.305317 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="16d26c33-9ea4-406c-b1d5-0cb773380404" containerName="revoke-keys" Oct 30 17:03:05.310714 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:05.310693 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f9f450-lxclp" Oct 30 17:03:05.326437 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:05.326409 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65f9f450-lxclp"] Oct 30 17:03:05.339204 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:05.339173 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81aedff4-a52f-4d6c-b0be-6905f9703a30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65f9f450-lxclp\" (UID: \"81aedff4-a52f-4d6c-b0be-6905f9703a30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f9f450-lxclp" Oct 30 17:03:05.339430 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:05.339406 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g29mv\" (UniqueName: \"kubernetes.io/projected/81aedff4-a52f-4d6c-b0be-6905f9703a30-kube-api-access-g29mv\") pod \"maas-api-revoke-sub-65f9f450-lxclp\" (UID: \"81aedff4-a52f-4d6c-b0be-6905f9703a30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f9f450-lxclp" Oct 30 17:03:05.339637 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:05.339621 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81aedff4-a52f-4d6c-b0be-6905f9703a30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65f9f450-lxclp\" (UID: \"81aedff4-a52f-4d6c-b0be-6905f9703a30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f9f450-lxclp" Oct 30 17:03:05.374646 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:05.374615 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:03:05.374646 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:05.374647 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:03:05.441137 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:05.441106 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81aedff4-a52f-4d6c-b0be-6905f9703a30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65f9f450-lxclp\" (UID: \"81aedff4-a52f-4d6c-b0be-6905f9703a30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f9f450-lxclp" Oct 30 17:03:05.441362 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:05.441168 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81aedff4-a52f-4d6c-b0be-6905f9703a30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65f9f450-lxclp\" (UID: \"81aedff4-a52f-4d6c-b0be-6905f9703a30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f9f450-lxclp" Oct 30 17:03:05.441362 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:05.441341 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g29mv\" (UniqueName: \"kubernetes.io/projected/81aedff4-a52f-4d6c-b0be-6905f9703a30-kube-api-access-g29mv\") pod \"maas-api-revoke-sub-65f9f450-lxclp\" (UID: \"81aedff4-a52f-4d6c-b0be-6905f9703a30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f9f450-lxclp" Oct 30 17:03:05.441774 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:05.441752 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81aedff4-a52f-4d6c-b0be-6905f9703a30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65f9f450-lxclp\" (UID: \"81aedff4-a52f-4d6c-b0be-6905f9703a30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f9f450-lxclp" Oct 30 17:03:05.443570 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:05.443549 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81aedff4-a52f-4d6c-b0be-6905f9703a30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65f9f450-lxclp\" (UID: \"81aedff4-a52f-4d6c-b0be-6905f9703a30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f9f450-lxclp" Oct 30 17:03:05.467017 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:05.466984 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g29mv\" (UniqueName: \"kubernetes.io/projected/81aedff4-a52f-4d6c-b0be-6905f9703a30-kube-api-access-g29mv\") pod \"maas-api-revoke-sub-65f9f450-lxclp\" (UID: \"81aedff4-a52f-4d6c-b0be-6905f9703a30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f9f450-lxclp" Oct 30 17:03:05.625868 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:05.625787 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f9f450-lxclp" Oct 30 17:03:05.663919 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:05.663884 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" Oct 30 17:03:05.678155 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:05.678125 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" Oct 30 17:03:05.901475 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:05.901404 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" Oct 30 17:03:05.935951 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:03:05.935924 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81aedff4_a52f_4d6c_b0be_6905f9703a30.slice/crio-c72f58e04acaa538bb2cd4e3182bddea15eb6c9d28eb539c73204ea32dfa3ca9 WatchSource:0}: Error finding container c72f58e04acaa538bb2cd4e3182bddea15eb6c9d28eb539c73204ea32dfa3ca9: Status 404 returned error can't find the container with id c72f58e04acaa538bb2cd4e3182bddea15eb6c9d28eb539c73204ea32dfa3ca9 Oct 30 17:03:05.937995 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:05.937974 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65f9f450-lxclp"] Oct 30 17:03:06.897679 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:06.897651 4522 generic.go:356] "Generic (PLEG): container finished" podID="81aedff4-a52f-4d6c-b0be-6905f9703a30" containerID="e15e04735a83948ae21f8c772527b25a72708264c120a6dc358ca3e7b41bebf0" exitCode=0 Oct 30 17:03:06.898088 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:06.897741 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f9f450-lxclp" event={"ID":"81aedff4-a52f-4d6c-b0be-6905f9703a30","Type":"ContainerDied","Data":"e15e04735a83948ae21f8c772527b25a72708264c120a6dc358ca3e7b41bebf0"} Oct 30 17:03:06.898088 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:06.897798 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f9f450-lxclp" event={"ID":"81aedff4-a52f-4d6c-b0be-6905f9703a30","Type":"ContainerStarted","Data":"c72f58e04acaa538bb2cd4e3182bddea15eb6c9d28eb539c73204ea32dfa3ca9"} Oct 30 17:03:07.251664 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:07.251628 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq"] Oct 30 17:03:07.255917 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:07.255894 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq" Oct 30 17:03:07.280550 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:07.280517 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq"] Oct 30 17:03:07.368522 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:07.368483 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77c327fe-cc04-492f-bc0e-efb01a43a67f-istio-data\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq\" (UID: \"77c327fe-cc04-492f-bc0e-efb01a43a67f\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq" Oct 30 17:03:07.368705 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:07.368571 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77c327fe-cc04-492f-bc0e-efb01a43a67f-workload-socket\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq\" (UID: \"77c327fe-cc04-492f-bc0e-efb01a43a67f\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq" Oct 30 17:03:07.368705 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:07.368608 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77c327fe-cc04-492f-bc0e-efb01a43a67f-istio-envoy\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq\" (UID: \"77c327fe-cc04-492f-bc0e-efb01a43a67f\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq" Oct 30 17:03:07.368705 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:07.368627 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77c327fe-cc04-492f-bc0e-efb01a43a67f-credential-socket\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq\" (UID: \"77c327fe-cc04-492f-bc0e-efb01a43a67f\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq" Oct 30 17:03:07.368705 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:07.368705 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77c327fe-cc04-492f-bc0e-efb01a43a67f-workload-certs\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq\" (UID: \"77c327fe-cc04-492f-bc0e-efb01a43a67f\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq" Oct 30 17:03:07.368876 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:07.368722 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77c327fe-cc04-492f-bc0e-efb01a43a67f-istiod-ca-cert\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq\" (UID: \"77c327fe-cc04-492f-bc0e-efb01a43a67f\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq" Oct 30 17:03:07.368876 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:07.368771 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77c327fe-cc04-492f-bc0e-efb01a43a67f-istio-token\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq\" (UID: \"77c327fe-cc04-492f-bc0e-efb01a43a67f\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq" Oct 30 17:03:07.368938 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:07.368910 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzvx7\" (UniqueName: \"kubernetes.io/projected/77c327fe-cc04-492f-bc0e-efb01a43a67f-kube-api-access-fzvx7\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq\" (UID: \"77c327fe-cc04-492f-bc0e-efb01a43a67f\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq" Oct 30 17:03:07.368972 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:07.368960 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77c327fe-cc04-492f-bc0e-efb01a43a67f-istio-podinfo\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq\" (UID: \"77c327fe-cc04-492f-bc0e-efb01a43a67f\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq" Oct 30 17:03:07.470295 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:07.470260 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77c327fe-cc04-492f-bc0e-efb01a43a67f-istio-data\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq\" (UID: \"77c327fe-cc04-492f-bc0e-efb01a43a67f\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq" Oct 30 17:03:07.470525 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:07.470346 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77c327fe-cc04-492f-bc0e-efb01a43a67f-workload-socket\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq\" (UID: \"77c327fe-cc04-492f-bc0e-efb01a43a67f\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq" Oct 30 17:03:07.470525 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:07.470386 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77c327fe-cc04-492f-bc0e-efb01a43a67f-istio-envoy\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq\" (UID: \"77c327fe-cc04-492f-bc0e-efb01a43a67f\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq" Oct 30 17:03:07.470525 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:07.470407 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77c327fe-cc04-492f-bc0e-efb01a43a67f-credential-socket\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq\" (UID: \"77c327fe-cc04-492f-bc0e-efb01a43a67f\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq" Oct 30 17:03:07.470525 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:07.470452 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77c327fe-cc04-492f-bc0e-efb01a43a67f-workload-certs\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq\" (UID: \"77c327fe-cc04-492f-bc0e-efb01a43a67f\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq" Oct 30 17:03:07.470525 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:07.470476 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77c327fe-cc04-492f-bc0e-efb01a43a67f-istiod-ca-cert\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq\" (UID: \"77c327fe-cc04-492f-bc0e-efb01a43a67f\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq" Oct 30 17:03:07.470525 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:07.470518 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77c327fe-cc04-492f-bc0e-efb01a43a67f-istio-token\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq\" (UID: \"77c327fe-cc04-492f-bc0e-efb01a43a67f\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq" Oct 30 17:03:07.470845 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:07.470695 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzvx7\" (UniqueName: \"kubernetes.io/projected/77c327fe-cc04-492f-bc0e-efb01a43a67f-kube-api-access-fzvx7\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq\" (UID: \"77c327fe-cc04-492f-bc0e-efb01a43a67f\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq" Oct 30 17:03:07.470845 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:07.470767 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77c327fe-cc04-492f-bc0e-efb01a43a67f-istio-podinfo\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq\" (UID: \"77c327fe-cc04-492f-bc0e-efb01a43a67f\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq" Oct 30 17:03:07.470845 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:07.470768 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77c327fe-cc04-492f-bc0e-efb01a43a67f-istio-data\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq\" (UID: \"77c327fe-cc04-492f-bc0e-efb01a43a67f\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq" Oct 30 17:03:07.470845 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:07.470822 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77c327fe-cc04-492f-bc0e-efb01a43a67f-workload-socket\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq\" (UID: \"77c327fe-cc04-492f-bc0e-efb01a43a67f\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq" Oct 30 17:03:07.471148 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:07.471123 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77c327fe-cc04-492f-bc0e-efb01a43a67f-credential-socket\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq\" (UID: \"77c327fe-cc04-492f-bc0e-efb01a43a67f\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq" Oct 30 17:03:07.471259 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:07.471238 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77c327fe-cc04-492f-bc0e-efb01a43a67f-istiod-ca-cert\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq\" (UID: \"77c327fe-cc04-492f-bc0e-efb01a43a67f\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq" Oct 30 17:03:07.471353 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:07.471334 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77c327fe-cc04-492f-bc0e-efb01a43a67f-workload-certs\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq\" (UID: \"77c327fe-cc04-492f-bc0e-efb01a43a67f\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq" Oct 30 17:03:07.472729 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:07.472707 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77c327fe-cc04-492f-bc0e-efb01a43a67f-istio-envoy\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq\" (UID: \"77c327fe-cc04-492f-bc0e-efb01a43a67f\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq" Oct 30 17:03:07.473105 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:07.473088 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77c327fe-cc04-492f-bc0e-efb01a43a67f-istio-podinfo\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq\" (UID: \"77c327fe-cc04-492f-bc0e-efb01a43a67f\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq" Oct 30 17:03:07.495830 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:07.495803 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77c327fe-cc04-492f-bc0e-efb01a43a67f-istio-token\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq\" (UID: \"77c327fe-cc04-492f-bc0e-efb01a43a67f\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq" Oct 30 17:03:07.495973 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:07.495897 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzvx7\" (UniqueName: \"kubernetes.io/projected/77c327fe-cc04-492f-bc0e-efb01a43a67f-kube-api-access-fzvx7\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq\" (UID: \"77c327fe-cc04-492f-bc0e-efb01a43a67f\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq" Oct 30 17:03:07.570462 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:07.570381 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq" Oct 30 17:03:07.934959 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:03:07.934911 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77c327fe_cc04_492f_bc0e_efb01a43a67f.slice/crio-99f143979e62bfac740520e8fa141397e6311d90a9a61b1da3ad699d48a3378c WatchSource:0}: Error finding container 99f143979e62bfac740520e8fa141397e6311d90a9a61b1da3ad699d48a3378c: Status 404 returned error can't find the container with id 99f143979e62bfac740520e8fa141397e6311d90a9a61b1da3ad699d48a3378c Oct 30 17:03:07.936052 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:07.936024 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq"] Oct 30 17:03:07.937979 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:07.937931 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:03:07.938114 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:07.938022 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:03:07.938114 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:07.938068 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:03:08.048648 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:08.048623 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f9f450-lxclp" Oct 30 17:03:08.084782 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:08.084740 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81aedff4-a52f-4d6c-b0be-6905f9703a30-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81aedff4-a52f-4d6c-b0be-6905f9703a30-maas-api-xks-ca\") pod \"81aedff4-a52f-4d6c-b0be-6905f9703a30\" (UID: \"81aedff4-a52f-4d6c-b0be-6905f9703a30\") " Oct 30 17:03:08.084984 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:08.084798 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81aedff4-a52f-4d6c-b0be-6905f9703a30-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81aedff4-a52f-4d6c-b0be-6905f9703a30-maas-api-openshift-ca\") pod \"81aedff4-a52f-4d6c-b0be-6905f9703a30\" (UID: \"81aedff4-a52f-4d6c-b0be-6905f9703a30\") " Oct 30 17:03:08.085049 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:08.085017 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81aedff4-a52f-4d6c-b0be-6905f9703a30-kube-api-access-g29mv\" (UniqueName: \"kubernetes.io/projected/81aedff4-a52f-4d6c-b0be-6905f9703a30-kube-api-access-g29mv\") pod \"81aedff4-a52f-4d6c-b0be-6905f9703a30\" (UID: \"81aedff4-a52f-4d6c-b0be-6905f9703a30\") " Oct 30 17:03:08.085270 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:08.085239 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81aedff4-a52f-4d6c-b0be-6905f9703a30-maas-api-openshift-ca" pod "81aedff4-a52f-4d6c-b0be-6905f9703a30" (UID: "81aedff4-a52f-4d6c-b0be-6905f9703a30"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:03:08.085717 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:08.085698 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81aedff4-a52f-4d6c-b0be-6905f9703a30-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:03:08.087012 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:08.086983 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81aedff4-a52f-4d6c-b0be-6905f9703a30-maas-api-xks-ca" pod "81aedff4-a52f-4d6c-b0be-6905f9703a30" (UID: "81aedff4-a52f-4d6c-b0be-6905f9703a30"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:03:08.087099 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:08.087065 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81aedff4-a52f-4d6c-b0be-6905f9703a30-kube-api-access-g29mv" pod "81aedff4-a52f-4d6c-b0be-6905f9703a30" (UID: "81aedff4-a52f-4d6c-b0be-6905f9703a30"). InnerVolumeSpecName "kube-api-access-g29mv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:03:08.187709 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:08.187602 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81aedff4-a52f-4d6c-b0be-6905f9703a30-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:03:08.187709 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:08.187645 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g29mv\" (UniqueName: \"kubernetes.io/projected/81aedff4-a52f-4d6c-b0be-6905f9703a30-kube-api-access-g29mv\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:03:08.927100 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:08.927019 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f9f450-lxclp" Oct 30 17:03:08.927100 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:08.927052 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f9f450-lxclp" event={"ID":"81aedff4-a52f-4d6c-b0be-6905f9703a30","Type":"ContainerDied","Data":"c72f58e04acaa538bb2cd4e3182bddea15eb6c9d28eb539c73204ea32dfa3ca9"} Oct 30 17:03:08.927345 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:08.927109 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c72f58e04acaa538bb2cd4e3182bddea15eb6c9d28eb539c73204ea32dfa3ca9" Oct 30 17:03:08.929035 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:08.929006 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq" event={"ID":"77c327fe-cc04-492f-bc0e-efb01a43a67f","Type":"ContainerStarted","Data":"5e7604ac9a3c4cbddd2348117d5f83dcef0bdf184cd82f06067a46d4d08d0fa2"} Oct 30 17:03:08.929167 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:08.929057 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq" event={"ID":"77c327fe-cc04-492f-bc0e-efb01a43a67f","Type":"ContainerStarted","Data":"99f143979e62bfac740520e8fa141397e6311d90a9a61b1da3ad699d48a3378c"} Oct 30 17:03:09.009331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:08.998619 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq" podStartSLOduration=1.998600757 podStartE2EDuration="1.998600757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:03:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:03:08.991483514 +0000 UTC m=+962.992238440" watchObservedRunningTime="2026-10-30 17:03:08.998600757 +0000 UTC m=+962.999355660" Oct 30 17:03:09.571065 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:09.571021 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq" Oct 30 17:03:09.572496 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:09.572467 4522 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 17:03:09.572627 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:09.572525 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq" podUID="77c327fe-cc04-492f-bc0e-efb01a43a67f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 17:03:09.734920 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:09.734885 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b48042d6-nbn57"] Oct 30 17:03:09.735538 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:09.735523 4522 cpu_manager.go:414] "containerMap: removing container" podUID="81aedff4-a52f-4d6c-b0be-6905f9703a30" containerName="revoke-keys" Oct 30 17:03:09.735579 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:09.735539 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81aedff4-a52f-4d6c-b0be-6905f9703a30" containerName="revoke-keys" Oct 30 17:03:09.735665 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:09.735656 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="81aedff4-a52f-4d6c-b0be-6905f9703a30" containerName="revoke-keys" Oct 30 17:03:09.739413 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:09.739394 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b48042d6-nbn57" Oct 30 17:03:09.759926 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:09.759896 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b48042d6-nbn57"] Oct 30 17:03:09.765395 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:09.765227 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:03:09.768213 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:09.768187 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:03:09.830194 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:09.830095 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e6051e6-4d38-43a6-a8ae-83a82263ac3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b48042d6-nbn57\" (UID: \"6e6051e6-4d38-43a6-a8ae-83a82263ac3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b48042d6-nbn57" Oct 30 17:03:09.830374 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:09.830284 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c2vb\" (UniqueName: \"kubernetes.io/projected/6e6051e6-4d38-43a6-a8ae-83a82263ac3f-kube-api-access-7c2vb\") pod \"maas-api-revoke-sub-b48042d6-nbn57\" (UID: \"6e6051e6-4d38-43a6-a8ae-83a82263ac3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b48042d6-nbn57" Oct 30 17:03:09.830615 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:09.830595 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e6051e6-4d38-43a6-a8ae-83a82263ac3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b48042d6-nbn57\" (UID: \"6e6051e6-4d38-43a6-a8ae-83a82263ac3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b48042d6-nbn57" Oct 30 17:03:09.936429 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:09.936375 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c2vb\" (UniqueName: \"kubernetes.io/projected/6e6051e6-4d38-43a6-a8ae-83a82263ac3f-kube-api-access-7c2vb\") pod \"maas-api-revoke-sub-b48042d6-nbn57\" (UID: \"6e6051e6-4d38-43a6-a8ae-83a82263ac3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b48042d6-nbn57" Oct 30 17:03:09.936857 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:09.936678 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e6051e6-4d38-43a6-a8ae-83a82263ac3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b48042d6-nbn57\" (UID: \"6e6051e6-4d38-43a6-a8ae-83a82263ac3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b48042d6-nbn57" Oct 30 17:03:09.936857 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:09.936768 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e6051e6-4d38-43a6-a8ae-83a82263ac3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b48042d6-nbn57\" (UID: \"6e6051e6-4d38-43a6-a8ae-83a82263ac3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b48042d6-nbn57" Oct 30 17:03:09.940495 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:09.940467 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e6051e6-4d38-43a6-a8ae-83a82263ac3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b48042d6-nbn57\" (UID: \"6e6051e6-4d38-43a6-a8ae-83a82263ac3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b48042d6-nbn57" Oct 30 17:03:09.947660 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:09.947634 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e6051e6-4d38-43a6-a8ae-83a82263ac3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b48042d6-nbn57\" (UID: \"6e6051e6-4d38-43a6-a8ae-83a82263ac3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b48042d6-nbn57" Oct 30 17:03:09.962758 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:09.962731 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c2vb\" (UniqueName: \"kubernetes.io/projected/6e6051e6-4d38-43a6-a8ae-83a82263ac3f-kube-api-access-7c2vb\") pod \"maas-api-revoke-sub-b48042d6-nbn57\" (UID: \"6e6051e6-4d38-43a6-a8ae-83a82263ac3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b48042d6-nbn57" Oct 30 17:03:10.053268 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:10.053237 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b48042d6-nbn57" Oct 30 17:03:10.384716 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:03:10.384654 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e6051e6_4d38_43a6_a8ae_83a82263ac3f.slice/crio-c3176d559408a0fee5ea34568b2ec4723c139dcf8a9e85e9b33b3036fc32d241 WatchSource:0}: Error finding container c3176d559408a0fee5ea34568b2ec4723c139dcf8a9e85e9b33b3036fc32d241: Status 404 returned error can't find the container with id c3176d559408a0fee5ea34568b2ec4723c139dcf8a9e85e9b33b3036fc32d241 Oct 30 17:03:10.387866 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:10.387838 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b48042d6-nbn57"] Oct 30 17:03:10.571935 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:10.571898 4522 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 17:03:10.572101 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:10.571952 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq" podUID="77c327fe-cc04-492f-bc0e-efb01a43a67f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 17:03:10.955563 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:10.955521 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b48042d6-nbn57" event={"ID":"6e6051e6-4d38-43a6-a8ae-83a82263ac3f","Type":"ContainerStarted","Data":"baf1aae4838cd5706963456dd3f421484c27d50c1f3b6ffb656a8ed073efbc3b"} Oct 30 17:03:10.955753 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:10.955594 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b48042d6-nbn57" event={"ID":"6e6051e6-4d38-43a6-a8ae-83a82263ac3f","Type":"ContainerStarted","Data":"c3176d559408a0fee5ea34568b2ec4723c139dcf8a9e85e9b33b3036fc32d241"} Oct 30 17:03:11.025755 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:11.025684 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b48042d6-nbn57" podStartSLOduration=2.025664729 podStartE2EDuration="2.025664729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:03:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:03:11.017667046 +0000 UTC m=+965.018421949" watchObservedRunningTime="2026-10-30 17:03:11.025664729 +0000 UTC m=+965.026419630" Oct 30 17:03:11.571002 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:11.570969 4522 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 17:03:11.571391 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:11.571023 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq" podUID="77c327fe-cc04-492f-bc0e-efb01a43a67f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 17:03:11.963776 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:11.963737 4522 generic.go:356] "Generic (PLEG): container finished" podID="6e6051e6-4d38-43a6-a8ae-83a82263ac3f" containerID="baf1aae4838cd5706963456dd3f421484c27d50c1f3b6ffb656a8ed073efbc3b" exitCode=0 Oct 30 17:03:11.963946 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:11.963818 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b48042d6-nbn57" event={"ID":"6e6051e6-4d38-43a6-a8ae-83a82263ac3f","Type":"ContainerDied","Data":"baf1aae4838cd5706963456dd3f421484c27d50c1f3b6ffb656a8ed073efbc3b"} Oct 30 17:03:12.571163 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:12.571122 4522 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 17:03:12.571690 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:12.571180 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq" podUID="77c327fe-cc04-492f-bc0e-efb01a43a67f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 17:03:12.955492 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:03:12.955455 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:03:13.168948 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:13.168923 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b48042d6-nbn57" Oct 30 17:03:13.328775 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:13.328666 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e6051e6-4d38-43a6-a8ae-83a82263ac3f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e6051e6-4d38-43a6-a8ae-83a82263ac3f-maas-api-openshift-ca\") pod \"6e6051e6-4d38-43a6-a8ae-83a82263ac3f\" (UID: \"6e6051e6-4d38-43a6-a8ae-83a82263ac3f\") " Oct 30 17:03:13.329120 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:13.329100 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e6051e6-4d38-43a6-a8ae-83a82263ac3f-kube-api-access-7c2vb\" (UniqueName: \"kubernetes.io/projected/6e6051e6-4d38-43a6-a8ae-83a82263ac3f-kube-api-access-7c2vb\") pod \"6e6051e6-4d38-43a6-a8ae-83a82263ac3f\" (UID: \"6e6051e6-4d38-43a6-a8ae-83a82263ac3f\") " Oct 30 17:03:13.329341 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:13.329324 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e6051e6-4d38-43a6-a8ae-83a82263ac3f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e6051e6-4d38-43a6-a8ae-83a82263ac3f-maas-api-xks-ca\") pod \"6e6051e6-4d38-43a6-a8ae-83a82263ac3f\" (UID: \"6e6051e6-4d38-43a6-a8ae-83a82263ac3f\") " Oct 30 17:03:13.330668 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:13.330625 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e6051e6-4d38-43a6-a8ae-83a82263ac3f-maas-api-openshift-ca" pod "6e6051e6-4d38-43a6-a8ae-83a82263ac3f" (UID: "6e6051e6-4d38-43a6-a8ae-83a82263ac3f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:03:13.333226 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:13.333189 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e6051e6-4d38-43a6-a8ae-83a82263ac3f-kube-api-access-7c2vb" pod "6e6051e6-4d38-43a6-a8ae-83a82263ac3f" (UID: "6e6051e6-4d38-43a6-a8ae-83a82263ac3f"). InnerVolumeSpecName "kube-api-access-7c2vb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:03:13.348250 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:13.348218 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e6051e6-4d38-43a6-a8ae-83a82263ac3f-maas-api-xks-ca" pod "6e6051e6-4d38-43a6-a8ae-83a82263ac3f" (UID: "6e6051e6-4d38-43a6-a8ae-83a82263ac3f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:03:13.430996 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:13.430965 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e6051e6-4d38-43a6-a8ae-83a82263ac3f-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:03:13.430996 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:13.430995 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c2vb\" (UniqueName: \"kubernetes.io/projected/6e6051e6-4d38-43a6-a8ae-83a82263ac3f-kube-api-access-7c2vb\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:03:13.431191 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:13.431006 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e6051e6-4d38-43a6-a8ae-83a82263ac3f-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:03:13.571645 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:13.571598 4522 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 17:03:13.572054 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:13.571674 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq" podUID="77c327fe-cc04-492f-bc0e-efb01a43a67f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 17:03:13.980513 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:13.980484 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b48042d6-nbn57" Oct 30 17:03:13.980694 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:13.980506 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b48042d6-nbn57" event={"ID":"6e6051e6-4d38-43a6-a8ae-83a82263ac3f","Type":"ContainerDied","Data":"c3176d559408a0fee5ea34568b2ec4723c139dcf8a9e85e9b33b3036fc32d241"} Oct 30 17:03:13.980694 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:13.980597 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3176d559408a0fee5ea34568b2ec4723c139dcf8a9e85e9b33b3036fc32d241" Oct 30 17:03:14.571399 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:14.571363 4522 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 17:03:14.571576 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:14.571437 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq" podUID="77c327fe-cc04-492f-bc0e-efb01a43a67f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 17:03:15.582329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:15.581536 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq" Oct 30 17:03:15.586332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:15.582949 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq" Oct 30 17:03:15.589168 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:15.586516 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq" Oct 30 17:03:17.663445 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:17.663402 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" podUID="8a6a85ad-c944-450f-b0bb-e3b64b0f56f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:19.362949 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:19.362914 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30d09eab-w7nmx"] Oct 30 17:03:19.363562 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:19.363548 4522 cpu_manager.go:414] "containerMap: removing container" podUID="6e6051e6-4d38-43a6-a8ae-83a82263ac3f" containerName="revoke-keys" Oct 30 17:03:19.363612 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:19.363562 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e6051e6-4d38-43a6-a8ae-83a82263ac3f" containerName="revoke-keys" Oct 30 17:03:19.363667 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:19.363658 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e6051e6-4d38-43a6-a8ae-83a82263ac3f" containerName="revoke-keys" Oct 30 17:03:19.367318 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:19.367291 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30d09eab-w7nmx" Oct 30 17:03:19.387230 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:19.387199 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30d09eab-w7nmx"] Oct 30 17:03:19.395515 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:19.395489 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:03:19.396222 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:19.396198 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:03:19.445763 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:19.445733 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d232f5be-69ed-4150-908c-b5a13b78e223-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30d09eab-w7nmx\" (UID: \"d232f5be-69ed-4150-908c-b5a13b78e223\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30d09eab-w7nmx" Oct 30 17:03:19.445944 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:19.445922 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d232f5be-69ed-4150-908c-b5a13b78e223-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30d09eab-w7nmx\" (UID: \"d232f5be-69ed-4150-908c-b5a13b78e223\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30d09eab-w7nmx" Oct 30 17:03:19.446091 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:19.446068 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5rls\" (UniqueName: \"kubernetes.io/projected/d232f5be-69ed-4150-908c-b5a13b78e223-kube-api-access-p5rls\") pod \"maas-api-revoke-sub-30d09eab-w7nmx\" (UID: \"d232f5be-69ed-4150-908c-b5a13b78e223\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30d09eab-w7nmx" Oct 30 17:03:19.547479 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:19.547444 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d232f5be-69ed-4150-908c-b5a13b78e223-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30d09eab-w7nmx\" (UID: \"d232f5be-69ed-4150-908c-b5a13b78e223\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30d09eab-w7nmx" Oct 30 17:03:19.547661 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:19.547619 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d232f5be-69ed-4150-908c-b5a13b78e223-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30d09eab-w7nmx\" (UID: \"d232f5be-69ed-4150-908c-b5a13b78e223\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30d09eab-w7nmx" Oct 30 17:03:19.547769 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:19.547751 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5rls\" (UniqueName: \"kubernetes.io/projected/d232f5be-69ed-4150-908c-b5a13b78e223-kube-api-access-p5rls\") pod \"maas-api-revoke-sub-30d09eab-w7nmx\" (UID: \"d232f5be-69ed-4150-908c-b5a13b78e223\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30d09eab-w7nmx" Oct 30 17:03:19.548547 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:19.548515 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d232f5be-69ed-4150-908c-b5a13b78e223-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30d09eab-w7nmx\" (UID: \"d232f5be-69ed-4150-908c-b5a13b78e223\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30d09eab-w7nmx" Oct 30 17:03:19.550175 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:19.550153 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d232f5be-69ed-4150-908c-b5a13b78e223-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30d09eab-w7nmx\" (UID: \"d232f5be-69ed-4150-908c-b5a13b78e223\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30d09eab-w7nmx" Oct 30 17:03:19.573390 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:19.573363 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5rls\" (UniqueName: \"kubernetes.io/projected/d232f5be-69ed-4150-908c-b5a13b78e223-kube-api-access-p5rls\") pod \"maas-api-revoke-sub-30d09eab-w7nmx\" (UID: \"d232f5be-69ed-4150-908c-b5a13b78e223\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30d09eab-w7nmx" Oct 30 17:03:19.681184 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:19.681092 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30d09eab-w7nmx" Oct 30 17:03:19.986944 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:03:19.986909 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd232f5be_69ed_4150_908c_b5a13b78e223.slice/crio-40544a40ddca8d09ef452599efc68abbe0eca849ad0b82c18482dc207af14fbe WatchSource:0}: Error finding container 40544a40ddca8d09ef452599efc68abbe0eca849ad0b82c18482dc207af14fbe: Status 404 returned error can't find the container with id 40544a40ddca8d09ef452599efc68abbe0eca849ad0b82c18482dc207af14fbe Oct 30 17:03:19.989578 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:19.989551 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30d09eab-w7nmx"] Oct 30 17:03:20.064466 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:20.064429 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30d09eab-w7nmx" event={"ID":"d232f5be-69ed-4150-908c-b5a13b78e223","Type":"ContainerStarted","Data":"40544a40ddca8d09ef452599efc68abbe0eca849ad0b82c18482dc207af14fbe"} Oct 30 17:03:21.073581 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:21.073551 4522 generic.go:356] "Generic (PLEG): container finished" podID="d232f5be-69ed-4150-908c-b5a13b78e223" containerID="abd24e3c6e73ec455d869be2d32619c6b0f0be4dcc234aa49bfdd0d9715b1018" exitCode=0 Oct 30 17:03:21.073959 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:21.073635 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30d09eab-w7nmx" event={"ID":"d232f5be-69ed-4150-908c-b5a13b78e223","Type":"ContainerDied","Data":"abd24e3c6e73ec455d869be2d32619c6b0f0be4dcc234aa49bfdd0d9715b1018"} Oct 30 17:03:22.260433 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:22.260398 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30d09eab-w7nmx" Oct 30 17:03:22.388545 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:22.388444 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d232f5be-69ed-4150-908c-b5a13b78e223-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d232f5be-69ed-4150-908c-b5a13b78e223-maas-api-xks-ca\") pod \"d232f5be-69ed-4150-908c-b5a13b78e223\" (UID: \"d232f5be-69ed-4150-908c-b5a13b78e223\") " Oct 30 17:03:22.388720 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:22.388566 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d232f5be-69ed-4150-908c-b5a13b78e223-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d232f5be-69ed-4150-908c-b5a13b78e223-maas-api-openshift-ca\") pod \"d232f5be-69ed-4150-908c-b5a13b78e223\" (UID: \"d232f5be-69ed-4150-908c-b5a13b78e223\") " Oct 30 17:03:22.388796 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:22.388751 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d232f5be-69ed-4150-908c-b5a13b78e223-kube-api-access-p5rls\" (UniqueName: \"kubernetes.io/projected/d232f5be-69ed-4150-908c-b5a13b78e223-kube-api-access-p5rls\") pod \"d232f5be-69ed-4150-908c-b5a13b78e223\" (UID: \"d232f5be-69ed-4150-908c-b5a13b78e223\") " Oct 30 17:03:22.388950 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:22.388921 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d232f5be-69ed-4150-908c-b5a13b78e223-maas-api-openshift-ca" pod "d232f5be-69ed-4150-908c-b5a13b78e223" (UID: "d232f5be-69ed-4150-908c-b5a13b78e223"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:03:22.389443 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:22.389420 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d232f5be-69ed-4150-908c-b5a13b78e223-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:03:22.391103 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:22.391077 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d232f5be-69ed-4150-908c-b5a13b78e223-maas-api-xks-ca" pod "d232f5be-69ed-4150-908c-b5a13b78e223" (UID: "d232f5be-69ed-4150-908c-b5a13b78e223"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:03:22.391103 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:22.391094 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d232f5be-69ed-4150-908c-b5a13b78e223-kube-api-access-p5rls" pod "d232f5be-69ed-4150-908c-b5a13b78e223" (UID: "d232f5be-69ed-4150-908c-b5a13b78e223"). InnerVolumeSpecName "kube-api-access-p5rls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:03:22.491259 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:22.491217 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d232f5be-69ed-4150-908c-b5a13b78e223-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:03:22.491259 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:22.491251 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5rls\" (UniqueName: \"kubernetes.io/projected/d232f5be-69ed-4150-908c-b5a13b78e223-kube-api-access-p5rls\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:03:23.088633 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:23.088605 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30d09eab-w7nmx" Oct 30 17:03:23.088806 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:23.088628 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30d09eab-w7nmx" event={"ID":"d232f5be-69ed-4150-908c-b5a13b78e223","Type":"ContainerDied","Data":"40544a40ddca8d09ef452599efc68abbe0eca849ad0b82c18482dc207af14fbe"} Oct 30 17:03:23.088806 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:23.088680 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40544a40ddca8d09ef452599efc68abbe0eca849ad0b82c18482dc207af14fbe" Oct 30 17:03:25.163295 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:25.163265 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6edf9460-x8x96"] Oct 30 17:03:25.164034 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:25.164014 4522 cpu_manager.go:414] "containerMap: removing container" podUID="d232f5be-69ed-4150-908c-b5a13b78e223" containerName="revoke-keys" Oct 30 17:03:25.164034 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:25.164033 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d232f5be-69ed-4150-908c-b5a13b78e223" containerName="revoke-keys" Oct 30 17:03:25.164173 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:25.164131 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="d232f5be-69ed-4150-908c-b5a13b78e223" containerName="revoke-keys" Oct 30 17:03:25.167784 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:25.167770 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6edf9460-x8x96" Oct 30 17:03:25.199742 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:25.199709 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:03:25.199981 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:25.199867 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:03:25.207880 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:25.207841 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6edf9460-x8x96"] Oct 30 17:03:25.228387 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:25.228355 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp5c4\" (UniqueName: \"kubernetes.io/projected/63b543bd-6567-43b7-8f6b-138ae297ddaa-kube-api-access-wp5c4\") pod \"maas-api-revoke-sub-6edf9460-x8x96\" (UID: \"63b543bd-6567-43b7-8f6b-138ae297ddaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6edf9460-x8x96" Oct 30 17:03:25.228703 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:25.228658 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63b543bd-6567-43b7-8f6b-138ae297ddaa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6edf9460-x8x96\" (UID: \"63b543bd-6567-43b7-8f6b-138ae297ddaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6edf9460-x8x96" Oct 30 17:03:25.229124 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:25.229099 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63b543bd-6567-43b7-8f6b-138ae297ddaa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6edf9460-x8x96\" (UID: \"63b543bd-6567-43b7-8f6b-138ae297ddaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6edf9460-x8x96" Oct 30 17:03:25.330649 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:25.330610 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp5c4\" (UniqueName: \"kubernetes.io/projected/63b543bd-6567-43b7-8f6b-138ae297ddaa-kube-api-access-wp5c4\") pod \"maas-api-revoke-sub-6edf9460-x8x96\" (UID: \"63b543bd-6567-43b7-8f6b-138ae297ddaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6edf9460-x8x96" Oct 30 17:03:25.330825 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:25.330767 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63b543bd-6567-43b7-8f6b-138ae297ddaa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6edf9460-x8x96\" (UID: \"63b543bd-6567-43b7-8f6b-138ae297ddaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6edf9460-x8x96" Oct 30 17:03:25.330890 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:25.330865 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63b543bd-6567-43b7-8f6b-138ae297ddaa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6edf9460-x8x96\" (UID: \"63b543bd-6567-43b7-8f6b-138ae297ddaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6edf9460-x8x96" Oct 30 17:03:25.331528 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:25.331507 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63b543bd-6567-43b7-8f6b-138ae297ddaa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6edf9460-x8x96\" (UID: \"63b543bd-6567-43b7-8f6b-138ae297ddaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6edf9460-x8x96" Oct 30 17:03:25.333171 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:25.333154 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63b543bd-6567-43b7-8f6b-138ae297ddaa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6edf9460-x8x96\" (UID: \"63b543bd-6567-43b7-8f6b-138ae297ddaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6edf9460-x8x96" Oct 30 17:03:25.355718 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:25.355687 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp5c4\" (UniqueName: \"kubernetes.io/projected/63b543bd-6567-43b7-8f6b-138ae297ddaa-kube-api-access-wp5c4\") pod \"maas-api-revoke-sub-6edf9460-x8x96\" (UID: \"63b543bd-6567-43b7-8f6b-138ae297ddaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6edf9460-x8x96" Oct 30 17:03:25.482617 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:25.482579 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6edf9460-x8x96" Oct 30 17:03:25.802322 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:25.799601 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6edf9460-x8x96"] Oct 30 17:03:26.112432 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:26.112389 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6edf9460-x8x96" event={"ID":"63b543bd-6567-43b7-8f6b-138ae297ddaa","Type":"ContainerStarted","Data":"dcc95aa156120478df9a2cfd89333305cbb129bba73539cef4688079417f0aa7"} Oct 30 17:03:26.159972 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:26.159926 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6edf9460-x8x96" podStartSLOduration=1.15991416 podStartE2EDuration="1.15991416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:03:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:03:26.15944687 +0000 UTC m=+980.160201791" watchObservedRunningTime="2026-10-30 17:03:26.15991416 +0000 UTC m=+980.160669062" Oct 30 17:03:27.120667 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:27.120634 4522 generic.go:356] "Generic (PLEG): container finished" podID="63b543bd-6567-43b7-8f6b-138ae297ddaa" containerID="1c5908797b81a32f971ba1ae8d082b8cf8e8acca8c8fa35e5542a6c2cc354b26" exitCode=0 Oct 30 17:03:27.121059 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:27.120701 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6edf9460-x8x96" event={"ID":"63b543bd-6567-43b7-8f6b-138ae297ddaa","Type":"ContainerDied","Data":"1c5908797b81a32f971ba1ae8d082b8cf8e8acca8c8fa35e5542a6c2cc354b26"} Oct 30 17:03:28.285574 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:28.285547 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6edf9460-x8x96" Oct 30 17:03:28.373719 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:28.373680 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63b543bd-6567-43b7-8f6b-138ae297ddaa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63b543bd-6567-43b7-8f6b-138ae297ddaa-maas-api-openshift-ca\") pod \"63b543bd-6567-43b7-8f6b-138ae297ddaa\" (UID: \"63b543bd-6567-43b7-8f6b-138ae297ddaa\") " Oct 30 17:03:28.373916 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:28.373808 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63b543bd-6567-43b7-8f6b-138ae297ddaa-kube-api-access-wp5c4\" (UniqueName: \"kubernetes.io/projected/63b543bd-6567-43b7-8f6b-138ae297ddaa-kube-api-access-wp5c4\") pod \"63b543bd-6567-43b7-8f6b-138ae297ddaa\" (UID: \"63b543bd-6567-43b7-8f6b-138ae297ddaa\") " Oct 30 17:03:28.374051 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:28.374030 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63b543bd-6567-43b7-8f6b-138ae297ddaa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63b543bd-6567-43b7-8f6b-138ae297ddaa-maas-api-xks-ca\") pod \"63b543bd-6567-43b7-8f6b-138ae297ddaa\" (UID: \"63b543bd-6567-43b7-8f6b-138ae297ddaa\") " Oct 30 17:03:28.374124 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:28.374088 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63b543bd-6567-43b7-8f6b-138ae297ddaa-maas-api-openshift-ca" pod "63b543bd-6567-43b7-8f6b-138ae297ddaa" (UID: "63b543bd-6567-43b7-8f6b-138ae297ddaa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:03:28.374725 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:28.374697 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63b543bd-6567-43b7-8f6b-138ae297ddaa-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:03:28.376159 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:28.376125 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63b543bd-6567-43b7-8f6b-138ae297ddaa-kube-api-access-wp5c4" pod "63b543bd-6567-43b7-8f6b-138ae297ddaa" (UID: "63b543bd-6567-43b7-8f6b-138ae297ddaa"). InnerVolumeSpecName "kube-api-access-wp5c4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:03:28.376580 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:28.376544 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63b543bd-6567-43b7-8f6b-138ae297ddaa-maas-api-xks-ca" pod "63b543bd-6567-43b7-8f6b-138ae297ddaa" (UID: "63b543bd-6567-43b7-8f6b-138ae297ddaa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:03:28.476262 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:28.476219 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wp5c4\" (UniqueName: \"kubernetes.io/projected/63b543bd-6567-43b7-8f6b-138ae297ddaa-kube-api-access-wp5c4\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:03:28.476262 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:28.476254 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63b543bd-6567-43b7-8f6b-138ae297ddaa-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:03:29.137200 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:29.137166 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6edf9460-x8x96" Oct 30 17:03:29.137433 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:29.137205 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6edf9460-x8x96" event={"ID":"63b543bd-6567-43b7-8f6b-138ae297ddaa","Type":"ContainerDied","Data":"dcc95aa156120478df9a2cfd89333305cbb129bba73539cef4688079417f0aa7"} Oct 30 17:03:29.137433 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:29.137258 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dcc95aa156120478df9a2cfd89333305cbb129bba73539cef4688079417f0aa7" Oct 30 17:03:32.179294 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:32.178540 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" podUID="2f1e3bbb-ab1c-455d-a05d-d62f72c5d009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:32.226371 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:32.215857 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" podUID="9bfbc6d3-91fd-41a7-9585-40a8f252f743" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 30 17:03:32.355197 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:32.355141 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" podUID="c9ad71b7-ad90-4c8c-bf27-0f10b23f1593" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:32.452512 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:32.452470 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" podUID="a3e5e732-2c67-4377-9755-e2178bf33772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:32.558529 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:32.558465 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" podUID="75321c5d-8bf8-45c6-8811-81981fe3cad4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:32.656484 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:32.656435 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" podUID="8a6a85ad-c944-450f-b0bb-e3b64b0f56f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:32.866503 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:32.866396 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" podUID="0b832781-3d41-4496-aa86-e4d92792e590" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:32.892742 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:32.892682 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" podUID="4b4ad752-e053-4cea-a3bc-fe2be6f92021" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:32.906333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:32.902388 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" podUID="4360e9dc-c001-4102-8af6-b55b692fcf2c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:33.151089 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:33.150680 4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" podUID="2f1e3bbb-ab1c-455d-a05d-d62f72c5d009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:33.189468 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:33.187434 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" podUID="2f1e3bbb-ab1c-455d-a05d-d62f72c5d009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 17:03:33.189468 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:33.188202 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" Oct 30 17:03:33.221332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:33.220391 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" podUID="9bfbc6d3-91fd-41a7-9585-40a8f252f743" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:33.221332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:33.221141 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" Oct 30 17:03:33.344331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:33.339440 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" podUID="43ca764d-ae0e-4858-9841-5bbd0ffdbb85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:33.367916 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:33.361838 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" podUID="c9ad71b7-ad90-4c8c-bf27-0f10b23f1593" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:33.403430 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:33.378865 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" Oct 30 17:03:33.469356 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:33.442801 4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" podUID="0b832781-3d41-4496-aa86-e4d92792e590" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": context deadline exceeded" Oct 30 17:03:33.509349 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:33.475432 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" podUID="a3e5e732-2c67-4377-9755-e2178bf33772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:33.509349 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:33.476082 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" Oct 30 17:03:33.611333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:33.573378 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" podUID="75321c5d-8bf8-45c6-8811-81981fe3cad4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:33.611333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:33.573805 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" Oct 30 17:03:33.696704 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:33.664813 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" podUID="8a6a85ad-c944-450f-b0bb-e3b64b0f56f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:33.696704 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:33.689354 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" Oct 30 17:03:33.927586 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:33.920429 4522 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:03:33.927586 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:33.920537 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn" podUID="015c4531-2766-4522-acf0-0427f1ba0a3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:34.076890 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:34.076834 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" podUID="0b832781-3d41-4496-aa86-e4d92792e590" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 17:03:34.408574 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:34.091529 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" podUID="4360e9dc-c001-4102-8af6-b55b692fcf2c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Oct 30 17:03:34.440511 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:34.173661 4522 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:03:34.440790 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:34.440760 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp" podUID="9eb030f0-e0a2-446e-b35f-4c497d9a3aa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:34.440943 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:34.254089 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" podUID="2f1e3bbb-ab1c-455d-a05d-d62f72c5d009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:34.441283 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:33.908550 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" podUID="4b4ad752-e053-4cea-a3bc-fe2be6f92021" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:34.441630 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:34.254159 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" podUID="9bfbc6d3-91fd-41a7-9585-40a8f252f743" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:34.441940 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:34.348411 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" podUID="43ca764d-ae0e-4858-9841-5bbd0ffdbb85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:34.442320 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:34.442277 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" podUID="c9ad71b7-ad90-4c8c-bf27-0f10b23f1593" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 17:03:34.481839 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:34.480238 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" podUID="a3e5e732-2c67-4377-9755-e2178bf33772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:34.614453 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:34.614398 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" podUID="75321c5d-8bf8-45c6-8811-81981fe3cad4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 17:03:34.648365 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:34.648294 4522 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:03:34.648659 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:34.648616 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw" podUID="0e123c50-767e-478c-a7e5-0d2476ccbc3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:34.667508 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:34.667404 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" podUID="8a6a85ad-c944-450f-b0bb-e3b64b0f56f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:34.966386 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:34.924611 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" Oct 30 17:03:35.029790 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:35.029757 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" Oct 30 17:03:35.038493 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:35.038454 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" Oct 30 17:03:35.038806 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:35.038788 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" Oct 30 17:03:35.038989 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:35.038976 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" Oct 30 17:03:35.286242 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:35.286155 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" Oct 30 17:03:35.287967 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:35.287945 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" Oct 30 17:03:35.288860 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:35.288807 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" Oct 30 17:03:35.292104 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:35.292061 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" Oct 30 17:03:35.293377 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:35.292847 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" Oct 30 17:03:35.293377 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:35.293125 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" Oct 30 17:03:35.294403 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:35.294385 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" Oct 30 17:03:35.295383 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:35.295365 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" Oct 30 17:03:35.317346 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:35.317318 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" Oct 30 17:03:38.001042 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:38.001006 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs"] Oct 30 17:03:38.001947 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:38.001926 4522 cpu_manager.go:414] "containerMap: removing container" podUID="63b543bd-6567-43b7-8f6b-138ae297ddaa" containerName="revoke-keys" Oct 30 17:03:38.001991 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:38.001952 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63b543bd-6567-43b7-8f6b-138ae297ddaa" containerName="revoke-keys" Oct 30 17:03:38.002131 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:38.002121 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="63b543bd-6567-43b7-8f6b-138ae297ddaa" containerName="revoke-keys" Oct 30 17:03:38.005987 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:38.005969 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs" Oct 30 17:03:38.041463 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:38.041435 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs"] Oct 30 17:03:38.079886 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:38.079854 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rwxw\" (UniqueName: \"kubernetes.io/projected/89585321-089e-477a-a6fa-d8b5a8478639-kube-api-access-7rwxw\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs\" (UID: \"89585321-089e-477a-a6fa-d8b5a8478639\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs" Oct 30 17:03:38.080085 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:38.079909 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89585321-089e-477a-a6fa-d8b5a8478639-workload-certs\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs\" (UID: \"89585321-089e-477a-a6fa-d8b5a8478639\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs" Oct 30 17:03:38.080142 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:38.080076 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89585321-089e-477a-a6fa-d8b5a8478639-istio-podinfo\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs\" (UID: \"89585321-089e-477a-a6fa-d8b5a8478639\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs" Oct 30 17:03:38.080142 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:38.080126 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89585321-089e-477a-a6fa-d8b5a8478639-workload-socket\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs\" (UID: \"89585321-089e-477a-a6fa-d8b5a8478639\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs" Oct 30 17:03:38.080333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:38.080282 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89585321-089e-477a-a6fa-d8b5a8478639-istio-envoy\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs\" (UID: \"89585321-089e-477a-a6fa-d8b5a8478639\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs" Oct 30 17:03:38.080413 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:38.080340 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89585321-089e-477a-a6fa-d8b5a8478639-istiod-ca-cert\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs\" (UID: \"89585321-089e-477a-a6fa-d8b5a8478639\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs" Oct 30 17:03:38.080458 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:38.080419 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89585321-089e-477a-a6fa-d8b5a8478639-istio-token\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs\" (UID: \"89585321-089e-477a-a6fa-d8b5a8478639\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs" Oct 30 17:03:38.080502 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:38.080484 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89585321-089e-477a-a6fa-d8b5a8478639-credential-socket\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs\" (UID: \"89585321-089e-477a-a6fa-d8b5a8478639\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs" Oct 30 17:03:38.080579 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:38.080560 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89585321-089e-477a-a6fa-d8b5a8478639-istio-data\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs\" (UID: \"89585321-089e-477a-a6fa-d8b5a8478639\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs" Oct 30 17:03:38.181603 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:38.181570 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89585321-089e-477a-a6fa-d8b5a8478639-istio-token\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs\" (UID: \"89585321-089e-477a-a6fa-d8b5a8478639\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs" Oct 30 17:03:38.181785 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:38.181644 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89585321-089e-477a-a6fa-d8b5a8478639-credential-socket\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs\" (UID: \"89585321-089e-477a-a6fa-d8b5a8478639\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs" Oct 30 17:03:38.181785 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:38.181679 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89585321-089e-477a-a6fa-d8b5a8478639-istio-data\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs\" (UID: \"89585321-089e-477a-a6fa-d8b5a8478639\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs" Oct 30 17:03:38.181785 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:38.181699 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rwxw\" (UniqueName: \"kubernetes.io/projected/89585321-089e-477a-a6fa-d8b5a8478639-kube-api-access-7rwxw\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs\" (UID: \"89585321-089e-477a-a6fa-d8b5a8478639\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs" Oct 30 17:03:38.181785 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:38.181728 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89585321-089e-477a-a6fa-d8b5a8478639-workload-certs\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs\" (UID: \"89585321-089e-477a-a6fa-d8b5a8478639\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs" Oct 30 17:03:38.182017 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:38.181845 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89585321-089e-477a-a6fa-d8b5a8478639-istio-podinfo\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs\" (UID: \"89585321-089e-477a-a6fa-d8b5a8478639\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs" Oct 30 17:03:38.182017 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:38.181890 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89585321-089e-477a-a6fa-d8b5a8478639-workload-socket\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs\" (UID: \"89585321-089e-477a-a6fa-d8b5a8478639\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs" Oct 30 17:03:38.182017 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:38.181993 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89585321-089e-477a-a6fa-d8b5a8478639-istio-envoy\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs\" (UID: \"89585321-089e-477a-a6fa-d8b5a8478639\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs" Oct 30 17:03:38.182190 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:38.182032 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89585321-089e-477a-a6fa-d8b5a8478639-istiod-ca-cert\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs\" (UID: \"89585321-089e-477a-a6fa-d8b5a8478639\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs" Oct 30 17:03:38.182190 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:38.182065 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89585321-089e-477a-a6fa-d8b5a8478639-credential-socket\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs\" (UID: \"89585321-089e-477a-a6fa-d8b5a8478639\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs" Oct 30 17:03:38.182299 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:38.182184 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89585321-089e-477a-a6fa-d8b5a8478639-workload-certs\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs\" (UID: \"89585321-089e-477a-a6fa-d8b5a8478639\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs" Oct 30 17:03:38.182299 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:38.182170 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89585321-089e-477a-a6fa-d8b5a8478639-istio-data\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs\" (UID: \"89585321-089e-477a-a6fa-d8b5a8478639\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs" Oct 30 17:03:38.182426 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:38.182302 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89585321-089e-477a-a6fa-d8b5a8478639-workload-socket\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs\" (UID: \"89585321-089e-477a-a6fa-d8b5a8478639\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs" Oct 30 17:03:38.182634 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:38.182615 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89585321-089e-477a-a6fa-d8b5a8478639-istiod-ca-cert\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs\" (UID: \"89585321-089e-477a-a6fa-d8b5a8478639\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs" Oct 30 17:03:38.184137 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:38.184117 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89585321-089e-477a-a6fa-d8b5a8478639-istio-envoy\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs\" (UID: \"89585321-089e-477a-a6fa-d8b5a8478639\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs" Oct 30 17:03:38.184485 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:38.184463 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89585321-089e-477a-a6fa-d8b5a8478639-istio-podinfo\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs\" (UID: \"89585321-089e-477a-a6fa-d8b5a8478639\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs" Oct 30 17:03:38.206701 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:38.206673 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89585321-089e-477a-a6fa-d8b5a8478639-istio-token\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs\" (UID: \"89585321-089e-477a-a6fa-d8b5a8478639\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs" Oct 30 17:03:38.206811 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:38.206789 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rwxw\" (UniqueName: \"kubernetes.io/projected/89585321-089e-477a-a6fa-d8b5a8478639-kube-api-access-7rwxw\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs\" (UID: \"89585321-089e-477a-a6fa-d8b5a8478639\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs" Oct 30 17:03:38.320199 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:38.320106 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs" Oct 30 17:03:38.659835 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:03:38.659791 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89585321_089e_477a_a6fa_d8b5a8478639.slice/crio-79a3d622a78ff539f26306bc0f8665f57bee81b352ba897e301ed66d30ccd49a WatchSource:0}: Error finding container 79a3d622a78ff539f26306bc0f8665f57bee81b352ba897e301ed66d30ccd49a: Status 404 returned error can't find the container with id 79a3d622a78ff539f26306bc0f8665f57bee81b352ba897e301ed66d30ccd49a Oct 30 17:03:38.662481 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:38.662454 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs"] Oct 30 17:03:38.663172 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:38.663137 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:03:38.663234 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:38.663218 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:03:38.663276 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:38.663263 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:03:38.912200 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:38.912101 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs" event={"ID":"89585321-089e-477a-a6fa-d8b5a8478639","Type":"ContainerStarted","Data":"c2d48bf0a257b79b6553b3f993854a291adf4d0a65f772ef1503224773dc8cdb"} Oct 30 17:03:38.912200 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:38.912160 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs" event={"ID":"89585321-089e-477a-a6fa-d8b5a8478639","Type":"ContainerStarted","Data":"79a3d622a78ff539f26306bc0f8665f57bee81b352ba897e301ed66d30ccd49a"} Oct 30 17:03:38.964476 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:38.964424 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs" podStartSLOduration=1.9644006539999999 podStartE2EDuration="1.964400654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:03:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:03:38.960110795 +0000 UTC m=+992.960865720" watchObservedRunningTime="2026-10-30 17:03:38.964400654 +0000 UTC m=+992.965155554" Oct 30 17:03:39.320388 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:39.320348 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs" Oct 30 17:03:39.322480 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:39.322449 4522 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 17:03:39.322616 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:39.322499 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs" podUID="89585321-089e-477a-a6fa-d8b5a8478639" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 17:03:39.743350 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:39.743315 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e802cc01-qxkv9"] Oct 30 17:03:39.750043 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:39.750005 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e802cc01-qxkv9" Oct 30 17:03:39.768776 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:39.768748 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e802cc01-qxkv9"] Oct 30 17:03:39.776516 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:39.776489 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:03:39.776646 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:39.776567 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:03:39.923570 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:39.923532 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89112b19-c140-40c2-8bd3-474c37608706-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e802cc01-qxkv9\" (UID: \"89112b19-c140-40c2-8bd3-474c37608706\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e802cc01-qxkv9" Oct 30 17:03:39.923746 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:39.923594 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89112b19-c140-40c2-8bd3-474c37608706-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e802cc01-qxkv9\" (UID: \"89112b19-c140-40c2-8bd3-474c37608706\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e802cc01-qxkv9" Oct 30 17:03:39.923746 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:39.923706 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72z88\" (UniqueName: \"kubernetes.io/projected/89112b19-c140-40c2-8bd3-474c37608706-kube-api-access-72z88\") pod \"maas-api-revoke-sub-e802cc01-qxkv9\" (UID: \"89112b19-c140-40c2-8bd3-474c37608706\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e802cc01-qxkv9" Oct 30 17:03:40.025708 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:40.025614 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72z88\" (UniqueName: \"kubernetes.io/projected/89112b19-c140-40c2-8bd3-474c37608706-kube-api-access-72z88\") pod \"maas-api-revoke-sub-e802cc01-qxkv9\" (UID: \"89112b19-c140-40c2-8bd3-474c37608706\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e802cc01-qxkv9" Oct 30 17:03:40.025901 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:40.025881 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89112b19-c140-40c2-8bd3-474c37608706-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e802cc01-qxkv9\" (UID: \"89112b19-c140-40c2-8bd3-474c37608706\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e802cc01-qxkv9" Oct 30 17:03:40.026026 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:40.026007 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89112b19-c140-40c2-8bd3-474c37608706-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e802cc01-qxkv9\" (UID: \"89112b19-c140-40c2-8bd3-474c37608706\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e802cc01-qxkv9" Oct 30 17:03:40.026532 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:40.026512 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89112b19-c140-40c2-8bd3-474c37608706-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e802cc01-qxkv9\" (UID: \"89112b19-c140-40c2-8bd3-474c37608706\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e802cc01-qxkv9" Oct 30 17:03:40.028452 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:40.028426 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89112b19-c140-40c2-8bd3-474c37608706-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e802cc01-qxkv9\" (UID: \"89112b19-c140-40c2-8bd3-474c37608706\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e802cc01-qxkv9" Oct 30 17:03:40.050777 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:40.050752 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72z88\" (UniqueName: \"kubernetes.io/projected/89112b19-c140-40c2-8bd3-474c37608706-kube-api-access-72z88\") pod \"maas-api-revoke-sub-e802cc01-qxkv9\" (UID: \"89112b19-c140-40c2-8bd3-474c37608706\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e802cc01-qxkv9" Oct 30 17:03:40.069466 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:40.069443 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e802cc01-qxkv9" Oct 30 17:03:40.322513 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:40.320676 4522 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 17:03:40.322513 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:40.320724 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs" podUID="89585321-089e-477a-a6fa-d8b5a8478639" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 17:03:40.443330 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:03:40.443280 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89112b19_c140_40c2_8bd3_474c37608706.slice/crio-abee0bf5fb682c13959461954922e6bd58c65b4224973a04d0d9782d80f32187 WatchSource:0}: Error finding container abee0bf5fb682c13959461954922e6bd58c65b4224973a04d0d9782d80f32187: Status 404 returned error can't find the container with id abee0bf5fb682c13959461954922e6bd58c65b4224973a04d0d9782d80f32187 Oct 30 17:03:40.446207 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:40.446179 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e802cc01-qxkv9"] Oct 30 17:03:40.930752 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:40.930648 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e802cc01-qxkv9" event={"ID":"89112b19-c140-40c2-8bd3-474c37608706","Type":"ContainerStarted","Data":"019384f535cac6f7f822049e1373ba022c0db5515e23ffa8110f7d0862425da0"} Oct 30 17:03:40.930752 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:40.930714 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e802cc01-qxkv9" event={"ID":"89112b19-c140-40c2-8bd3-474c37608706","Type":"ContainerStarted","Data":"abee0bf5fb682c13959461954922e6bd58c65b4224973a04d0d9782d80f32187"} Oct 30 17:03:40.978993 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:40.978937 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e802cc01-qxkv9" podStartSLOduration=1.9789243970000001 podStartE2EDuration="1.978924397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:03:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:03:40.976946458 +0000 UTC m=+994.977701380" watchObservedRunningTime="2026-10-30 17:03:40.978924397 +0000 UTC m=+994.979679298" Oct 30 17:03:41.321349 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:41.321302 4522 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 17:03:41.321630 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:41.321595 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs" podUID="89585321-089e-477a-a6fa-d8b5a8478639" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 17:03:41.940256 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:41.940219 4522 generic.go:356] "Generic (PLEG): container finished" podID="89112b19-c140-40c2-8bd3-474c37608706" containerID="019384f535cac6f7f822049e1373ba022c0db5515e23ffa8110f7d0862425da0" exitCode=0 Oct 30 17:03:41.940735 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:41.940314 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e802cc01-qxkv9" event={"ID":"89112b19-c140-40c2-8bd3-474c37608706","Type":"ContainerDied","Data":"019384f535cac6f7f822049e1373ba022c0db5515e23ffa8110f7d0862425da0"} Oct 30 17:03:42.321495 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:42.321464 4522 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 17:03:42.321697 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:42.321518 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs" podUID="89585321-089e-477a-a6fa-d8b5a8478639" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 17:03:42.965028 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:03:42.964825 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:03:43.168034 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:43.168005 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e802cc01-qxkv9" Oct 30 17:03:43.271464 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:43.271386 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89112b19-c140-40c2-8bd3-474c37608706-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89112b19-c140-40c2-8bd3-474c37608706-maas-api-openshift-ca\") pod \"89112b19-c140-40c2-8bd3-474c37608706\" (UID: \"89112b19-c140-40c2-8bd3-474c37608706\") " Oct 30 17:03:43.271613 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:43.271597 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89112b19-c140-40c2-8bd3-474c37608706-kube-api-access-72z88\" (UniqueName: \"kubernetes.io/projected/89112b19-c140-40c2-8bd3-474c37608706-kube-api-access-72z88\") pod \"89112b19-c140-40c2-8bd3-474c37608706\" (UID: \"89112b19-c140-40c2-8bd3-474c37608706\") " Oct 30 17:03:43.271654 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:43.271630 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89112b19-c140-40c2-8bd3-474c37608706-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89112b19-c140-40c2-8bd3-474c37608706-maas-api-xks-ca\") pod \"89112b19-c140-40c2-8bd3-474c37608706\" (UID: \"89112b19-c140-40c2-8bd3-474c37608706\") " Oct 30 17:03:43.271767 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:43.271735 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89112b19-c140-40c2-8bd3-474c37608706-maas-api-openshift-ca" pod "89112b19-c140-40c2-8bd3-474c37608706" (UID: "89112b19-c140-40c2-8bd3-474c37608706"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:03:43.272208 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:43.272183 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89112b19-c140-40c2-8bd3-474c37608706-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:03:43.273760 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:43.273739 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89112b19-c140-40c2-8bd3-474c37608706-kube-api-access-72z88" pod "89112b19-c140-40c2-8bd3-474c37608706" (UID: "89112b19-c140-40c2-8bd3-474c37608706"). InnerVolumeSpecName "kube-api-access-72z88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:03:43.273839 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:43.273742 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89112b19-c140-40c2-8bd3-474c37608706-maas-api-xks-ca" pod "89112b19-c140-40c2-8bd3-474c37608706" (UID: "89112b19-c140-40c2-8bd3-474c37608706"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:03:43.321404 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:43.321361 4522 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 17:03:43.321573 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:43.321425 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs" podUID="89585321-089e-477a-a6fa-d8b5a8478639" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 17:03:43.373441 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:43.373409 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72z88\" (UniqueName: \"kubernetes.io/projected/89112b19-c140-40c2-8bd3-474c37608706-kube-api-access-72z88\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:03:43.373441 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:43.373437 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89112b19-c140-40c2-8bd3-474c37608706-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:03:43.962142 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:43.962105 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e802cc01-qxkv9" event={"ID":"89112b19-c140-40c2-8bd3-474c37608706","Type":"ContainerDied","Data":"abee0bf5fb682c13959461954922e6bd58c65b4224973a04d0d9782d80f32187"} Oct 30 17:03:43.962142 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:43.962138 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e802cc01-qxkv9" Oct 30 17:03:43.962367 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:43.962163 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="abee0bf5fb682c13959461954922e6bd58c65b4224973a04d0d9782d80f32187" Oct 30 17:03:44.320997 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:44.320894 4522 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 17:03:44.320997 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:44.320952 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs" podUID="89585321-089e-477a-a6fa-d8b5a8478639" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 17:03:44.745376 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:44.745340 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-205ecb34-t9bsr"] Oct 30 17:03:44.745975 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:44.745962 4522 cpu_manager.go:414] "containerMap: removing container" podUID="89112b19-c140-40c2-8bd3-474c37608706" containerName="revoke-keys" Oct 30 17:03:44.746039 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:44.745976 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89112b19-c140-40c2-8bd3-474c37608706" containerName="revoke-keys" Oct 30 17:03:44.746103 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:44.746094 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="89112b19-c140-40c2-8bd3-474c37608706" containerName="revoke-keys" Oct 30 17:03:44.750023 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:44.749995 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-205ecb34-t9bsr" Oct 30 17:03:44.770809 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:44.770781 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-205ecb34-t9bsr"] Oct 30 17:03:44.778217 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:44.778190 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:03:44.778385 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:44.778299 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:03:44.805400 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:44.805359 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxsxq\" (UniqueName: \"kubernetes.io/projected/54ff10e5-bdc9-4077-89eb-de38fe313bd3-kube-api-access-cxsxq\") pod \"maas-api-revoke-sub-205ecb34-t9bsr\" (UID: \"54ff10e5-bdc9-4077-89eb-de38fe313bd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-205ecb34-t9bsr" Oct 30 17:03:44.805623 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:44.805424 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54ff10e5-bdc9-4077-89eb-de38fe313bd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-205ecb34-t9bsr\" (UID: \"54ff10e5-bdc9-4077-89eb-de38fe313bd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-205ecb34-t9bsr" Oct 30 17:03:44.805734 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:44.805704 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54ff10e5-bdc9-4077-89eb-de38fe313bd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-205ecb34-t9bsr\" (UID: \"54ff10e5-bdc9-4077-89eb-de38fe313bd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-205ecb34-t9bsr" Oct 30 17:03:44.907158 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:44.907114 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54ff10e5-bdc9-4077-89eb-de38fe313bd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-205ecb34-t9bsr\" (UID: \"54ff10e5-bdc9-4077-89eb-de38fe313bd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-205ecb34-t9bsr" Oct 30 17:03:44.907380 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:44.907252 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxsxq\" (UniqueName: \"kubernetes.io/projected/54ff10e5-bdc9-4077-89eb-de38fe313bd3-kube-api-access-cxsxq\") pod \"maas-api-revoke-sub-205ecb34-t9bsr\" (UID: \"54ff10e5-bdc9-4077-89eb-de38fe313bd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-205ecb34-t9bsr" Oct 30 17:03:44.907380 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:44.907286 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54ff10e5-bdc9-4077-89eb-de38fe313bd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-205ecb34-t9bsr\" (UID: \"54ff10e5-bdc9-4077-89eb-de38fe313bd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-205ecb34-t9bsr" Oct 30 17:03:44.907934 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:44.907911 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54ff10e5-bdc9-4077-89eb-de38fe313bd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-205ecb34-t9bsr\" (UID: \"54ff10e5-bdc9-4077-89eb-de38fe313bd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-205ecb34-t9bsr" Oct 30 17:03:44.909830 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:44.909803 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54ff10e5-bdc9-4077-89eb-de38fe313bd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-205ecb34-t9bsr\" (UID: \"54ff10e5-bdc9-4077-89eb-de38fe313bd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-205ecb34-t9bsr" Oct 30 17:03:44.932670 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:44.932634 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxsxq\" (UniqueName: \"kubernetes.io/projected/54ff10e5-bdc9-4077-89eb-de38fe313bd3-kube-api-access-cxsxq\") pod \"maas-api-revoke-sub-205ecb34-t9bsr\" (UID: \"54ff10e5-bdc9-4077-89eb-de38fe313bd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-205ecb34-t9bsr" Oct 30 17:03:45.063622 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:45.063528 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-205ecb34-t9bsr" Oct 30 17:03:45.320968 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:45.320883 4522 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 17:03:45.320968 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:45.320946 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs" podUID="89585321-089e-477a-a6fa-d8b5a8478639" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 17:03:45.370773 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:03:45.370736 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54ff10e5_bdc9_4077_89eb_de38fe313bd3.slice/crio-dab605013fd6c1b3713018f44a23ac4235404d61d4388e1a5b6f293f489e1e67 WatchSource:0}: Error finding container dab605013fd6c1b3713018f44a23ac4235404d61d4388e1a5b6f293f489e1e67: Status 404 returned error can't find the container with id dab605013fd6c1b3713018f44a23ac4235404d61d4388e1a5b6f293f489e1e67 Oct 30 17:03:45.373028 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:45.372996 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-205ecb34-t9bsr"] Oct 30 17:03:45.980978 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:45.980878 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-205ecb34-t9bsr" event={"ID":"54ff10e5-bdc9-4077-89eb-de38fe313bd3","Type":"ContainerStarted","Data":"9c93624faf76105aa4f7314a9be7276cc4df64933f8a67f3de93068248489b29"} Oct 30 17:03:45.981162 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:45.980984 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-205ecb34-t9bsr" event={"ID":"54ff10e5-bdc9-4077-89eb-de38fe313bd3","Type":"ContainerStarted","Data":"dab605013fd6c1b3713018f44a23ac4235404d61d4388e1a5b6f293f489e1e67"} Oct 30 17:03:46.028818 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:46.028745 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-205ecb34-t9bsr" podStartSLOduration=2.028731087 podStartE2EDuration="2.028731087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:03:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:03:46.024681167 +0000 UTC m=+1000.025436071" watchObservedRunningTime="2026-10-30 17:03:46.028731087 +0000 UTC m=+1000.029485988" Oct 30 17:03:46.335357 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:46.335237 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs" Oct 30 17:03:46.336182 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:46.336160 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs" Oct 30 17:03:46.337187 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:46.337166 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs" Oct 30 17:03:46.991486 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:46.991451 4522 generic.go:356] "Generic (PLEG): container finished" podID="54ff10e5-bdc9-4077-89eb-de38fe313bd3" containerID="9c93624faf76105aa4f7314a9be7276cc4df64933f8a67f3de93068248489b29" exitCode=0 Oct 30 17:03:46.991672 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:46.991542 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-205ecb34-t9bsr" event={"ID":"54ff10e5-bdc9-4077-89eb-de38fe313bd3","Type":"ContainerDied","Data":"9c93624faf76105aa4f7314a9be7276cc4df64933f8a67f3de93068248489b29"} Oct 30 17:03:48.148845 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:48.148820 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-205ecb34-t9bsr" Oct 30 17:03:48.256540 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:48.256454 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54ff10e5-bdc9-4077-89eb-de38fe313bd3-kube-api-access-cxsxq\" (UniqueName: \"kubernetes.io/projected/54ff10e5-bdc9-4077-89eb-de38fe313bd3-kube-api-access-cxsxq\") pod \"54ff10e5-bdc9-4077-89eb-de38fe313bd3\" (UID: \"54ff10e5-bdc9-4077-89eb-de38fe313bd3\") " Oct 30 17:03:48.256540 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:48.256503 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54ff10e5-bdc9-4077-89eb-de38fe313bd3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54ff10e5-bdc9-4077-89eb-de38fe313bd3-maas-api-xks-ca\") pod \"54ff10e5-bdc9-4077-89eb-de38fe313bd3\" (UID: \"54ff10e5-bdc9-4077-89eb-de38fe313bd3\") " Oct 30 17:03:48.256731 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:48.256640 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54ff10e5-bdc9-4077-89eb-de38fe313bd3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54ff10e5-bdc9-4077-89eb-de38fe313bd3-maas-api-openshift-ca\") pod \"54ff10e5-bdc9-4077-89eb-de38fe313bd3\" (UID: \"54ff10e5-bdc9-4077-89eb-de38fe313bd3\") " Oct 30 17:03:48.257011 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:48.256982 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54ff10e5-bdc9-4077-89eb-de38fe313bd3-maas-api-openshift-ca" pod "54ff10e5-bdc9-4077-89eb-de38fe313bd3" (UID: "54ff10e5-bdc9-4077-89eb-de38fe313bd3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:03:48.258563 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:48.258540 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54ff10e5-bdc9-4077-89eb-de38fe313bd3-kube-api-access-cxsxq" pod "54ff10e5-bdc9-4077-89eb-de38fe313bd3" (UID: "54ff10e5-bdc9-4077-89eb-de38fe313bd3"). InnerVolumeSpecName "kube-api-access-cxsxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:03:48.258676 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:48.258640 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54ff10e5-bdc9-4077-89eb-de38fe313bd3-maas-api-xks-ca" pod "54ff10e5-bdc9-4077-89eb-de38fe313bd3" (UID: "54ff10e5-bdc9-4077-89eb-de38fe313bd3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:03:48.358263 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:48.358232 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54ff10e5-bdc9-4077-89eb-de38fe313bd3-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:03:48.358263 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:48.358262 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxsxq\" (UniqueName: \"kubernetes.io/projected/54ff10e5-bdc9-4077-89eb-de38fe313bd3-kube-api-access-cxsxq\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:03:48.358263 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:48.358277 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54ff10e5-bdc9-4077-89eb-de38fe313bd3-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:03:49.007202 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:49.007173 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-205ecb34-t9bsr" Oct 30 17:03:49.007395 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:49.007201 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-205ecb34-t9bsr" event={"ID":"54ff10e5-bdc9-4077-89eb-de38fe313bd3","Type":"ContainerDied","Data":"dab605013fd6c1b3713018f44a23ac4235404d61d4388e1a5b6f293f489e1e67"} Oct 30 17:03:49.007395 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:49.007274 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dab605013fd6c1b3713018f44a23ac4235404d61d4388e1a5b6f293f489e1e67" Oct 30 17:03:55.297331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:55.296433 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" podUID="43ca764d-ae0e-4858-9841-5bbd0ffdbb85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:55.297331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:55.296963 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" podUID="4360e9dc-c001-4102-8af6-b55b692fcf2c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Oct 30 17:03:55.321333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:55.320544 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" podUID="4b4ad752-e053-4cea-a3bc-fe2be6f92021" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:55.342331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:55.334969 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" podUID="0b832781-3d41-4496-aa86-e4d92792e590" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 17:03:55.346330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:55.343392 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" podUID="2f1e3bbb-ab1c-455d-a05d-d62f72c5d009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:55.447331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:55.446507 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" podUID="c9ad71b7-ad90-4c8c-bf27-0f10b23f1593" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:55.447331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:55.446456 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" podUID="9bfbc6d3-91fd-41a7-9585-40a8f252f743" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:55.486331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:55.485382 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" podUID="a3e5e732-2c67-4377-9755-e2178bf33772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:55.532331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:55.531365 4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" podUID="43ca764d-ae0e-4858-9841-5bbd0ffdbb85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:55.618393 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:55.617642 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" podUID="75321c5d-8bf8-45c6-8811-81981fe3cad4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 17:03:55.678332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:55.677332 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" podUID="8a6a85ad-c944-450f-b0bb-e3b64b0f56f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:55.757332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:55.754937 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f6793c9-kd4rj"] Oct 30 17:03:55.757332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:55.755896 4522 cpu_manager.go:414] "containerMap: removing container" podUID="54ff10e5-bdc9-4077-89eb-de38fe313bd3" containerName="revoke-keys" Oct 30 17:03:55.757332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:55.755915 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54ff10e5-bdc9-4077-89eb-de38fe313bd3" containerName="revoke-keys" Oct 30 17:03:55.757332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:55.756081 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="54ff10e5-bdc9-4077-89eb-de38fe313bd3" containerName="revoke-keys" Oct 30 17:03:55.773376 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:55.773240 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6793c9-kd4rj" Oct 30 17:03:55.812732 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:55.808617 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:03:55.876382 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:55.871819 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:03:55.957333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:55.950356 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f6793c9-kd4rj"] Oct 30 17:03:55.986335 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:55.974986 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b55e2d8-0ec6-44c4-ab7d-5e2a74aa8096-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f6793c9-kd4rj\" (UID: \"1b55e2d8-0ec6-44c4-ab7d-5e2a74aa8096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6793c9-kd4rj" Oct 30 17:03:55.986335 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:55.975110 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b55e2d8-0ec6-44c4-ab7d-5e2a74aa8096-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f6793c9-kd4rj\" (UID: \"1b55e2d8-0ec6-44c4-ab7d-5e2a74aa8096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6793c9-kd4rj" Oct 30 17:03:55.986335 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:55.975288 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fzvq\" (UniqueName: \"kubernetes.io/projected/1b55e2d8-0ec6-44c4-ab7d-5e2a74aa8096-kube-api-access-7fzvq\") pod \"maas-api-revoke-sub-8f6793c9-kd4rj\" (UID: \"1b55e2d8-0ec6-44c4-ab7d-5e2a74aa8096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6793c9-kd4rj" Oct 30 17:03:56.100332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:56.097325 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fzvq\" (UniqueName: \"kubernetes.io/projected/1b55e2d8-0ec6-44c4-ab7d-5e2a74aa8096-kube-api-access-7fzvq\") pod \"maas-api-revoke-sub-8f6793c9-kd4rj\" (UID: \"1b55e2d8-0ec6-44c4-ab7d-5e2a74aa8096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6793c9-kd4rj" Oct 30 17:03:56.100332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:56.097545 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b55e2d8-0ec6-44c4-ab7d-5e2a74aa8096-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f6793c9-kd4rj\" (UID: \"1b55e2d8-0ec6-44c4-ab7d-5e2a74aa8096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6793c9-kd4rj" Oct 30 17:03:56.100332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:56.097629 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b55e2d8-0ec6-44c4-ab7d-5e2a74aa8096-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f6793c9-kd4rj\" (UID: \"1b55e2d8-0ec6-44c4-ab7d-5e2a74aa8096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6793c9-kd4rj" Oct 30 17:03:56.128383 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:56.119540 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b55e2d8-0ec6-44c4-ab7d-5e2a74aa8096-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f6793c9-kd4rj\" (UID: \"1b55e2d8-0ec6-44c4-ab7d-5e2a74aa8096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6793c9-kd4rj" Oct 30 17:03:56.182651 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:56.182565 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b55e2d8-0ec6-44c4-ab7d-5e2a74aa8096-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f6793c9-kd4rj\" (UID: \"1b55e2d8-0ec6-44c4-ab7d-5e2a74aa8096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6793c9-kd4rj" Oct 30 17:03:56.343722 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:56.323395 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" podUID="4b4ad752-e053-4cea-a3bc-fe2be6f92021" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:56.343722 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:56.330902 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" podUID="4360e9dc-c001-4102-8af6-b55b692fcf2c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:56.343722 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:56.331167 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" podUID="43ca764d-ae0e-4858-9841-5bbd0ffdbb85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 17:03:56.367343 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:56.366073 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" podUID="0b832781-3d41-4496-aa86-e4d92792e590" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:56.421411 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:56.371436 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" podUID="2f1e3bbb-ab1c-455d-a05d-d62f72c5d009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 17:03:56.477326 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:56.464525 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" podUID="c9ad71b7-ad90-4c8c-bf27-0f10b23f1593" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:56.477326 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:56.466957 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" podUID="9bfbc6d3-91fd-41a7-9585-40a8f252f743" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:56.501330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:56.488185 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" podUID="a3e5e732-2c67-4377-9755-e2178bf33772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:56.515328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:56.501902 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" Oct 30 17:03:56.515328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:56.501972 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" Oct 30 17:03:56.515328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:56.502006 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" Oct 30 17:03:56.515328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:56.502043 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" Oct 30 17:03:56.515328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:56.502077 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" Oct 30 17:03:56.515328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:56.502110 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" Oct 30 17:03:56.515328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:56.502144 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" Oct 30 17:03:56.515328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:56.502178 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" Oct 30 17:03:56.847239 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:56.847194 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" podUID="75321c5d-8bf8-45c6-8811-81981fe3cad4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:56.960334 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:56.926256 4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" podUID="4b4ad752-e053-4cea-a3bc-fe2be6f92021" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/health\": context deadline exceeded" Oct 30 17:03:56.960334 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:56.929393 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" podUID="8a6a85ad-c944-450f-b0bb-e3b64b0f56f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:56.960334 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:56.959492 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" Oct 30 17:03:56.960334 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:56.959798 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" Oct 30 17:03:57.042340 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:57.039648 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fzvq\" (UniqueName: \"kubernetes.io/projected/1b55e2d8-0ec6-44c4-ab7d-5e2a74aa8096-kube-api-access-7fzvq\") pod \"maas-api-revoke-sub-8f6793c9-kd4rj\" (UID: \"1b55e2d8-0ec6-44c4-ab7d-5e2a74aa8096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6793c9-kd4rj" Oct 30 17:03:57.069496 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:57.069450 4522 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" podUID="4360e9dc-c001-4102-8af6-b55b692fcf2c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": context deadline exceeded" Oct 30 17:03:57.444170 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:57.074881 4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" podUID="a3e5e732-2c67-4377-9755-e2178bf33772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": context deadline exceeded" Oct 30 17:03:57.444797 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:57.103035 4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" podUID="75321c5d-8bf8-45c6-8811-81981fe3cad4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": context deadline exceeded" Oct 30 17:03:57.472598 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:57.468649 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" Oct 30 17:03:57.510508 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:57.494639 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6793c9-kd4rj" Oct 30 17:03:57.526455 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:57.521659 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" podUID="9bfbc6d3-91fd-41a7-9585-40a8f252f743" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:57.526455 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:57.524105 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" podUID="4b4ad752-e053-4cea-a3bc-fe2be6f92021" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:57.550626 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:57.528023 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" podUID="c9ad71b7-ad90-4c8c-bf27-0f10b23f1593" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:57.588892 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:57.584875 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" Oct 30 17:03:57.592328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:57.589814 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" Oct 30 17:03:57.592328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:57.589882 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" Oct 30 17:03:57.611072 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:57.611029 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" podUID="a3e5e732-2c67-4377-9755-e2178bf33772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:57.619179 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:57.619152 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" Oct 30 17:03:57.695645 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:57.695568 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" Oct 30 17:03:57.714814 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:57.696503 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" Oct 30 17:03:57.729140 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:57.729115 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" Oct 30 17:03:57.729413 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:57.729399 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" Oct 30 17:03:57.729540 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:57.729530 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" Oct 30 17:03:58.065460 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:03:58.065428 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b55e2d8_0ec6_44c4_ab7d_5e2a74aa8096.slice/crio-2507d4123b09ce330834feea21aaf91ddf0f9effd8a388d7f27db3ed8f1a1229 WatchSource:0}: Error finding container 2507d4123b09ce330834feea21aaf91ddf0f9effd8a388d7f27db3ed8f1a1229: Status 404 returned error can't find the container with id 2507d4123b09ce330834feea21aaf91ddf0f9effd8a388d7f27db3ed8f1a1229 Oct 30 17:03:58.067177 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:58.067149 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f6793c9-kd4rj"] Oct 30 17:03:58.773659 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:58.773606 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6793c9-kd4rj" event={"ID":"1b55e2d8-0ec6-44c4-ab7d-5e2a74aa8096","Type":"ContainerStarted","Data":"5097e7baa61fb493fe09b367f6e74424fd7fb91f31cddbd8ccc33ae14756d127"} Oct 30 17:03:58.774108 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:58.773673 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6793c9-kd4rj" event={"ID":"1b55e2d8-0ec6-44c4-ab7d-5e2a74aa8096","Type":"ContainerStarted","Data":"2507d4123b09ce330834feea21aaf91ddf0f9effd8a388d7f27db3ed8f1a1229"} Oct 30 17:03:58.828661 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:58.828610 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6793c9-kd4rj" podStartSLOduration=3.828595504 podStartE2EDuration="3.828595504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:03:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:03:58.825001654 +0000 UTC m=+1012.825756556" watchObservedRunningTime="2026-10-30 17:03:58.828595504 +0000 UTC m=+1012.829350430" Oct 30 17:03:59.783623 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:59.783590 4522 generic.go:356] "Generic (PLEG): container finished" podID="1b55e2d8-0ec6-44c4-ab7d-5e2a74aa8096" containerID="5097e7baa61fb493fe09b367f6e74424fd7fb91f31cddbd8ccc33ae14756d127" exitCode=0 Oct 30 17:03:59.784104 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:03:59.783671 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6793c9-kd4rj" event={"ID":"1b55e2d8-0ec6-44c4-ab7d-5e2a74aa8096","Type":"ContainerDied","Data":"5097e7baa61fb493fe09b367f6e74424fd7fb91f31cddbd8ccc33ae14756d127"} Oct 30 17:04:00.953230 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:00.953202 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6793c9-kd4rj" Oct 30 17:04:01.115828 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:01.115729 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b55e2d8-0ec6-44c4-ab7d-5e2a74aa8096-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b55e2d8-0ec6-44c4-ab7d-5e2a74aa8096-maas-api-openshift-ca\") pod \"1b55e2d8-0ec6-44c4-ab7d-5e2a74aa8096\" (UID: \"1b55e2d8-0ec6-44c4-ab7d-5e2a74aa8096\") " Oct 30 17:04:01.116009 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:01.115920 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b55e2d8-0ec6-44c4-ab7d-5e2a74aa8096-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b55e2d8-0ec6-44c4-ab7d-5e2a74aa8096-maas-api-xks-ca\") pod \"1b55e2d8-0ec6-44c4-ab7d-5e2a74aa8096\" (UID: \"1b55e2d8-0ec6-44c4-ab7d-5e2a74aa8096\") " Oct 30 17:04:01.116071 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:01.116047 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b55e2d8-0ec6-44c4-ab7d-5e2a74aa8096-maas-api-openshift-ca" pod "1b55e2d8-0ec6-44c4-ab7d-5e2a74aa8096" (UID: "1b55e2d8-0ec6-44c4-ab7d-5e2a74aa8096"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:04:01.116172 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:01.116157 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b55e2d8-0ec6-44c4-ab7d-5e2a74aa8096-kube-api-access-7fzvq\" (UniqueName: \"kubernetes.io/projected/1b55e2d8-0ec6-44c4-ab7d-5e2a74aa8096-kube-api-access-7fzvq\") pod \"1b55e2d8-0ec6-44c4-ab7d-5e2a74aa8096\" (UID: \"1b55e2d8-0ec6-44c4-ab7d-5e2a74aa8096\") " Oct 30 17:04:01.116667 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:01.116645 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b55e2d8-0ec6-44c4-ab7d-5e2a74aa8096-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:04:01.117990 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:01.117966 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b55e2d8-0ec6-44c4-ab7d-5e2a74aa8096-maas-api-xks-ca" pod "1b55e2d8-0ec6-44c4-ab7d-5e2a74aa8096" (UID: "1b55e2d8-0ec6-44c4-ab7d-5e2a74aa8096"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:04:01.118115 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:01.118097 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b55e2d8-0ec6-44c4-ab7d-5e2a74aa8096-kube-api-access-7fzvq" pod "1b55e2d8-0ec6-44c4-ab7d-5e2a74aa8096" (UID: "1b55e2d8-0ec6-44c4-ab7d-5e2a74aa8096"). InnerVolumeSpecName "kube-api-access-7fzvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:04:01.218689 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:01.218655 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b55e2d8-0ec6-44c4-ab7d-5e2a74aa8096-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:04:01.218689 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:01.218684 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fzvq\" (UniqueName: \"kubernetes.io/projected/1b55e2d8-0ec6-44c4-ab7d-5e2a74aa8096-kube-api-access-7fzvq\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:04:01.605113 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:01.605079 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43acf099-5pc5b"] Oct 30 17:04:01.606047 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:01.606023 4522 cpu_manager.go:414] "containerMap: removing container" podUID="1b55e2d8-0ec6-44c4-ab7d-5e2a74aa8096" containerName="revoke-keys" Oct 30 17:04:01.606047 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:01.606047 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b55e2d8-0ec6-44c4-ab7d-5e2a74aa8096" containerName="revoke-keys" Oct 30 17:04:01.606291 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:01.606278 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b55e2d8-0ec6-44c4-ab7d-5e2a74aa8096" containerName="revoke-keys" Oct 30 17:04:01.610686 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:01.610656 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43acf099-5pc5b" Oct 30 17:04:01.631499 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:01.631469 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43acf099-5pc5b"] Oct 30 17:04:01.726785 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:01.726736 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5b86\" (UniqueName: \"kubernetes.io/projected/08f2d7b7-67d4-4ffa-a773-bd28a9befe03-kube-api-access-c5b86\") pod \"maas-api-revoke-sub-43acf099-5pc5b\" (UID: \"08f2d7b7-67d4-4ffa-a773-bd28a9befe03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43acf099-5pc5b" Oct 30 17:04:01.726961 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:01.726916 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08f2d7b7-67d4-4ffa-a773-bd28a9befe03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43acf099-5pc5b\" (UID: \"08f2d7b7-67d4-4ffa-a773-bd28a9befe03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43acf099-5pc5b" Oct 30 17:04:01.727045 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:01.726996 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08f2d7b7-67d4-4ffa-a773-bd28a9befe03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43acf099-5pc5b\" (UID: \"08f2d7b7-67d4-4ffa-a773-bd28a9befe03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43acf099-5pc5b" Oct 30 17:04:01.802078 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:01.802044 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6793c9-kd4rj" event={"ID":"1b55e2d8-0ec6-44c4-ab7d-5e2a74aa8096","Type":"ContainerDied","Data":"2507d4123b09ce330834feea21aaf91ddf0f9effd8a388d7f27db3ed8f1a1229"} Oct 30 17:04:01.802078 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:01.802063 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6793c9-kd4rj" Oct 30 17:04:01.802274 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:01.802093 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2507d4123b09ce330834feea21aaf91ddf0f9effd8a388d7f27db3ed8f1a1229" Oct 30 17:04:01.828433 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:01.828394 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08f2d7b7-67d4-4ffa-a773-bd28a9befe03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43acf099-5pc5b\" (UID: \"08f2d7b7-67d4-4ffa-a773-bd28a9befe03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43acf099-5pc5b" Oct 30 17:04:01.828603 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:01.828464 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08f2d7b7-67d4-4ffa-a773-bd28a9befe03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43acf099-5pc5b\" (UID: \"08f2d7b7-67d4-4ffa-a773-bd28a9befe03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43acf099-5pc5b" Oct 30 17:04:01.828645 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:01.828607 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5b86\" (UniqueName: \"kubernetes.io/projected/08f2d7b7-67d4-4ffa-a773-bd28a9befe03-kube-api-access-c5b86\") pod \"maas-api-revoke-sub-43acf099-5pc5b\" (UID: \"08f2d7b7-67d4-4ffa-a773-bd28a9befe03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43acf099-5pc5b" Oct 30 17:04:01.829212 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:01.829184 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08f2d7b7-67d4-4ffa-a773-bd28a9befe03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43acf099-5pc5b\" (UID: \"08f2d7b7-67d4-4ffa-a773-bd28a9befe03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43acf099-5pc5b" Oct 30 17:04:01.830846 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:01.830824 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08f2d7b7-67d4-4ffa-a773-bd28a9befe03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43acf099-5pc5b\" (UID: \"08f2d7b7-67d4-4ffa-a773-bd28a9befe03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43acf099-5pc5b" Oct 30 17:04:01.854995 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:01.854961 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5b86\" (UniqueName: \"kubernetes.io/projected/08f2d7b7-67d4-4ffa-a773-bd28a9befe03-kube-api-access-c5b86\") pod \"maas-api-revoke-sub-43acf099-5pc5b\" (UID: \"08f2d7b7-67d4-4ffa-a773-bd28a9befe03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43acf099-5pc5b" Oct 30 17:04:01.941183 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:01.941100 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43acf099-5pc5b" Oct 30 17:04:02.270058 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:04:02.270021 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08f2d7b7_67d4_4ffa_a773_bd28a9befe03.slice/crio-bd943f13f241b6833359986ce11a98edbac9f4b6c14b2c8cec1e233194d67067 WatchSource:0}: Error finding container bd943f13f241b6833359986ce11a98edbac9f4b6c14b2c8cec1e233194d67067: Status 404 returned error can't find the container with id bd943f13f241b6833359986ce11a98edbac9f4b6c14b2c8cec1e233194d67067 Oct 30 17:04:02.274149 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:02.274125 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43acf099-5pc5b"] Oct 30 17:04:02.811162 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:02.811128 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43acf099-5pc5b" event={"ID":"08f2d7b7-67d4-4ffa-a773-bd28a9befe03","Type":"ContainerStarted","Data":"7960ee5f6ea9dccaae9b475f800c4409b021763be44246b67b29564704e4715f"} Oct 30 17:04:02.811369 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:02.811178 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43acf099-5pc5b" event={"ID":"08f2d7b7-67d4-4ffa-a773-bd28a9befe03","Type":"ContainerStarted","Data":"bd943f13f241b6833359986ce11a98edbac9f4b6c14b2c8cec1e233194d67067"} Oct 30 17:04:02.859723 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:02.859602 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43acf099-5pc5b" podStartSLOduration=1.859588103 podStartE2EDuration="1.859588103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:04:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:04:02.856982649 +0000 UTC m=+1016.857737573" watchObservedRunningTime="2026-10-30 17:04:02.859588103 +0000 UTC m=+1016.860343019" Oct 30 17:04:03.820037 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:03.820005 4522 generic.go:356] "Generic (PLEG): container finished" podID="08f2d7b7-67d4-4ffa-a773-bd28a9befe03" containerID="7960ee5f6ea9dccaae9b475f800c4409b021763be44246b67b29564704e4715f" exitCode=0 Oct 30 17:04:03.820469 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:03.820072 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43acf099-5pc5b" event={"ID":"08f2d7b7-67d4-4ffa-a773-bd28a9befe03","Type":"ContainerDied","Data":"7960ee5f6ea9dccaae9b475f800c4409b021763be44246b67b29564704e4715f"} Oct 30 17:04:04.996538 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:04.996510 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43acf099-5pc5b" Oct 30 17:04:05.075527 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:05.075493 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08f2d7b7-67d4-4ffa-a773-bd28a9befe03-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08f2d7b7-67d4-4ffa-a773-bd28a9befe03-maas-api-openshift-ca\") pod \"08f2d7b7-67d4-4ffa-a773-bd28a9befe03\" (UID: \"08f2d7b7-67d4-4ffa-a773-bd28a9befe03\") " Oct 30 17:04:05.075723 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:05.075536 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08f2d7b7-67d4-4ffa-a773-bd28a9befe03-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08f2d7b7-67d4-4ffa-a773-bd28a9befe03-maas-api-xks-ca\") pod \"08f2d7b7-67d4-4ffa-a773-bd28a9befe03\" (UID: \"08f2d7b7-67d4-4ffa-a773-bd28a9befe03\") " Oct 30 17:04:05.075723 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:05.075629 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08f2d7b7-67d4-4ffa-a773-bd28a9befe03-kube-api-access-c5b86\" (UniqueName: \"kubernetes.io/projected/08f2d7b7-67d4-4ffa-a773-bd28a9befe03-kube-api-access-c5b86\") pod \"08f2d7b7-67d4-4ffa-a773-bd28a9befe03\" (UID: \"08f2d7b7-67d4-4ffa-a773-bd28a9befe03\") " Oct 30 17:04:05.075942 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:05.075917 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08f2d7b7-67d4-4ffa-a773-bd28a9befe03-maas-api-openshift-ca" pod "08f2d7b7-67d4-4ffa-a773-bd28a9befe03" (UID: "08f2d7b7-67d4-4ffa-a773-bd28a9befe03"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:04:05.076242 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:05.076222 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08f2d7b7-67d4-4ffa-a773-bd28a9befe03-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:04:05.077831 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:05.077804 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08f2d7b7-67d4-4ffa-a773-bd28a9befe03-maas-api-xks-ca" pod "08f2d7b7-67d4-4ffa-a773-bd28a9befe03" (UID: "08f2d7b7-67d4-4ffa-a773-bd28a9befe03"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:04:05.077956 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:05.077940 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08f2d7b7-67d4-4ffa-a773-bd28a9befe03-kube-api-access-c5b86" pod "08f2d7b7-67d4-4ffa-a773-bd28a9befe03" (UID: "08f2d7b7-67d4-4ffa-a773-bd28a9befe03"). InnerVolumeSpecName "kube-api-access-c5b86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:04:05.177411 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:05.177382 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5b86\" (UniqueName: \"kubernetes.io/projected/08f2d7b7-67d4-4ffa-a773-bd28a9befe03-kube-api-access-c5b86\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:04:05.177411 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:05.177410 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08f2d7b7-67d4-4ffa-a773-bd28a9befe03-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:04:05.851622 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:05.851578 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43acf099-5pc5b" Oct 30 17:04:05.851806 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:05.851588 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43acf099-5pc5b" event={"ID":"08f2d7b7-67d4-4ffa-a773-bd28a9befe03","Type":"ContainerDied","Data":"bd943f13f241b6833359986ce11a98edbac9f4b6c14b2c8cec1e233194d67067"} Oct 30 17:04:05.851806 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:05.851713 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd943f13f241b6833359986ce11a98edbac9f4b6c14b2c8cec1e233194d67067" Oct 30 17:04:09.250472 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:09.250438 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b299bf6b-7kj9h"] Oct 30 17:04:09.251078 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:09.251064 4522 cpu_manager.go:414] "containerMap: removing container" podUID="08f2d7b7-67d4-4ffa-a773-bd28a9befe03" containerName="revoke-keys" Oct 30 17:04:09.251078 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:09.251079 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08f2d7b7-67d4-4ffa-a773-bd28a9befe03" containerName="revoke-keys" Oct 30 17:04:09.251195 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:09.251185 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="08f2d7b7-67d4-4ffa-a773-bd28a9befe03" containerName="revoke-keys" Oct 30 17:04:09.255010 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:09.254987 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b299bf6b-7kj9h" Oct 30 17:04:09.275915 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:09.275882 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b299bf6b-7kj9h"] Oct 30 17:04:09.285079 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:09.285053 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:04:09.285079 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:09.285056 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:04:09.335682 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:09.335651 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndchd\" (UniqueName: \"kubernetes.io/projected/b9b1a1bc-e0d2-4325-8090-9857d12b8046-kube-api-access-ndchd\") pod \"maas-api-revoke-sub-b299bf6b-7kj9h\" (UID: \"b9b1a1bc-e0d2-4325-8090-9857d12b8046\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b299bf6b-7kj9h" Oct 30 17:04:09.335861 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:09.335763 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9b1a1bc-e0d2-4325-8090-9857d12b8046-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b299bf6b-7kj9h\" (UID: \"b9b1a1bc-e0d2-4325-8090-9857d12b8046\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b299bf6b-7kj9h" Oct 30 17:04:09.335861 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:09.335846 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9b1a1bc-e0d2-4325-8090-9857d12b8046-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b299bf6b-7kj9h\" (UID: \"b9b1a1bc-e0d2-4325-8090-9857d12b8046\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b299bf6b-7kj9h" Oct 30 17:04:09.437398 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:09.437359 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndchd\" (UniqueName: \"kubernetes.io/projected/b9b1a1bc-e0d2-4325-8090-9857d12b8046-kube-api-access-ndchd\") pod \"maas-api-revoke-sub-b299bf6b-7kj9h\" (UID: \"b9b1a1bc-e0d2-4325-8090-9857d12b8046\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b299bf6b-7kj9h" Oct 30 17:04:09.437585 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:09.437438 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9b1a1bc-e0d2-4325-8090-9857d12b8046-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b299bf6b-7kj9h\" (UID: \"b9b1a1bc-e0d2-4325-8090-9857d12b8046\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b299bf6b-7kj9h" Oct 30 17:04:09.437585 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:09.437492 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9b1a1bc-e0d2-4325-8090-9857d12b8046-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b299bf6b-7kj9h\" (UID: \"b9b1a1bc-e0d2-4325-8090-9857d12b8046\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b299bf6b-7kj9h" Oct 30 17:04:09.438159 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:09.438131 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9b1a1bc-e0d2-4325-8090-9857d12b8046-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b299bf6b-7kj9h\" (UID: \"b9b1a1bc-e0d2-4325-8090-9857d12b8046\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b299bf6b-7kj9h" Oct 30 17:04:09.439932 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:09.439907 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9b1a1bc-e0d2-4325-8090-9857d12b8046-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b299bf6b-7kj9h\" (UID: \"b9b1a1bc-e0d2-4325-8090-9857d12b8046\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b299bf6b-7kj9h" Oct 30 17:04:09.463877 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:09.463849 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndchd\" (UniqueName: \"kubernetes.io/projected/b9b1a1bc-e0d2-4325-8090-9857d12b8046-kube-api-access-ndchd\") pod \"maas-api-revoke-sub-b299bf6b-7kj9h\" (UID: \"b9b1a1bc-e0d2-4325-8090-9857d12b8046\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b299bf6b-7kj9h" Oct 30 17:04:09.570984 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:09.570893 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b299bf6b-7kj9h" Oct 30 17:04:09.885478 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:04:09.885442 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9b1a1bc_e0d2_4325_8090_9857d12b8046.slice/crio-60ef2ef6f5517fcce2c0a93ed13f8598d6de1a4ba7be8182b6f650300144a338 WatchSource:0}: Error finding container 60ef2ef6f5517fcce2c0a93ed13f8598d6de1a4ba7be8182b6f650300144a338: Status 404 returned error can't find the container with id 60ef2ef6f5517fcce2c0a93ed13f8598d6de1a4ba7be8182b6f650300144a338 Oct 30 17:04:09.888289 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:09.888269 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b299bf6b-7kj9h"] Oct 30 17:04:10.893251 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:10.893218 4522 generic.go:356] "Generic (PLEG): container finished" podID="b9b1a1bc-e0d2-4325-8090-9857d12b8046" containerID="40f15f9eaabc54cdc360c3d14031fc4d99ead13d7e0d2967aa89f45e2202f19b" exitCode=0 Oct 30 17:04:10.893683 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:10.893271 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b299bf6b-7kj9h" event={"ID":"b9b1a1bc-e0d2-4325-8090-9857d12b8046","Type":"ContainerDied","Data":"40f15f9eaabc54cdc360c3d14031fc4d99ead13d7e0d2967aa89f45e2202f19b"} Oct 30 17:04:10.893683 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:10.893361 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b299bf6b-7kj9h" event={"ID":"b9b1a1bc-e0d2-4325-8090-9857d12b8046","Type":"ContainerStarted","Data":"60ef2ef6f5517fcce2c0a93ed13f8598d6de1a4ba7be8182b6f650300144a338"} Oct 30 17:04:12.076104 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:12.076079 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b299bf6b-7kj9h" Oct 30 17:04:12.177819 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:12.177723 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9b1a1bc-e0d2-4325-8090-9857d12b8046-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9b1a1bc-e0d2-4325-8090-9857d12b8046-maas-api-openshift-ca\") pod \"b9b1a1bc-e0d2-4325-8090-9857d12b8046\" (UID: \"b9b1a1bc-e0d2-4325-8090-9857d12b8046\") " Oct 30 17:04:12.178024 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:12.177920 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9b1a1bc-e0d2-4325-8090-9857d12b8046-kube-api-access-ndchd\" (UniqueName: \"kubernetes.io/projected/b9b1a1bc-e0d2-4325-8090-9857d12b8046-kube-api-access-ndchd\") pod \"b9b1a1bc-e0d2-4325-8090-9857d12b8046\" (UID: \"b9b1a1bc-e0d2-4325-8090-9857d12b8046\") " Oct 30 17:04:12.178024 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:12.178009 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9b1a1bc-e0d2-4325-8090-9857d12b8046-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9b1a1bc-e0d2-4325-8090-9857d12b8046-maas-api-xks-ca\") pod \"b9b1a1bc-e0d2-4325-8090-9857d12b8046\" (UID: \"b9b1a1bc-e0d2-4325-8090-9857d12b8046\") " Oct 30 17:04:12.178173 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:12.178139 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9b1a1bc-e0d2-4325-8090-9857d12b8046-maas-api-openshift-ca" pod "b9b1a1bc-e0d2-4325-8090-9857d12b8046" (UID: "b9b1a1bc-e0d2-4325-8090-9857d12b8046"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:04:12.178778 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:12.178751 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9b1a1bc-e0d2-4325-8090-9857d12b8046-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:04:12.180113 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:12.180092 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9b1a1bc-e0d2-4325-8090-9857d12b8046-maas-api-xks-ca" pod "b9b1a1bc-e0d2-4325-8090-9857d12b8046" (UID: "b9b1a1bc-e0d2-4325-8090-9857d12b8046"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:04:12.180385 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:12.180363 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9b1a1bc-e0d2-4325-8090-9857d12b8046-kube-api-access-ndchd" pod "b9b1a1bc-e0d2-4325-8090-9857d12b8046" (UID: "b9b1a1bc-e0d2-4325-8090-9857d12b8046"). InnerVolumeSpecName "kube-api-access-ndchd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:04:12.280199 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:12.280167 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9b1a1bc-e0d2-4325-8090-9857d12b8046-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:04:12.280199 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:12.280198 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndchd\" (UniqueName: \"kubernetes.io/projected/b9b1a1bc-e0d2-4325-8090-9857d12b8046-kube-api-access-ndchd\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:04:12.910594 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:12.910564 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b299bf6b-7kj9h" Oct 30 17:04:12.910792 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:12.910646 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b299bf6b-7kj9h" event={"ID":"b9b1a1bc-e0d2-4325-8090-9857d12b8046","Type":"ContainerDied","Data":"60ef2ef6f5517fcce2c0a93ed13f8598d6de1a4ba7be8182b6f650300144a338"} Oct 30 17:04:12.910792 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:12.910729 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60ef2ef6f5517fcce2c0a93ed13f8598d6de1a4ba7be8182b6f650300144a338" Oct 30 17:04:12.936028 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:04:12.935992 4522 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9b1a1bc_e0d2_4325_8090_9857d12b8046.slice\": RecentStats: unable to find data in memory cache]" Oct 30 17:04:12.949761 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:04:12.949727 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:04:16.126460 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:16.126420 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1fa217-jhcsb"] Oct 30 17:04:16.127102 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:16.127087 4522 cpu_manager.go:414] "containerMap: removing container" podUID="b9b1a1bc-e0d2-4325-8090-9857d12b8046" containerName="revoke-keys" Oct 30 17:04:16.127173 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:16.127103 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9b1a1bc-e0d2-4325-8090-9857d12b8046" containerName="revoke-keys" Oct 30 17:04:16.127213 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:16.127199 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9b1a1bc-e0d2-4325-8090-9857d12b8046" containerName="revoke-keys" Oct 30 17:04:16.131278 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:16.131260 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1fa217-jhcsb" Oct 30 17:04:16.162874 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:16.162839 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1fa217-jhcsb"] Oct 30 17:04:16.180709 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:16.180659 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:04:16.180709 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:16.180693 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:04:16.238239 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:16.238206 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42542b00-e51a-4f16-8987-26f02d7c4f85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1fa217-jhcsb\" (UID: \"42542b00-e51a-4f16-8987-26f02d7c4f85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1fa217-jhcsb" Oct 30 17:04:16.238629 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:16.238544 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42542b00-e51a-4f16-8987-26f02d7c4f85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1fa217-jhcsb\" (UID: \"42542b00-e51a-4f16-8987-26f02d7c4f85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1fa217-jhcsb" Oct 30 17:04:16.239038 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:16.239007 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thjv5\" (UniqueName: \"kubernetes.io/projected/42542b00-e51a-4f16-8987-26f02d7c4f85-kube-api-access-thjv5\") pod \"maas-api-revoke-sub-2e1fa217-jhcsb\" (UID: \"42542b00-e51a-4f16-8987-26f02d7c4f85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1fa217-jhcsb" Oct 30 17:04:16.340395 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:16.340350 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thjv5\" (UniqueName: \"kubernetes.io/projected/42542b00-e51a-4f16-8987-26f02d7c4f85-kube-api-access-thjv5\") pod \"maas-api-revoke-sub-2e1fa217-jhcsb\" (UID: \"42542b00-e51a-4f16-8987-26f02d7c4f85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1fa217-jhcsb" Oct 30 17:04:16.340595 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:16.340420 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42542b00-e51a-4f16-8987-26f02d7c4f85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1fa217-jhcsb\" (UID: \"42542b00-e51a-4f16-8987-26f02d7c4f85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1fa217-jhcsb" Oct 30 17:04:16.340595 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:16.340580 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42542b00-e51a-4f16-8987-26f02d7c4f85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1fa217-jhcsb\" (UID: \"42542b00-e51a-4f16-8987-26f02d7c4f85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1fa217-jhcsb" Oct 30 17:04:16.341108 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:16.341075 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42542b00-e51a-4f16-8987-26f02d7c4f85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1fa217-jhcsb\" (UID: \"42542b00-e51a-4f16-8987-26f02d7c4f85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1fa217-jhcsb" Oct 30 17:04:16.342866 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:16.342841 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42542b00-e51a-4f16-8987-26f02d7c4f85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1fa217-jhcsb\" (UID: \"42542b00-e51a-4f16-8987-26f02d7c4f85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1fa217-jhcsb" Oct 30 17:04:16.367012 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:16.366986 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thjv5\" (UniqueName: \"kubernetes.io/projected/42542b00-e51a-4f16-8987-26f02d7c4f85-kube-api-access-thjv5\") pod \"maas-api-revoke-sub-2e1fa217-jhcsb\" (UID: \"42542b00-e51a-4f16-8987-26f02d7c4f85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1fa217-jhcsb" Oct 30 17:04:16.447604 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:16.447510 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1fa217-jhcsb" Oct 30 17:04:16.759048 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:04:16.759017 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42542b00_e51a_4f16_8987_26f02d7c4f85.slice/crio-4dc90ef43eaa8105e65d109e61e5764d1412b45b0f425c04d6705857295c71de WatchSource:0}: Error finding container 4dc90ef43eaa8105e65d109e61e5764d1412b45b0f425c04d6705857295c71de: Status 404 returned error can't find the container with id 4dc90ef43eaa8105e65d109e61e5764d1412b45b0f425c04d6705857295c71de Oct 30 17:04:16.762155 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:16.762129 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1fa217-jhcsb"] Oct 30 17:04:16.940847 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:16.940811 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1fa217-jhcsb" event={"ID":"42542b00-e51a-4f16-8987-26f02d7c4f85","Type":"ContainerStarted","Data":"4dc90ef43eaa8105e65d109e61e5764d1412b45b0f425c04d6705857295c71de"} Oct 30 17:04:17.950243 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:17.950141 4522 generic.go:356] "Generic (PLEG): container finished" podID="42542b00-e51a-4f16-8987-26f02d7c4f85" containerID="4a51773c7fa60abee216fe1723f125470f4c5fd3c2e84ccabf9870a3067daf0c" exitCode=0 Oct 30 17:04:17.950243 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:17.950199 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1fa217-jhcsb" event={"ID":"42542b00-e51a-4f16-8987-26f02d7c4f85","Type":"ContainerDied","Data":"4a51773c7fa60abee216fe1723f125470f4c5fd3c2e84ccabf9870a3067daf0c"} Oct 30 17:04:19.766372 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:19.742344 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" podUID="0b832781-3d41-4496-aa86-e4d92792e590" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:19.936636 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:19.936142 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1fa217-jhcsb" Oct 30 17:04:20.054332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:20.042055 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42542b00-e51a-4f16-8987-26f02d7c4f85-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42542b00-e51a-4f16-8987-26f02d7c4f85-maas-api-xks-ca\") pod \"42542b00-e51a-4f16-8987-26f02d7c4f85\" (UID: \"42542b00-e51a-4f16-8987-26f02d7c4f85\") " Oct 30 17:04:20.054332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:20.042124 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42542b00-e51a-4f16-8987-26f02d7c4f85-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42542b00-e51a-4f16-8987-26f02d7c4f85-maas-api-openshift-ca\") pod \"42542b00-e51a-4f16-8987-26f02d7c4f85\" (UID: \"42542b00-e51a-4f16-8987-26f02d7c4f85\") " Oct 30 17:04:20.054332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:20.042185 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42542b00-e51a-4f16-8987-26f02d7c4f85-kube-api-access-thjv5\" (UniqueName: \"kubernetes.io/projected/42542b00-e51a-4f16-8987-26f02d7c4f85-kube-api-access-thjv5\") pod \"42542b00-e51a-4f16-8987-26f02d7c4f85\" (UID: \"42542b00-e51a-4f16-8987-26f02d7c4f85\") " Oct 30 17:04:20.114354 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:20.109325 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42542b00-e51a-4f16-8987-26f02d7c4f85-maas-api-openshift-ca" pod "42542b00-e51a-4f16-8987-26f02d7c4f85" (UID: "42542b00-e51a-4f16-8987-26f02d7c4f85"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:04:20.189441 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:20.160393 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42542b00-e51a-4f16-8987-26f02d7c4f85-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:04:20.339330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:20.319379 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1fa217-jhcsb" event={"ID":"42542b00-e51a-4f16-8987-26f02d7c4f85","Type":"ContainerDied","Data":"4dc90ef43eaa8105e65d109e61e5764d1412b45b0f425c04d6705857295c71de"} Oct 30 17:04:20.339330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:20.319443 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4dc90ef43eaa8105e65d109e61e5764d1412b45b0f425c04d6705857295c71de" Oct 30 17:04:20.339330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:20.319511 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1fa217-jhcsb" Oct 30 17:04:20.339330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:20.332525 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42542b00-e51a-4f16-8987-26f02d7c4f85-kube-api-access-thjv5" pod "42542b00-e51a-4f16-8987-26f02d7c4f85" (UID: "42542b00-e51a-4f16-8987-26f02d7c4f85"). InnerVolumeSpecName "kube-api-access-thjv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:04:20.346486 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:20.346451 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thjv5\" (UniqueName: \"kubernetes.io/projected/42542b00-e51a-4f16-8987-26f02d7c4f85-kube-api-access-thjv5\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:04:20.351723 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:20.351694 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42542b00-e51a-4f16-8987-26f02d7c4f85-maas-api-xks-ca" pod "42542b00-e51a-4f16-8987-26f02d7c4f85" (UID: "42542b00-e51a-4f16-8987-26f02d7c4f85"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:04:20.457398 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:20.450341 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42542b00-e51a-4f16-8987-26f02d7c4f85-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:04:20.493471 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:20.493400 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" podUID="4360e9dc-c001-4102-8af6-b55b692fcf2c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:20.579330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:20.578400 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" podUID="c9ad71b7-ad90-4c8c-bf27-0f10b23f1593" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:20.698329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:20.698098 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" podUID="9bfbc6d3-91fd-41a7-9585-40a8f252f743" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:20.720498 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:20.720408 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" podUID="a3e5e732-2c67-4377-9755-e2178bf33772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:20.722824 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:20.721357 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" podUID="8a6a85ad-c944-450f-b0bb-e3b64b0f56f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 17:04:20.722824 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:20.721394 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" podUID="4b4ad752-e053-4cea-a3bc-fe2be6f92021" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:20.722824 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:20.721360 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" podUID="2f1e3bbb-ab1c-455d-a05d-d62f72c5d009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:20.752330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:20.736537 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" podUID="43ca764d-ae0e-4858-9841-5bbd0ffdbb85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:20.752330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:20.751414 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" podUID="0b832781-3d41-4496-aa86-e4d92792e590" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:20.752330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:20.751789 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" Oct 30 17:04:21.519332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:21.519006 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" podUID="4360e9dc-c001-4102-8af6-b55b692fcf2c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Oct 30 17:04:21.554332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:21.525489 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" Oct 30 17:04:21.758333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:21.757456 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" podUID="0b832781-3d41-4496-aa86-e4d92792e590" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:21.800328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:21.758869 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" podUID="9bfbc6d3-91fd-41a7-9585-40a8f252f743" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:21.800328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:21.759372 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" Oct 30 17:04:21.800328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:21.759404 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" podUID="4b4ad752-e053-4cea-a3bc-fe2be6f92021" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:21.800328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:21.759744 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" podUID="8a6a85ad-c944-450f-b0bb-e3b64b0f56f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 17:04:21.800328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:21.760041 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" Oct 30 17:04:21.800328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:21.760094 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" Oct 30 17:04:21.800328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:21.761146 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" podUID="a3e5e732-2c67-4377-9755-e2178bf33772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 17:04:21.800328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:21.761437 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" Oct 30 17:04:21.800328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:21.761688 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" podUID="2f1e3bbb-ab1c-455d-a05d-d62f72c5d009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:21.800328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:21.761968 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" Oct 30 17:04:21.800328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:21.762538 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" podUID="43ca764d-ae0e-4858-9841-5bbd0ffdbb85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:21.800328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:21.762805 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" Oct 30 17:04:21.831332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:21.823843 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" podUID="75321c5d-8bf8-45c6-8811-81981fe3cad4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:21.831332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:21.823941 4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" podUID="9bfbc6d3-91fd-41a7-9585-40a8f252f743" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:21.831332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:21.824288 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" podUID="c9ad71b7-ad90-4c8c-bf27-0f10b23f1593" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 17:04:21.849331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:21.833375 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" Oct 30 17:04:22.521774 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:22.521723 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" podUID="4360e9dc-c001-4102-8af6-b55b692fcf2c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:22.946443 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:22.916422 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" podUID="c9ad71b7-ad90-4c8c-bf27-0f10b23f1593" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:22.946443 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:22.920085 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" podUID="75321c5d-8bf8-45c6-8811-81981fe3cad4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:22.946443 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:22.931667 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" podUID="9bfbc6d3-91fd-41a7-9585-40a8f252f743" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:22.946443 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:22.932013 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" podUID="2f1e3bbb-ab1c-455d-a05d-d62f72c5d009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 17:04:22.946443 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:22.932203 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" podUID="0b832781-3d41-4496-aa86-e4d92792e590" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 17:04:22.946443 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:22.939124 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" podUID="4b4ad752-e053-4cea-a3bc-fe2be6f92021" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Oct 30 17:04:22.946443 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:22.939485 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" podUID="8a6a85ad-c944-450f-b0bb-e3b64b0f56f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 17:04:22.946443 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:22.939741 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" podUID="a3e5e732-2c67-4377-9755-e2178bf33772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 17:04:22.946443 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:22.939977 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" podUID="43ca764d-ae0e-4858-9841-5bbd0ffdbb85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 17:04:23.535246 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:23.535208 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" podUID="4360e9dc-c001-4102-8af6-b55b692fcf2c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:23.557600 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:04:23.554910 4522 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.377s" Oct 30 17:04:23.557600 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:23.555005 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" Oct 30 17:04:23.557600 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:23.555173 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" Oct 30 17:04:23.557600 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:23.555297 4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" podUID="0b832781-3d41-4496-aa86-e4d92792e590" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:23.561079 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:23.558795 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" Oct 30 17:04:23.565237 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:23.565209 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" Oct 30 17:04:23.565413 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:23.565270 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" Oct 30 17:04:23.565413 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:23.565322 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" Oct 30 17:04:23.565413 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:23.565356 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" Oct 30 17:04:23.565413 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:23.565384 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" Oct 30 17:04:23.565644 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:23.565415 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" Oct 30 17:04:23.565644 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:23.565455 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" Oct 30 17:04:23.582642 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:23.582363 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" Oct 30 17:04:24.945902 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:24.945869 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fd22682-dvzrn"] Oct 30 17:04:24.946772 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:24.946753 4522 cpu_manager.go:414] "containerMap: removing container" podUID="42542b00-e51a-4f16-8987-26f02d7c4f85" containerName="revoke-keys" Oct 30 17:04:24.946823 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:24.946774 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42542b00-e51a-4f16-8987-26f02d7c4f85" containerName="revoke-keys" Oct 30 17:04:24.946960 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:24.946949 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="42542b00-e51a-4f16-8987-26f02d7c4f85" containerName="revoke-keys" Oct 30 17:04:24.951179 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:24.951158 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd22682-dvzrn" Oct 30 17:04:24.968582 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:24.968552 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fd22682-dvzrn"] Oct 30 17:04:24.971144 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:24.971122 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:04:24.977631 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:24.977604 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:04:25.020499 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:25.020468 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e15c46d-5a43-4f41-8645-43bdb8ed2845-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9fd22682-dvzrn\" (UID: \"2e15c46d-5a43-4f41-8645-43bdb8ed2845\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd22682-dvzrn" Oct 30 17:04:25.020651 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:25.020534 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e15c46d-5a43-4f41-8645-43bdb8ed2845-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9fd22682-dvzrn\" (UID: \"2e15c46d-5a43-4f41-8645-43bdb8ed2845\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd22682-dvzrn" Oct 30 17:04:25.020651 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:25.020639 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7czsx\" (UniqueName: \"kubernetes.io/projected/2e15c46d-5a43-4f41-8645-43bdb8ed2845-kube-api-access-7czsx\") pod \"maas-api-revoke-sub-9fd22682-dvzrn\" (UID: \"2e15c46d-5a43-4f41-8645-43bdb8ed2845\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd22682-dvzrn" Oct 30 17:04:25.122164 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:25.122124 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e15c46d-5a43-4f41-8645-43bdb8ed2845-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9fd22682-dvzrn\" (UID: \"2e15c46d-5a43-4f41-8645-43bdb8ed2845\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd22682-dvzrn" Oct 30 17:04:25.122377 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:25.122198 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e15c46d-5a43-4f41-8645-43bdb8ed2845-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9fd22682-dvzrn\" (UID: \"2e15c46d-5a43-4f41-8645-43bdb8ed2845\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd22682-dvzrn" Oct 30 17:04:25.122377 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:25.122247 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7czsx\" (UniqueName: \"kubernetes.io/projected/2e15c46d-5a43-4f41-8645-43bdb8ed2845-kube-api-access-7czsx\") pod \"maas-api-revoke-sub-9fd22682-dvzrn\" (UID: \"2e15c46d-5a43-4f41-8645-43bdb8ed2845\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd22682-dvzrn" Oct 30 17:04:25.122954 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:25.122935 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e15c46d-5a43-4f41-8645-43bdb8ed2845-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9fd22682-dvzrn\" (UID: \"2e15c46d-5a43-4f41-8645-43bdb8ed2845\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd22682-dvzrn" Oct 30 17:04:25.124532 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:25.124505 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e15c46d-5a43-4f41-8645-43bdb8ed2845-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9fd22682-dvzrn\" (UID: \"2e15c46d-5a43-4f41-8645-43bdb8ed2845\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd22682-dvzrn" Oct 30 17:04:25.149144 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:25.149108 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7czsx\" (UniqueName: \"kubernetes.io/projected/2e15c46d-5a43-4f41-8645-43bdb8ed2845-kube-api-access-7czsx\") pod \"maas-api-revoke-sub-9fd22682-dvzrn\" (UID: \"2e15c46d-5a43-4f41-8645-43bdb8ed2845\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd22682-dvzrn" Oct 30 17:04:25.273742 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:25.273710 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd22682-dvzrn" Oct 30 17:04:25.580368 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:04:25.580338 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e15c46d_5a43_4f41_8645_43bdb8ed2845.slice/crio-d52c4f656110cb2daffe423e370189241d12b11884358bddaf56112a3ccc82c6 WatchSource:0}: Error finding container d52c4f656110cb2daffe423e370189241d12b11884358bddaf56112a3ccc82c6: Status 404 returned error can't find the container with id d52c4f656110cb2daffe423e370189241d12b11884358bddaf56112a3ccc82c6 Oct 30 17:04:25.583159 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:25.583058 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fd22682-dvzrn"] Oct 30 17:04:26.577191 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:26.577160 4522 generic.go:356] "Generic (PLEG): container finished" podID="2e15c46d-5a43-4f41-8645-43bdb8ed2845" containerID="0a944c591ac537cd870e1944f6a7114fc3b367115234c514807ba57c93f77a7a" exitCode=0 Oct 30 17:04:26.577605 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:26.577209 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd22682-dvzrn" event={"ID":"2e15c46d-5a43-4f41-8645-43bdb8ed2845","Type":"ContainerDied","Data":"0a944c591ac537cd870e1944f6a7114fc3b367115234c514807ba57c93f77a7a"} Oct 30 17:04:26.577605 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:26.577258 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd22682-dvzrn" event={"ID":"2e15c46d-5a43-4f41-8645-43bdb8ed2845","Type":"ContainerStarted","Data":"d52c4f656110cb2daffe423e370189241d12b11884358bddaf56112a3ccc82c6"} Oct 30 17:04:27.748112 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:27.748086 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd22682-dvzrn" Oct 30 17:04:27.861929 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:27.861840 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e15c46d-5a43-4f41-8645-43bdb8ed2845-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e15c46d-5a43-4f41-8645-43bdb8ed2845-maas-api-xks-ca\") pod \"2e15c46d-5a43-4f41-8645-43bdb8ed2845\" (UID: \"2e15c46d-5a43-4f41-8645-43bdb8ed2845\") " Oct 30 17:04:27.862103 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:27.862025 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e15c46d-5a43-4f41-8645-43bdb8ed2845-kube-api-access-7czsx\" (UniqueName: \"kubernetes.io/projected/2e15c46d-5a43-4f41-8645-43bdb8ed2845-kube-api-access-7czsx\") pod \"2e15c46d-5a43-4f41-8645-43bdb8ed2845\" (UID: \"2e15c46d-5a43-4f41-8645-43bdb8ed2845\") " Oct 30 17:04:27.862187 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:27.862173 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e15c46d-5a43-4f41-8645-43bdb8ed2845-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e15c46d-5a43-4f41-8645-43bdb8ed2845-maas-api-openshift-ca\") pod \"2e15c46d-5a43-4f41-8645-43bdb8ed2845\" (UID: \"2e15c46d-5a43-4f41-8645-43bdb8ed2845\") " Oct 30 17:04:27.862577 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:27.862547 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e15c46d-5a43-4f41-8645-43bdb8ed2845-maas-api-openshift-ca" pod "2e15c46d-5a43-4f41-8645-43bdb8ed2845" (UID: "2e15c46d-5a43-4f41-8645-43bdb8ed2845"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:04:27.863987 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:27.863964 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e15c46d-5a43-4f41-8645-43bdb8ed2845-maas-api-xks-ca" pod "2e15c46d-5a43-4f41-8645-43bdb8ed2845" (UID: "2e15c46d-5a43-4f41-8645-43bdb8ed2845"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:04:27.864175 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:27.864144 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e15c46d-5a43-4f41-8645-43bdb8ed2845-kube-api-access-7czsx" pod "2e15c46d-5a43-4f41-8645-43bdb8ed2845" (UID: "2e15c46d-5a43-4f41-8645-43bdb8ed2845"). InnerVolumeSpecName "kube-api-access-7czsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:04:27.964134 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:27.964100 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7czsx\" (UniqueName: \"kubernetes.io/projected/2e15c46d-5a43-4f41-8645-43bdb8ed2845-kube-api-access-7czsx\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:04:27.964134 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:27.964128 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e15c46d-5a43-4f41-8645-43bdb8ed2845-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:04:27.964134 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:27.964137 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e15c46d-5a43-4f41-8645-43bdb8ed2845-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:04:28.593723 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:28.593696 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd22682-dvzrn" Oct 30 17:04:28.593919 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:28.593689 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd22682-dvzrn" event={"ID":"2e15c46d-5a43-4f41-8645-43bdb8ed2845","Type":"ContainerDied","Data":"d52c4f656110cb2daffe423e370189241d12b11884358bddaf56112a3ccc82c6"} Oct 30 17:04:28.593919 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:28.593818 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d52c4f656110cb2daffe423e370189241d12b11884358bddaf56112a3ccc82c6" Oct 30 17:04:32.640944 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:32.640837 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1322290-9cgh5"] Oct 30 17:04:32.649015 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:32.641471 4522 cpu_manager.go:414] "containerMap: removing container" podUID="2e15c46d-5a43-4f41-8645-43bdb8ed2845" containerName="revoke-keys" Oct 30 17:04:32.649015 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:32.641484 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e15c46d-5a43-4f41-8645-43bdb8ed2845" containerName="revoke-keys" Oct 30 17:04:32.649015 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:32.641602 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e15c46d-5a43-4f41-8645-43bdb8ed2845" containerName="revoke-keys" Oct 30 17:04:32.649015 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:32.645529 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1322290-9cgh5" Oct 30 17:04:32.666494 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:32.666462 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1322290-9cgh5"] Oct 30 17:04:32.674884 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:32.674858 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:04:32.675010 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:32.674966 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:04:32.737231 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:32.737195 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ab92463-8644-45d5-a8dc-dde087cc8f4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1322290-9cgh5\" (UID: \"1ab92463-8644-45d5-a8dc-dde087cc8f4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1322290-9cgh5" Oct 30 17:04:32.737419 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:32.737252 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ab92463-8644-45d5-a8dc-dde087cc8f4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1322290-9cgh5\" (UID: \"1ab92463-8644-45d5-a8dc-dde087cc8f4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1322290-9cgh5" Oct 30 17:04:32.737497 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:32.737477 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlnkz\" (UniqueName: \"kubernetes.io/projected/1ab92463-8644-45d5-a8dc-dde087cc8f4e-kube-api-access-jlnkz\") pod \"maas-api-revoke-sub-f1322290-9cgh5\" (UID: \"1ab92463-8644-45d5-a8dc-dde087cc8f4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1322290-9cgh5" Oct 30 17:04:32.838765 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:32.838730 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlnkz\" (UniqueName: \"kubernetes.io/projected/1ab92463-8644-45d5-a8dc-dde087cc8f4e-kube-api-access-jlnkz\") pod \"maas-api-revoke-sub-f1322290-9cgh5\" (UID: \"1ab92463-8644-45d5-a8dc-dde087cc8f4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1322290-9cgh5" Oct 30 17:04:32.838932 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:32.838844 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ab92463-8644-45d5-a8dc-dde087cc8f4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1322290-9cgh5\" (UID: \"1ab92463-8644-45d5-a8dc-dde087cc8f4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1322290-9cgh5" Oct 30 17:04:32.838932 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:32.838884 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ab92463-8644-45d5-a8dc-dde087cc8f4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1322290-9cgh5\" (UID: \"1ab92463-8644-45d5-a8dc-dde087cc8f4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1322290-9cgh5" Oct 30 17:04:32.839512 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:32.839491 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ab92463-8644-45d5-a8dc-dde087cc8f4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1322290-9cgh5\" (UID: \"1ab92463-8644-45d5-a8dc-dde087cc8f4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1322290-9cgh5" Oct 30 17:04:32.841398 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:32.841373 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ab92463-8644-45d5-a8dc-dde087cc8f4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1322290-9cgh5\" (UID: \"1ab92463-8644-45d5-a8dc-dde087cc8f4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1322290-9cgh5" Oct 30 17:04:32.865413 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:32.865382 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlnkz\" (UniqueName: \"kubernetes.io/projected/1ab92463-8644-45d5-a8dc-dde087cc8f4e-kube-api-access-jlnkz\") pod \"maas-api-revoke-sub-f1322290-9cgh5\" (UID: \"1ab92463-8644-45d5-a8dc-dde087cc8f4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1322290-9cgh5" Oct 30 17:04:32.968408 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:32.968368 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1322290-9cgh5" Oct 30 17:04:33.279829 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:04:33.279800 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ab92463_8644_45d5_a8dc_dde087cc8f4e.slice/crio-3d9e5897717c9d0b300ba8403589c50b293e61a81ebcf8be9e3d6e7b9ac0acd6 WatchSource:0}: Error finding container 3d9e5897717c9d0b300ba8403589c50b293e61a81ebcf8be9e3d6e7b9ac0acd6: Status 404 returned error can't find the container with id 3d9e5897717c9d0b300ba8403589c50b293e61a81ebcf8be9e3d6e7b9ac0acd6 Oct 30 17:04:33.285001 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:33.284970 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1322290-9cgh5"] Oct 30 17:04:33.630856 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:33.630822 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1322290-9cgh5" event={"ID":"1ab92463-8644-45d5-a8dc-dde087cc8f4e","Type":"ContainerStarted","Data":"b955334b78e41bc6f8553b9870e1f0c61ce2840485b3c8742d3dbf766e1c6dd0"} Oct 30 17:04:33.631010 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:33.630875 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1322290-9cgh5" event={"ID":"1ab92463-8644-45d5-a8dc-dde087cc8f4e","Type":"ContainerStarted","Data":"3d9e5897717c9d0b300ba8403589c50b293e61a81ebcf8be9e3d6e7b9ac0acd6"} Oct 30 17:04:33.682183 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:33.682124 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1322290-9cgh5" podStartSLOduration=1.6821084659999999 podStartE2EDuration="1.682108466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:04:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:04:33.678500518 +0000 UTC m=+1047.679255431" watchObservedRunningTime="2026-10-30 17:04:33.682108466 +0000 UTC m=+1047.682863366" Oct 30 17:04:34.640075 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:34.640042 4522 generic.go:356] "Generic (PLEG): container finished" podID="1ab92463-8644-45d5-a8dc-dde087cc8f4e" containerID="b955334b78e41bc6f8553b9870e1f0c61ce2840485b3c8742d3dbf766e1c6dd0" exitCode=0 Oct 30 17:04:34.640262 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:34.640134 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1322290-9cgh5" event={"ID":"1ab92463-8644-45d5-a8dc-dde087cc8f4e","Type":"ContainerDied","Data":"b955334b78e41bc6f8553b9870e1f0c61ce2840485b3c8742d3dbf766e1c6dd0"} Oct 30 17:04:35.838638 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:35.838612 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1322290-9cgh5" Oct 30 17:04:35.991848 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:35.991815 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ab92463-8644-45d5-a8dc-dde087cc8f4e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ab92463-8644-45d5-a8dc-dde087cc8f4e-maas-api-xks-ca\") pod \"1ab92463-8644-45d5-a8dc-dde087cc8f4e\" (UID: \"1ab92463-8644-45d5-a8dc-dde087cc8f4e\") " Oct 30 17:04:35.992034 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:35.991876 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ab92463-8644-45d5-a8dc-dde087cc8f4e-kube-api-access-jlnkz\" (UniqueName: \"kubernetes.io/projected/1ab92463-8644-45d5-a8dc-dde087cc8f4e-kube-api-access-jlnkz\") pod \"1ab92463-8644-45d5-a8dc-dde087cc8f4e\" (UID: \"1ab92463-8644-45d5-a8dc-dde087cc8f4e\") " Oct 30 17:04:35.992034 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:35.991953 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ab92463-8644-45d5-a8dc-dde087cc8f4e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ab92463-8644-45d5-a8dc-dde087cc8f4e-maas-api-openshift-ca\") pod \"1ab92463-8644-45d5-a8dc-dde087cc8f4e\" (UID: \"1ab92463-8644-45d5-a8dc-dde087cc8f4e\") " Oct 30 17:04:35.992375 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:35.992352 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ab92463-8644-45d5-a8dc-dde087cc8f4e-maas-api-openshift-ca" pod "1ab92463-8644-45d5-a8dc-dde087cc8f4e" (UID: "1ab92463-8644-45d5-a8dc-dde087cc8f4e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:04:35.992668 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:35.992648 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ab92463-8644-45d5-a8dc-dde087cc8f4e-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:04:35.994091 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:35.994067 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ab92463-8644-45d5-a8dc-dde087cc8f4e-maas-api-xks-ca" pod "1ab92463-8644-45d5-a8dc-dde087cc8f4e" (UID: "1ab92463-8644-45d5-a8dc-dde087cc8f4e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:04:35.994173 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:35.994072 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ab92463-8644-45d5-a8dc-dde087cc8f4e-kube-api-access-jlnkz" pod "1ab92463-8644-45d5-a8dc-dde087cc8f4e" (UID: "1ab92463-8644-45d5-a8dc-dde087cc8f4e"). InnerVolumeSpecName "kube-api-access-jlnkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:04:36.094437 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:36.094400 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ab92463-8644-45d5-a8dc-dde087cc8f4e-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:04:36.094437 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:36.094435 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlnkz\" (UniqueName: \"kubernetes.io/projected/1ab92463-8644-45d5-a8dc-dde087cc8f4e-kube-api-access-jlnkz\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:04:36.658633 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:36.658602 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1322290-9cgh5" Oct 30 17:04:36.658831 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:36.658638 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1322290-9cgh5" event={"ID":"1ab92463-8644-45d5-a8dc-dde087cc8f4e","Type":"ContainerDied","Data":"3d9e5897717c9d0b300ba8403589c50b293e61a81ebcf8be9e3d6e7b9ac0acd6"} Oct 30 17:04:36.658831 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:36.658686 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d9e5897717c9d0b300ba8403589c50b293e61a81ebcf8be9e3d6e7b9ac0acd6" Oct 30 17:04:38.253108 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:38.253077 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fc76a9f-c9rfl"] Oct 30 17:04:38.253737 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:38.253723 4522 cpu_manager.go:414] "containerMap: removing container" podUID="1ab92463-8644-45d5-a8dc-dde087cc8f4e" containerName="revoke-keys" Oct 30 17:04:38.253801 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:38.253738 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ab92463-8644-45d5-a8dc-dde087cc8f4e" containerName="revoke-keys" Oct 30 17:04:38.253840 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:38.253828 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ab92463-8644-45d5-a8dc-dde087cc8f4e" containerName="revoke-keys" Oct 30 17:04:38.257547 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:38.257531 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc76a9f-c9rfl" Oct 30 17:04:38.281100 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:38.281065 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fc76a9f-c9rfl"] Oct 30 17:04:38.286754 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:38.286724 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:04:38.286891 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:38.286774 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:04:38.428655 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:38.428623 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk78x\" (UniqueName: \"kubernetes.io/projected/cf80b052-02c6-43a3-87f5-6fd7c59a7c41-kube-api-access-pk78x\") pod \"maas-api-revoke-sub-5fc76a9f-c9rfl\" (UID: \"cf80b052-02c6-43a3-87f5-6fd7c59a7c41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc76a9f-c9rfl" Oct 30 17:04:38.428853 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:38.428683 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf80b052-02c6-43a3-87f5-6fd7c59a7c41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fc76a9f-c9rfl\" (UID: \"cf80b052-02c6-43a3-87f5-6fd7c59a7c41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc76a9f-c9rfl" Oct 30 17:04:38.428917 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:38.428865 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf80b052-02c6-43a3-87f5-6fd7c59a7c41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fc76a9f-c9rfl\" (UID: \"cf80b052-02c6-43a3-87f5-6fd7c59a7c41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc76a9f-c9rfl" Oct 30 17:04:38.530157 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:38.530066 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk78x\" (UniqueName: \"kubernetes.io/projected/cf80b052-02c6-43a3-87f5-6fd7c59a7c41-kube-api-access-pk78x\") pod \"maas-api-revoke-sub-5fc76a9f-c9rfl\" (UID: \"cf80b052-02c6-43a3-87f5-6fd7c59a7c41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc76a9f-c9rfl" Oct 30 17:04:38.530157 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:38.530149 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf80b052-02c6-43a3-87f5-6fd7c59a7c41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fc76a9f-c9rfl\" (UID: \"cf80b052-02c6-43a3-87f5-6fd7c59a7c41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc76a9f-c9rfl" Oct 30 17:04:38.530449 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:38.530279 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf80b052-02c6-43a3-87f5-6fd7c59a7c41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fc76a9f-c9rfl\" (UID: \"cf80b052-02c6-43a3-87f5-6fd7c59a7c41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc76a9f-c9rfl" Oct 30 17:04:38.531178 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:38.531151 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf80b052-02c6-43a3-87f5-6fd7c59a7c41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fc76a9f-c9rfl\" (UID: \"cf80b052-02c6-43a3-87f5-6fd7c59a7c41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc76a9f-c9rfl" Oct 30 17:04:38.533010 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:38.532986 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf80b052-02c6-43a3-87f5-6fd7c59a7c41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fc76a9f-c9rfl\" (UID: \"cf80b052-02c6-43a3-87f5-6fd7c59a7c41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc76a9f-c9rfl" Oct 30 17:04:38.556850 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:38.556825 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk78x\" (UniqueName: \"kubernetes.io/projected/cf80b052-02c6-43a3-87f5-6fd7c59a7c41-kube-api-access-pk78x\") pod \"maas-api-revoke-sub-5fc76a9f-c9rfl\" (UID: \"cf80b052-02c6-43a3-87f5-6fd7c59a7c41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc76a9f-c9rfl" Oct 30 17:04:38.572222 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:38.572198 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc76a9f-c9rfl" Oct 30 17:04:38.877528 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:04:38.877498 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf80b052_02c6_43a3_87f5_6fd7c59a7c41.slice/crio-2c76e566ea8d1dd81967c57388475643718b1ec5db45938bf5a1f4f8d593e492 WatchSource:0}: Error finding container 2c76e566ea8d1dd81967c57388475643718b1ec5db45938bf5a1f4f8d593e492: Status 404 returned error can't find the container with id 2c76e566ea8d1dd81967c57388475643718b1ec5db45938bf5a1f4f8d593e492 Oct 30 17:04:38.880585 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:38.880561 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fc76a9f-c9rfl"] Oct 30 17:04:39.697927 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:39.697842 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc76a9f-c9rfl" event={"ID":"cf80b052-02c6-43a3-87f5-6fd7c59a7c41","Type":"ContainerStarted","Data":"2923fd24d9c676cbe5acec5373d13b1b088485143583cd36ec3d2b85de8aeb70"} Oct 30 17:04:39.697927 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:39.697897 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc76a9f-c9rfl" event={"ID":"cf80b052-02c6-43a3-87f5-6fd7c59a7c41","Type":"ContainerStarted","Data":"2c76e566ea8d1dd81967c57388475643718b1ec5db45938bf5a1f4f8d593e492"} Oct 30 17:04:39.744906 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:39.744857 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc76a9f-c9rfl" podStartSLOduration=1.744844037 podStartE2EDuration="1.744844037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:04:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:04:39.742636843 +0000 UTC m=+1053.743391744" watchObservedRunningTime="2026-10-30 17:04:39.744844037 +0000 UTC m=+1053.745598953" Oct 30 17:04:40.721328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:40.719215 4522 generic.go:356] "Generic (PLEG): container finished" podID="cf80b052-02c6-43a3-87f5-6fd7c59a7c41" containerID="2923fd24d9c676cbe5acec5373d13b1b088485143583cd36ec3d2b85de8aeb70" exitCode=0 Oct 30 17:04:40.721328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:40.719420 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc76a9f-c9rfl" event={"ID":"cf80b052-02c6-43a3-87f5-6fd7c59a7c41","Type":"ContainerDied","Data":"2923fd24d9c676cbe5acec5373d13b1b088485143583cd36ec3d2b85de8aeb70"} Oct 30 17:04:40.733176 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:40.733103 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad0ce664-jlzrk"] Oct 30 17:04:40.743949 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:40.743922 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad0ce664-jlzrk" Oct 30 17:04:40.760652 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:40.760544 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad0ce664-jlzrk"] Oct 30 17:04:40.869595 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:40.869560 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28e852d6-7b4b-41c6-9c45-437db3f4beca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad0ce664-jlzrk\" (UID: \"28e852d6-7b4b-41c6-9c45-437db3f4beca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad0ce664-jlzrk" Oct 30 17:04:40.869766 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:40.869610 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-987wp\" (UniqueName: \"kubernetes.io/projected/28e852d6-7b4b-41c6-9c45-437db3f4beca-kube-api-access-987wp\") pod \"maas-api-revoke-sub-ad0ce664-jlzrk\" (UID: \"28e852d6-7b4b-41c6-9c45-437db3f4beca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad0ce664-jlzrk" Oct 30 17:04:40.869896 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:40.869872 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28e852d6-7b4b-41c6-9c45-437db3f4beca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad0ce664-jlzrk\" (UID: \"28e852d6-7b4b-41c6-9c45-437db3f4beca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad0ce664-jlzrk" Oct 30 17:04:40.990381 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:40.986346 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28e852d6-7b4b-41c6-9c45-437db3f4beca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad0ce664-jlzrk\" (UID: \"28e852d6-7b4b-41c6-9c45-437db3f4beca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad0ce664-jlzrk" Oct 30 17:04:40.990381 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:40.986544 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28e852d6-7b4b-41c6-9c45-437db3f4beca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad0ce664-jlzrk\" (UID: \"28e852d6-7b4b-41c6-9c45-437db3f4beca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad0ce664-jlzrk" Oct 30 17:04:40.990381 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:40.986582 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-987wp\" (UniqueName: \"kubernetes.io/projected/28e852d6-7b4b-41c6-9c45-437db3f4beca-kube-api-access-987wp\") pod \"maas-api-revoke-sub-ad0ce664-jlzrk\" (UID: \"28e852d6-7b4b-41c6-9c45-437db3f4beca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad0ce664-jlzrk" Oct 30 17:04:41.002335 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:40.991177 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28e852d6-7b4b-41c6-9c45-437db3f4beca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad0ce664-jlzrk\" (UID: \"28e852d6-7b4b-41c6-9c45-437db3f4beca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad0ce664-jlzrk" Oct 30 17:04:41.002335 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:41.000349 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28e852d6-7b4b-41c6-9c45-437db3f4beca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad0ce664-jlzrk\" (UID: \"28e852d6-7b4b-41c6-9c45-437db3f4beca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad0ce664-jlzrk" Oct 30 17:04:41.019667 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:41.013907 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-987wp\" (UniqueName: \"kubernetes.io/projected/28e852d6-7b4b-41c6-9c45-437db3f4beca-kube-api-access-987wp\") pod \"maas-api-revoke-sub-ad0ce664-jlzrk\" (UID: \"28e852d6-7b4b-41c6-9c45-437db3f4beca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad0ce664-jlzrk" Oct 30 17:04:41.068472 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:41.068424 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad0ce664-jlzrk" Oct 30 17:04:41.647332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:41.615866 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad0ce664-jlzrk"] Oct 30 17:04:41.731703 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:04:41.712174 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28e852d6_7b4b_41c6_9c45_437db3f4beca.slice/crio-76af9b6c48ebc718df2736f2171c09cafd7293105881803e52d14f78aa5ccd21 WatchSource:0}: Error finding container 76af9b6c48ebc718df2736f2171c09cafd7293105881803e52d14f78aa5ccd21: Status 404 returned error can't find the container with id 76af9b6c48ebc718df2736f2171c09cafd7293105881803e52d14f78aa5ccd21 Oct 30 17:04:42.064328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:42.060359 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-500d6aa3-klfgm"] Oct 30 17:04:42.105644 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:42.105611 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-500d6aa3-klfgm" Oct 30 17:04:42.282328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:42.274764 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-500d6aa3-klfgm"] Oct 30 17:04:42.282328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:42.282251 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d9871ea-340a-4768-8640-9980529335ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-500d6aa3-klfgm\" (UID: \"7d9871ea-340a-4768-8640-9980529335ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-500d6aa3-klfgm" Oct 30 17:04:42.282576 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:42.282334 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sxdw\" (UniqueName: \"kubernetes.io/projected/7d9871ea-340a-4768-8640-9980529335ef-kube-api-access-5sxdw\") pod \"maas-api-revoke-sub-500d6aa3-klfgm\" (UID: \"7d9871ea-340a-4768-8640-9980529335ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-500d6aa3-klfgm" Oct 30 17:04:42.282576 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:42.282473 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d9871ea-340a-4768-8640-9980529335ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-500d6aa3-klfgm\" (UID: \"7d9871ea-340a-4768-8640-9980529335ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-500d6aa3-klfgm" Oct 30 17:04:42.389714 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:42.386137 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d9871ea-340a-4768-8640-9980529335ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-500d6aa3-klfgm\" (UID: \"7d9871ea-340a-4768-8640-9980529335ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-500d6aa3-klfgm" Oct 30 17:04:42.389714 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:42.386211 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sxdw\" (UniqueName: \"kubernetes.io/projected/7d9871ea-340a-4768-8640-9980529335ef-kube-api-access-5sxdw\") pod \"maas-api-revoke-sub-500d6aa3-klfgm\" (UID: \"7d9871ea-340a-4768-8640-9980529335ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-500d6aa3-klfgm" Oct 30 17:04:42.389714 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:42.386352 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d9871ea-340a-4768-8640-9980529335ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-500d6aa3-klfgm\" (UID: \"7d9871ea-340a-4768-8640-9980529335ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-500d6aa3-klfgm" Oct 30 17:04:42.389714 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:42.387788 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d9871ea-340a-4768-8640-9980529335ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-500d6aa3-klfgm\" (UID: \"7d9871ea-340a-4768-8640-9980529335ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-500d6aa3-klfgm" Oct 30 17:04:42.486375 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:42.483120 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d9871ea-340a-4768-8640-9980529335ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-500d6aa3-klfgm\" (UID: \"7d9871ea-340a-4768-8640-9980529335ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-500d6aa3-klfgm" Oct 30 17:04:42.557742 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:42.557694 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" podUID="2f1e3bbb-ab1c-455d-a05d-d62f72c5d009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:42.592761 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:42.582515 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" podUID="4b4ad752-e053-4cea-a3bc-fe2be6f92021" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:42.652389 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:42.644189 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" podUID="4360e9dc-c001-4102-8af6-b55b692fcf2c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:42.994330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:42.976874 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" podUID="43ca764d-ae0e-4858-9841-5bbd0ffdbb85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 17:04:42.994330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:42.977228 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" podUID="a3e5e732-2c67-4377-9755-e2178bf33772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 17:04:42.994330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:42.977501 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" podUID="8a6a85ad-c944-450f-b0bb-e3b64b0f56f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 17:04:42.994330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:42.977730 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" podUID="9bfbc6d3-91fd-41a7-9585-40a8f252f743" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:43.104815 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:43.093821 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" podUID="75321c5d-8bf8-45c6-8811-81981fe3cad4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:43.174334 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:43.172240 4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" podUID="2f1e3bbb-ab1c-455d-a05d-d62f72c5d009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:43.202875 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:43.189363 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" podUID="0b832781-3d41-4496-aa86-e4d92792e590" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 17:04:43.221609 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:43.208932 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sxdw\" (UniqueName: \"kubernetes.io/projected/7d9871ea-340a-4768-8640-9980529335ef-kube-api-access-5sxdw\") pod \"maas-api-revoke-sub-500d6aa3-klfgm\" (UID: \"7d9871ea-340a-4768-8640-9980529335ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-500d6aa3-klfgm" Oct 30 17:04:43.239658 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:43.185052 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad0ce664-jlzrk" event={"ID":"28e852d6-7b4b-41c6-9c45-437db3f4beca","Type":"ContainerStarted","Data":"76af9b6c48ebc718df2736f2171c09cafd7293105881803e52d14f78aa5ccd21"} Oct 30 17:04:43.245431 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:43.240348 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-500d6aa3-klfgm" Oct 30 17:04:43.348168 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:43.348127 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" podUID="c9ad71b7-ad90-4c8c-bf27-0f10b23f1593" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:43.448245 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:43.448192 4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" podUID="0b832781-3d41-4496-aa86-e4d92792e590" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:43.634088 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:43.634000 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc76a9f-c9rfl" Oct 30 17:04:43.638493 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:43.638463 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf80b052-02c6-43a3-87f5-6fd7c59a7c41-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf80b052-02c6-43a3-87f5-6fd7c59a7c41-maas-api-xks-ca\") pod \"cf80b052-02c6-43a3-87f5-6fd7c59a7c41\" (UID: \"cf80b052-02c6-43a3-87f5-6fd7c59a7c41\") " Oct 30 17:04:43.638752 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:43.638730 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf80b052-02c6-43a3-87f5-6fd7c59a7c41-kube-api-access-pk78x\" (UniqueName: \"kubernetes.io/projected/cf80b052-02c6-43a3-87f5-6fd7c59a7c41-kube-api-access-pk78x\") pod \"cf80b052-02c6-43a3-87f5-6fd7c59a7c41\" (UID: \"cf80b052-02c6-43a3-87f5-6fd7c59a7c41\") " Oct 30 17:04:43.638947 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:43.638929 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf80b052-02c6-43a3-87f5-6fd7c59a7c41-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf80b052-02c6-43a3-87f5-6fd7c59a7c41-maas-api-openshift-ca\") pod \"cf80b052-02c6-43a3-87f5-6fd7c59a7c41\" (UID: \"cf80b052-02c6-43a3-87f5-6fd7c59a7c41\") " Oct 30 17:04:43.651986 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:43.651942 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf80b052-02c6-43a3-87f5-6fd7c59a7c41-maas-api-openshift-ca" pod "cf80b052-02c6-43a3-87f5-6fd7c59a7c41" (UID: "cf80b052-02c6-43a3-87f5-6fd7c59a7c41"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:04:43.727875 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:43.727817 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf80b052-02c6-43a3-87f5-6fd7c59a7c41-maas-api-xks-ca" pod "cf80b052-02c6-43a3-87f5-6fd7c59a7c41" (UID: "cf80b052-02c6-43a3-87f5-6fd7c59a7c41"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:04:43.778331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:43.750529 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf80b052-02c6-43a3-87f5-6fd7c59a7c41-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:04:43.778605 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:43.778578 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf80b052-02c6-43a3-87f5-6fd7c59a7c41-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:04:43.851332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:43.848319 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf80b052-02c6-43a3-87f5-6fd7c59a7c41-kube-api-access-pk78x" pod "cf80b052-02c6-43a3-87f5-6fd7c59a7c41" (UID: "cf80b052-02c6-43a3-87f5-6fd7c59a7c41"). InnerVolumeSpecName "kube-api-access-pk78x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:04:43.948022 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:43.912039 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pk78x\" (UniqueName: \"kubernetes.io/projected/cf80b052-02c6-43a3-87f5-6fd7c59a7c41-kube-api-access-pk78x\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:04:44.055926 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:43.978728 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" podUID="9bfbc6d3-91fd-41a7-9585-40a8f252f743" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:44.055926 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:43.979370 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" podUID="8a6a85ad-c944-450f-b0bb-e3b64b0f56f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 17:04:44.055926 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:43.979994 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" Oct 30 17:04:44.055926 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:43.980064 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" Oct 30 17:04:44.112350 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:44.103906 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" podUID="75321c5d-8bf8-45c6-8811-81981fe3cad4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:44.125702 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:44.113646 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" Oct 30 17:04:44.236433 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:44.233714 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" podUID="0b832781-3d41-4496-aa86-e4d92792e590" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:44.375374 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:44.353561 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" podUID="c9ad71b7-ad90-4c8c-bf27-0f10b23f1593" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 17:04:44.469389 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:04:44.468990 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:04:44.559567 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:44.559453 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" podUID="4360e9dc-c001-4102-8af6-b55b692fcf2c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:44.598840 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:44.587399 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" podUID="2f1e3bbb-ab1c-455d-a05d-d62f72c5d009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:44.599667 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:44.587459 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" podUID="4b4ad752-e053-4cea-a3bc-fe2be6f92021" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:44.632998 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:44.632967 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc76a9f-c9rfl" Oct 30 17:04:44.670345 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:44.668765 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" Oct 30 17:04:44.670345 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:44.668826 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" Oct 30 17:04:44.670345 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:44.668844 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc76a9f-c9rfl" event={"ID":"cf80b052-02c6-43a3-87f5-6fd7c59a7c41","Type":"ContainerDied","Data":"2c76e566ea8d1dd81967c57388475643718b1ec5db45938bf5a1f4f8d593e492"} Oct 30 17:04:44.670345 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:44.668882 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c76e566ea8d1dd81967c57388475643718b1ec5db45938bf5a1f4f8d593e492" Oct 30 17:04:44.775349 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:44.751392 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad0ce664-jlzrk" event={"ID":"28e852d6-7b4b-41c6-9c45-437db3f4beca","Type":"ContainerStarted","Data":"6ba783db16a28d987c813cc7365c685286eea5908c8e352ed861e0effbe646be"} Oct 30 17:04:44.982915 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:44.982705 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" podUID="8a6a85ad-c944-450f-b0bb-e3b64b0f56f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:44.993383 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:44.983272 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" podUID="43ca764d-ae0e-4858-9841-5bbd0ffdbb85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:45.060364 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:44.993942 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" podUID="9bfbc6d3-91fd-41a7-9585-40a8f252f743" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 30 17:04:45.065373 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:45.014445 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" podUID="a3e5e732-2c67-4377-9755-e2178bf33772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:45.116175 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:45.116073 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" podUID="75321c5d-8bf8-45c6-8811-81981fe3cad4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:45.255331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:45.238428 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" podUID="0b832781-3d41-4496-aa86-e4d92792e590" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:45.358445 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:45.358399 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" podUID="c9ad71b7-ad90-4c8c-bf27-0f10b23f1593" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:45.560909 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:45.533654 4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" podUID="43ca764d-ae0e-4858-9841-5bbd0ffdbb85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": context deadline exceeded" Oct 30 17:04:45.613849 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:45.603437 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" podUID="2f1e3bbb-ab1c-455d-a05d-d62f72c5d009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:45.613849 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:45.603832 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" Oct 30 17:04:45.637114 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:45.621639 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" podUID="4b4ad752-e053-4cea-a3bc-fe2be6f92021" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Oct 30 17:04:45.637114 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:45.621967 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" Oct 30 17:04:45.637114 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:45.630465 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" podUID="4360e9dc-c001-4102-8af6-b55b692fcf2c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:45.637114 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:45.630771 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" Oct 30 17:04:45.791332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:45.783895 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" Oct 30 17:04:45.817536 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:45.812567 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" Oct 30 17:04:45.838990 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:45.838959 4522 generic.go:356] "Generic (PLEG): container finished" podID="28e852d6-7b4b-41c6-9c45-437db3f4beca" containerID="6ba783db16a28d987c813cc7365c685286eea5908c8e352ed861e0effbe646be" exitCode=0 Oct 30 17:04:45.840502 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:45.840469 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad0ce664-jlzrk" event={"ID":"28e852d6-7b4b-41c6-9c45-437db3f4beca","Type":"ContainerDied","Data":"6ba783db16a28d987c813cc7365c685286eea5908c8e352ed861e0effbe646be"} Oct 30 17:04:45.866613 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:04:45.866112 4522 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28e852d6_7b4b_41c6_9c45_437db3f4beca.slice/crio-conmon-6ba783db16a28d987c813cc7365c685286eea5908c8e352ed861e0effbe646be.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf80b052_02c6_43a3_87f5_6fd7c59a7c41.slice\": RecentStats: unable to find data in memory cache]" Oct 30 17:04:46.082133 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:46.082037 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" podUID="a3e5e732-2c67-4377-9755-e2178bf33772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:46.085349 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:46.084172 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" podUID="9bfbc6d3-91fd-41a7-9585-40a8f252f743" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 30 17:04:46.085349 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:46.085077 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" Oct 30 17:04:46.129034 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:46.128926 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" podUID="75321c5d-8bf8-45c6-8811-81981fe3cad4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:46.241155 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:46.240725 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" podUID="0b832781-3d41-4496-aa86-e4d92792e590" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:46.342870 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:46.342677 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" Oct 30 17:04:46.343089 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:46.343073 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" Oct 30 17:04:46.343204 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:46.343195 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" Oct 30 17:04:46.343378 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:46.343366 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" Oct 30 17:04:46.361343 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:46.361290 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" podUID="c9ad71b7-ad90-4c8c-bf27-0f10b23f1593" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:46.361870 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:46.361771 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-500d6aa3-klfgm"] Oct 30 17:04:46.410132 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:46.410104 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" Oct 30 17:04:46.411708 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:46.411686 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" Oct 30 17:04:46.413554 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:46.413533 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" Oct 30 17:04:46.441716 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:46.441696 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad0ce664-jlzrk" Oct 30 17:04:46.595328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:46.595221 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28e852d6-7b4b-41c6-9c45-437db3f4beca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28e852d6-7b4b-41c6-9c45-437db3f4beca-maas-api-xks-ca\") pod \"28e852d6-7b4b-41c6-9c45-437db3f4beca\" (UID: \"28e852d6-7b4b-41c6-9c45-437db3f4beca\") " Oct 30 17:04:46.595492 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:46.595426 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28e852d6-7b4b-41c6-9c45-437db3f4beca-kube-api-access-987wp\" (UniqueName: \"kubernetes.io/projected/28e852d6-7b4b-41c6-9c45-437db3f4beca-kube-api-access-987wp\") pod \"28e852d6-7b4b-41c6-9c45-437db3f4beca\" (UID: \"28e852d6-7b4b-41c6-9c45-437db3f4beca\") " Oct 30 17:04:46.595492 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:46.595465 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28e852d6-7b4b-41c6-9c45-437db3f4beca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28e852d6-7b4b-41c6-9c45-437db3f4beca-maas-api-openshift-ca\") pod \"28e852d6-7b4b-41c6-9c45-437db3f4beca\" (UID: \"28e852d6-7b4b-41c6-9c45-437db3f4beca\") " Oct 30 17:04:46.596515 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:46.596486 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28e852d6-7b4b-41c6-9c45-437db3f4beca-maas-api-openshift-ca" pod "28e852d6-7b4b-41c6-9c45-437db3f4beca" (UID: "28e852d6-7b4b-41c6-9c45-437db3f4beca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:04:46.598400 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:46.598280 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28e852d6-7b4b-41c6-9c45-437db3f4beca-maas-api-xks-ca" pod "28e852d6-7b4b-41c6-9c45-437db3f4beca" (UID: "28e852d6-7b4b-41c6-9c45-437db3f4beca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:04:46.598524 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:46.598497 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28e852d6-7b4b-41c6-9c45-437db3f4beca-kube-api-access-987wp" pod "28e852d6-7b4b-41c6-9c45-437db3f4beca" (UID: "28e852d6-7b4b-41c6-9c45-437db3f4beca"). InnerVolumeSpecName "kube-api-access-987wp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:04:46.697674 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:46.697641 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28e852d6-7b4b-41c6-9c45-437db3f4beca-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:04:46.697674 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:46.697676 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-987wp\" (UniqueName: \"kubernetes.io/projected/28e852d6-7b4b-41c6-9c45-437db3f4beca-kube-api-access-987wp\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:04:46.697891 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:46.697693 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28e852d6-7b4b-41c6-9c45-437db3f4beca-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:04:46.850493 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:46.850394 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-500d6aa3-klfgm" event={"ID":"7d9871ea-340a-4768-8640-9980529335ef","Type":"ContainerStarted","Data":"51eea853c3ed5b0566a65665a435f912cb68ccde063ae2fd3d6b30a19e2243b5"} Oct 30 17:04:46.850493 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:46.850461 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-500d6aa3-klfgm" event={"ID":"7d9871ea-340a-4768-8640-9980529335ef","Type":"ContainerStarted","Data":"40ad43865f2c5c1f51aa424af32030f3e106cfd4af22b7d4e6fc9d2a8494dde8"} Oct 30 17:04:46.852040 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:46.852017 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad0ce664-jlzrk" Oct 30 17:04:46.852173 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:46.852053 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad0ce664-jlzrk" event={"ID":"28e852d6-7b4b-41c6-9c45-437db3f4beca","Type":"ContainerDied","Data":"76af9b6c48ebc718df2736f2171c09cafd7293105881803e52d14f78aa5ccd21"} Oct 30 17:04:46.852173 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:46.852094 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76af9b6c48ebc718df2736f2171c09cafd7293105881803e52d14f78aa5ccd21" Oct 30 17:04:46.896831 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:46.896777 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-500d6aa3-klfgm" podStartSLOduration=5.896762466 podStartE2EDuration="5.896762466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:04:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:04:46.891976323 +0000 UTC m=+1060.892731251" watchObservedRunningTime="2026-10-30 17:04:46.896762466 +0000 UTC m=+1060.897517428" Oct 30 17:04:47.860924 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:47.860890 4522 generic.go:356] "Generic (PLEG): container finished" podID="7d9871ea-340a-4768-8640-9980529335ef" containerID="51eea853c3ed5b0566a65665a435f912cb68ccde063ae2fd3d6b30a19e2243b5" exitCode=0 Oct 30 17:04:47.861345 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:47.860936 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-500d6aa3-klfgm" event={"ID":"7d9871ea-340a-4768-8640-9980529335ef","Type":"ContainerDied","Data":"51eea853c3ed5b0566a65665a435f912cb68ccde063ae2fd3d6b30a19e2243b5"} Oct 30 17:04:49.032105 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:49.032079 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-500d6aa3-klfgm" Oct 30 17:04:49.130809 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:49.130717 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d9871ea-340a-4768-8640-9980529335ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d9871ea-340a-4768-8640-9980529335ef-maas-api-openshift-ca\") pod \"7d9871ea-340a-4768-8640-9980529335ef\" (UID: \"7d9871ea-340a-4768-8640-9980529335ef\") " Oct 30 17:04:49.130809 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:49.130766 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d9871ea-340a-4768-8640-9980529335ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d9871ea-340a-4768-8640-9980529335ef-maas-api-xks-ca\") pod \"7d9871ea-340a-4768-8640-9980529335ef\" (UID: \"7d9871ea-340a-4768-8640-9980529335ef\") " Oct 30 17:04:49.131050 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:49.130894 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d9871ea-340a-4768-8640-9980529335ef-kube-api-access-5sxdw\" (UniqueName: \"kubernetes.io/projected/7d9871ea-340a-4768-8640-9980529335ef-kube-api-access-5sxdw\") pod \"7d9871ea-340a-4768-8640-9980529335ef\" (UID: \"7d9871ea-340a-4768-8640-9980529335ef\") " Oct 30 17:04:49.131162 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:49.131132 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d9871ea-340a-4768-8640-9980529335ef-maas-api-openshift-ca" pod "7d9871ea-340a-4768-8640-9980529335ef" (UID: "7d9871ea-340a-4768-8640-9980529335ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:04:49.131352 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:49.131334 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d9871ea-340a-4768-8640-9980529335ef-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:04:49.132972 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:49.132948 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d9871ea-340a-4768-8640-9980529335ef-kube-api-access-5sxdw" pod "7d9871ea-340a-4768-8640-9980529335ef" (UID: "7d9871ea-340a-4768-8640-9980529335ef"). InnerVolumeSpecName "kube-api-access-5sxdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:04:49.133126 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:49.133110 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d9871ea-340a-4768-8640-9980529335ef-maas-api-xks-ca" pod "7d9871ea-340a-4768-8640-9980529335ef" (UID: "7d9871ea-340a-4768-8640-9980529335ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:04:49.232919 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:49.232889 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sxdw\" (UniqueName: \"kubernetes.io/projected/7d9871ea-340a-4768-8640-9980529335ef-kube-api-access-5sxdw\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:04:49.232919 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:49.232916 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d9871ea-340a-4768-8640-9980529335ef-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:04:49.879139 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:49.879114 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-500d6aa3-klfgm" Oct 30 17:04:49.879139 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:49.879114 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-500d6aa3-klfgm" event={"ID":"7d9871ea-340a-4768-8640-9980529335ef","Type":"ContainerDied","Data":"40ad43865f2c5c1f51aa424af32030f3e106cfd4af22b7d4e6fc9d2a8494dde8"} Oct 30 17:04:49.879370 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:49.879157 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40ad43865f2c5c1f51aa424af32030f3e106cfd4af22b7d4e6fc9d2a8494dde8" Oct 30 17:04:51.083162 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:51.083130 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e52a6c23-kxgkf"] Oct 30 17:04:51.083847 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:51.083830 4522 cpu_manager.go:414] "containerMap: removing container" podUID="28e852d6-7b4b-41c6-9c45-437db3f4beca" containerName="revoke-keys" Oct 30 17:04:51.083903 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:51.083848 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28e852d6-7b4b-41c6-9c45-437db3f4beca" containerName="revoke-keys" Oct 30 17:04:51.083903 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:51.083873 4522 cpu_manager.go:414] "containerMap: removing container" podUID="cf80b052-02c6-43a3-87f5-6fd7c59a7c41" containerName="revoke-keys" Oct 30 17:04:51.083903 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:51.083878 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf80b052-02c6-43a3-87f5-6fd7c59a7c41" containerName="revoke-keys" Oct 30 17:04:51.083903 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:51.083886 4522 cpu_manager.go:414] "containerMap: removing container" podUID="7d9871ea-340a-4768-8640-9980529335ef" containerName="revoke-keys" Oct 30 17:04:51.083903 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:51.083891 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d9871ea-340a-4768-8640-9980529335ef" containerName="revoke-keys" Oct 30 17:04:51.084055 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:51.083980 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="28e852d6-7b4b-41c6-9c45-437db3f4beca" containerName="revoke-keys" Oct 30 17:04:51.084055 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:51.083991 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf80b052-02c6-43a3-87f5-6fd7c59a7c41" containerName="revoke-keys" Oct 30 17:04:51.084055 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:51.083998 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d9871ea-340a-4768-8640-9980529335ef" containerName="revoke-keys" Oct 30 17:04:51.087961 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:51.087939 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e52a6c23-kxgkf" Oct 30 17:04:51.109944 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:51.109912 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e52a6c23-kxgkf"] Oct 30 17:04:51.116392 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:51.116367 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:04:51.116549 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:51.116532 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:04:51.158333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:51.158287 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxdtt\" (UniqueName: \"kubernetes.io/projected/e5803ca4-1b00-4cf8-a466-77c4314d10fc-kube-api-access-kxdtt\") pod \"maas-api-revoke-sub-e52a6c23-kxgkf\" (UID: \"e5803ca4-1b00-4cf8-a466-77c4314d10fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e52a6c23-kxgkf" Oct 30 17:04:51.158587 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:51.158563 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5803ca4-1b00-4cf8-a466-77c4314d10fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e52a6c23-kxgkf\" (UID: \"e5803ca4-1b00-4cf8-a466-77c4314d10fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e52a6c23-kxgkf" Oct 30 17:04:51.158635 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:51.158604 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5803ca4-1b00-4cf8-a466-77c4314d10fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e52a6c23-kxgkf\" (UID: \"e5803ca4-1b00-4cf8-a466-77c4314d10fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e52a6c23-kxgkf" Oct 30 17:04:51.260412 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:51.260372 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxdtt\" (UniqueName: \"kubernetes.io/projected/e5803ca4-1b00-4cf8-a466-77c4314d10fc-kube-api-access-kxdtt\") pod \"maas-api-revoke-sub-e52a6c23-kxgkf\" (UID: \"e5803ca4-1b00-4cf8-a466-77c4314d10fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e52a6c23-kxgkf" Oct 30 17:04:51.260625 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:51.260587 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5803ca4-1b00-4cf8-a466-77c4314d10fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e52a6c23-kxgkf\" (UID: \"e5803ca4-1b00-4cf8-a466-77c4314d10fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e52a6c23-kxgkf" Oct 30 17:04:51.260625 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:51.260616 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5803ca4-1b00-4cf8-a466-77c4314d10fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e52a6c23-kxgkf\" (UID: \"e5803ca4-1b00-4cf8-a466-77c4314d10fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e52a6c23-kxgkf" Oct 30 17:04:51.261340 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:51.261284 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5803ca4-1b00-4cf8-a466-77c4314d10fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e52a6c23-kxgkf\" (UID: \"e5803ca4-1b00-4cf8-a466-77c4314d10fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e52a6c23-kxgkf" Oct 30 17:04:51.262975 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:51.262953 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5803ca4-1b00-4cf8-a466-77c4314d10fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e52a6c23-kxgkf\" (UID: \"e5803ca4-1b00-4cf8-a466-77c4314d10fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e52a6c23-kxgkf" Oct 30 17:04:51.286902 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:51.286877 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxdtt\" (UniqueName: \"kubernetes.io/projected/e5803ca4-1b00-4cf8-a466-77c4314d10fc-kube-api-access-kxdtt\") pod \"maas-api-revoke-sub-e52a6c23-kxgkf\" (UID: \"e5803ca4-1b00-4cf8-a466-77c4314d10fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e52a6c23-kxgkf" Oct 30 17:04:51.404903 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:51.404816 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e52a6c23-kxgkf" Oct 30 17:04:51.711590 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:04:51.711554 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5803ca4_1b00_4cf8_a466_77c4314d10fc.slice/crio-be796ef386e73b876aca5242626fef1da372db384087764e52e09c230f2e5e8c WatchSource:0}: Error finding container be796ef386e73b876aca5242626fef1da372db384087764e52e09c230f2e5e8c: Status 404 returned error can't find the container with id be796ef386e73b876aca5242626fef1da372db384087764e52e09c230f2e5e8c Oct 30 17:04:51.714836 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:51.714814 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e52a6c23-kxgkf"] Oct 30 17:04:51.896067 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:51.896036 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e52a6c23-kxgkf" event={"ID":"e5803ca4-1b00-4cf8-a466-77c4314d10fc","Type":"ContainerStarted","Data":"be796ef386e73b876aca5242626fef1da372db384087764e52e09c230f2e5e8c"} Oct 30 17:04:52.096014 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:52.095980 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff01347a-plj8r"] Oct 30 17:04:52.100577 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:52.100558 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff01347a-plj8r" Oct 30 17:04:52.121598 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:52.121570 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff01347a-plj8r"] Oct 30 17:04:52.174350 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:52.174298 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29f6p\" (UniqueName: \"kubernetes.io/projected/b5e2c1aa-a72b-4fdf-ad50-6714890937b1-kube-api-access-29f6p\") pod \"maas-api-revoke-sub-ff01347a-plj8r\" (UID: \"b5e2c1aa-a72b-4fdf-ad50-6714890937b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff01347a-plj8r" Oct 30 17:04:52.174538 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:52.174365 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5e2c1aa-a72b-4fdf-ad50-6714890937b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff01347a-plj8r\" (UID: \"b5e2c1aa-a72b-4fdf-ad50-6714890937b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff01347a-plj8r" Oct 30 17:04:52.174711 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:52.174689 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5e2c1aa-a72b-4fdf-ad50-6714890937b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff01347a-plj8r\" (UID: \"b5e2c1aa-a72b-4fdf-ad50-6714890937b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff01347a-plj8r" Oct 30 17:04:52.276660 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:52.276563 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5e2c1aa-a72b-4fdf-ad50-6714890937b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff01347a-plj8r\" (UID: \"b5e2c1aa-a72b-4fdf-ad50-6714890937b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff01347a-plj8r" Oct 30 17:04:52.276824 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:52.276690 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29f6p\" (UniqueName: \"kubernetes.io/projected/b5e2c1aa-a72b-4fdf-ad50-6714890937b1-kube-api-access-29f6p\") pod \"maas-api-revoke-sub-ff01347a-plj8r\" (UID: \"b5e2c1aa-a72b-4fdf-ad50-6714890937b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff01347a-plj8r" Oct 30 17:04:52.276824 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:52.276724 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5e2c1aa-a72b-4fdf-ad50-6714890937b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff01347a-plj8r\" (UID: \"b5e2c1aa-a72b-4fdf-ad50-6714890937b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff01347a-plj8r" Oct 30 17:04:52.277342 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:52.277317 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5e2c1aa-a72b-4fdf-ad50-6714890937b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff01347a-plj8r\" (UID: \"b5e2c1aa-a72b-4fdf-ad50-6714890937b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff01347a-plj8r" Oct 30 17:04:52.279270 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:52.279245 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5e2c1aa-a72b-4fdf-ad50-6714890937b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff01347a-plj8r\" (UID: \"b5e2c1aa-a72b-4fdf-ad50-6714890937b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff01347a-plj8r" Oct 30 17:04:52.303421 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:52.303384 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29f6p\" (UniqueName: \"kubernetes.io/projected/b5e2c1aa-a72b-4fdf-ad50-6714890937b1-kube-api-access-29f6p\") pod \"maas-api-revoke-sub-ff01347a-plj8r\" (UID: \"b5e2c1aa-a72b-4fdf-ad50-6714890937b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff01347a-plj8r" Oct 30 17:04:52.414516 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:52.414467 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff01347a-plj8r" Oct 30 17:04:52.664239 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:52.664200 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-04248e/distinct-w1-04248e-kserve-5859d49965-kgq46"] Oct 30 17:04:52.669497 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:52.669471 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-04248e/distinct-w1-04248e-kserve-5859d49965-kgq46" Oct 30 17:04:52.688231 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:52.688202 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-04248e/distinct-w1-04248e-kserve-5859d49965-kgq46"] Oct 30 17:04:52.688808 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:52.688733 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-04248e\"/\"openshift-service-ca.crt\"" Oct 30 17:04:52.688941 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:52.688847 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-04248e\"/\"distinct-w1-04248e-kserve-self-signed-certs\"" Oct 30 17:04:52.688941 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:52.688856 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-04248e\"/\"default-dockercfg-hpx7q\"" Oct 30 17:04:52.688941 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:52.688856 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-04248e\"/\"kube-root-ca.crt\"" Oct 30 17:04:52.759710 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:04:52.759668 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5e2c1aa_a72b_4fdf_ad50_6714890937b1.slice/crio-0882bb3d82c851a377bd03363ddbb0a601e80357c34aa2dd926f986ed8c79747 WatchSource:0}: Error finding container 0882bb3d82c851a377bd03363ddbb0a601e80357c34aa2dd926f986ed8c79747: Status 404 returned error can't find the container with id 0882bb3d82c851a377bd03363ddbb0a601e80357c34aa2dd926f986ed8c79747 Oct 30 17:04:52.762447 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:52.762427 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff01347a-plj8r"] Oct 30 17:04:52.784479 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:52.784455 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dda5ac4-30dd-432e-81eb-cb6d39800628-home\") pod \"distinct-w1-04248e-kserve-5859d49965-kgq46\" (UID: \"8dda5ac4-30dd-432e-81eb-cb6d39800628\") " pod="e2e-models-e2e-worker-w1-04248e/distinct-w1-04248e-kserve-5859d49965-kgq46" Oct 30 17:04:52.784601 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:52.784584 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dda5ac4-30dd-432e-81eb-cb6d39800628-tls-certs\") pod \"distinct-w1-04248e-kserve-5859d49965-kgq46\" (UID: \"8dda5ac4-30dd-432e-81eb-cb6d39800628\") " pod="e2e-models-e2e-worker-w1-04248e/distinct-w1-04248e-kserve-5859d49965-kgq46" Oct 30 17:04:52.784672 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:52.784653 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrkfp\" (UniqueName: \"kubernetes.io/projected/8dda5ac4-30dd-432e-81eb-cb6d39800628-kube-api-access-vrkfp\") pod \"distinct-w1-04248e-kserve-5859d49965-kgq46\" (UID: \"8dda5ac4-30dd-432e-81eb-cb6d39800628\") " pod="e2e-models-e2e-worker-w1-04248e/distinct-w1-04248e-kserve-5859d49965-kgq46" Oct 30 17:04:52.784785 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:52.784774 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dda5ac4-30dd-432e-81eb-cb6d39800628-tmp-dir\") pod \"distinct-w1-04248e-kserve-5859d49965-kgq46\" (UID: \"8dda5ac4-30dd-432e-81eb-cb6d39800628\") " pod="e2e-models-e2e-worker-w1-04248e/distinct-w1-04248e-kserve-5859d49965-kgq46" Oct 30 17:04:52.784913 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:52.784900 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dda5ac4-30dd-432e-81eb-cb6d39800628-dshm\") pod \"distinct-w1-04248e-kserve-5859d49965-kgq46\" (UID: \"8dda5ac4-30dd-432e-81eb-cb6d39800628\") " pod="e2e-models-e2e-worker-w1-04248e/distinct-w1-04248e-kserve-5859d49965-kgq46" Oct 30 17:04:52.784946 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:52.784921 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dda5ac4-30dd-432e-81eb-cb6d39800628-model-cache\") pod \"distinct-w1-04248e-kserve-5859d49965-kgq46\" (UID: \"8dda5ac4-30dd-432e-81eb-cb6d39800628\") " pod="e2e-models-e2e-worker-w1-04248e/distinct-w1-04248e-kserve-5859d49965-kgq46" Oct 30 17:04:52.886945 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:52.886842 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dda5ac4-30dd-432e-81eb-cb6d39800628-dshm\") pod \"distinct-w1-04248e-kserve-5859d49965-kgq46\" (UID: \"8dda5ac4-30dd-432e-81eb-cb6d39800628\") " pod="e2e-models-e2e-worker-w1-04248e/distinct-w1-04248e-kserve-5859d49965-kgq46" Oct 30 17:04:52.886945 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:52.886890 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dda5ac4-30dd-432e-81eb-cb6d39800628-model-cache\") pod \"distinct-w1-04248e-kserve-5859d49965-kgq46\" (UID: \"8dda5ac4-30dd-432e-81eb-cb6d39800628\") " pod="e2e-models-e2e-worker-w1-04248e/distinct-w1-04248e-kserve-5859d49965-kgq46" Oct 30 17:04:52.886945 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:52.886922 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dda5ac4-30dd-432e-81eb-cb6d39800628-home\") pod \"distinct-w1-04248e-kserve-5859d49965-kgq46\" (UID: \"8dda5ac4-30dd-432e-81eb-cb6d39800628\") " pod="e2e-models-e2e-worker-w1-04248e/distinct-w1-04248e-kserve-5859d49965-kgq46" Oct 30 17:04:52.887223 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:52.887048 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dda5ac4-30dd-432e-81eb-cb6d39800628-tls-certs\") pod \"distinct-w1-04248e-kserve-5859d49965-kgq46\" (UID: \"8dda5ac4-30dd-432e-81eb-cb6d39800628\") " pod="e2e-models-e2e-worker-w1-04248e/distinct-w1-04248e-kserve-5859d49965-kgq46" Oct 30 17:04:52.887223 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:52.887096 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrkfp\" (UniqueName: \"kubernetes.io/projected/8dda5ac4-30dd-432e-81eb-cb6d39800628-kube-api-access-vrkfp\") pod \"distinct-w1-04248e-kserve-5859d49965-kgq46\" (UID: \"8dda5ac4-30dd-432e-81eb-cb6d39800628\") " pod="e2e-models-e2e-worker-w1-04248e/distinct-w1-04248e-kserve-5859d49965-kgq46" Oct 30 17:04:52.887368 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:52.887237 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dda5ac4-30dd-432e-81eb-cb6d39800628-tmp-dir\") pod \"distinct-w1-04248e-kserve-5859d49965-kgq46\" (UID: \"8dda5ac4-30dd-432e-81eb-cb6d39800628\") " pod="e2e-models-e2e-worker-w1-04248e/distinct-w1-04248e-kserve-5859d49965-kgq46" Oct 30 17:04:52.888068 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:52.888034 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dda5ac4-30dd-432e-81eb-cb6d39800628-model-cache\") pod \"distinct-w1-04248e-kserve-5859d49965-kgq46\" (UID: \"8dda5ac4-30dd-432e-81eb-cb6d39800628\") " pod="e2e-models-e2e-worker-w1-04248e/distinct-w1-04248e-kserve-5859d49965-kgq46" Oct 30 17:04:52.888272 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:52.888239 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dda5ac4-30dd-432e-81eb-cb6d39800628-tmp-dir\") pod \"distinct-w1-04248e-kserve-5859d49965-kgq46\" (UID: \"8dda5ac4-30dd-432e-81eb-cb6d39800628\") " pod="e2e-models-e2e-worker-w1-04248e/distinct-w1-04248e-kserve-5859d49965-kgq46" Oct 30 17:04:52.888580 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:52.888496 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dda5ac4-30dd-432e-81eb-cb6d39800628-home\") pod \"distinct-w1-04248e-kserve-5859d49965-kgq46\" (UID: \"8dda5ac4-30dd-432e-81eb-cb6d39800628\") " pod="e2e-models-e2e-worker-w1-04248e/distinct-w1-04248e-kserve-5859d49965-kgq46" Oct 30 17:04:52.890656 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:52.890630 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dda5ac4-30dd-432e-81eb-cb6d39800628-dshm\") pod \"distinct-w1-04248e-kserve-5859d49965-kgq46\" (UID: \"8dda5ac4-30dd-432e-81eb-cb6d39800628\") " pod="e2e-models-e2e-worker-w1-04248e/distinct-w1-04248e-kserve-5859d49965-kgq46" Oct 30 17:04:52.890949 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:52.890924 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dda5ac4-30dd-432e-81eb-cb6d39800628-tls-certs\") pod \"distinct-w1-04248e-kserve-5859d49965-kgq46\" (UID: \"8dda5ac4-30dd-432e-81eb-cb6d39800628\") " pod="e2e-models-e2e-worker-w1-04248e/distinct-w1-04248e-kserve-5859d49965-kgq46" Oct 30 17:04:52.908640 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:52.908611 4522 generic.go:356] "Generic (PLEG): container finished" podID="e5803ca4-1b00-4cf8-a466-77c4314d10fc" containerID="acc2bbce2d157acaca9c817854f37aa397f0cc943d172ae5dc677fdc848f2566" exitCode=0 Oct 30 17:04:52.908778 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:52.908700 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e52a6c23-kxgkf" event={"ID":"e5803ca4-1b00-4cf8-a466-77c4314d10fc","Type":"ContainerDied","Data":"acc2bbce2d157acaca9c817854f37aa397f0cc943d172ae5dc677fdc848f2566"} Oct 30 17:04:52.913216 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:52.911605 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff01347a-plj8r" event={"ID":"b5e2c1aa-a72b-4fdf-ad50-6714890937b1","Type":"ContainerStarted","Data":"0882bb3d82c851a377bd03363ddbb0a601e80357c34aa2dd926f986ed8c79747"} Oct 30 17:04:52.926041 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:52.925763 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrkfp\" (UniqueName: \"kubernetes.io/projected/8dda5ac4-30dd-432e-81eb-cb6d39800628-kube-api-access-vrkfp\") pod \"distinct-w1-04248e-kserve-5859d49965-kgq46\" (UID: \"8dda5ac4-30dd-432e-81eb-cb6d39800628\") " pod="e2e-models-e2e-worker-w1-04248e/distinct-w1-04248e-kserve-5859d49965-kgq46" Oct 30 17:04:52.986105 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:52.986070 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-04248e/distinct-w1-04248e-kserve-5859d49965-kgq46" Oct 30 17:04:53.306580 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:04:53.306550 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8dda5ac4_30dd_432e_81eb_cb6d39800628.slice/crio-6230ec148b5348c7196d8a846d8baaec77975bb33548d680f33794ec680846ec WatchSource:0}: Error finding container 6230ec148b5348c7196d8a846d8baaec77975bb33548d680f33794ec680846ec: Status 404 returned error can't find the container with id 6230ec148b5348c7196d8a846d8baaec77975bb33548d680f33794ec680846ec Oct 30 17:04:53.309823 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:53.309795 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-04248e/distinct-w1-04248e-kserve-5859d49965-kgq46"] Oct 30 17:04:53.921156 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:53.921120 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-04248e/distinct-w1-04248e-kserve-5859d49965-kgq46" event={"ID":"8dda5ac4-30dd-432e-81eb-cb6d39800628","Type":"ContainerStarted","Data":"26b7d27f48006df06e7014bee645a72ebe5ba485469fb4b64fe85d7fe30c94d9"} Oct 30 17:04:53.921346 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:53.921167 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-04248e/distinct-w1-04248e-kserve-5859d49965-kgq46" event={"ID":"8dda5ac4-30dd-432e-81eb-cb6d39800628","Type":"ContainerStarted","Data":"6230ec148b5348c7196d8a846d8baaec77975bb33548d680f33794ec680846ec"} Oct 30 17:04:53.922802 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:53.922776 4522 generic.go:356] "Generic (PLEG): container finished" podID="b5e2c1aa-a72b-4fdf-ad50-6714890937b1" containerID="d68ddbd3391a428c3d9ff3ad458209be35ee5a438b325b63c88b8430c5d6c808" exitCode=0 Oct 30 17:04:53.922935 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:53.922829 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff01347a-plj8r" event={"ID":"b5e2c1aa-a72b-4fdf-ad50-6714890937b1","Type":"ContainerDied","Data":"d68ddbd3391a428c3d9ff3ad458209be35ee5a438b325b63c88b8430c5d6c808"} Oct 30 17:04:53.969329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:53.969219 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-04248e/distinct-w1-04248e-kserve-5859d49965-kgq46" podStartSLOduration=1.969204451 podStartE2EDuration="1.969204451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:04:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:04:53.966154799 +0000 UTC m=+1067.966909699" watchObservedRunningTime="2026-10-30 17:04:53.969204451 +0000 UTC m=+1067.969959352" Oct 30 17:04:53.986231 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:53.986188 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04248e/distinct-w1-04248e-kserve-5859d49965-kgq46" Oct 30 17:04:54.073698 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:54.073670 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e52a6c23-kxgkf" Oct 30 17:04:54.206621 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:54.206586 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5803ca4-1b00-4cf8-a466-77c4314d10fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5803ca4-1b00-4cf8-a466-77c4314d10fc-maas-api-openshift-ca\") pod \"e5803ca4-1b00-4cf8-a466-77c4314d10fc\" (UID: \"e5803ca4-1b00-4cf8-a466-77c4314d10fc\") " Oct 30 17:04:54.206817 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:54.206633 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5803ca4-1b00-4cf8-a466-77c4314d10fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5803ca4-1b00-4cf8-a466-77c4314d10fc-maas-api-xks-ca\") pod \"e5803ca4-1b00-4cf8-a466-77c4314d10fc\" (UID: \"e5803ca4-1b00-4cf8-a466-77c4314d10fc\") " Oct 30 17:04:54.206817 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:54.206681 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5803ca4-1b00-4cf8-a466-77c4314d10fc-kube-api-access-kxdtt\" (UniqueName: \"kubernetes.io/projected/e5803ca4-1b00-4cf8-a466-77c4314d10fc-kube-api-access-kxdtt\") pod \"e5803ca4-1b00-4cf8-a466-77c4314d10fc\" (UID: \"e5803ca4-1b00-4cf8-a466-77c4314d10fc\") " Oct 30 17:04:54.207044 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:54.207019 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5803ca4-1b00-4cf8-a466-77c4314d10fc-maas-api-openshift-ca" pod "e5803ca4-1b00-4cf8-a466-77c4314d10fc" (UID: "e5803ca4-1b00-4cf8-a466-77c4314d10fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:04:54.207382 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:54.207359 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5803ca4-1b00-4cf8-a466-77c4314d10fc-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:04:54.208885 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:54.208865 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5803ca4-1b00-4cf8-a466-77c4314d10fc-maas-api-xks-ca" pod "e5803ca4-1b00-4cf8-a466-77c4314d10fc" (UID: "e5803ca4-1b00-4cf8-a466-77c4314d10fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:04:54.208954 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:54.208932 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5803ca4-1b00-4cf8-a466-77c4314d10fc-kube-api-access-kxdtt" pod "e5803ca4-1b00-4cf8-a466-77c4314d10fc" (UID: "e5803ca4-1b00-4cf8-a466-77c4314d10fc"). InnerVolumeSpecName "kube-api-access-kxdtt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:04:54.308465 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:54.308428 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxdtt\" (UniqueName: \"kubernetes.io/projected/e5803ca4-1b00-4cf8-a466-77c4314d10fc-kube-api-access-kxdtt\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:04:54.308465 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:54.308456 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5803ca4-1b00-4cf8-a466-77c4314d10fc-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:04:54.932027 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:54.931987 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e52a6c23-kxgkf" Oct 30 17:04:54.932027 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:54.932012 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e52a6c23-kxgkf" event={"ID":"e5803ca4-1b00-4cf8-a466-77c4314d10fc","Type":"ContainerDied","Data":"be796ef386e73b876aca5242626fef1da372db384087764e52e09c230f2e5e8c"} Oct 30 17:04:54.932260 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:54.932062 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be796ef386e73b876aca5242626fef1da372db384087764e52e09c230f2e5e8c" Oct 30 17:04:55.095036 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:55.095010 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff01347a-plj8r" Oct 30 17:04:55.220682 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:55.220650 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5e2c1aa-a72b-4fdf-ad50-6714890937b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5e2c1aa-a72b-4fdf-ad50-6714890937b1-maas-api-openshift-ca\") pod \"b5e2c1aa-a72b-4fdf-ad50-6714890937b1\" (UID: \"b5e2c1aa-a72b-4fdf-ad50-6714890937b1\") " Oct 30 17:04:55.220851 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:55.220734 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5e2c1aa-a72b-4fdf-ad50-6714890937b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5e2c1aa-a72b-4fdf-ad50-6714890937b1-maas-api-xks-ca\") pod \"b5e2c1aa-a72b-4fdf-ad50-6714890937b1\" (UID: \"b5e2c1aa-a72b-4fdf-ad50-6714890937b1\") " Oct 30 17:04:55.220918 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:55.220889 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5e2c1aa-a72b-4fdf-ad50-6714890937b1-kube-api-access-29f6p\" (UniqueName: \"kubernetes.io/projected/b5e2c1aa-a72b-4fdf-ad50-6714890937b1-kube-api-access-29f6p\") pod \"b5e2c1aa-a72b-4fdf-ad50-6714890937b1\" (UID: \"b5e2c1aa-a72b-4fdf-ad50-6714890937b1\") " Oct 30 17:04:55.221027 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:55.221005 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5e2c1aa-a72b-4fdf-ad50-6714890937b1-maas-api-openshift-ca" pod "b5e2c1aa-a72b-4fdf-ad50-6714890937b1" (UID: "b5e2c1aa-a72b-4fdf-ad50-6714890937b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:04:55.221444 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:55.221409 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5e2c1aa-a72b-4fdf-ad50-6714890937b1-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:04:55.223062 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:55.223033 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5e2c1aa-a72b-4fdf-ad50-6714890937b1-maas-api-xks-ca" pod "b5e2c1aa-a72b-4fdf-ad50-6714890937b1" (UID: "b5e2c1aa-a72b-4fdf-ad50-6714890937b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:04:55.223149 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:55.223113 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5e2c1aa-a72b-4fdf-ad50-6714890937b1-kube-api-access-29f6p" pod "b5e2c1aa-a72b-4fdf-ad50-6714890937b1" (UID: "b5e2c1aa-a72b-4fdf-ad50-6714890937b1"). InnerVolumeSpecName "kube-api-access-29f6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:04:55.323014 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:55.322982 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5e2c1aa-a72b-4fdf-ad50-6714890937b1-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:04:55.323014 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:55.323009 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29f6p\" (UniqueName: \"kubernetes.io/projected/b5e2c1aa-a72b-4fdf-ad50-6714890937b1-kube-api-access-29f6p\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:04:55.940718 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:55.940675 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff01347a-plj8r" event={"ID":"b5e2c1aa-a72b-4fdf-ad50-6714890937b1","Type":"ContainerDied","Data":"0882bb3d82c851a377bd03363ddbb0a601e80357c34aa2dd926f986ed8c79747"} Oct 30 17:04:55.940925 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:55.940742 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff01347a-plj8r" Oct 30 17:04:55.940925 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:55.940761 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0882bb3d82c851a377bd03363ddbb0a601e80357c34aa2dd926f986ed8c79747" Oct 30 17:04:57.406957 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:57.406922 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d4057dd-j9pc9"] Oct 30 17:04:57.407803 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:57.407782 4522 cpu_manager.go:414] "containerMap: removing container" podUID="e5803ca4-1b00-4cf8-a466-77c4314d10fc" containerName="revoke-keys" Oct 30 17:04:57.407906 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:57.407805 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5803ca4-1b00-4cf8-a466-77c4314d10fc" containerName="revoke-keys" Oct 30 17:04:57.407906 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:57.407842 4522 cpu_manager.go:414] "containerMap: removing container" podUID="b5e2c1aa-a72b-4fdf-ad50-6714890937b1" containerName="revoke-keys" Oct 30 17:04:57.407906 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:57.407850 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5e2c1aa-a72b-4fdf-ad50-6714890937b1" containerName="revoke-keys" Oct 30 17:04:57.408075 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:57.408015 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5e2c1aa-a72b-4fdf-ad50-6714890937b1" containerName="revoke-keys" Oct 30 17:04:57.408075 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:57.408040 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5803ca4-1b00-4cf8-a466-77c4314d10fc" containerName="revoke-keys" Oct 30 17:04:57.412146 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:57.412114 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d4057dd-j9pc9" Oct 30 17:04:57.439253 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:57.439215 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d4057dd-j9pc9"] Oct 30 17:04:57.439647 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:57.439627 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:04:57.439874 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:57.439791 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:04:57.560145 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:57.560111 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfvd8\" (UniqueName: \"kubernetes.io/projected/4b05eaa8-4779-420d-8add-1c9bb962d90a-kube-api-access-tfvd8\") pod \"maas-api-revoke-sub-4d4057dd-j9pc9\" (UID: \"4b05eaa8-4779-420d-8add-1c9bb962d90a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d4057dd-j9pc9" Oct 30 17:04:57.560365 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:57.560162 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b05eaa8-4779-420d-8add-1c9bb962d90a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d4057dd-j9pc9\" (UID: \"4b05eaa8-4779-420d-8add-1c9bb962d90a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d4057dd-j9pc9" Oct 30 17:04:57.560491 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:57.560468 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b05eaa8-4779-420d-8add-1c9bb962d90a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d4057dd-j9pc9\" (UID: \"4b05eaa8-4779-420d-8add-1c9bb962d90a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d4057dd-j9pc9" Oct 30 17:04:57.662192 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:57.662105 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b05eaa8-4779-420d-8add-1c9bb962d90a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d4057dd-j9pc9\" (UID: \"4b05eaa8-4779-420d-8add-1c9bb962d90a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d4057dd-j9pc9" Oct 30 17:04:57.662390 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:57.662214 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfvd8\" (UniqueName: \"kubernetes.io/projected/4b05eaa8-4779-420d-8add-1c9bb962d90a-kube-api-access-tfvd8\") pod \"maas-api-revoke-sub-4d4057dd-j9pc9\" (UID: \"4b05eaa8-4779-420d-8add-1c9bb962d90a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d4057dd-j9pc9" Oct 30 17:04:57.662390 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:57.662249 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b05eaa8-4779-420d-8add-1c9bb962d90a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d4057dd-j9pc9\" (UID: \"4b05eaa8-4779-420d-8add-1c9bb962d90a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d4057dd-j9pc9" Oct 30 17:04:57.662881 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:57.662857 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b05eaa8-4779-420d-8add-1c9bb962d90a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d4057dd-j9pc9\" (UID: \"4b05eaa8-4779-420d-8add-1c9bb962d90a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d4057dd-j9pc9" Oct 30 17:04:57.664513 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:57.664494 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b05eaa8-4779-420d-8add-1c9bb962d90a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d4057dd-j9pc9\" (UID: \"4b05eaa8-4779-420d-8add-1c9bb962d90a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d4057dd-j9pc9" Oct 30 17:04:57.686896 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:57.686871 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfvd8\" (UniqueName: \"kubernetes.io/projected/4b05eaa8-4779-420d-8add-1c9bb962d90a-kube-api-access-tfvd8\") pod \"maas-api-revoke-sub-4d4057dd-j9pc9\" (UID: \"4b05eaa8-4779-420d-8add-1c9bb962d90a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d4057dd-j9pc9" Oct 30 17:04:57.727924 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:57.727894 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d4057dd-j9pc9" Oct 30 17:04:58.045092 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:58.045052 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d4057dd-j9pc9"] Oct 30 17:04:59.070500 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:59.070458 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d4057dd-j9pc9" event={"ID":"4b05eaa8-4779-420d-8add-1c9bb962d90a","Type":"ContainerStarted","Data":"a8c6583a4fe72c4667be38f28da2f2e9e374792ab0882f60db71285d78931f14"} Oct 30 17:04:59.071148 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:04:59.071121 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d4057dd-j9pc9" event={"ID":"4b05eaa8-4779-420d-8add-1c9bb962d90a","Type":"ContainerStarted","Data":"0c249bae9d9f0d9f7176386c7bed05a35de67ff1522c714003dab27de597264f"} Oct 30 17:05:00.130395 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:00.129724 4522 generic.go:356] "Generic (PLEG): container finished" podID="4b05eaa8-4779-420d-8add-1c9bb962d90a" containerID="a8c6583a4fe72c4667be38f28da2f2e9e374792ab0882f60db71285d78931f14" exitCode=0 Oct 30 17:05:00.130395 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:00.129812 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d4057dd-j9pc9" event={"ID":"4b05eaa8-4779-420d-8add-1c9bb962d90a","Type":"ContainerDied","Data":"a8c6583a4fe72c4667be38f28da2f2e9e374792ab0882f60db71285d78931f14"} Oct 30 17:05:01.016735 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:00.796247 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" podUID="2f1e3bbb-ab1c-455d-a05d-d62f72c5d009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:01.160327 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:01.017374 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" podUID="8a6a85ad-c944-450f-b0bb-e3b64b0f56f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:01.160327 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:01.018369 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" podUID="43ca764d-ae0e-4858-9841-5bbd0ffdbb85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 17:05:01.256332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:01.237721 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" podUID="4360e9dc-c001-4102-8af6-b55b692fcf2c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:01.256332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:01.245547 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" podUID="0b832781-3d41-4496-aa86-e4d92792e590" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:01.324756 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:01.324629 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" podUID="75321c5d-8bf8-45c6-8811-81981fe3cad4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:01.358900 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:01.353683 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" podUID="a3e5e732-2c67-4377-9755-e2178bf33772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:01.406497 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:01.406444 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" podUID="c9ad71b7-ad90-4c8c-bf27-0f10b23f1593" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 17:05:01.447347 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:01.439909 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" podUID="4b4ad752-e053-4cea-a3bc-fe2be6f92021" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:01.925388 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:01.920255 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d4057dd-j9pc9" Oct 30 17:05:01.925388 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:01.920867 4522 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:05:01.925388 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:01.920918 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m" podUID="232a5af4-ecfb-4e38-a937-fc555a9b946b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:02.029331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:02.023693 4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" podUID="9bfbc6d3-91fd-41a7-9585-40a8f252f743" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:02.036817 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:02.036783 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b05eaa8-4779-420d-8add-1c9bb962d90a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b05eaa8-4779-420d-8add-1c9bb962d90a-maas-api-openshift-ca\") pod \"4b05eaa8-4779-420d-8add-1c9bb962d90a\" (UID: \"4b05eaa8-4779-420d-8add-1c9bb962d90a\") " Oct 30 17:05:02.037272 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:02.037251 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b05eaa8-4779-420d-8add-1c9bb962d90a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b05eaa8-4779-420d-8add-1c9bb962d90a-maas-api-xks-ca\") pod \"4b05eaa8-4779-420d-8add-1c9bb962d90a\" (UID: \"4b05eaa8-4779-420d-8add-1c9bb962d90a\") " Oct 30 17:05:02.037493 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:02.037477 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b05eaa8-4779-420d-8add-1c9bb962d90a-kube-api-access-tfvd8\" (UniqueName: \"kubernetes.io/projected/4b05eaa8-4779-420d-8add-1c9bb962d90a-kube-api-access-tfvd8\") pod \"4b05eaa8-4779-420d-8add-1c9bb962d90a\" (UID: \"4b05eaa8-4779-420d-8add-1c9bb962d90a\") " Oct 30 17:05:02.096569 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:02.052725 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b05eaa8-4779-420d-8add-1c9bb962d90a-maas-api-openshift-ca" pod "4b05eaa8-4779-420d-8add-1c9bb962d90a" (UID: "4b05eaa8-4779-420d-8add-1c9bb962d90a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:05:02.097559 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:02.067414 4522 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:05:02.106831 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:02.097831 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq" podUID="46027d41-98ad-409d-a66f-4bc050bc2648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:02.117695 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:02.117668 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b05eaa8-4779-420d-8add-1c9bb962d90a-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:05:02.142339 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:02.117616 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" podUID="9bfbc6d3-91fd-41a7-9585-40a8f252f743" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:02.338650 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:02.186294 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" podUID="8a6a85ad-c944-450f-b0bb-e3b64b0f56f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:02.338650 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:02.187090 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" podUID="43ca764d-ae0e-4858-9841-5bbd0ffdbb85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:02.338650 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:02.241947 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" podUID="4360e9dc-c001-4102-8af6-b55b692fcf2c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:02.338650 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:02.266409 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" podUID="0b832781-3d41-4496-aa86-e4d92792e590" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 17:05:02.338650 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:02.286664 4522 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:05:02.338650 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:02.286731 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz" podUID="b2010977-7978-4ee0-9b32-3449329c0b4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:02.338650 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:02.334496 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" podUID="75321c5d-8bf8-45c6-8811-81981fe3cad4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 17:05:02.418210 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:02.371603 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" podUID="a3e5e732-2c67-4377-9755-e2178bf33772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:02.418210 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:02.397130 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b05eaa8-4779-420d-8add-1c9bb962d90a-maas-api-xks-ca" pod "4b05eaa8-4779-420d-8add-1c9bb962d90a" (UID: "4b05eaa8-4779-420d-8add-1c9bb962d90a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:05:02.423745 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:02.421658 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" Oct 30 17:05:02.427220 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:02.423711 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" Oct 30 17:05:02.427220 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:02.424005 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" Oct 30 17:05:02.427220 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:02.424042 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" Oct 30 17:05:02.427220 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:02.424081 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" Oct 30 17:05:02.427220 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:02.424116 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" Oct 30 17:05:02.427220 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:02.424146 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" Oct 30 17:05:02.428344 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:02.428288 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b05eaa8-4779-420d-8add-1c9bb962d90a-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:05:02.471370 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:02.429171 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" Oct 30 17:05:02.471370 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:02.445937 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" podUID="4b4ad752-e053-4cea-a3bc-fe2be6f92021" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:02.471370 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:02.446333 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" Oct 30 17:05:02.471370 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:02.447770 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" podUID="c9ad71b7-ad90-4c8c-bf27-0f10b23f1593" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 17:05:02.471370 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:02.448203 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" Oct 30 17:05:02.634065 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:02.632345 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b05eaa8-4779-420d-8add-1c9bb962d90a-kube-api-access-tfvd8" pod "4b05eaa8-4779-420d-8add-1c9bb962d90a" (UID: "4b05eaa8-4779-420d-8add-1c9bb962d90a"). InnerVolumeSpecName "kube-api-access-tfvd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:05:02.635329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:02.635219 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" Oct 30 17:05:02.804327 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:02.784278 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" Oct 30 17:05:02.894881 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:02.894572 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" Oct 30 17:05:02.927878 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:02.920377 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfvd8\" (UniqueName: \"kubernetes.io/projected/4b05eaa8-4779-420d-8add-1c9bb962d90a-kube-api-access-tfvd8\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:05:02.960743 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:02.960705 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" Oct 30 17:05:02.968972 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:02.968948 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" Oct 30 17:05:02.974613 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:02.974593 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" Oct 30 17:05:02.974828 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:02.974811 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d4057dd-j9pc9" event={"ID":"4b05eaa8-4779-420d-8add-1c9bb962d90a","Type":"ContainerDied","Data":"0c249bae9d9f0d9f7176386c7bed05a35de67ff1522c714003dab27de597264f"} Oct 30 17:05:02.979797 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:02.978541 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c249bae9d9f0d9f7176386c7bed05a35de67ff1522c714003dab27de597264f" Oct 30 17:05:02.979797 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:02.973927 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d4057dd-j9pc9" Oct 30 17:05:02.986596 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:02.986573 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-04248e/distinct-w1-04248e-kserve-5859d49965-kgq46" Oct 30 17:05:02.999425 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:02.999270 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-04248e/distinct-w1-04248e-kserve-5859d49965-kgq46" Oct 30 17:05:03.995136 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:03.995107 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04248e/distinct-w1-04248e-kserve-5859d49965-kgq46" Oct 30 17:05:06.028512 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:06.028474 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8644-cc17f3"] Oct 30 17:05:06.029352 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:06.029329 4522 cpu_manager.go:414] "containerMap: removing container" podUID="4b05eaa8-4779-420d-8add-1c9bb962d90a" containerName="revoke-keys" Oct 30 17:05:06.029352 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:06.029351 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b05eaa8-4779-420d-8add-1c9bb962d90a" containerName="revoke-keys" Oct 30 17:05:06.029550 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:06.029536 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b05eaa8-4779-420d-8add-1c9bb962d90a" containerName="revoke-keys" Oct 30 17:05:06.033571 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:06.033552 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8644-cc17f3" Oct 30 17:05:06.051866 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:06.051837 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8644-cc17f3"] Oct 30 17:05:06.052220 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:06.052201 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-wddsm\"" Oct 30 17:05:06.086558 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:06.086524 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvwln\" (UniqueName: \"kubernetes.io/projected/48db6c9b-7254-44a5-946f-bad4ce831ec8-kube-api-access-jvwln\") pod \"test-curl-8644-cc17f3\" (UID: \"48db6c9b-7254-44a5-946f-bad4ce831ec8\") " pod="opendatahub/test-curl-8644-cc17f3" Oct 30 17:05:06.188643 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:06.188613 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvwln\" (UniqueName: \"kubernetes.io/projected/48db6c9b-7254-44a5-946f-bad4ce831ec8-kube-api-access-jvwln\") pod \"test-curl-8644-cc17f3\" (UID: \"48db6c9b-7254-44a5-946f-bad4ce831ec8\") " pod="opendatahub/test-curl-8644-cc17f3" Oct 30 17:05:06.214854 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:06.214819 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvwln\" (UniqueName: \"kubernetes.io/projected/48db6c9b-7254-44a5-946f-bad4ce831ec8-kube-api-access-jvwln\") pod \"test-curl-8644-cc17f3\" (UID: \"48db6c9b-7254-44a5-946f-bad4ce831ec8\") " pod="opendatahub/test-curl-8644-cc17f3" Oct 30 17:05:06.363220 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:06.363143 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-wddsm\"" Oct 30 17:05:06.367207 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:06.367187 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8644-cc17f3" Oct 30 17:05:06.669659 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:05:06.669627 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod48db6c9b_7254_44a5_946f_bad4ce831ec8.slice/crio-1c422e60c2886664df554aa6401848e9a2dc3a1846cecb2989855cc132b8f87f WatchSource:0}: Error finding container 1c422e60c2886664df554aa6401848e9a2dc3a1846cecb2989855cc132b8f87f: Status 404 returned error can't find the container with id 1c422e60c2886664df554aa6401848e9a2dc3a1846cecb2989855cc132b8f87f Oct 30 17:05:06.671764 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:06.671737 4522 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:05:06.673037 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:06.673015 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8644-cc17f3"] Oct 30 17:05:07.007208 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:07.007176 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8644-cc17f3" event={"ID":"48db6c9b-7254-44a5-946f-bad4ce831ec8","Type":"ContainerStarted","Data":"1c422e60c2886664df554aa6401848e9a2dc3a1846cecb2989855cc132b8f87f"} Oct 30 17:05:07.593843 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:07.593804 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e98948d5-wz288"] Oct 30 17:05:07.598370 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:07.598348 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98948d5-wz288" Oct 30 17:05:07.622404 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:07.622378 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e98948d5-wz288"] Oct 30 17:05:07.629789 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:07.629764 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:05:07.629936 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:07.629813 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:05:07.713462 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:07.713420 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4clmz\" (UniqueName: \"kubernetes.io/projected/81dd4cb6-82cd-4ba7-832d-fd43c4432044-kube-api-access-4clmz\") pod \"maas-api-revoke-sub-e98948d5-wz288\" (UID: \"81dd4cb6-82cd-4ba7-832d-fd43c4432044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98948d5-wz288" Oct 30 17:05:07.713652 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:07.713552 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81dd4cb6-82cd-4ba7-832d-fd43c4432044-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e98948d5-wz288\" (UID: \"81dd4cb6-82cd-4ba7-832d-fd43c4432044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98948d5-wz288" Oct 30 17:05:07.713652 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:07.713630 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81dd4cb6-82cd-4ba7-832d-fd43c4432044-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e98948d5-wz288\" (UID: \"81dd4cb6-82cd-4ba7-832d-fd43c4432044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98948d5-wz288" Oct 30 17:05:07.815811 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:07.815774 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4clmz\" (UniqueName: \"kubernetes.io/projected/81dd4cb6-82cd-4ba7-832d-fd43c4432044-kube-api-access-4clmz\") pod \"maas-api-revoke-sub-e98948d5-wz288\" (UID: \"81dd4cb6-82cd-4ba7-832d-fd43c4432044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98948d5-wz288" Oct 30 17:05:07.815988 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:07.815889 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81dd4cb6-82cd-4ba7-832d-fd43c4432044-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e98948d5-wz288\" (UID: \"81dd4cb6-82cd-4ba7-832d-fd43c4432044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98948d5-wz288" Oct 30 17:05:07.815988 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:07.815950 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81dd4cb6-82cd-4ba7-832d-fd43c4432044-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e98948d5-wz288\" (UID: \"81dd4cb6-82cd-4ba7-832d-fd43c4432044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98948d5-wz288" Oct 30 17:05:07.817128 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:07.817090 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81dd4cb6-82cd-4ba7-832d-fd43c4432044-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e98948d5-wz288\" (UID: \"81dd4cb6-82cd-4ba7-832d-fd43c4432044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98948d5-wz288" Oct 30 17:05:07.819768 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:07.819719 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81dd4cb6-82cd-4ba7-832d-fd43c4432044-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e98948d5-wz288\" (UID: \"81dd4cb6-82cd-4ba7-832d-fd43c4432044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98948d5-wz288" Oct 30 17:05:07.843344 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:07.843287 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4clmz\" (UniqueName: \"kubernetes.io/projected/81dd4cb6-82cd-4ba7-832d-fd43c4432044-kube-api-access-4clmz\") pod \"maas-api-revoke-sub-e98948d5-wz288\" (UID: \"81dd4cb6-82cd-4ba7-832d-fd43c4432044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98948d5-wz288" Oct 30 17:05:07.916678 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:07.916602 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98948d5-wz288" Oct 30 17:05:08.244174 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:08.244142 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e98948d5-wz288"] Oct 30 17:05:08.248345 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:05:08.248294 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81dd4cb6_82cd_4ba7_832d_fd43c4432044.slice/crio-bc24de65e4da0bc37ff81cb1843b4bf07ac4e336593112d2b0f59cdc3c428898 WatchSource:0}: Error finding container bc24de65e4da0bc37ff81cb1843b4bf07ac4e336593112d2b0f59cdc3c428898: Status 404 returned error can't find the container with id bc24de65e4da0bc37ff81cb1843b4bf07ac4e336593112d2b0f59cdc3c428898 Oct 30 17:05:09.026511 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:09.026475 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98948d5-wz288" event={"ID":"81dd4cb6-82cd-4ba7-832d-fd43c4432044","Type":"ContainerStarted","Data":"2c65870447f84761427395653edde21d0fc029d1143bb4ff7ab26bd4e92b0e73"} Oct 30 17:05:09.026942 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:09.026535 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98948d5-wz288" event={"ID":"81dd4cb6-82cd-4ba7-832d-fd43c4432044","Type":"ContainerStarted","Data":"bc24de65e4da0bc37ff81cb1843b4bf07ac4e336593112d2b0f59cdc3c428898"} Oct 30 17:05:09.028186 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:09.028158 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8644-cc17f3" event={"ID":"48db6c9b-7254-44a5-946f-bad4ce831ec8","Type":"ContainerStarted","Data":"cb4e1deb56e59f5673f58e194239056eb5481b378c59b6b6a8956feec08f4ac1"} Oct 30 17:05:09.072963 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:09.072922 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98948d5-wz288" podStartSLOduration=2.072909905 podStartE2EDuration="2.072909905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:05:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:05:09.072513163 +0000 UTC m=+1083.073268079" watchObservedRunningTime="2026-10-30 17:05:09.072909905 +0000 UTC m=+1083.073664806" Oct 30 17:05:09.116881 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:09.116830 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8644-cc17f3" podStartSLOduration=0.899357354 podStartE2EDuration="3.116816406s" totalImagesPullingTime="2.217459052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:05:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:05:09.11558962 +0000 UTC m=+1083.116344536" watchObservedRunningTime="2026-10-30 17:05:09.116816406 +0000 UTC m=+1083.117571307" Oct 30 17:05:10.037221 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:10.037183 4522 generic.go:356] "Generic (PLEG): container finished" podID="81dd4cb6-82cd-4ba7-832d-fd43c4432044" containerID="2c65870447f84761427395653edde21d0fc029d1143bb4ff7ab26bd4e92b0e73" exitCode=0 Oct 30 17:05:10.037649 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:10.037220 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98948d5-wz288" event={"ID":"81dd4cb6-82cd-4ba7-832d-fd43c4432044","Type":"ContainerDied","Data":"2c65870447f84761427395653edde21d0fc029d1143bb4ff7ab26bd4e92b0e73"} Oct 30 17:05:10.075024 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:10.074997 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8644-cc17f3"] Oct 30 17:05:10.079735 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:10.079707 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8644-cc17f3"] Oct 30 17:05:10.237913 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:10.237879 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8644-dd7736"] Oct 30 17:05:10.238554 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:10.238540 4522 cpu_manager.go:414] "containerMap: removing container" podUID="48db6c9b-7254-44a5-946f-bad4ce831ec8" containerName="test-curl-8644-cc17f3" Oct 30 17:05:10.238623 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:10.238555 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48db6c9b-7254-44a5-946f-bad4ce831ec8" containerName="test-curl-8644-cc17f3" Oct 30 17:05:10.238661 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:10.238655 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="48db6c9b-7254-44a5-946f-bad4ce831ec8" containerName="test-curl-8644-cc17f3" Oct 30 17:05:10.242450 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:10.242433 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8644-dd7736" Oct 30 17:05:10.261034 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:10.261010 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8644-dd7736"] Oct 30 17:05:10.360598 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:10.360496 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6br7\" (UniqueName: \"kubernetes.io/projected/5dd248b5-6345-4d7a-b977-4de2a531732e-kube-api-access-h6br7\") pod \"test-curl-8644-dd7736\" (UID: \"5dd248b5-6345-4d7a-b977-4de2a531732e\") " pod="opendatahub/test-curl-8644-dd7736" Oct 30 17:05:10.461793 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:10.461762 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6br7\" (UniqueName: \"kubernetes.io/projected/5dd248b5-6345-4d7a-b977-4de2a531732e-kube-api-access-h6br7\") pod \"test-curl-8644-dd7736\" (UID: \"5dd248b5-6345-4d7a-b977-4de2a531732e\") " pod="opendatahub/test-curl-8644-dd7736" Oct 30 17:05:10.490417 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:10.490389 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6br7\" (UniqueName: \"kubernetes.io/projected/5dd248b5-6345-4d7a-b977-4de2a531732e-kube-api-access-h6br7\") pod \"test-curl-8644-dd7736\" (UID: \"5dd248b5-6345-4d7a-b977-4de2a531732e\") " pod="opendatahub/test-curl-8644-dd7736" Oct 30 17:05:10.555333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:10.555274 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8644-dd7736" Oct 30 17:05:10.860060 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:05:10.860022 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5dd248b5_6345_4d7a_b977_4de2a531732e.slice/crio-f960bcf0d7c656ee9e6ad55ee86ede6a9280b9aef8c4dd14aa5f270588c1b3b7 WatchSource:0}: Error finding container f960bcf0d7c656ee9e6ad55ee86ede6a9280b9aef8c4dd14aa5f270588c1b3b7: Status 404 returned error can't find the container with id f960bcf0d7c656ee9e6ad55ee86ede6a9280b9aef8c4dd14aa5f270588c1b3b7 Oct 30 17:05:10.865882 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:10.865805 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8644-dd7736"] Oct 30 17:05:11.046595 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:11.046558 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8644-dd7736" event={"ID":"5dd248b5-6345-4d7a-b977-4de2a531732e","Type":"ContainerStarted","Data":"f0c4abdde41026163b0638e4c6972388cc5a92213e3e92b7381a39bdb4cfd007"} Oct 30 17:05:11.046595 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:11.046603 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8644-dd7736" event={"ID":"5dd248b5-6345-4d7a-b977-4de2a531732e","Type":"ContainerStarted","Data":"f960bcf0d7c656ee9e6ad55ee86ede6a9280b9aef8c4dd14aa5f270588c1b3b7"} Oct 30 17:05:11.047143 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:11.046961 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8644-cc17f3" podUID="48db6c9b-7254-44a5-946f-bad4ce831ec8" containerName="test-curl-8644-cc17f3" containerID="cri-o://cb4e1deb56e59f5673f58e194239056eb5481b378c59b6b6a8956feec08f4ac1" gracePeriod=2 Oct 30 17:05:11.092882 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:11.092829 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8644-dd7736" podStartSLOduration=1.092813231 podStartE2EDuration="1.092813231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:05:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:05:11.090147928 +0000 UTC m=+1085.090902843" watchObservedRunningTime="2026-10-30 17:05:11.092813231 +0000 UTC m=+1085.093568132" Oct 30 17:05:11.192297 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:11.192269 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98948d5-wz288" Oct 30 17:05:11.274925 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:11.274891 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81dd4cb6-82cd-4ba7-832d-fd43c4432044-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81dd4cb6-82cd-4ba7-832d-fd43c4432044-maas-api-openshift-ca\") pod \"81dd4cb6-82cd-4ba7-832d-fd43c4432044\" (UID: \"81dd4cb6-82cd-4ba7-832d-fd43c4432044\") " Oct 30 17:05:11.275160 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:11.275006 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81dd4cb6-82cd-4ba7-832d-fd43c4432044-kube-api-access-4clmz\" (UniqueName: \"kubernetes.io/projected/81dd4cb6-82cd-4ba7-832d-fd43c4432044-kube-api-access-4clmz\") pod \"81dd4cb6-82cd-4ba7-832d-fd43c4432044\" (UID: \"81dd4cb6-82cd-4ba7-832d-fd43c4432044\") " Oct 30 17:05:11.275353 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:11.275322 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81dd4cb6-82cd-4ba7-832d-fd43c4432044-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81dd4cb6-82cd-4ba7-832d-fd43c4432044-maas-api-xks-ca\") pod \"81dd4cb6-82cd-4ba7-832d-fd43c4432044\" (UID: \"81dd4cb6-82cd-4ba7-832d-fd43c4432044\") " Oct 30 17:05:11.275763 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:11.275737 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81dd4cb6-82cd-4ba7-832d-fd43c4432044-maas-api-openshift-ca" pod "81dd4cb6-82cd-4ba7-832d-fd43c4432044" (UID: "81dd4cb6-82cd-4ba7-832d-fd43c4432044"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:05:11.276591 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:11.276562 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81dd4cb6-82cd-4ba7-832d-fd43c4432044-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:05:11.278755 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:11.278713 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81dd4cb6-82cd-4ba7-832d-fd43c4432044-kube-api-access-4clmz" pod "81dd4cb6-82cd-4ba7-832d-fd43c4432044" (UID: "81dd4cb6-82cd-4ba7-832d-fd43c4432044"). InnerVolumeSpecName "kube-api-access-4clmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:05:11.287349 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:11.281451 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81dd4cb6-82cd-4ba7-832d-fd43c4432044-maas-api-xks-ca" pod "81dd4cb6-82cd-4ba7-832d-fd43c4432044" (UID: "81dd4cb6-82cd-4ba7-832d-fd43c4432044"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:05:11.378086 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:11.378055 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81dd4cb6-82cd-4ba7-832d-fd43c4432044-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:05:11.378086 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:11.378082 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4clmz\" (UniqueName: \"kubernetes.io/projected/81dd4cb6-82cd-4ba7-832d-fd43c4432044-kube-api-access-4clmz\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:05:12.032570 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:12.029745 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8644-dd7736"] Oct 30 17:05:12.034557 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:12.034531 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8644-dd7736"] Oct 30 17:05:12.055919 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:12.055890 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98948d5-wz288" Oct 30 17:05:12.056392 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:12.055913 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98948d5-wz288" event={"ID":"81dd4cb6-82cd-4ba7-832d-fd43c4432044","Type":"ContainerDied","Data":"bc24de65e4da0bc37ff81cb1843b4bf07ac4e336593112d2b0f59cdc3c428898"} Oct 30 17:05:12.056392 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:12.056007 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc24de65e4da0bc37ff81cb1843b4bf07ac4e336593112d2b0f59cdc3c428898" Oct 30 17:05:12.952660 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:05:12.952568 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:05:13.073684 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:13.073657 4522 generic.go:356] "Generic (PLEG): container finished" podID="48db6c9b-7254-44a5-946f-bad4ce831ec8" containerID="cb4e1deb56e59f5673f58e194239056eb5481b378c59b6b6a8956feec08f4ac1" exitCode=137 Oct 30 17:05:13.074088 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:13.073956 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8644-dd7736" podUID="5dd248b5-6345-4d7a-b977-4de2a531732e" containerName="test-curl-8644-dd7736" containerID="cri-o://f0c4abdde41026163b0638e4c6972388cc5a92213e3e92b7381a39bdb4cfd007" gracePeriod=2 Oct 30 17:05:13.253665 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:13.253639 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8644-cc17f3" Oct 30 17:05:13.411758 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:13.411726 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48db6c9b-7254-44a5-946f-bad4ce831ec8-kube-api-access-jvwln\" (UniqueName: \"kubernetes.io/projected/48db6c9b-7254-44a5-946f-bad4ce831ec8-kube-api-access-jvwln\") pod \"48db6c9b-7254-44a5-946f-bad4ce831ec8\" (UID: \"48db6c9b-7254-44a5-946f-bad4ce831ec8\") " Oct 30 17:05:13.414558 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:13.414529 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48db6c9b-7254-44a5-946f-bad4ce831ec8-kube-api-access-jvwln" pod "48db6c9b-7254-44a5-946f-bad4ce831ec8" (UID: "48db6c9b-7254-44a5-946f-bad4ce831ec8"). InnerVolumeSpecName "kube-api-access-jvwln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:05:13.523480 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:13.515498 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvwln\" (UniqueName: \"kubernetes.io/projected/48db6c9b-7254-44a5-946f-bad4ce831ec8-kube-api-access-jvwln\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:05:14.083578 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:14.083548 4522 scope.go:122] "RemoveContainer" containerID="cb4e1deb56e59f5673f58e194239056eb5481b378c59b6b6a8956feec08f4ac1" Oct 30 17:05:14.083578 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:14.083559 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8644-cc17f3" Oct 30 17:05:14.108622 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:05:14.108590 4522 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8644-cc17f3\" is forbidden: User \"system:node:ip-10-0-128-218.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-218.ec2.internal' and this object" podUID="48db6c9b-7254-44a5-946f-bad4ce831ec8" pod="opendatahub/test-curl-8644-cc17f3" Oct 30 17:05:14.146180 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:14.146147 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48db6c9b-7254-44a5-946f-bad4ce831ec8" path="/var/lib/kubelet/pods/48db6c9b-7254-44a5-946f-bad4ce831ec8/volumes" Oct 30 17:05:14.527226 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:14.522117 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43e9434f-nzkjb"] Oct 30 17:05:14.527226 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:14.523559 4522 cpu_manager.go:414] "containerMap: removing container" podUID="5dd248b5-6345-4d7a-b977-4de2a531732e" containerName="test-curl-8644-dd7736" Oct 30 17:05:14.527226 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:14.523583 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dd248b5-6345-4d7a-b977-4de2a531732e" containerName="test-curl-8644-dd7736" Oct 30 17:05:14.527226 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:14.523726 4522 cpu_manager.go:414] "containerMap: removing container" podUID="81dd4cb6-82cd-4ba7-832d-fd43c4432044" containerName="revoke-keys" Oct 30 17:05:14.527226 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:14.523734 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81dd4cb6-82cd-4ba7-832d-fd43c4432044" containerName="revoke-keys" Oct 30 17:05:14.527226 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:14.523996 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="81dd4cb6-82cd-4ba7-832d-fd43c4432044" containerName="revoke-keys" Oct 30 17:05:14.527226 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:14.524019 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dd248b5-6345-4d7a-b977-4de2a531732e" containerName="test-curl-8644-dd7736" Oct 30 17:05:14.530734 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:14.530709 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43e9434f-nzkjb" Oct 30 17:05:14.545617 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:14.545589 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43e9434f-nzkjb"] Oct 30 17:05:14.553595 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:14.553565 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:05:14.553713 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:14.553645 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:05:14.632024 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:14.631985 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bfc83a8-a240-4f65-8034-6dc828a8c14e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43e9434f-nzkjb\" (UID: \"2bfc83a8-a240-4f65-8034-6dc828a8c14e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43e9434f-nzkjb" Oct 30 17:05:14.632164 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:14.632148 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6z6f\" (UniqueName: \"kubernetes.io/projected/2bfc83a8-a240-4f65-8034-6dc828a8c14e-kube-api-access-c6z6f\") pod \"maas-api-revoke-sub-43e9434f-nzkjb\" (UID: \"2bfc83a8-a240-4f65-8034-6dc828a8c14e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43e9434f-nzkjb" Oct 30 17:05:14.632291 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:14.632273 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bfc83a8-a240-4f65-8034-6dc828a8c14e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43e9434f-nzkjb\" (UID: \"2bfc83a8-a240-4f65-8034-6dc828a8c14e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43e9434f-nzkjb" Oct 30 17:05:14.734175 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:14.734136 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bfc83a8-a240-4f65-8034-6dc828a8c14e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43e9434f-nzkjb\" (UID: \"2bfc83a8-a240-4f65-8034-6dc828a8c14e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43e9434f-nzkjb" Oct 30 17:05:14.734394 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:14.734279 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6z6f\" (UniqueName: \"kubernetes.io/projected/2bfc83a8-a240-4f65-8034-6dc828a8c14e-kube-api-access-c6z6f\") pod \"maas-api-revoke-sub-43e9434f-nzkjb\" (UID: \"2bfc83a8-a240-4f65-8034-6dc828a8c14e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43e9434f-nzkjb" Oct 30 17:05:14.734459 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:14.734399 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bfc83a8-a240-4f65-8034-6dc828a8c14e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43e9434f-nzkjb\" (UID: \"2bfc83a8-a240-4f65-8034-6dc828a8c14e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43e9434f-nzkjb" Oct 30 17:05:14.734768 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:14.734743 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bfc83a8-a240-4f65-8034-6dc828a8c14e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43e9434f-nzkjb\" (UID: \"2bfc83a8-a240-4f65-8034-6dc828a8c14e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43e9434f-nzkjb" Oct 30 17:05:14.736669 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:14.736647 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bfc83a8-a240-4f65-8034-6dc828a8c14e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43e9434f-nzkjb\" (UID: \"2bfc83a8-a240-4f65-8034-6dc828a8c14e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43e9434f-nzkjb" Oct 30 17:05:14.760518 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:14.760490 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6z6f\" (UniqueName: \"kubernetes.io/projected/2bfc83a8-a240-4f65-8034-6dc828a8c14e-kube-api-access-c6z6f\") pod \"maas-api-revoke-sub-43e9434f-nzkjb\" (UID: \"2bfc83a8-a240-4f65-8034-6dc828a8c14e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43e9434f-nzkjb" Oct 30 17:05:14.846121 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:14.846045 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43e9434f-nzkjb" Oct 30 17:05:15.047675 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:15.047647 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-b4f406/distinct-w2-b4f406-kserve-5db4cc6969-7n846"] Oct 30 17:05:15.053618 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:15.053596 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b4f406/distinct-w2-b4f406-kserve-5db4cc6969-7n846" Oct 30 17:05:15.071096 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:15.070925 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-b4f406\"/\"distinct-w2-b4f406-kserve-self-signed-certs\"" Oct 30 17:05:15.071096 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:15.070940 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-b4f406\"/\"default-dockercfg-gf4d8\"" Oct 30 17:05:15.071096 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:15.070940 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-b4f406\"/\"openshift-service-ca.crt\"" Oct 30 17:05:15.071096 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:15.070932 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-b4f406\"/\"kube-root-ca.crt\"" Oct 30 17:05:15.071497 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:15.071480 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-b4f406/distinct-w2-b4f406-kserve-5db4cc6969-7n846"] Oct 30 17:05:15.140320 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:15.140279 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8810dbc-b769-46a4-90b9-72a68592bc1a-home\") pod \"distinct-w2-b4f406-kserve-5db4cc6969-7n846\" (UID: \"b8810dbc-b769-46a4-90b9-72a68592bc1a\") " pod="e2e-models-e2e-worker-w2-b4f406/distinct-w2-b4f406-kserve-5db4cc6969-7n846" Oct 30 17:05:15.140771 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:15.140359 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5gwv\" (UniqueName: \"kubernetes.io/projected/b8810dbc-b769-46a4-90b9-72a68592bc1a-kube-api-access-t5gwv\") pod \"distinct-w2-b4f406-kserve-5db4cc6969-7n846\" (UID: \"b8810dbc-b769-46a4-90b9-72a68592bc1a\") " pod="e2e-models-e2e-worker-w2-b4f406/distinct-w2-b4f406-kserve-5db4cc6969-7n846" Oct 30 17:05:15.140771 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:15.140513 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8810dbc-b769-46a4-90b9-72a68592bc1a-model-cache\") pod \"distinct-w2-b4f406-kserve-5db4cc6969-7n846\" (UID: \"b8810dbc-b769-46a4-90b9-72a68592bc1a\") " pod="e2e-models-e2e-worker-w2-b4f406/distinct-w2-b4f406-kserve-5db4cc6969-7n846" Oct 30 17:05:15.140771 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:15.140562 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8810dbc-b769-46a4-90b9-72a68592bc1a-tls-certs\") pod \"distinct-w2-b4f406-kserve-5db4cc6969-7n846\" (UID: \"b8810dbc-b769-46a4-90b9-72a68592bc1a\") " pod="e2e-models-e2e-worker-w2-b4f406/distinct-w2-b4f406-kserve-5db4cc6969-7n846" Oct 30 17:05:15.140771 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:15.140625 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8810dbc-b769-46a4-90b9-72a68592bc1a-tmp-dir\") pod \"distinct-w2-b4f406-kserve-5db4cc6969-7n846\" (UID: \"b8810dbc-b769-46a4-90b9-72a68592bc1a\") " pod="e2e-models-e2e-worker-w2-b4f406/distinct-w2-b4f406-kserve-5db4cc6969-7n846" Oct 30 17:05:15.140771 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:15.140696 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8810dbc-b769-46a4-90b9-72a68592bc1a-dshm\") pod \"distinct-w2-b4f406-kserve-5db4cc6969-7n846\" (UID: \"b8810dbc-b769-46a4-90b9-72a68592bc1a\") " pod="e2e-models-e2e-worker-w2-b4f406/distinct-w2-b4f406-kserve-5db4cc6969-7n846" Oct 30 17:05:15.152855 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:05:15.152822 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bfc83a8_a240_4f65_8034_6dc828a8c14e.slice/crio-e0588affc0f9858f67abeee7d39086560a1e2135b3fc07ee1239226a96cf0352 WatchSource:0}: Error finding container e0588affc0f9858f67abeee7d39086560a1e2135b3fc07ee1239226a96cf0352: Status 404 returned error can't find the container with id e0588affc0f9858f67abeee7d39086560a1e2135b3fc07ee1239226a96cf0352 Oct 30 17:05:15.155789 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:15.155637 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43e9434f-nzkjb"] Oct 30 17:05:15.253145 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:15.252593 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8810dbc-b769-46a4-90b9-72a68592bc1a-dshm\") pod \"distinct-w2-b4f406-kserve-5db4cc6969-7n846\" (UID: \"b8810dbc-b769-46a4-90b9-72a68592bc1a\") " pod="e2e-models-e2e-worker-w2-b4f406/distinct-w2-b4f406-kserve-5db4cc6969-7n846" Oct 30 17:05:15.253145 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:15.252650 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8810dbc-b769-46a4-90b9-72a68592bc1a-home\") pod \"distinct-w2-b4f406-kserve-5db4cc6969-7n846\" (UID: \"b8810dbc-b769-46a4-90b9-72a68592bc1a\") " pod="e2e-models-e2e-worker-w2-b4f406/distinct-w2-b4f406-kserve-5db4cc6969-7n846" Oct 30 17:05:15.253145 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:15.252696 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5gwv\" (UniqueName: \"kubernetes.io/projected/b8810dbc-b769-46a4-90b9-72a68592bc1a-kube-api-access-t5gwv\") pod \"distinct-w2-b4f406-kserve-5db4cc6969-7n846\" (UID: \"b8810dbc-b769-46a4-90b9-72a68592bc1a\") " pod="e2e-models-e2e-worker-w2-b4f406/distinct-w2-b4f406-kserve-5db4cc6969-7n846" Oct 30 17:05:15.253145 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:15.252920 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8810dbc-b769-46a4-90b9-72a68592bc1a-model-cache\") pod \"distinct-w2-b4f406-kserve-5db4cc6969-7n846\" (UID: \"b8810dbc-b769-46a4-90b9-72a68592bc1a\") " pod="e2e-models-e2e-worker-w2-b4f406/distinct-w2-b4f406-kserve-5db4cc6969-7n846" Oct 30 17:05:15.253145 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:15.253013 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8810dbc-b769-46a4-90b9-72a68592bc1a-tls-certs\") pod \"distinct-w2-b4f406-kserve-5db4cc6969-7n846\" (UID: \"b8810dbc-b769-46a4-90b9-72a68592bc1a\") " pod="e2e-models-e2e-worker-w2-b4f406/distinct-w2-b4f406-kserve-5db4cc6969-7n846" Oct 30 17:05:15.253145 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:15.253078 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8810dbc-b769-46a4-90b9-72a68592bc1a-tmp-dir\") pod \"distinct-w2-b4f406-kserve-5db4cc6969-7n846\" (UID: \"b8810dbc-b769-46a4-90b9-72a68592bc1a\") " pod="e2e-models-e2e-worker-w2-b4f406/distinct-w2-b4f406-kserve-5db4cc6969-7n846" Oct 30 17:05:15.253713 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:15.253599 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8810dbc-b769-46a4-90b9-72a68592bc1a-tmp-dir\") pod \"distinct-w2-b4f406-kserve-5db4cc6969-7n846\" (UID: \"b8810dbc-b769-46a4-90b9-72a68592bc1a\") " pod="e2e-models-e2e-worker-w2-b4f406/distinct-w2-b4f406-kserve-5db4cc6969-7n846" Oct 30 17:05:15.262068 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:15.258443 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8810dbc-b769-46a4-90b9-72a68592bc1a-dshm\") pod \"distinct-w2-b4f406-kserve-5db4cc6969-7n846\" (UID: \"b8810dbc-b769-46a4-90b9-72a68592bc1a\") " pod="e2e-models-e2e-worker-w2-b4f406/distinct-w2-b4f406-kserve-5db4cc6969-7n846" Oct 30 17:05:15.269339 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:15.262921 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8810dbc-b769-46a4-90b9-72a68592bc1a-model-cache\") pod \"distinct-w2-b4f406-kserve-5db4cc6969-7n846\" (UID: \"b8810dbc-b769-46a4-90b9-72a68592bc1a\") " pod="e2e-models-e2e-worker-w2-b4f406/distinct-w2-b4f406-kserve-5db4cc6969-7n846" Oct 30 17:05:15.269691 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:15.269667 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8810dbc-b769-46a4-90b9-72a68592bc1a-home\") pod \"distinct-w2-b4f406-kserve-5db4cc6969-7n846\" (UID: \"b8810dbc-b769-46a4-90b9-72a68592bc1a\") " pod="e2e-models-e2e-worker-w2-b4f406/distinct-w2-b4f406-kserve-5db4cc6969-7n846" Oct 30 17:05:15.283359 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:15.281136 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8810dbc-b769-46a4-90b9-72a68592bc1a-tls-certs\") pod \"distinct-w2-b4f406-kserve-5db4cc6969-7n846\" (UID: \"b8810dbc-b769-46a4-90b9-72a68592bc1a\") " pod="e2e-models-e2e-worker-w2-b4f406/distinct-w2-b4f406-kserve-5db4cc6969-7n846" Oct 30 17:05:15.285782 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:15.284481 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5gwv\" (UniqueName: \"kubernetes.io/projected/b8810dbc-b769-46a4-90b9-72a68592bc1a-kube-api-access-t5gwv\") pod \"distinct-w2-b4f406-kserve-5db4cc6969-7n846\" (UID: \"b8810dbc-b769-46a4-90b9-72a68592bc1a\") " pod="e2e-models-e2e-worker-w2-b4f406/distinct-w2-b4f406-kserve-5db4cc6969-7n846" Oct 30 17:05:15.368742 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:15.368641 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b4f406/distinct-w2-b4f406-kserve-5db4cc6969-7n846" Oct 30 17:05:15.402998 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:15.402958 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8644-dd7736" Oct 30 17:05:15.558191 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:15.558149 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dd248b5-6345-4d7a-b977-4de2a531732e-kube-api-access-h6br7\" (UniqueName: \"kubernetes.io/projected/5dd248b5-6345-4d7a-b977-4de2a531732e-kube-api-access-h6br7\") pod \"5dd248b5-6345-4d7a-b977-4de2a531732e\" (UID: \"5dd248b5-6345-4d7a-b977-4de2a531732e\") " Oct 30 17:05:15.560807 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:15.560775 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dd248b5-6345-4d7a-b977-4de2a531732e-kube-api-access-h6br7" pod "5dd248b5-6345-4d7a-b977-4de2a531732e" (UID: "5dd248b5-6345-4d7a-b977-4de2a531732e"). InnerVolumeSpecName "kube-api-access-h6br7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:05:15.660821 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:15.660727 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6br7\" (UniqueName: \"kubernetes.io/projected/5dd248b5-6345-4d7a-b977-4de2a531732e-kube-api-access-h6br7\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:05:15.698558 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:05:15.698518 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb8810dbc_b769_46a4_90b9_72a68592bc1a.slice/crio-e62870ba1362c4acd0e97b3c959da66020ce7cfbecf949c23062b22ad1be0dbc WatchSource:0}: Error finding container e62870ba1362c4acd0e97b3c959da66020ce7cfbecf949c23062b22ad1be0dbc: Status 404 returned error can't find the container with id e62870ba1362c4acd0e97b3c959da66020ce7cfbecf949c23062b22ad1be0dbc Oct 30 17:05:15.702581 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:15.702549 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-b4f406/distinct-w2-b4f406-kserve-5db4cc6969-7n846"] Oct 30 17:05:16.107126 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:16.107084 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b4f406/distinct-w2-b4f406-kserve-5db4cc6969-7n846" event={"ID":"b8810dbc-b769-46a4-90b9-72a68592bc1a","Type":"ContainerStarted","Data":"c7b60475ed24d290180c64665644a3145ff12c2e5b4f2a6b0a432df5aa22bb6e"} Oct 30 17:05:16.107366 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:16.107137 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b4f406/distinct-w2-b4f406-kserve-5db4cc6969-7n846" event={"ID":"b8810dbc-b769-46a4-90b9-72a68592bc1a","Type":"ContainerStarted","Data":"e62870ba1362c4acd0e97b3c959da66020ce7cfbecf949c23062b22ad1be0dbc"} Oct 30 17:05:16.109363 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:16.109300 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43e9434f-nzkjb" event={"ID":"2bfc83a8-a240-4f65-8034-6dc828a8c14e","Type":"ContainerStarted","Data":"73ebb99cddb09eb2591357918620a190199d0bc8fd4ca16cbb572ffbed0fc7b1"} Oct 30 17:05:16.109495 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:16.109389 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43e9434f-nzkjb" event={"ID":"2bfc83a8-a240-4f65-8034-6dc828a8c14e","Type":"ContainerStarted","Data":"e0588affc0f9858f67abeee7d39086560a1e2135b3fc07ee1239226a96cf0352"} Oct 30 17:05:16.110672 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:16.110644 4522 generic.go:356] "Generic (PLEG): container finished" podID="5dd248b5-6345-4d7a-b977-4de2a531732e" containerID="f0c4abdde41026163b0638e4c6972388cc5a92213e3e92b7381a39bdb4cfd007" exitCode=137 Oct 30 17:05:16.110812 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:16.110685 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8644-dd7736" Oct 30 17:05:16.110812 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:16.110710 4522 scope.go:122] "RemoveContainer" containerID="f0c4abdde41026163b0638e4c6972388cc5a92213e3e92b7381a39bdb4cfd007" Oct 30 17:05:16.121771 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:16.121727 4522 scope.go:122] "RemoveContainer" containerID="f0c4abdde41026163b0638e4c6972388cc5a92213e3e92b7381a39bdb4cfd007" Oct 30 17:05:16.122093 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:05:16.122067 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0c4abdde41026163b0638e4c6972388cc5a92213e3e92b7381a39bdb4cfd007\": container with ID starting with f0c4abdde41026163b0638e4c6972388cc5a92213e3e92b7381a39bdb4cfd007 not found: ID does not exist" containerID="f0c4abdde41026163b0638e4c6972388cc5a92213e3e92b7381a39bdb4cfd007" Oct 30 17:05:16.122159 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:16.122104 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0c4abdde41026163b0638e4c6972388cc5a92213e3e92b7381a39bdb4cfd007"} err="failed to get container status \"f0c4abdde41026163b0638e4c6972388cc5a92213e3e92b7381a39bdb4cfd007\": rpc error: code = NotFound desc = could not find container \"f0c4abdde41026163b0638e4c6972388cc5a92213e3e92b7381a39bdb4cfd007\": container with ID starting with f0c4abdde41026163b0638e4c6972388cc5a92213e3e92b7381a39bdb4cfd007 not found: ID does not exist" Oct 30 17:05:16.148712 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:16.148679 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dd248b5-6345-4d7a-b977-4de2a531732e" path="/var/lib/kubelet/pods/5dd248b5-6345-4d7a-b977-4de2a531732e/volumes" Oct 30 17:05:16.158378 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:16.158328 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-b4f406/distinct-w2-b4f406-kserve-5db4cc6969-7n846" podStartSLOduration=1.158299075 podStartE2EDuration="1.158299075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:05:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:05:16.153993178 +0000 UTC m=+1090.154748104" watchObservedRunningTime="2026-10-30 17:05:16.158299075 +0000 UTC m=+1090.159053976" Oct 30 17:05:16.206574 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:16.206504 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43e9434f-nzkjb" podStartSLOduration=2.206486338 podStartE2EDuration="2.206486338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:05:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:05:16.200001348 +0000 UTC m=+1090.200756273" watchObservedRunningTime="2026-10-30 17:05:16.206486338 +0000 UTC m=+1090.207241239" Oct 30 17:05:16.369926 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:16.369830 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b4f406/distinct-w2-b4f406-kserve-5db4cc6969-7n846" Oct 30 17:05:17.137875 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:17.137832 4522 generic.go:356] "Generic (PLEG): container finished" podID="2bfc83a8-a240-4f65-8034-6dc828a8c14e" containerID="73ebb99cddb09eb2591357918620a190199d0bc8fd4ca16cbb572ffbed0fc7b1" exitCode=0 Oct 30 17:05:17.138108 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:17.138068 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43e9434f-nzkjb" event={"ID":"2bfc83a8-a240-4f65-8034-6dc828a8c14e","Type":"ContainerDied","Data":"73ebb99cddb09eb2591357918620a190199d0bc8fd4ca16cbb572ffbed0fc7b1"} Oct 30 17:05:18.317632 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:18.317608 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43e9434f-nzkjb" Oct 30 17:05:18.507827 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:18.507794 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bfc83a8-a240-4f65-8034-6dc828a8c14e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bfc83a8-a240-4f65-8034-6dc828a8c14e-maas-api-openshift-ca\") pod \"2bfc83a8-a240-4f65-8034-6dc828a8c14e\" (UID: \"2bfc83a8-a240-4f65-8034-6dc828a8c14e\") " Oct 30 17:05:18.507980 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:18.507887 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bfc83a8-a240-4f65-8034-6dc828a8c14e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bfc83a8-a240-4f65-8034-6dc828a8c14e-maas-api-xks-ca\") pod \"2bfc83a8-a240-4f65-8034-6dc828a8c14e\" (UID: \"2bfc83a8-a240-4f65-8034-6dc828a8c14e\") " Oct 30 17:05:18.508039 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:18.508030 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bfc83a8-a240-4f65-8034-6dc828a8c14e-kube-api-access-c6z6f\" (UniqueName: \"kubernetes.io/projected/2bfc83a8-a240-4f65-8034-6dc828a8c14e-kube-api-access-c6z6f\") pod \"2bfc83a8-a240-4f65-8034-6dc828a8c14e\" (UID: \"2bfc83a8-a240-4f65-8034-6dc828a8c14e\") " Oct 30 17:05:18.508206 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:18.508177 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bfc83a8-a240-4f65-8034-6dc828a8c14e-maas-api-openshift-ca" pod "2bfc83a8-a240-4f65-8034-6dc828a8c14e" (UID: "2bfc83a8-a240-4f65-8034-6dc828a8c14e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:05:18.508624 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:18.508603 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bfc83a8-a240-4f65-8034-6dc828a8c14e-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:05:18.510010 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:18.509989 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bfc83a8-a240-4f65-8034-6dc828a8c14e-maas-api-xks-ca" pod "2bfc83a8-a240-4f65-8034-6dc828a8c14e" (UID: "2bfc83a8-a240-4f65-8034-6dc828a8c14e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:05:18.510060 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:18.510037 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bfc83a8-a240-4f65-8034-6dc828a8c14e-kube-api-access-c6z6f" pod "2bfc83a8-a240-4f65-8034-6dc828a8c14e" (UID: "2bfc83a8-a240-4f65-8034-6dc828a8c14e"). InnerVolumeSpecName "kube-api-access-c6z6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:05:18.609569 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:18.609529 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6z6f\" (UniqueName: \"kubernetes.io/projected/2bfc83a8-a240-4f65-8034-6dc828a8c14e-kube-api-access-c6z6f\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:05:18.609569 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:18.609564 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bfc83a8-a240-4f65-8034-6dc828a8c14e-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:05:18.965176 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:18.965146 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7eda4908-7886d476c-zj8zq"] Oct 30 17:05:18.965797 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:18.965782 4522 cpu_manager.go:414] "containerMap: removing container" podUID="2bfc83a8-a240-4f65-8034-6dc828a8c14e" containerName="revoke-keys" Oct 30 17:05:18.965859 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:18.965797 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bfc83a8-a240-4f65-8034-6dc828a8c14e" containerName="revoke-keys" Oct 30 17:05:18.965921 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:18.965913 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bfc83a8-a240-4f65-8034-6dc828a8c14e" containerName="revoke-keys" Oct 30 17:05:18.969646 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:18.969627 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7eda4908-7886d476c-zj8zq" Oct 30 17:05:18.990471 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:18.990440 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7eda4908-7886d476c-zj8zq"] Oct 30 17:05:18.992828 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:18.992803 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-7eda4908\"" Oct 30 17:05:18.998479 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:18.998457 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-7eda4908-dockercfg-kcjcm\"" Oct 30 17:05:19.013606 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:19.013580 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wltqb\" (UniqueName: \"kubernetes.io/projected/a9d0d135-98cd-40ca-aefd-9a13aafbbd68-kube-api-access-wltqb\") pod \"payload-pre-processing-e2e-ait-7eda4908-7886d476c-zj8zq\" (UID: \"a9d0d135-98cd-40ca-aefd-9a13aafbbd68\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7eda4908-7886d476c-zj8zq" Oct 30 17:05:19.013795 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:19.013780 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9d0d135-98cd-40ca-aefd-9a13aafbbd68-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7eda4908-7886d476c-zj8zq\" (UID: \"a9d0d135-98cd-40ca-aefd-9a13aafbbd68\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7eda4908-7886d476c-zj8zq" Oct 30 17:05:19.126399 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:19.121576 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9d0d135-98cd-40ca-aefd-9a13aafbbd68-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7eda4908-7886d476c-zj8zq\" (UID: \"a9d0d135-98cd-40ca-aefd-9a13aafbbd68\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7eda4908-7886d476c-zj8zq" Oct 30 17:05:19.126399 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:19.121920 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wltqb\" (UniqueName: \"kubernetes.io/projected/a9d0d135-98cd-40ca-aefd-9a13aafbbd68-kube-api-access-wltqb\") pod \"payload-pre-processing-e2e-ait-7eda4908-7886d476c-zj8zq\" (UID: \"a9d0d135-98cd-40ca-aefd-9a13aafbbd68\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7eda4908-7886d476c-zj8zq" Oct 30 17:05:19.126399 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:19.123849 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9d0d135-98cd-40ca-aefd-9a13aafbbd68-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7eda4908-7886d476c-zj8zq\" (UID: \"a9d0d135-98cd-40ca-aefd-9a13aafbbd68\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7eda4908-7886d476c-zj8zq" Oct 30 17:05:19.160546 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:19.143395 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7eda4908-59dccc9fbb-j26p6"] Oct 30 17:05:19.189402 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:19.184390 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wltqb\" (UniqueName: \"kubernetes.io/projected/a9d0d135-98cd-40ca-aefd-9a13aafbbd68-kube-api-access-wltqb\") pod \"payload-pre-processing-e2e-ait-7eda4908-7886d476c-zj8zq\" (UID: \"a9d0d135-98cd-40ca-aefd-9a13aafbbd68\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7eda4908-7886d476c-zj8zq" Oct 30 17:05:19.209761 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:19.209730 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7eda4908-59dccc9fbb-j26p6" Oct 30 17:05:19.216471 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:19.216360 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7eda4908-59dccc9fbb-j26p6"] Oct 30 17:05:19.222558 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:19.220266 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43e9434f-nzkjb" event={"ID":"2bfc83a8-a240-4f65-8034-6dc828a8c14e","Type":"ContainerDied","Data":"e0588affc0f9858f67abeee7d39086560a1e2135b3fc07ee1239226a96cf0352"} Oct 30 17:05:19.222558 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:19.220326 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0588affc0f9858f67abeee7d39086560a1e2135b3fc07ee1239226a96cf0352" Oct 30 17:05:19.222558 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:19.220394 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43e9434f-nzkjb" Oct 30 17:05:19.284415 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:19.284377 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7eda4908-7886d476c-zj8zq" Oct 30 17:05:19.325809 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:19.325769 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9aac958f-8ecb-4ac3-8c02-d00e448f74a5-plugins-config-volume\") pod \"payload-processing-e2e-ait-7eda4908-59dccc9fbb-j26p6\" (UID: \"9aac958f-8ecb-4ac3-8c02-d00e448f74a5\") " pod="openshift-ingress/payload-processing-e2e-ait-7eda4908-59dccc9fbb-j26p6" Oct 30 17:05:19.326235 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:19.325931 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grxwm\" (UniqueName: \"kubernetes.io/projected/9aac958f-8ecb-4ac3-8c02-d00e448f74a5-kube-api-access-grxwm\") pod \"payload-processing-e2e-ait-7eda4908-59dccc9fbb-j26p6\" (UID: \"9aac958f-8ecb-4ac3-8c02-d00e448f74a5\") " pod="openshift-ingress/payload-processing-e2e-ait-7eda4908-59dccc9fbb-j26p6" Oct 30 17:05:19.427187 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:19.427148 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9aac958f-8ecb-4ac3-8c02-d00e448f74a5-plugins-config-volume\") pod \"payload-processing-e2e-ait-7eda4908-59dccc9fbb-j26p6\" (UID: \"9aac958f-8ecb-4ac3-8c02-d00e448f74a5\") " pod="openshift-ingress/payload-processing-e2e-ait-7eda4908-59dccc9fbb-j26p6" Oct 30 17:05:19.427417 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:19.427258 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grxwm\" (UniqueName: \"kubernetes.io/projected/9aac958f-8ecb-4ac3-8c02-d00e448f74a5-kube-api-access-grxwm\") pod \"payload-processing-e2e-ait-7eda4908-59dccc9fbb-j26p6\" (UID: \"9aac958f-8ecb-4ac3-8c02-d00e448f74a5\") " pod="openshift-ingress/payload-processing-e2e-ait-7eda4908-59dccc9fbb-j26p6" Oct 30 17:05:19.428107 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:19.428081 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9aac958f-8ecb-4ac3-8c02-d00e448f74a5-plugins-config-volume\") pod \"payload-processing-e2e-ait-7eda4908-59dccc9fbb-j26p6\" (UID: \"9aac958f-8ecb-4ac3-8c02-d00e448f74a5\") " pod="openshift-ingress/payload-processing-e2e-ait-7eda4908-59dccc9fbb-j26p6" Oct 30 17:05:19.455006 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:19.454976 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grxwm\" (UniqueName: \"kubernetes.io/projected/9aac958f-8ecb-4ac3-8c02-d00e448f74a5-kube-api-access-grxwm\") pod \"payload-processing-e2e-ait-7eda4908-59dccc9fbb-j26p6\" (UID: \"9aac958f-8ecb-4ac3-8c02-d00e448f74a5\") " pod="openshift-ingress/payload-processing-e2e-ait-7eda4908-59dccc9fbb-j26p6" Oct 30 17:05:19.547544 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:19.547462 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7eda4908-59dccc9fbb-j26p6" Oct 30 17:05:19.599180 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:05:19.599142 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9d0d135_98cd_40ca_aefd_9a13aafbbd68.slice/crio-2886e3cc329491a9707798de6afd3018bb3b9641a139ba58be4e8b577e555bdc WatchSource:0}: Error finding container 2886e3cc329491a9707798de6afd3018bb3b9641a139ba58be4e8b577e555bdc: Status 404 returned error can't find the container with id 2886e3cc329491a9707798de6afd3018bb3b9641a139ba58be4e8b577e555bdc Oct 30 17:05:19.601094 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:19.601067 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7eda4908-7886d476c-zj8zq"] Oct 30 17:05:19.863422 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:05:19.863384 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aac958f_8ecb_4ac3_8c02_d00e448f74a5.slice/crio-4037ba745474d61d56108d2d20c0158bc7267790c593b17bf37da9290f4e41ce WatchSource:0}: Error finding container 4037ba745474d61d56108d2d20c0158bc7267790c593b17bf37da9290f4e41ce: Status 404 returned error can't find the container with id 4037ba745474d61d56108d2d20c0158bc7267790c593b17bf37da9290f4e41ce Oct 30 17:05:19.865106 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:19.865079 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7eda4908-59dccc9fbb-j26p6"] Oct 30 17:05:20.229852 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:20.229811 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7eda4908-59dccc9fbb-j26p6" event={"ID":"9aac958f-8ecb-4ac3-8c02-d00e448f74a5","Type":"ContainerStarted","Data":"d34acdaad727fb05695cb566dfe4b81dea0937eaeac2a2cf3af2e089a7fe0484"} Oct 30 17:05:20.230024 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:20.229876 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7eda4908-59dccc9fbb-j26p6" event={"ID":"9aac958f-8ecb-4ac3-8c02-d00e448f74a5","Type":"ContainerStarted","Data":"4037ba745474d61d56108d2d20c0158bc7267790c593b17bf37da9290f4e41ce"} Oct 30 17:05:20.230024 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:20.229909 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-7eda4908-59dccc9fbb-j26p6" Oct 30 17:05:20.231609 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:20.231581 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7eda4908-7886d476c-zj8zq" event={"ID":"a9d0d135-98cd-40ca-aefd-9a13aafbbd68","Type":"ContainerStarted","Data":"d5a7ded6360a5e2a345446efa5bdeb747cf1909edaba7f6ebb5053ebab2c1002"} Oct 30 17:05:20.231739 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:20.231627 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7eda4908-7886d476c-zj8zq" event={"ID":"a9d0d135-98cd-40ca-aefd-9a13aafbbd68","Type":"ContainerStarted","Data":"2886e3cc329491a9707798de6afd3018bb3b9641a139ba58be4e8b577e555bdc"} Oct 30 17:05:20.231739 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:20.231677 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-7eda4908-7886d476c-zj8zq" Oct 30 17:05:20.277824 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:20.277774 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-7eda4908-59dccc9fbb-j26p6" podStartSLOduration=1.277760596 podStartE2EDuration="1.277760596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:05:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:05:20.276108421 +0000 UTC m=+1094.276863342" watchObservedRunningTime="2026-10-30 17:05:20.277760596 +0000 UTC m=+1094.278515505" Oct 30 17:05:20.326011 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:20.325963 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-7eda4908-7886d476c-zj8zq" podStartSLOduration=2.325950173 podStartE2EDuration="2.325950173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:05:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:05:20.324350704 +0000 UTC m=+1094.325105602" watchObservedRunningTime="2026-10-30 17:05:20.325950173 +0000 UTC m=+1094.326705104" Oct 30 17:05:22.597059 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:22.597022 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-952bc16e-6865b69dc5-zqc5b"] Oct 30 17:05:22.602129 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:22.602109 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-952bc16e-6865b69dc5-zqc5b" Oct 30 17:05:22.623596 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:22.623567 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-952bc16e-6865b69dc5-zqc5b"] Oct 30 17:05:22.631698 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:22.631670 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-952bc16e\"" Oct 30 17:05:22.631946 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:22.631909 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-952bc16e-dockercfg-fjfzl\"" Oct 30 17:05:22.672684 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:22.672651 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ddc647-b2ef-445a-a05a-96893865f100-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-952bc16e-6865b69dc5-zqc5b\" (UID: \"64ddc647-b2ef-445a-a05a-96893865f100\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-952bc16e-6865b69dc5-zqc5b" Oct 30 17:05:22.672874 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:22.672854 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrc9n\" (UniqueName: \"kubernetes.io/projected/64ddc647-b2ef-445a-a05a-96893865f100-kube-api-access-jrc9n\") pod \"payload-pre-processing-e2e-ait-952bc16e-6865b69dc5-zqc5b\" (UID: \"64ddc647-b2ef-445a-a05a-96893865f100\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-952bc16e-6865b69dc5-zqc5b" Oct 30 17:05:22.712607 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:22.712571 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-952bc16e-5cdf6db644-hm4mt"] Oct 30 17:05:22.730325 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:22.718709 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-952bc16e-5cdf6db644-hm4mt" Oct 30 17:05:22.746275 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:22.746247 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-952bc16e-5cdf6db644-hm4mt"] Oct 30 17:05:22.828886 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:22.828844 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ddc647-b2ef-445a-a05a-96893865f100-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-952bc16e-6865b69dc5-zqc5b\" (UID: \"64ddc647-b2ef-445a-a05a-96893865f100\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-952bc16e-6865b69dc5-zqc5b" Oct 30 17:05:22.829442 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:22.829412 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77f5ee2d-25a4-486c-965a-324c473dd9b7-plugins-config-volume\") pod \"payload-processing-e2e-ait-952bc16e-5cdf6db644-hm4mt\" (UID: \"77f5ee2d-25a4-486c-965a-324c473dd9b7\") " pod="openshift-ingress/payload-processing-e2e-ait-952bc16e-5cdf6db644-hm4mt" Oct 30 17:05:22.829652 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:22.829632 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrc9n\" (UniqueName: \"kubernetes.io/projected/64ddc647-b2ef-445a-a05a-96893865f100-kube-api-access-jrc9n\") pod \"payload-pre-processing-e2e-ait-952bc16e-6865b69dc5-zqc5b\" (UID: \"64ddc647-b2ef-445a-a05a-96893865f100\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-952bc16e-6865b69dc5-zqc5b" Oct 30 17:05:22.830037 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:22.829929 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlpt7\" (UniqueName: \"kubernetes.io/projected/77f5ee2d-25a4-486c-965a-324c473dd9b7-kube-api-access-qlpt7\") pod \"payload-processing-e2e-ait-952bc16e-5cdf6db644-hm4mt\" (UID: \"77f5ee2d-25a4-486c-965a-324c473dd9b7\") " pod="openshift-ingress/payload-processing-e2e-ait-952bc16e-5cdf6db644-hm4mt" Oct 30 17:05:22.830271 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:22.830240 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ddc647-b2ef-445a-a05a-96893865f100-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-952bc16e-6865b69dc5-zqc5b\" (UID: \"64ddc647-b2ef-445a-a05a-96893865f100\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-952bc16e-6865b69dc5-zqc5b" Oct 30 17:05:22.914068 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:22.913979 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrc9n\" (UniqueName: \"kubernetes.io/projected/64ddc647-b2ef-445a-a05a-96893865f100-kube-api-access-jrc9n\") pod \"payload-pre-processing-e2e-ait-952bc16e-6865b69dc5-zqc5b\" (UID: \"64ddc647-b2ef-445a-a05a-96893865f100\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-952bc16e-6865b69dc5-zqc5b" Oct 30 17:05:22.916715 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:22.916681 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-952bc16e-6865b69dc5-zqc5b" Oct 30 17:05:22.932151 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:22.932115 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77f5ee2d-25a4-486c-965a-324c473dd9b7-plugins-config-volume\") pod \"payload-processing-e2e-ait-952bc16e-5cdf6db644-hm4mt\" (UID: \"77f5ee2d-25a4-486c-965a-324c473dd9b7\") " pod="openshift-ingress/payload-processing-e2e-ait-952bc16e-5cdf6db644-hm4mt" Oct 30 17:05:22.932285 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:22.932197 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlpt7\" (UniqueName: \"kubernetes.io/projected/77f5ee2d-25a4-486c-965a-324c473dd9b7-kube-api-access-qlpt7\") pod \"payload-processing-e2e-ait-952bc16e-5cdf6db644-hm4mt\" (UID: \"77f5ee2d-25a4-486c-965a-324c473dd9b7\") " pod="openshift-ingress/payload-processing-e2e-ait-952bc16e-5cdf6db644-hm4mt" Oct 30 17:05:22.932824 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:22.932799 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77f5ee2d-25a4-486c-965a-324c473dd9b7-plugins-config-volume\") pod \"payload-processing-e2e-ait-952bc16e-5cdf6db644-hm4mt\" (UID: \"77f5ee2d-25a4-486c-965a-324c473dd9b7\") " pod="openshift-ingress/payload-processing-e2e-ait-952bc16e-5cdf6db644-hm4mt" Oct 30 17:05:22.958604 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:22.958578 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlpt7\" (UniqueName: \"kubernetes.io/projected/77f5ee2d-25a4-486c-965a-324c473dd9b7-kube-api-access-qlpt7\") pod \"payload-processing-e2e-ait-952bc16e-5cdf6db644-hm4mt\" (UID: \"77f5ee2d-25a4-486c-965a-324c473dd9b7\") " pod="openshift-ingress/payload-processing-e2e-ait-952bc16e-5cdf6db644-hm4mt" Oct 30 17:05:23.055880 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:23.055836 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-952bc16e-5cdf6db644-hm4mt" Oct 30 17:05:23.232955 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:05:23.232923 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64ddc647_b2ef_445a_a05a_96893865f100.slice/crio-af573360279f9cc89b0ca1411ad7a76ed2324961226366a67822ec39e541e864 WatchSource:0}: Error finding container af573360279f9cc89b0ca1411ad7a76ed2324961226366a67822ec39e541e864: Status 404 returned error can't find the container with id af573360279f9cc89b0ca1411ad7a76ed2324961226366a67822ec39e541e864 Oct 30 17:05:23.236476 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:23.236450 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-952bc16e-6865b69dc5-zqc5b"] Oct 30 17:05:23.256860 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:23.256831 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-952bc16e-6865b69dc5-zqc5b" event={"ID":"64ddc647-b2ef-445a-a05a-96893865f100","Type":"ContainerStarted","Data":"af573360279f9cc89b0ca1411ad7a76ed2324961226366a67822ec39e541e864"} Oct 30 17:05:23.373072 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:05:23.373040 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77f5ee2d_25a4_486c_965a_324c473dd9b7.slice/crio-1fc84898be2776b368e26a36a390707254b47ff417efacdd4f0b87eece5e25ca WatchSource:0}: Error finding container 1fc84898be2776b368e26a36a390707254b47ff417efacdd4f0b87eece5e25ca: Status 404 returned error can't find the container with id 1fc84898be2776b368e26a36a390707254b47ff417efacdd4f0b87eece5e25ca Oct 30 17:05:23.376355 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:23.376336 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-952bc16e-5cdf6db644-hm4mt"] Oct 30 17:05:24.266363 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:24.266325 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-952bc16e-6865b69dc5-zqc5b" event={"ID":"64ddc647-b2ef-445a-a05a-96893865f100","Type":"ContainerStarted","Data":"bae4f1bf4fcc3682226d6dd85c5c1a1aa1138ec77ce216842d5ee14b16c8ed2e"} Oct 30 17:05:24.266807 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:24.266393 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-952bc16e-6865b69dc5-zqc5b" Oct 30 17:05:24.268211 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:24.268186 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-952bc16e-5cdf6db644-hm4mt" event={"ID":"77f5ee2d-25a4-486c-965a-324c473dd9b7","Type":"ContainerStarted","Data":"7800ceadf8d9edb241e828e57b3eb76c8826c2e76df077a69939c4a3c40d6b2d"} Oct 30 17:05:24.268353 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:24.268223 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-952bc16e-5cdf6db644-hm4mt" event={"ID":"77f5ee2d-25a4-486c-965a-324c473dd9b7","Type":"ContainerStarted","Data":"1fc84898be2776b368e26a36a390707254b47ff417efacdd4f0b87eece5e25ca"} Oct 30 17:05:24.268353 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:24.268312 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-952bc16e-5cdf6db644-hm4mt" Oct 30 17:05:24.313571 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:24.313517 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-952bc16e-6865b69dc5-zqc5b" podStartSLOduration=2.313503971 podStartE2EDuration="2.313503971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:05:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:05:24.312837684 +0000 UTC m=+1098.313592585" watchObservedRunningTime="2026-10-30 17:05:24.313503971 +0000 UTC m=+1098.314258915" Oct 30 17:05:24.364246 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:24.364194 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-952bc16e-5cdf6db644-hm4mt" podStartSLOduration=2.364180203 podStartE2EDuration="2.364180203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:05:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:05:24.35904529 +0000 UTC m=+1098.359800192" watchObservedRunningTime="2026-10-30 17:05:24.364180203 +0000 UTC m=+1098.364935104" Oct 30 17:05:24.886923 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:24.886886 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc707697-hbvdm"] Oct 30 17:05:24.891554 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:24.891533 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc707697-hbvdm" Oct 30 17:05:24.909820 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:24.909780 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:05:24.909820 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:24.909805 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:05:24.912741 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:24.912715 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc707697-hbvdm"] Oct 30 17:05:24.965752 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:24.965505 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/677fd30b-2b0d-42ff-8351-8d6e801adf5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc707697-hbvdm\" (UID: \"677fd30b-2b0d-42ff-8351-8d6e801adf5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc707697-hbvdm" Oct 30 17:05:24.965752 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:24.965740 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/677fd30b-2b0d-42ff-8351-8d6e801adf5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc707697-hbvdm\" (UID: \"677fd30b-2b0d-42ff-8351-8d6e801adf5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc707697-hbvdm" Oct 30 17:05:24.965972 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:24.965832 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxdmv\" (UniqueName: \"kubernetes.io/projected/677fd30b-2b0d-42ff-8351-8d6e801adf5f-kube-api-access-vxdmv\") pod \"maas-api-revoke-sub-dc707697-hbvdm\" (UID: \"677fd30b-2b0d-42ff-8351-8d6e801adf5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc707697-hbvdm" Oct 30 17:05:25.067818 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:25.067776 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/677fd30b-2b0d-42ff-8351-8d6e801adf5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc707697-hbvdm\" (UID: \"677fd30b-2b0d-42ff-8351-8d6e801adf5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc707697-hbvdm" Oct 30 17:05:25.068020 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:25.067873 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxdmv\" (UniqueName: \"kubernetes.io/projected/677fd30b-2b0d-42ff-8351-8d6e801adf5f-kube-api-access-vxdmv\") pod \"maas-api-revoke-sub-dc707697-hbvdm\" (UID: \"677fd30b-2b0d-42ff-8351-8d6e801adf5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc707697-hbvdm" Oct 30 17:05:25.068086 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:25.068029 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/677fd30b-2b0d-42ff-8351-8d6e801adf5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc707697-hbvdm\" (UID: \"677fd30b-2b0d-42ff-8351-8d6e801adf5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc707697-hbvdm" Oct 30 17:05:25.068578 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:25.068553 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/677fd30b-2b0d-42ff-8351-8d6e801adf5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc707697-hbvdm\" (UID: \"677fd30b-2b0d-42ff-8351-8d6e801adf5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc707697-hbvdm" Oct 30 17:05:25.070508 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:25.070485 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/677fd30b-2b0d-42ff-8351-8d6e801adf5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc707697-hbvdm\" (UID: \"677fd30b-2b0d-42ff-8351-8d6e801adf5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc707697-hbvdm" Oct 30 17:05:25.096130 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:25.096095 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxdmv\" (UniqueName: \"kubernetes.io/projected/677fd30b-2b0d-42ff-8351-8d6e801adf5f-kube-api-access-vxdmv\") pod \"maas-api-revoke-sub-dc707697-hbvdm\" (UID: \"677fd30b-2b0d-42ff-8351-8d6e801adf5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc707697-hbvdm" Oct 30 17:05:25.212346 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:25.212314 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc707697-hbvdm" Oct 30 17:05:25.369657 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:25.369625 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-b4f406/distinct-w2-b4f406-kserve-5db4cc6969-7n846" Oct 30 17:05:25.390299 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:25.390261 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-b4f406/distinct-w2-b4f406-kserve-5db4cc6969-7n846" Oct 30 17:05:25.530711 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:05:25.530676 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod677fd30b_2b0d_42ff_8351_8d6e801adf5f.slice/crio-2112df3daa3af35a7267d224322eea3f739b20a71d8f33fda96b9459fc8eeead WatchSource:0}: Error finding container 2112df3daa3af35a7267d224322eea3f739b20a71d8f33fda96b9459fc8eeead: Status 404 returned error can't find the container with id 2112df3daa3af35a7267d224322eea3f739b20a71d8f33fda96b9459fc8eeead Oct 30 17:05:25.534003 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:25.533975 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc707697-hbvdm"] Oct 30 17:05:26.380332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:26.379424 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc707697-hbvdm" event={"ID":"677fd30b-2b0d-42ff-8351-8d6e801adf5f","Type":"ContainerStarted","Data":"734f2ae61c97009cf9b9862a641563555211e36619d7be1efc805a17c7a20c16"} Oct 30 17:05:26.380332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:26.379486 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc707697-hbvdm" event={"ID":"677fd30b-2b0d-42ff-8351-8d6e801adf5f","Type":"ContainerStarted","Data":"2112df3daa3af35a7267d224322eea3f739b20a71d8f33fda96b9459fc8eeead"} Oct 30 17:05:26.496328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:26.485218 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc707697-hbvdm" podStartSLOduration=2.485196155 podStartE2EDuration="2.485196155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:05:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:05:26.459679756 +0000 UTC m=+1100.460434662" watchObservedRunningTime="2026-10-30 17:05:26.485196155 +0000 UTC m=+1100.485951058" Oct 30 17:05:27.413441 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:27.408718 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/distinct-w2-b4f406-kserve-5db4cc6969-7n846" podUID="b8810dbc-b769-46a4-90b9-72a68592bc1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:27.459330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:27.457968 4522 generic.go:356] "Generic (PLEG): container finished" podID="677fd30b-2b0d-42ff-8351-8d6e801adf5f" containerID="734f2ae61c97009cf9b9862a641563555211e36619d7be1efc805a17c7a20c16" exitCode=0 Oct 30 17:05:27.459330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:27.458085 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc707697-hbvdm" event={"ID":"677fd30b-2b0d-42ff-8351-8d6e801adf5f","Type":"ContainerDied","Data":"734f2ae61c97009cf9b9862a641563555211e36619d7be1efc805a17c7a20c16"} Oct 30 17:05:27.792330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:27.791460 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" podUID="4360e9dc-c001-4102-8af6-b55b692fcf2c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:27.797329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:27.793198 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" podUID="2f1e3bbb-ab1c-455d-a05d-d62f72c5d009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:27.932331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:27.929662 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" podUID="0b832781-3d41-4496-aa86-e4d92792e590" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:27.959330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:27.956187 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" podUID="8a6a85ad-c944-450f-b0bb-e3b64b0f56f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:27.983337 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:27.972778 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" podUID="c9ad71b7-ad90-4c8c-bf27-0f10b23f1593" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:27.983337 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:27.974867 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" podUID="75321c5d-8bf8-45c6-8811-81981fe3cad4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:27.987328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:27.986473 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/distinct-w1-04248e-kserve-5859d49965-kgq46" podUID="8dda5ac4-30dd-432e-81eb-cb6d39800628" containerName="main" probeResult="failure" output="Get \"https://10.134.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:28.087385 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:28.086394 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" podUID="9bfbc6d3-91fd-41a7-9585-40a8f252f743" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:28.339388 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:28.330403 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" podUID="a3e5e732-2c67-4377-9755-e2178bf33772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:28.415329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:28.414433 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/distinct-w2-b4f406-kserve-5db4cc6969-7n846" podUID="b8810dbc-b769-46a4-90b9-72a68592bc1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:28.438330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:28.433446 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" podUID="43ca764d-ae0e-4858-9841-5bbd0ffdbb85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:28.705328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:28.702402 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" podUID="4b4ad752-e053-4cea-a3bc-fe2be6f92021" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:28.798336 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:28.792393 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" podUID="4360e9dc-c001-4102-8af6-b55b692fcf2c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:28.798336 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:28.793009 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" Oct 30 17:05:28.826147 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:28.804421 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" podUID="2f1e3bbb-ab1c-455d-a05d-d62f72c5d009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:28.826147 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:28.805199 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" Oct 30 17:05:29.073646 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:28.934053 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" podUID="0b832781-3d41-4496-aa86-e4d92792e590" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:29.073646 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:28.934676 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" Oct 30 17:05:29.073646 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:28.982365 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" podUID="75321c5d-8bf8-45c6-8811-81981fe3cad4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:29.073646 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:29.016526 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" podUID="c9ad71b7-ad90-4c8c-bf27-0f10b23f1593" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 17:05:29.073646 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:29.017299 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" Oct 30 17:05:29.073646 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:29.017384 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" Oct 30 17:05:29.073646 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:29.022252 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" podUID="8a6a85ad-c944-450f-b0bb-e3b64b0f56f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:29.073646 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:29.022646 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" Oct 30 17:05:29.073646 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:29.044387 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/distinct-w1-04248e-kserve-5859d49965-kgq46" podUID="8dda5ac4-30dd-432e-81eb-cb6d39800628" containerName="main" probeResult="failure" output="Get \"https://10.134.0.108:8000/ready\": context deadline exceeded" Oct 30 17:05:29.073646 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:29.044935 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04248e/distinct-w1-04248e-kserve-5859d49965-kgq46" Oct 30 17:05:29.099672 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:29.098424 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" podUID="9bfbc6d3-91fd-41a7-9585-40a8f252f743" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:29.099672 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:29.098973 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" Oct 30 17:05:29.340381 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:29.337769 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" podUID="a3e5e732-2c67-4377-9755-e2178bf33772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:29.373408 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:29.367591 4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" podUID="c9ad71b7-ad90-4c8c-bf27-0f10b23f1593" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": context deadline exceeded" Oct 30 17:05:29.373408 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:29.367972 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" Oct 30 17:05:29.464363 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:29.458407 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/distinct-w2-b4f406-kserve-5db4cc6969-7n846" podUID="b8810dbc-b769-46a4-90b9-72a68592bc1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:29.464363 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:29.461749 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" podUID="43ca764d-ae0e-4858-9841-5bbd0ffdbb85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:29.465336 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:29.465092 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" Oct 30 17:05:29.752331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:29.735658 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" podUID="4b4ad752-e053-4cea-a3bc-fe2be6f92021" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:29.752331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:29.743956 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" Oct 30 17:05:29.807668 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:29.807623 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" podUID="2f1e3bbb-ab1c-455d-a05d-d62f72c5d009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:29.898963 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:29.809236 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" podUID="4360e9dc-c001-4102-8af6-b55b692fcf2c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Oct 30 17:05:29.970611 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:29.911851 4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" podUID="8a6a85ad-c944-450f-b0bb-e3b64b0f56f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:29.970969 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:29.935710 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" podUID="0b832781-3d41-4496-aa86-e4d92792e590" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 17:05:29.990339 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:29.988520 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" podUID="75321c5d-8bf8-45c6-8811-81981fe3cad4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:30.017537 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:30.017484 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" podUID="c9ad71b7-ad90-4c8c-bf27-0f10b23f1593" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:30.044947 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:30.044906 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" podUID="8a6a85ad-c944-450f-b0bb-e3b64b0f56f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 17:05:30.071037 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:30.070992 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/distinct-w1-04248e-kserve-5859d49965-kgq46" podUID="8dda5ac4-30dd-432e-81eb-cb6d39800628" containerName="main" probeResult="failure" output="Get \"https://10.134.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:30.100453 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:30.100395 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" podUID="9bfbc6d3-91fd-41a7-9585-40a8f252f743" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:30.413388 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:30.367560 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" podUID="a3e5e732-2c67-4377-9755-e2178bf33772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:30.607390 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:30.599560 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" podUID="43ca764d-ae0e-4858-9841-5bbd0ffdbb85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 17:05:30.624433 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:30.624376 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/distinct-w2-b4f406-kserve-5db4cc6969-7n846" podUID="b8810dbc-b769-46a4-90b9-72a68592bc1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": context deadline exceeded" Oct 30 17:05:30.754331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:30.753563 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc707697-hbvdm" Oct 30 17:05:30.787329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:30.787233 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" podUID="4b4ad752-e053-4cea-a3bc-fe2be6f92021" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:30.896857 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:30.815109 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" podUID="2f1e3bbb-ab1c-455d-a05d-d62f72c5d009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:30.972719 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:30.972674 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" podUID="0b832781-3d41-4496-aa86-e4d92792e590" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:30.990330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:30.977455 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" podUID="4360e9dc-c001-4102-8af6-b55b692fcf2c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Oct 30 17:05:31.019441 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:30.991736 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" podUID="75321c5d-8bf8-45c6-8811-81981fe3cad4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:31.177384 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:31.177242 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/distinct-w1-04248e-kserve-5859d49965-kgq46" podUID="8dda5ac4-30dd-432e-81eb-cb6d39800628" containerName="main" probeResult="failure" output="Get \"https://10.134.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:31.242332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:31.204857 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" podUID="8a6a85ad-c944-450f-b0bb-e3b64b0f56f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:31.242332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:31.240700 4522 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:05:31.242332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:31.240756 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs" podUID="2ecae9c7-ada3-4612-909f-1c1493369daf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:31.291089 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:31.291039 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" podUID="9bfbc6d3-91fd-41a7-9585-40a8f252f743" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:31.372416 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:31.343074 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/677fd30b-2b0d-42ff-8351-8d6e801adf5f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/677fd30b-2b0d-42ff-8351-8d6e801adf5f-maas-api-openshift-ca\") pod \"677fd30b-2b0d-42ff-8351-8d6e801adf5f\" (UID: \"677fd30b-2b0d-42ff-8351-8d6e801adf5f\") " Oct 30 17:05:31.372416 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:31.371544 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/677fd30b-2b0d-42ff-8351-8d6e801adf5f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/677fd30b-2b0d-42ff-8351-8d6e801adf5f-maas-api-xks-ca\") pod \"677fd30b-2b0d-42ff-8351-8d6e801adf5f\" (UID: \"677fd30b-2b0d-42ff-8351-8d6e801adf5f\") " Oct 30 17:05:31.372416 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:31.371835 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/677fd30b-2b0d-42ff-8351-8d6e801adf5f-kube-api-access-vxdmv\" (UniqueName: \"kubernetes.io/projected/677fd30b-2b0d-42ff-8351-8d6e801adf5f-kube-api-access-vxdmv\") pod \"677fd30b-2b0d-42ff-8351-8d6e801adf5f\" (UID: \"677fd30b-2b0d-42ff-8351-8d6e801adf5f\") " Oct 30 17:05:31.421129 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:31.369735 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/677fd30b-2b0d-42ff-8351-8d6e801adf5f-maas-api-openshift-ca" pod "677fd30b-2b0d-42ff-8351-8d6e801adf5f" (UID: "677fd30b-2b0d-42ff-8351-8d6e801adf5f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:05:31.421409 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:31.369809 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" podUID="a3e5e732-2c67-4377-9755-e2178bf33772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:31.423423 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:31.423382 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" podUID="c9ad71b7-ad90-4c8c-bf27-0f10b23f1593" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:31.509351 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:31.479114 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/677fd30b-2b0d-42ff-8351-8d6e801adf5f-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:05:31.509351 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:05:31.505476 4522 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.368s" Oct 30 17:05:31.509351 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:31.505591 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" Oct 30 17:05:31.509351 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:31.505780 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-7eda4908-59dccc9fbb-j26p6" Oct 30 17:05:31.512005 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:31.511968 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/677fd30b-2b0d-42ff-8351-8d6e801adf5f-maas-api-xks-ca" pod "677fd30b-2b0d-42ff-8351-8d6e801adf5f" (UID: "677fd30b-2b0d-42ff-8351-8d6e801adf5f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:05:31.521619 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:31.521587 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-7eda4908-7886d476c-zj8zq" Oct 30 17:05:31.560337 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:31.559344 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc707697-hbvdm" event={"ID":"677fd30b-2b0d-42ff-8351-8d6e801adf5f","Type":"ContainerDied","Data":"2112df3daa3af35a7267d224322eea3f739b20a71d8f33fda96b9459fc8eeead"} Oct 30 17:05:31.560337 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:31.559412 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2112df3daa3af35a7267d224322eea3f739b20a71d8f33fda96b9459fc8eeead" Oct 30 17:05:31.560337 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:31.559512 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc707697-hbvdm" Oct 30 17:05:31.636988 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:31.627829 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" Oct 30 17:05:31.655484 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:31.641112 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" podUID="43ca764d-ae0e-4858-9841-5bbd0ffdbb85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:31.729586 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:31.729531 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/distinct-w2-b4f406-kserve-5db4cc6969-7n846" podUID="b8810dbc-b769-46a4-90b9-72a68592bc1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:31.752366 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:31.742989 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/677fd30b-2b0d-42ff-8351-8d6e801adf5f-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:05:31.798417 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:31.790392 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" podUID="4b4ad752-e053-4cea-a3bc-fe2be6f92021" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:31.927976 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:31.840135 4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" podUID="9bfbc6d3-91fd-41a7-9585-40a8f252f743" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:31.939736 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:31.875486 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/677fd30b-2b0d-42ff-8351-8d6e801adf5f-kube-api-access-vxdmv" pod "677fd30b-2b0d-42ff-8351-8d6e801adf5f" (UID: "677fd30b-2b0d-42ff-8351-8d6e801adf5f"). InnerVolumeSpecName "kube-api-access-vxdmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:05:31.939962 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:31.926458 4522 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:05:31.940099 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:31.940070 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m" podUID="232a5af4-ecfb-4e38-a937-fc555a9b946b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:31.940496 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:31.884002 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" Oct 30 17:05:31.940725 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:31.905752 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" podUID="2f1e3bbb-ab1c-455d-a05d-d62f72c5d009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:31.965336 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:31.950443 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" Oct 30 17:05:32.002342 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:31.991891 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" Oct 30 17:05:32.003940 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:32.002932 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" Oct 30 17:05:32.004532 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:32.004506 4522 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:05:32.004715 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:32.004690 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb" podUID="534aeebe-4edf-425e-a16c-712cc498154b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:32.036335 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:32.027397 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" podUID="0b832781-3d41-4496-aa86-e4d92792e590" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:32.036335 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:32.029446 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxdmv\" (UniqueName: \"kubernetes.io/projected/677fd30b-2b0d-42ff-8351-8d6e801adf5f-kube-api-access-vxdmv\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:05:32.241419 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:32.184634 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/distinct-w1-04248e-kserve-5859d49965-kgq46" podUID="8dda5ac4-30dd-432e-81eb-cb6d39800628" containerName="main" probeResult="failure" output="Get \"https://10.134.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:32.413677 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:32.413643 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" Oct 30 17:05:32.416449 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:32.416427 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b4f406/distinct-w2-b4f406-kserve-5db4cc6969-7n846" Oct 30 17:05:32.418009 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:32.417988 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" Oct 30 17:05:32.447328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:32.428222 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" podUID="c9ad71b7-ad90-4c8c-bf27-0f10b23f1593" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:32.447328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:32.428612 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" Oct 30 17:05:32.593425 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:32.592333 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" Oct 30 17:05:32.598461 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:32.598190 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" Oct 30 17:05:32.616602 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:32.616557 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04248e/distinct-w1-04248e-kserve-5859d49965-kgq46" Oct 30 17:05:32.632749 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:32.629499 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" Oct 30 17:05:35.285161 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:35.285135 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-952bc16e-5cdf6db644-hm4mt" Oct 30 17:05:35.285553 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:35.285205 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-952bc16e-6865b69dc5-zqc5b" Oct 30 17:05:36.758163 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:36.758130 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-04248e/distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq"] Oct 30 17:05:36.758819 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:36.758801 4522 cpu_manager.go:414] "containerMap: removing container" podUID="677fd30b-2b0d-42ff-8351-8d6e801adf5f" containerName="revoke-keys" Oct 30 17:05:36.758819 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:36.758818 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="677fd30b-2b0d-42ff-8351-8d6e801adf5f" containerName="revoke-keys" Oct 30 17:05:36.758973 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:36.758924 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="677fd30b-2b0d-42ff-8351-8d6e801adf5f" containerName="revoke-keys" Oct 30 17:05:36.762722 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:36.762704 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-04248e/distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq" Oct 30 17:05:36.783875 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:36.783836 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-04248e/distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq"] Oct 30 17:05:36.790720 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:36.790697 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-04248e\"/\"distinct-2-w1-04248e-kserve-self-signed-certs\"" Oct 30 17:05:36.940120 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:36.940074 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f18303fb-6d27-4ff1-9cc9-ed032e43d098-home\") pod \"distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq\" (UID: \"f18303fb-6d27-4ff1-9cc9-ed032e43d098\") " pod="e2e-models-e2e-worker-w1-04248e/distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq" Oct 30 17:05:36.940979 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:36.940907 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br48l\" (UniqueName: \"kubernetes.io/projected/f18303fb-6d27-4ff1-9cc9-ed032e43d098-kube-api-access-br48l\") pod \"distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq\" (UID: \"f18303fb-6d27-4ff1-9cc9-ed032e43d098\") " pod="e2e-models-e2e-worker-w1-04248e/distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq" Oct 30 17:05:36.941184 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:36.941163 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f18303fb-6d27-4ff1-9cc9-ed032e43d098-tls-certs\") pod \"distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq\" (UID: \"f18303fb-6d27-4ff1-9cc9-ed032e43d098\") " pod="e2e-models-e2e-worker-w1-04248e/distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq" Oct 30 17:05:36.941494 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:36.941472 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f18303fb-6d27-4ff1-9cc9-ed032e43d098-dshm\") pod \"distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq\" (UID: \"f18303fb-6d27-4ff1-9cc9-ed032e43d098\") " pod="e2e-models-e2e-worker-w1-04248e/distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq" Oct 30 17:05:36.941869 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:36.941849 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f18303fb-6d27-4ff1-9cc9-ed032e43d098-model-cache\") pod \"distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq\" (UID: \"f18303fb-6d27-4ff1-9cc9-ed032e43d098\") " pod="e2e-models-e2e-worker-w1-04248e/distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq" Oct 30 17:05:36.941963 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:36.941913 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f18303fb-6d27-4ff1-9cc9-ed032e43d098-tmp-dir\") pod \"distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq\" (UID: \"f18303fb-6d27-4ff1-9cc9-ed032e43d098\") " pod="e2e-models-e2e-worker-w1-04248e/distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq" Oct 30 17:05:37.043773 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:37.043684 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f18303fb-6d27-4ff1-9cc9-ed032e43d098-model-cache\") pod \"distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq\" (UID: \"f18303fb-6d27-4ff1-9cc9-ed032e43d098\") " pod="e2e-models-e2e-worker-w1-04248e/distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq" Oct 30 17:05:37.043773 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:37.043724 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f18303fb-6d27-4ff1-9cc9-ed032e43d098-tmp-dir\") pod \"distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq\" (UID: \"f18303fb-6d27-4ff1-9cc9-ed032e43d098\") " pod="e2e-models-e2e-worker-w1-04248e/distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq" Oct 30 17:05:37.043773 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:37.043742 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f18303fb-6d27-4ff1-9cc9-ed032e43d098-home\") pod \"distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq\" (UID: \"f18303fb-6d27-4ff1-9cc9-ed032e43d098\") " pod="e2e-models-e2e-worker-w1-04248e/distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq" Oct 30 17:05:37.044041 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:37.043824 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br48l\" (UniqueName: \"kubernetes.io/projected/f18303fb-6d27-4ff1-9cc9-ed032e43d098-kube-api-access-br48l\") pod \"distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq\" (UID: \"f18303fb-6d27-4ff1-9cc9-ed032e43d098\") " pod="e2e-models-e2e-worker-w1-04248e/distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq" Oct 30 17:05:37.044041 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:37.043863 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f18303fb-6d27-4ff1-9cc9-ed032e43d098-tls-certs\") pod \"distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq\" (UID: \"f18303fb-6d27-4ff1-9cc9-ed032e43d098\") " pod="e2e-models-e2e-worker-w1-04248e/distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq" Oct 30 17:05:37.044041 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:37.043899 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f18303fb-6d27-4ff1-9cc9-ed032e43d098-dshm\") pod \"distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq\" (UID: \"f18303fb-6d27-4ff1-9cc9-ed032e43d098\") " pod="e2e-models-e2e-worker-w1-04248e/distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq" Oct 30 17:05:37.044169 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:37.044148 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f18303fb-6d27-4ff1-9cc9-ed032e43d098-model-cache\") pod \"distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq\" (UID: \"f18303fb-6d27-4ff1-9cc9-ed032e43d098\") " pod="e2e-models-e2e-worker-w1-04248e/distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq" Oct 30 17:05:37.044231 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:37.044202 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f18303fb-6d27-4ff1-9cc9-ed032e43d098-home\") pod \"distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq\" (UID: \"f18303fb-6d27-4ff1-9cc9-ed032e43d098\") " pod="e2e-models-e2e-worker-w1-04248e/distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq" Oct 30 17:05:37.044231 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:37.044223 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f18303fb-6d27-4ff1-9cc9-ed032e43d098-tmp-dir\") pod \"distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq\" (UID: \"f18303fb-6d27-4ff1-9cc9-ed032e43d098\") " pod="e2e-models-e2e-worker-w1-04248e/distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq" Oct 30 17:05:37.046181 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:37.046156 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f18303fb-6d27-4ff1-9cc9-ed032e43d098-dshm\") pod \"distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq\" (UID: \"f18303fb-6d27-4ff1-9cc9-ed032e43d098\") " pod="e2e-models-e2e-worker-w1-04248e/distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq" Oct 30 17:05:37.046367 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:37.046348 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f18303fb-6d27-4ff1-9cc9-ed032e43d098-tls-certs\") pod \"distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq\" (UID: \"f18303fb-6d27-4ff1-9cc9-ed032e43d098\") " pod="e2e-models-e2e-worker-w1-04248e/distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq" Oct 30 17:05:37.069927 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:37.069895 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-br48l\" (UniqueName: \"kubernetes.io/projected/f18303fb-6d27-4ff1-9cc9-ed032e43d098-kube-api-access-br48l\") pod \"distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq\" (UID: \"f18303fb-6d27-4ff1-9cc9-ed032e43d098\") " pod="e2e-models-e2e-worker-w1-04248e/distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq" Oct 30 17:05:37.076172 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:37.076142 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-04248e/distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq" Oct 30 17:05:37.393294 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:05:37.393264 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf18303fb_6d27_4ff1_9cc9_ed032e43d098.slice/crio-e000f0143fe2a1164f158a948e00aadbcb465d982540c55cebdeda53b6717584 WatchSource:0}: Error finding container e000f0143fe2a1164f158a948e00aadbcb465d982540c55cebdeda53b6717584: Status 404 returned error can't find the container with id e000f0143fe2a1164f158a948e00aadbcb465d982540c55cebdeda53b6717584 Oct 30 17:05:37.396185 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:37.396162 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-04248e/distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq"] Oct 30 17:05:37.648179 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:37.648082 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-04248e/distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq" event={"ID":"f18303fb-6d27-4ff1-9cc9-ed032e43d098","Type":"ContainerStarted","Data":"fe4e8732b1fc8a2bc0dddc27106aa9615f6269fa006fec9fcea562011fa00f1c"} Oct 30 17:05:37.648179 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:37.648129 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-04248e/distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq" event={"ID":"f18303fb-6d27-4ff1-9cc9-ed032e43d098","Type":"ContainerStarted","Data":"e000f0143fe2a1164f158a948e00aadbcb465d982540c55cebdeda53b6717584"} Oct 30 17:05:37.695784 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:37.695734 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-04248e/distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq" podStartSLOduration=1.6957189339999998 podStartE2EDuration="1.695718934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:05:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:05:37.693973564 +0000 UTC m=+1111.694728466" watchObservedRunningTime="2026-10-30 17:05:37.695718934 +0000 UTC m=+1111.696473848" Oct 30 17:05:37.760563 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:37.760528 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc"] Oct 30 17:05:37.764136 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:37.764113 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc" Oct 30 17:05:37.795367 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:37.795335 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc"] Oct 30 17:05:37.856710 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:37.856677 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c99fb3fd-1362-414e-82bd-88f20c055166-workload-socket\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc\" (UID: \"c99fb3fd-1362-414e-82bd-88f20c055166\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc" Oct 30 17:05:37.856914 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:37.856728 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c99fb3fd-1362-414e-82bd-88f20c055166-istiod-ca-cert\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc\" (UID: \"c99fb3fd-1362-414e-82bd-88f20c055166\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc" Oct 30 17:05:37.856914 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:37.856834 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c99fb3fd-1362-414e-82bd-88f20c055166-istio-token\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc\" (UID: \"c99fb3fd-1362-414e-82bd-88f20c055166\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc" Oct 30 17:05:37.856998 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:37.856987 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx2gn\" (UniqueName: \"kubernetes.io/projected/c99fb3fd-1362-414e-82bd-88f20c055166-kube-api-access-qx2gn\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc\" (UID: \"c99fb3fd-1362-414e-82bd-88f20c055166\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc" Oct 30 17:05:37.857060 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:37.857047 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c99fb3fd-1362-414e-82bd-88f20c055166-istio-podinfo\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc\" (UID: \"c99fb3fd-1362-414e-82bd-88f20c055166\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc" Oct 30 17:05:37.857177 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:37.857158 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c99fb3fd-1362-414e-82bd-88f20c055166-istio-envoy\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc\" (UID: \"c99fb3fd-1362-414e-82bd-88f20c055166\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc" Oct 30 17:05:37.857245 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:37.857227 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c99fb3fd-1362-414e-82bd-88f20c055166-istio-data\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc\" (UID: \"c99fb3fd-1362-414e-82bd-88f20c055166\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc" Oct 30 17:05:37.857365 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:37.857340 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c99fb3fd-1362-414e-82bd-88f20c055166-workload-certs\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc\" (UID: \"c99fb3fd-1362-414e-82bd-88f20c055166\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc" Oct 30 17:05:37.857516 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:37.857394 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c99fb3fd-1362-414e-82bd-88f20c055166-credential-socket\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc\" (UID: \"c99fb3fd-1362-414e-82bd-88f20c055166\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc" Oct 30 17:05:37.959278 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:37.959225 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c99fb3fd-1362-414e-82bd-88f20c055166-workload-certs\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc\" (UID: \"c99fb3fd-1362-414e-82bd-88f20c055166\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc" Oct 30 17:05:37.959521 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:37.959288 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c99fb3fd-1362-414e-82bd-88f20c055166-credential-socket\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc\" (UID: \"c99fb3fd-1362-414e-82bd-88f20c055166\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc" Oct 30 17:05:37.959521 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:37.959410 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c99fb3fd-1362-414e-82bd-88f20c055166-workload-socket\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc\" (UID: \"c99fb3fd-1362-414e-82bd-88f20c055166\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc" Oct 30 17:05:37.959521 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:37.959458 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c99fb3fd-1362-414e-82bd-88f20c055166-istiod-ca-cert\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc\" (UID: \"c99fb3fd-1362-414e-82bd-88f20c055166\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc" Oct 30 17:05:37.959521 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:37.959489 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c99fb3fd-1362-414e-82bd-88f20c055166-istio-token\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc\" (UID: \"c99fb3fd-1362-414e-82bd-88f20c055166\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc" Oct 30 17:05:37.959741 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:37.959630 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx2gn\" (UniqueName: \"kubernetes.io/projected/c99fb3fd-1362-414e-82bd-88f20c055166-kube-api-access-qx2gn\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc\" (UID: \"c99fb3fd-1362-414e-82bd-88f20c055166\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc" Oct 30 17:05:37.959741 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:37.959726 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c99fb3fd-1362-414e-82bd-88f20c055166-istio-podinfo\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc\" (UID: \"c99fb3fd-1362-414e-82bd-88f20c055166\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc" Oct 30 17:05:37.959866 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:37.959752 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c99fb3fd-1362-414e-82bd-88f20c055166-workload-certs\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc\" (UID: \"c99fb3fd-1362-414e-82bd-88f20c055166\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc" Oct 30 17:05:37.959866 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:37.959767 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c99fb3fd-1362-414e-82bd-88f20c055166-istio-envoy\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc\" (UID: \"c99fb3fd-1362-414e-82bd-88f20c055166\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc" Oct 30 17:05:37.959866 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:37.959783 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c99fb3fd-1362-414e-82bd-88f20c055166-workload-socket\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc\" (UID: \"c99fb3fd-1362-414e-82bd-88f20c055166\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc" Oct 30 17:05:37.959866 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:37.959823 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c99fb3fd-1362-414e-82bd-88f20c055166-credential-socket\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc\" (UID: \"c99fb3fd-1362-414e-82bd-88f20c055166\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc" Oct 30 17:05:37.959866 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:37.959850 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c99fb3fd-1362-414e-82bd-88f20c055166-istio-data\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc\" (UID: \"c99fb3fd-1362-414e-82bd-88f20c055166\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc" Oct 30 17:05:37.960115 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:37.960089 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c99fb3fd-1362-414e-82bd-88f20c055166-istio-data\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc\" (UID: \"c99fb3fd-1362-414e-82bd-88f20c055166\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc" Oct 30 17:05:37.960242 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:37.960215 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c99fb3fd-1362-414e-82bd-88f20c055166-istiod-ca-cert\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc\" (UID: \"c99fb3fd-1362-414e-82bd-88f20c055166\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc" Oct 30 17:05:37.962125 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:37.962097 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c99fb3fd-1362-414e-82bd-88f20c055166-istio-envoy\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc\" (UID: \"c99fb3fd-1362-414e-82bd-88f20c055166\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc" Oct 30 17:05:37.962354 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:37.962336 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c99fb3fd-1362-414e-82bd-88f20c055166-istio-podinfo\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc\" (UID: \"c99fb3fd-1362-414e-82bd-88f20c055166\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc" Oct 30 17:05:37.995110 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:37.995078 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx2gn\" (UniqueName: \"kubernetes.io/projected/c99fb3fd-1362-414e-82bd-88f20c055166-kube-api-access-qx2gn\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc\" (UID: \"c99fb3fd-1362-414e-82bd-88f20c055166\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc" Oct 30 17:05:37.998700 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:37.998673 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c99fb3fd-1362-414e-82bd-88f20c055166-istio-token\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc\" (UID: \"c99fb3fd-1362-414e-82bd-88f20c055166\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc" Oct 30 17:05:38.076917 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:38.076886 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04248e/distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq" Oct 30 17:05:38.079255 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:38.079228 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc" Oct 30 17:05:38.407088 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:05:38.407052 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc99fb3fd_1362_414e_82bd_88f20c055166.slice/crio-3fdf04d56684b580224bfe104161a2d90a2ba160d9c46bcb788ac4ecfdb74961 WatchSource:0}: Error finding container 3fdf04d56684b580224bfe104161a2d90a2ba160d9c46bcb788ac4ecfdb74961: Status 404 returned error can't find the container with id 3fdf04d56684b580224bfe104161a2d90a2ba160d9c46bcb788ac4ecfdb74961 Oct 30 17:05:38.410115 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:38.410075 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:05:38.410219 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:38.410166 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:05:38.410219 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:38.410208 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:05:38.411440 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:38.411419 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc"] Oct 30 17:05:38.660105 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:38.660003 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc" event={"ID":"c99fb3fd-1362-414e-82bd-88f20c055166","Type":"ContainerStarted","Data":"a72e5b4dcef303d00a23928b773b6cce77fedcebe220e88403df980275c177bd"} Oct 30 17:05:38.660105 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:38.660068 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc" event={"ID":"c99fb3fd-1362-414e-82bd-88f20c055166","Type":"ContainerStarted","Data":"3fdf04d56684b580224bfe104161a2d90a2ba160d9c46bcb788ac4ecfdb74961"} Oct 30 17:05:38.710323 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:38.710263 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc" podStartSLOduration=1.710249626 podStartE2EDuration="1.710249626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:05:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:05:38.708666888 +0000 UTC m=+1112.709421789" watchObservedRunningTime="2026-10-30 17:05:38.710249626 +0000 UTC m=+1112.711004527" Oct 30 17:05:39.087271 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:39.087235 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc" Oct 30 17:05:39.087995 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:39.087298 4522 patch_prober.go:29] interesting pod/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 30 17:05:39.087995 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:39.087393 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc" podUID="c99fb3fd-1362-414e-82bd-88f20c055166" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 30 17:05:40.080199 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:40.080165 4522 patch_prober.go:29] interesting pod/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 30 17:05:40.080417 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:40.080228 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc" podUID="c99fb3fd-1362-414e-82bd-88f20c055166" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 30 17:05:41.081588 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:41.081368 4522 patch_prober.go:29] interesting pod/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 30 17:05:41.081588 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:41.081448 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc" podUID="c99fb3fd-1362-414e-82bd-88f20c055166" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 30 17:05:42.080223 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:42.080184 4522 patch_prober.go:29] interesting pod/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 30 17:05:42.080431 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:42.080251 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc" podUID="c99fb3fd-1362-414e-82bd-88f20c055166" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 30 17:05:42.733467 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:42.733430 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh"] Oct 30 17:05:42.739947 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:42.739923 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh" Oct 30 17:05:42.763706 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:42.761659 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ce5ef44-9364-43ae-9c43-b46885be7b4f-workload-certs\") pod \"e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh\" (UID: \"8ce5ef44-9364-43ae-9c43-b46885be7b4f\") " pod="openshift-ingress/e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh" Oct 30 17:05:42.763706 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:42.761719 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce5ef44-9364-43ae-9c43-b46885be7b4f-workload-socket\") pod \"e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh\" (UID: \"8ce5ef44-9364-43ae-9c43-b46885be7b4f\") " pod="openshift-ingress/e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh" Oct 30 17:05:42.763706 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:42.762042 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ce5ef44-9364-43ae-9c43-b46885be7b4f-istiod-ca-cert\") pod \"e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh\" (UID: \"8ce5ef44-9364-43ae-9c43-b46885be7b4f\") " pod="openshift-ingress/e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh" Oct 30 17:05:42.763706 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:42.762083 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce5ef44-9364-43ae-9c43-b46885be7b4f-credential-socket\") pod \"e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh\" (UID: \"8ce5ef44-9364-43ae-9c43-b46885be7b4f\") " pod="openshift-ingress/e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh" Oct 30 17:05:42.763706 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:42.762109 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6km9\" (UniqueName: \"kubernetes.io/projected/8ce5ef44-9364-43ae-9c43-b46885be7b4f-kube-api-access-f6km9\") pod \"e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh\" (UID: \"8ce5ef44-9364-43ae-9c43-b46885be7b4f\") " pod="openshift-ingress/e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh" Oct 30 17:05:42.763706 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:42.762146 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ce5ef44-9364-43ae-9c43-b46885be7b4f-istio-token\") pod \"e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh\" (UID: \"8ce5ef44-9364-43ae-9c43-b46885be7b4f\") " pod="openshift-ingress/e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh" Oct 30 17:05:42.763706 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:42.762354 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ce5ef44-9364-43ae-9c43-b46885be7b4f-istio-podinfo\") pod \"e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh\" (UID: \"8ce5ef44-9364-43ae-9c43-b46885be7b4f\") " pod="openshift-ingress/e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh" Oct 30 17:05:42.763706 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:42.762429 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ce5ef44-9364-43ae-9c43-b46885be7b4f-istio-data\") pod \"e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh\" (UID: \"8ce5ef44-9364-43ae-9c43-b46885be7b4f\") " pod="openshift-ingress/e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh" Oct 30 17:05:42.763706 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:42.762555 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ce5ef44-9364-43ae-9c43-b46885be7b4f-istio-envoy\") pod \"e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh\" (UID: \"8ce5ef44-9364-43ae-9c43-b46885be7b4f\") " pod="openshift-ingress/e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh" Oct 30 17:05:42.765490 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:42.765466 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh"] Oct 30 17:05:42.771030 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:42.771000 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-1cdec0e0-gw-openshift-default-dockercfg-b2c6g\"" Oct 30 17:05:42.865353 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:42.865273 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ce5ef44-9364-43ae-9c43-b46885be7b4f-istio-podinfo\") pod \"e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh\" (UID: \"8ce5ef44-9364-43ae-9c43-b46885be7b4f\") " pod="openshift-ingress/e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh" Oct 30 17:05:42.866260 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:42.866236 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ce5ef44-9364-43ae-9c43-b46885be7b4f-istio-data\") pod \"e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh\" (UID: \"8ce5ef44-9364-43ae-9c43-b46885be7b4f\") " pod="openshift-ingress/e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh" Oct 30 17:05:42.866941 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:42.866920 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ce5ef44-9364-43ae-9c43-b46885be7b4f-istio-envoy\") pod \"e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh\" (UID: \"8ce5ef44-9364-43ae-9c43-b46885be7b4f\") " pod="openshift-ingress/e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh" Oct 30 17:05:42.867505 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:42.867486 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ce5ef44-9364-43ae-9c43-b46885be7b4f-workload-certs\") pod \"e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh\" (UID: \"8ce5ef44-9364-43ae-9c43-b46885be7b4f\") " pod="openshift-ingress/e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh" Oct 30 17:05:42.868097 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:42.868080 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce5ef44-9364-43ae-9c43-b46885be7b4f-workload-socket\") pod \"e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh\" (UID: \"8ce5ef44-9364-43ae-9c43-b46885be7b4f\") " pod="openshift-ingress/e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh" Oct 30 17:05:42.868584 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:42.868567 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ce5ef44-9364-43ae-9c43-b46885be7b4f-istiod-ca-cert\") pod \"e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh\" (UID: \"8ce5ef44-9364-43ae-9c43-b46885be7b4f\") " pod="openshift-ingress/e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh" Oct 30 17:05:42.870656 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:42.869437 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce5ef44-9364-43ae-9c43-b46885be7b4f-credential-socket\") pod \"e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh\" (UID: \"8ce5ef44-9364-43ae-9c43-b46885be7b4f\") " pod="openshift-ingress/e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh" Oct 30 17:05:42.870825 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:42.870807 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6km9\" (UniqueName: \"kubernetes.io/projected/8ce5ef44-9364-43ae-9c43-b46885be7b4f-kube-api-access-f6km9\") pod \"e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh\" (UID: \"8ce5ef44-9364-43ae-9c43-b46885be7b4f\") " pod="openshift-ingress/e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh" Oct 30 17:05:42.871247 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:42.871229 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ce5ef44-9364-43ae-9c43-b46885be7b4f-istio-token\") pod \"e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh\" (UID: \"8ce5ef44-9364-43ae-9c43-b46885be7b4f\") " pod="openshift-ingress/e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh" Oct 30 17:05:42.871717 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:42.866768 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ce5ef44-9364-43ae-9c43-b46885be7b4f-istio-data\") pod \"e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh\" (UID: \"8ce5ef44-9364-43ae-9c43-b46885be7b4f\") " pod="openshift-ingress/e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh" Oct 30 17:05:42.871846 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:42.868496 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce5ef44-9364-43ae-9c43-b46885be7b4f-workload-socket\") pod \"e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh\" (UID: \"8ce5ef44-9364-43ae-9c43-b46885be7b4f\") " pod="openshift-ingress/e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh" Oct 30 17:05:42.872173 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:42.870042 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce5ef44-9364-43ae-9c43-b46885be7b4f-credential-socket\") pod \"e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh\" (UID: \"8ce5ef44-9364-43ae-9c43-b46885be7b4f\") " pod="openshift-ingress/e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh" Oct 30 17:05:42.872292 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:42.870236 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ce5ef44-9364-43ae-9c43-b46885be7b4f-istio-envoy\") pod \"e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh\" (UID: \"8ce5ef44-9364-43ae-9c43-b46885be7b4f\") " pod="openshift-ingress/e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh" Oct 30 17:05:42.872454 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:42.867951 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ce5ef44-9364-43ae-9c43-b46885be7b4f-workload-certs\") pod \"e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh\" (UID: \"8ce5ef44-9364-43ae-9c43-b46885be7b4f\") " pod="openshift-ingress/e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh" Oct 30 17:05:42.872576 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:42.869366 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ce5ef44-9364-43ae-9c43-b46885be7b4f-istiod-ca-cert\") pod \"e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh\" (UID: \"8ce5ef44-9364-43ae-9c43-b46885be7b4f\") " pod="openshift-ingress/e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh" Oct 30 17:05:42.884231 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:42.884158 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ce5ef44-9364-43ae-9c43-b46885be7b4f-istio-podinfo\") pod \"e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh\" (UID: \"8ce5ef44-9364-43ae-9c43-b46885be7b4f\") " pod="openshift-ingress/e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh" Oct 30 17:05:42.934488 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:42.928062 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6km9\" (UniqueName: \"kubernetes.io/projected/8ce5ef44-9364-43ae-9c43-b46885be7b4f-kube-api-access-f6km9\") pod \"e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh\" (UID: \"8ce5ef44-9364-43ae-9c43-b46885be7b4f\") " pod="openshift-ingress/e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh" Oct 30 17:05:42.972328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:42.966928 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ce5ef44-9364-43ae-9c43-b46885be7b4f-istio-token\") pod \"e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh\" (UID: \"8ce5ef44-9364-43ae-9c43-b46885be7b4f\") " pod="openshift-ingress/e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh" Oct 30 17:05:43.048783 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:05:43.048687 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:05:43.055179 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:43.055146 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh" Oct 30 17:05:43.082986 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:43.082955 4522 patch_prober.go:29] interesting pod/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 30 17:05:43.083241 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:43.083210 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc" podUID="c99fb3fd-1362-414e-82bd-88f20c055166" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 30 17:05:43.443971 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:43.443928 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:05:43.444139 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:43.444036 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:05:43.444139 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:43.444084 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:05:43.450717 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:43.449058 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh"] Oct 30 17:05:43.738932 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:43.738897 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh" event={"ID":"8ce5ef44-9364-43ae-9c43-b46885be7b4f","Type":"ContainerStarted","Data":"6acc416bcf0088a8c8d2927f7a62519bf2217ea0fb732e585cd27dae98881a91"} Oct 30 17:05:43.739435 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:43.738949 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh" event={"ID":"8ce5ef44-9364-43ae-9c43-b46885be7b4f","Type":"ContainerStarted","Data":"4ecc31ea3691bef3d4162d62997cf45057ae35e4a84c77979423c4b5e0c85baa"} Oct 30 17:05:43.795333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:43.791689 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh" podStartSLOduration=1.7916689730000002 podStartE2EDuration="1.791668973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:05:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:05:43.787821073 +0000 UTC m=+1117.788576000" watchObservedRunningTime="2026-10-30 17:05:43.791668973 +0000 UTC m=+1117.792423875" Oct 30 17:05:44.056363 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:44.056228 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh" Oct 30 17:05:44.058620 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:44.058592 4522 patch_prober.go:29] interesting pod/e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 30 17:05:44.058745 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:44.058640 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh" podUID="8ce5ef44-9364-43ae-9c43-b46885be7b4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 30 17:05:44.080124 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:44.080092 4522 patch_prober.go:29] interesting pod/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 30 17:05:44.080288 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:44.080139 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc" podUID="c99fb3fd-1362-414e-82bd-88f20c055166" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 30 17:05:45.060495 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:45.060461 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh" Oct 30 17:05:45.294750 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:45.294720 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc" Oct 30 17:05:45.295122 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:45.295099 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc" Oct 30 17:05:45.295989 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:45.295967 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc" Oct 30 17:05:45.759761 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:45.759729 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh" Oct 30 17:05:45.760733 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:45.760712 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh" Oct 30 17:05:47.076640 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:47.076600 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-04248e/distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq" Oct 30 17:05:47.089770 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:47.089744 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-04248e/distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq" Oct 30 17:05:47.486675 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:47.486643 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-1cdec0e0-798d84688c-mtx4b"] Oct 30 17:05:47.490789 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:47.490773 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1cdec0e0-798d84688c-mtx4b" Oct 30 17:05:47.507384 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:47.507352 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-1cdec0e0\"" Oct 30 17:05:47.507538 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:47.507475 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-1cdec0e0-dockercfg-q9sxb\"" Oct 30 17:05:47.514403 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:47.514371 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-1cdec0e0-798d84688c-mtx4b"] Oct 30 17:05:47.560832 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:47.560794 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1cdec0e0-fb59dd5bb-g5xq4"] Oct 30 17:05:47.566383 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:47.566356 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-1cdec0e0-fb59dd5bb-g5xq4" Oct 30 17:05:47.586554 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:47.586022 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/780734db-39cc-4c52-bd39-efefc4ad7e72-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-1cdec0e0-798d84688c-mtx4b\" (UID: \"780734db-39cc-4c52-bd39-efefc4ad7e72\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-1cdec0e0-798d84688c-mtx4b" Oct 30 17:05:47.586554 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:47.586052 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjchd\" (UniqueName: \"kubernetes.io/projected/780734db-39cc-4c52-bd39-efefc4ad7e72-kube-api-access-bjchd\") pod \"payload-pre-processing-e2e-migrate-1cdec0e0-798d84688c-mtx4b\" (UID: \"780734db-39cc-4c52-bd39-efefc4ad7e72\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-1cdec0e0-798d84688c-mtx4b" Oct 30 17:05:47.687175 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:47.684382 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1cdec0e0-fb59dd5bb-g5xq4"] Oct 30 17:05:47.766884 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:47.766791 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/780734db-39cc-4c52-bd39-efefc4ad7e72-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-1cdec0e0-798d84688c-mtx4b\" (UID: \"780734db-39cc-4c52-bd39-efefc4ad7e72\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-1cdec0e0-798d84688c-mtx4b" Oct 30 17:05:47.766884 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:47.766824 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjchd\" (UniqueName: \"kubernetes.io/projected/780734db-39cc-4c52-bd39-efefc4ad7e72-kube-api-access-bjchd\") pod \"payload-pre-processing-e2e-migrate-1cdec0e0-798d84688c-mtx4b\" (UID: \"780734db-39cc-4c52-bd39-efefc4ad7e72\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-1cdec0e0-798d84688c-mtx4b" Oct 30 17:05:47.766884 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:47.766854 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkj98\" (UniqueName: \"kubernetes.io/projected/cf5eb021-2e73-46db-a0db-ab5dd73076e1-kube-api-access-fkj98\") pod \"payload-processing-e2e-migrate-1cdec0e0-fb59dd5bb-g5xq4\" (UID: \"cf5eb021-2e73-46db-a0db-ab5dd73076e1\") " pod="openshift-ingress/payload-processing-e2e-migrate-1cdec0e0-fb59dd5bb-g5xq4" Oct 30 17:05:47.767219 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:47.766949 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf5eb021-2e73-46db-a0db-ab5dd73076e1-plugins-config-volume\") pod \"payload-processing-e2e-migrate-1cdec0e0-fb59dd5bb-g5xq4\" (UID: \"cf5eb021-2e73-46db-a0db-ab5dd73076e1\") " pod="openshift-ingress/payload-processing-e2e-migrate-1cdec0e0-fb59dd5bb-g5xq4" Oct 30 17:05:47.769107 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:47.769081 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/780734db-39cc-4c52-bd39-efefc4ad7e72-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-1cdec0e0-798d84688c-mtx4b\" (UID: \"780734db-39cc-4c52-bd39-efefc4ad7e72\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-1cdec0e0-798d84688c-mtx4b" Oct 30 17:05:47.792089 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:47.792056 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjchd\" (UniqueName: \"kubernetes.io/projected/780734db-39cc-4c52-bd39-efefc4ad7e72-kube-api-access-bjchd\") pod \"payload-pre-processing-e2e-migrate-1cdec0e0-798d84688c-mtx4b\" (UID: \"780734db-39cc-4c52-bd39-efefc4ad7e72\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-1cdec0e0-798d84688c-mtx4b" Oct 30 17:05:47.792385 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:47.792366 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04248e/distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq" Oct 30 17:05:47.805839 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:47.805815 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1cdec0e0-798d84688c-mtx4b" Oct 30 17:05:47.870455 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:47.869544 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkj98\" (UniqueName: \"kubernetes.io/projected/cf5eb021-2e73-46db-a0db-ab5dd73076e1-kube-api-access-fkj98\") pod \"payload-processing-e2e-migrate-1cdec0e0-fb59dd5bb-g5xq4\" (UID: \"cf5eb021-2e73-46db-a0db-ab5dd73076e1\") " pod="openshift-ingress/payload-processing-e2e-migrate-1cdec0e0-fb59dd5bb-g5xq4" Oct 30 17:05:47.870455 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:47.869749 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf5eb021-2e73-46db-a0db-ab5dd73076e1-plugins-config-volume\") pod \"payload-processing-e2e-migrate-1cdec0e0-fb59dd5bb-g5xq4\" (UID: \"cf5eb021-2e73-46db-a0db-ab5dd73076e1\") " pod="openshift-ingress/payload-processing-e2e-migrate-1cdec0e0-fb59dd5bb-g5xq4" Oct 30 17:05:47.871331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:47.870800 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf5eb021-2e73-46db-a0db-ab5dd73076e1-plugins-config-volume\") pod \"payload-processing-e2e-migrate-1cdec0e0-fb59dd5bb-g5xq4\" (UID: \"cf5eb021-2e73-46db-a0db-ab5dd73076e1\") " pod="openshift-ingress/payload-processing-e2e-migrate-1cdec0e0-fb59dd5bb-g5xq4" Oct 30 17:05:47.901352 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:47.897355 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkj98\" (UniqueName: \"kubernetes.io/projected/cf5eb021-2e73-46db-a0db-ab5dd73076e1-kube-api-access-fkj98\") pod \"payload-processing-e2e-migrate-1cdec0e0-fb59dd5bb-g5xq4\" (UID: \"cf5eb021-2e73-46db-a0db-ab5dd73076e1\") " pod="openshift-ingress/payload-processing-e2e-migrate-1cdec0e0-fb59dd5bb-g5xq4" Oct 30 17:05:48.001350 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:47.997755 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-1cdec0e0-fb59dd5bb-g5xq4" Oct 30 17:05:48.134026 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:05:48.133980 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod780734db_39cc_4c52_bd39_efefc4ad7e72.slice/crio-8e0c46328abf4597a413c1f6584d88a2f2f8637e9ed6096608d829d116c551fe WatchSource:0}: Error finding container 8e0c46328abf4597a413c1f6584d88a2f2f8637e9ed6096608d829d116c551fe: Status 404 returned error can't find the container with id 8e0c46328abf4597a413c1f6584d88a2f2f8637e9ed6096608d829d116c551fe Oct 30 17:05:48.191580 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:48.191548 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-1cdec0e0-798d84688c-mtx4b"] Oct 30 17:05:48.341155 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:05:48.341118 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf5eb021_2e73_46db_a0db_ab5dd73076e1.slice/crio-0fe0a94ca0033a7438978dc533d9943daa8bb983a6780d5913a963fd810485ea WatchSource:0}: Error finding container 0fe0a94ca0033a7438978dc533d9943daa8bb983a6780d5913a963fd810485ea: Status 404 returned error can't find the container with id 0fe0a94ca0033a7438978dc533d9943daa8bb983a6780d5913a963fd810485ea Oct 30 17:05:48.345556 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:48.345533 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1cdec0e0-fb59dd5bb-g5xq4"] Oct 30 17:05:48.791479 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:48.791442 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1cdec0e0-798d84688c-mtx4b" event={"ID":"780734db-39cc-4c52-bd39-efefc4ad7e72","Type":"ContainerStarted","Data":"c6fcbb8259641d6b40e6faceea46cd958b839e7cde4a18b01c7897d9feacbe38"} Oct 30 17:05:48.791681 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:48.791522 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1cdec0e0-798d84688c-mtx4b" event={"ID":"780734db-39cc-4c52-bd39-efefc4ad7e72","Type":"ContainerStarted","Data":"8e0c46328abf4597a413c1f6584d88a2f2f8637e9ed6096608d829d116c551fe"} Oct 30 17:05:48.791681 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:48.791567 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1cdec0e0-798d84688c-mtx4b" Oct 30 17:05:48.797156 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:48.797127 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-1cdec0e0-fb59dd5bb-g5xq4" event={"ID":"cf5eb021-2e73-46db-a0db-ab5dd73076e1","Type":"ContainerStarted","Data":"57ea9f6405603940c16ca6f49d47e4e63f631c3520db909248211538b5218d61"} Oct 30 17:05:48.797289 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:48.797178 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-1cdec0e0-fb59dd5bb-g5xq4" event={"ID":"cf5eb021-2e73-46db-a0db-ab5dd73076e1","Type":"ContainerStarted","Data":"0fe0a94ca0033a7438978dc533d9943daa8bb983a6780d5913a963fd810485ea"} Oct 30 17:05:48.797289 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:48.797207 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-1cdec0e0-fb59dd5bb-g5xq4" Oct 30 17:05:48.835983 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:48.835934 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1cdec0e0-798d84688c-mtx4b" podStartSLOduration=1.835922562 podStartE2EDuration="1.835922562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:05:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:05:48.833783818 +0000 UTC m=+1122.834538732" watchObservedRunningTime="2026-10-30 17:05:48.835922562 +0000 UTC m=+1122.836677463" Oct 30 17:05:48.879238 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:48.879186 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-1cdec0e0-fb59dd5bb-g5xq4" podStartSLOduration=1.8791727809999998 podStartE2EDuration="1.879172781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:05:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:05:48.875431494 +0000 UTC m=+1122.876186416" watchObservedRunningTime="2026-10-30 17:05:48.879172781 +0000 UTC m=+1122.879927876" Oct 30 17:05:53.048902 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:53.048868 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d03a028-7cmmw"] Oct 30 17:05:53.061953 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:53.061919 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d03a028-7cmmw"] Oct 30 17:05:53.457391 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:53.457360 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6"] Oct 30 17:05:53.462606 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:53.462585 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6" Oct 30 17:05:53.487567 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:53.487536 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6"] Oct 30 17:05:53.575711 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:53.575674 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9c903cd-d636-4311-9afe-e51e4cc3db8e-credential-socket\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6\" (UID: \"c9c903cd-d636-4311-9afe-e51e4cc3db8e\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6" Oct 30 17:05:53.575711 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:53.575710 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9c903cd-d636-4311-9afe-e51e4cc3db8e-istio-envoy\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6\" (UID: \"c9c903cd-d636-4311-9afe-e51e4cc3db8e\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6" Oct 30 17:05:53.575920 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:53.575793 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9c903cd-d636-4311-9afe-e51e4cc3db8e-istiod-ca-cert\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6\" (UID: \"c9c903cd-d636-4311-9afe-e51e4cc3db8e\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6" Oct 30 17:05:53.575920 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:53.575827 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9c903cd-d636-4311-9afe-e51e4cc3db8e-istio-data\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6\" (UID: \"c9c903cd-d636-4311-9afe-e51e4cc3db8e\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6" Oct 30 17:05:53.575989 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:53.575949 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9c903cd-d636-4311-9afe-e51e4cc3db8e-workload-certs\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6\" (UID: \"c9c903cd-d636-4311-9afe-e51e4cc3db8e\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6" Oct 30 17:05:53.576021 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:53.575995 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9c903cd-d636-4311-9afe-e51e4cc3db8e-istio-token\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6\" (UID: \"c9c903cd-d636-4311-9afe-e51e4cc3db8e\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6" Oct 30 17:05:53.576051 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:53.576021 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9c903cd-d636-4311-9afe-e51e4cc3db8e-istio-podinfo\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6\" (UID: \"c9c903cd-d636-4311-9afe-e51e4cc3db8e\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6" Oct 30 17:05:53.576139 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:53.576128 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9c903cd-d636-4311-9afe-e51e4cc3db8e-workload-socket\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6\" (UID: \"c9c903cd-d636-4311-9afe-e51e4cc3db8e\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6" Oct 30 17:05:53.576190 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:53.576166 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd42w\" (UniqueName: \"kubernetes.io/projected/c9c903cd-d636-4311-9afe-e51e4cc3db8e-kube-api-access-zd42w\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6\" (UID: \"c9c903cd-d636-4311-9afe-e51e4cc3db8e\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6" Oct 30 17:05:53.677855 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:53.677817 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9c903cd-d636-4311-9afe-e51e4cc3db8e-workload-certs\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6\" (UID: \"c9c903cd-d636-4311-9afe-e51e4cc3db8e\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6" Oct 30 17:05:53.678055 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:53.677887 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9c903cd-d636-4311-9afe-e51e4cc3db8e-istio-token\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6\" (UID: \"c9c903cd-d636-4311-9afe-e51e4cc3db8e\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6" Oct 30 17:05:53.678055 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:53.677918 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9c903cd-d636-4311-9afe-e51e4cc3db8e-istio-podinfo\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6\" (UID: \"c9c903cd-d636-4311-9afe-e51e4cc3db8e\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6" Oct 30 17:05:53.678055 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:53.677993 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9c903cd-d636-4311-9afe-e51e4cc3db8e-workload-socket\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6\" (UID: \"c9c903cd-d636-4311-9afe-e51e4cc3db8e\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6" Oct 30 17:05:53.678055 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:53.678049 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd42w\" (UniqueName: \"kubernetes.io/projected/c9c903cd-d636-4311-9afe-e51e4cc3db8e-kube-api-access-zd42w\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6\" (UID: \"c9c903cd-d636-4311-9afe-e51e4cc3db8e\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6" Oct 30 17:05:53.678265 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:53.678236 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9c903cd-d636-4311-9afe-e51e4cc3db8e-credential-socket\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6\" (UID: \"c9c903cd-d636-4311-9afe-e51e4cc3db8e\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6" Oct 30 17:05:53.678366 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:53.678267 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9c903cd-d636-4311-9afe-e51e4cc3db8e-istio-envoy\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6\" (UID: \"c9c903cd-d636-4311-9afe-e51e4cc3db8e\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6" Oct 30 17:05:53.678366 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:53.678238 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9c903cd-d636-4311-9afe-e51e4cc3db8e-workload-certs\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6\" (UID: \"c9c903cd-d636-4311-9afe-e51e4cc3db8e\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6" Oct 30 17:05:53.678366 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:53.678342 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9c903cd-d636-4311-9afe-e51e4cc3db8e-istiod-ca-cert\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6\" (UID: \"c9c903cd-d636-4311-9afe-e51e4cc3db8e\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6" Oct 30 17:05:53.678366 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:53.678355 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9c903cd-d636-4311-9afe-e51e4cc3db8e-workload-socket\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6\" (UID: \"c9c903cd-d636-4311-9afe-e51e4cc3db8e\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6" Oct 30 17:05:53.678563 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:53.678373 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9c903cd-d636-4311-9afe-e51e4cc3db8e-istio-data\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6\" (UID: \"c9c903cd-d636-4311-9afe-e51e4cc3db8e\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6" Oct 30 17:05:53.678626 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:53.678592 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9c903cd-d636-4311-9afe-e51e4cc3db8e-credential-socket\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6\" (UID: \"c9c903cd-d636-4311-9afe-e51e4cc3db8e\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6" Oct 30 17:05:53.678713 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:53.678693 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9c903cd-d636-4311-9afe-e51e4cc3db8e-istio-data\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6\" (UID: \"c9c903cd-d636-4311-9afe-e51e4cc3db8e\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6" Oct 30 17:05:53.678980 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:53.678963 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9c903cd-d636-4311-9afe-e51e4cc3db8e-istiod-ca-cert\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6\" (UID: \"c9c903cd-d636-4311-9afe-e51e4cc3db8e\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6" Oct 30 17:05:53.680527 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:53.680495 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9c903cd-d636-4311-9afe-e51e4cc3db8e-istio-envoy\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6\" (UID: \"c9c903cd-d636-4311-9afe-e51e4cc3db8e\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6" Oct 30 17:05:53.680682 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:53.680662 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9c903cd-d636-4311-9afe-e51e4cc3db8e-istio-podinfo\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6\" (UID: \"c9c903cd-d636-4311-9afe-e51e4cc3db8e\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6" Oct 30 17:05:53.702165 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:53.702139 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9c903cd-d636-4311-9afe-e51e4cc3db8e-istio-token\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6\" (UID: \"c9c903cd-d636-4311-9afe-e51e4cc3db8e\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6" Oct 30 17:05:53.702274 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:53.702175 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd42w\" (UniqueName: \"kubernetes.io/projected/c9c903cd-d636-4311-9afe-e51e4cc3db8e-kube-api-access-zd42w\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6\" (UID: \"c9c903cd-d636-4311-9afe-e51e4cc3db8e\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6" Oct 30 17:05:53.778233 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:53.778144 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6" Oct 30 17:05:54.119407 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:05:54.119368 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9c903cd_d636_4311_9afe_e51e4cc3db8e.slice/crio-ea8640a8778e159d35d77c7a15914642bc337acabb3917faca104500d40b670f WatchSource:0}: Error finding container ea8640a8778e159d35d77c7a15914642bc337acabb3917faca104500d40b670f: Status 404 returned error can't find the container with id ea8640a8778e159d35d77c7a15914642bc337acabb3917faca104500d40b670f Oct 30 17:05:54.122516 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:54.122485 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:05:54.122590 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:54.122528 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6"] Oct 30 17:05:54.122590 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:54.122551 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:05:54.122590 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:54.122579 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:05:54.147179 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:54.147153 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="338f53d4-163f-49cf-a772-bc10a691ef68" path="/var/lib/kubelet/pods/338f53d4-163f-49cf-a772-bc10a691ef68/volumes" Oct 30 17:05:54.850740 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:54.850706 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6" event={"ID":"c9c903cd-d636-4311-9afe-e51e4cc3db8e","Type":"ContainerStarted","Data":"73b47ad2ff23e446c1c18cea703cd9198beb3a34408083ae602202c9a326d39f"} Oct 30 17:05:54.850925 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:54.850769 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6" event={"ID":"c9c903cd-d636-4311-9afe-e51e4cc3db8e","Type":"ContainerStarted","Data":"ea8640a8778e159d35d77c7a15914642bc337acabb3917faca104500d40b670f"} Oct 30 17:05:55.778397 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:55.778361 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6" Oct 30 17:05:55.779989 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:55.779958 4522 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 17:05:55.780131 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:55.780020 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6" podUID="c9c903cd-d636-4311-9afe-e51e4cc3db8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 17:05:56.779565 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:56.779522 4522 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 17:05:56.780044 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:56.779592 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6" podUID="c9c903cd-d636-4311-9afe-e51e4cc3db8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 17:05:57.778724 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:57.778683 4522 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 17:05:57.778901 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:57.778742 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6" podUID="c9c903cd-d636-4311-9afe-e51e4cc3db8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 17:05:58.778804 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:58.778769 4522 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 17:05:58.779187 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:58.778821 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6" podUID="c9c903cd-d636-4311-9afe-e51e4cc3db8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 17:05:59.028559 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:59.028507 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6" podStartSLOduration=6.028493645 podStartE2EDuration="6.028493645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:05:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:05:54.90647792 +0000 UTC m=+1128.907232832" watchObservedRunningTime="2026-10-30 17:05:59.028493645 +0000 UTC m=+1133.029248544" Oct 30 17:05:59.034178 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:59.034112 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c27b1e00-5kx5t"] Oct 30 17:05:59.047226 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:59.047202 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c27b1e00-5kx5t"] Oct 30 17:05:59.778659 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:59.778627 4522 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 17:05:59.778835 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:59.778689 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6" podUID="c9c903cd-d636-4311-9afe-e51e4cc3db8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 17:05:59.817433 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:59.817400 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-1cdec0e0-fb59dd5bb-g5xq4" Oct 30 17:05:59.818536 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:05:59.818513 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1cdec0e0-798d84688c-mtx4b" Oct 30 17:06:00.146709 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:00.146615 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="111a5d49-933a-4434-87a1-d6714c0a49ed" path="/var/lib/kubelet/pods/111a5d49-933a-4434-87a1-d6714c0a49ed/volumes" Oct 30 17:06:00.780332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:00.779553 4522 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 17:06:00.780332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:00.779662 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6" podUID="c9c903cd-d636-4311-9afe-e51e4cc3db8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 17:06:02.065097 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:02.065062 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6" Oct 30 17:06:02.067028 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:02.066999 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6" Oct 30 17:06:02.068429 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:02.068394 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6" Oct 30 17:06:03.082750 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:03.082715 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7df5a782-xmptt"] Oct 30 17:06:03.118332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:03.100565 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7df5a782-xmptt"] Oct 30 17:06:03.973437 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:03.973390 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" podUID="4360e9dc-c001-4102-8af6-b55b692fcf2c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Oct 30 17:06:03.974070 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:03.973698 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" podUID="0b832781-3d41-4496-aa86-e4d92792e590" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 17:06:03.992331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:03.988440 4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-04248e/distinct-w1-04248e-kserve-5859d49965-kgq46" podUID="8dda5ac4-30dd-432e-81eb-cb6d39800628" containerName="main" probeResult="failure" output="Get \"https://10.134.0.108:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:06:03.992647 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:03.992610 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" podUID="75321c5d-8bf8-45c6-8811-81981fe3cad4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:06:04.103448 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:04.103398 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" podUID="9bfbc6d3-91fd-41a7-9585-40a8f252f743" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:06:04.373386 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:04.350661 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f87925c2-329d-49b5-a1fd-4b74f2859a2a" path="/var/lib/kubelet/pods/f87925c2-329d-49b5-a1fd-4b74f2859a2a/volumes" Oct 30 17:06:04.404372 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:04.404323 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/distinct-w2-b4f406-kserve-5db4cc6969-7n846" podUID="b8810dbc-b769-46a4-90b9-72a68592bc1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": context deadline exceeded" Oct 30 17:06:04.445332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:04.423430 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" podUID="c9ad71b7-ad90-4c8c-bf27-0f10b23f1593" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 17:06:04.445332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:04.428413 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" podUID="a3e5e732-2c67-4377-9755-e2178bf33772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 17:06:04.609552 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:04.609501 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" podUID="4b4ad752-e053-4cea-a3bc-fe2be6f92021" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:06:04.666612 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:04.666482 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" podUID="8a6a85ad-c944-450f-b0bb-e3b64b0f56f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:06:04.820625 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:04.820586 4522 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:06:04.868361 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:04.833432 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw" podUID="0e123c50-767e-478c-a7e5-0d2476ccbc3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:06:04.868361 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:04.821785 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" podUID="2f1e3bbb-ab1c-455d-a05d-d62f72c5d009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:06:04.868361 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:04.834966 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq" podUID="f18303fb-6d27-4ff1-9cc9-ed032e43d098" containerName="main" probeResult="failure" output="Get \"https://10.134.0.120:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:06:04.868915 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:04.868890 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7eda4908-55hf8"] Oct 30 17:06:04.904760 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:04.903622 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7eda4908-55hf8" Oct 30 17:06:04.904968 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:04.903533 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7eda4908-55hf8"] Oct 30 17:06:04.945321 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:04.924805 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx48x\" (UniqueName: \"kubernetes.io/projected/ad3969c5-75de-45a6-a598-d7b62a296dd5-kube-api-access-vx48x\") pod \"maas-api-revoke-keys-e2e-ait-7eda4908-55hf8\" (UID: \"ad3969c5-75de-45a6-a598-d7b62a296dd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7eda4908-55hf8" Oct 30 17:06:04.945321 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:04.924893 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad3969c5-75de-45a6-a598-d7b62a296dd5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-7eda4908-55hf8\" (UID: \"ad3969c5-75de-45a6-a598-d7b62a296dd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7eda4908-55hf8" Oct 30 17:06:04.945321 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:04.924954 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad3969c5-75de-45a6-a598-d7b62a296dd5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-7eda4908-55hf8\" (UID: \"ad3969c5-75de-45a6-a598-d7b62a296dd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7eda4908-55hf8" Oct 30 17:06:04.977327 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:04.976884 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" podUID="0b832781-3d41-4496-aa86-e4d92792e590" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:06:04.981327 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:04.978172 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" Oct 30 17:06:04.981327 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:04.979785 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:06:04.981327 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:04.979867 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:06:05.077021 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:05.076119 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx48x\" (UniqueName: \"kubernetes.io/projected/ad3969c5-75de-45a6-a598-d7b62a296dd5-kube-api-access-vx48x\") pod \"maas-api-revoke-keys-e2e-ait-7eda4908-55hf8\" (UID: \"ad3969c5-75de-45a6-a598-d7b62a296dd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7eda4908-55hf8" Oct 30 17:06:05.077021 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:05.076227 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad3969c5-75de-45a6-a598-d7b62a296dd5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-7eda4908-55hf8\" (UID: \"ad3969c5-75de-45a6-a598-d7b62a296dd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7eda4908-55hf8" Oct 30 17:06:05.077021 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:05.076339 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad3969c5-75de-45a6-a598-d7b62a296dd5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-7eda4908-55hf8\" (UID: \"ad3969c5-75de-45a6-a598-d7b62a296dd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7eda4908-55hf8" Oct 30 17:06:05.089695 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:05.089669 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" Oct 30 17:06:05.093837 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:05.093355 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad3969c5-75de-45a6-a598-d7b62a296dd5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-7eda4908-55hf8\" (UID: \"ad3969c5-75de-45a6-a598-d7b62a296dd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7eda4908-55hf8" Oct 30 17:06:05.093837 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:05.093705 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad3969c5-75de-45a6-a598-d7b62a296dd5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-7eda4908-55hf8\" (UID: \"ad3969c5-75de-45a6-a598-d7b62a296dd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7eda4908-55hf8" Oct 30 17:06:05.179535 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:05.179501 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx48x\" (UniqueName: \"kubernetes.io/projected/ad3969c5-75de-45a6-a598-d7b62a296dd5-kube-api-access-vx48x\") pod \"maas-api-revoke-keys-e2e-ait-7eda4908-55hf8\" (UID: \"ad3969c5-75de-45a6-a598-d7b62a296dd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7eda4908-55hf8" Oct 30 17:06:05.300486 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:05.298631 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7eda4908-55hf8" Oct 30 17:06:05.676010 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:06:05.675975 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad3969c5_75de_45a6_a598_d7b62a296dd5.slice/crio-ddbccba1a4e5b9460a2d9d66ecc1aa44329b009f599114ccb1701ffff267af70 WatchSource:0}: Error finding container ddbccba1a4e5b9460a2d9d66ecc1aa44329b009f599114ccb1701ffff267af70: Status 404 returned error can't find the container with id ddbccba1a4e5b9460a2d9d66ecc1aa44329b009f599114ccb1701ffff267af70 Oct 30 17:06:05.678599 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:05.678565 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7eda4908-55hf8"] Oct 30 17:06:06.110459 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:06.110424 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7eda4908-55hf8" event={"ID":"ad3969c5-75de-45a6-a598-d7b62a296dd5","Type":"ContainerStarted","Data":"2ff694d90998f96a8da26f7eb9973b6b4f27f48179e082bc271ef5206b197a81"} Oct 30 17:06:06.110631 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:06.110483 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7eda4908-55hf8" event={"ID":"ad3969c5-75de-45a6-a598-d7b62a296dd5","Type":"ContainerStarted","Data":"ddbccba1a4e5b9460a2d9d66ecc1aa44329b009f599114ccb1701ffff267af70"} Oct 30 17:06:06.157166 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:06.157106 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7eda4908-55hf8" podStartSLOduration=2.157092508 podStartE2EDuration="2.157092508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:06:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:06:06.153004034 +0000 UTC m=+1140.153758967" watchObservedRunningTime="2026-10-30 17:06:06.157092508 +0000 UTC m=+1140.157847408" Oct 30 17:06:07.035636 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:07.035606 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1be7fd0-x6sgq"] Oct 30 17:06:07.048901 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:07.048872 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1be7fd0-x6sgq"] Oct 30 17:06:07.125364 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:07.125333 4522 generic.go:356] "Generic (PLEG): container finished" podID="ad3969c5-75de-45a6-a598-d7b62a296dd5" containerID="2ff694d90998f96a8da26f7eb9973b6b4f27f48179e082bc271ef5206b197a81" exitCode=0 Oct 30 17:06:07.125558 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:07.125467 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7eda4908-55hf8" event={"ID":"ad3969c5-75de-45a6-a598-d7b62a296dd5","Type":"ContainerDied","Data":"2ff694d90998f96a8da26f7eb9973b6b4f27f48179e082bc271ef5206b197a81"} Oct 30 17:06:08.148530 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:08.148494 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35684f3d-77e1-42cf-8c59-633053012794" path="/var/lib/kubelet/pods/35684f3d-77e1-42cf-8c59-633053012794/volumes" Oct 30 17:06:08.329813 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:08.329787 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7eda4908-55hf8" Oct 30 17:06:08.441718 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:08.441635 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad3969c5-75de-45a6-a598-d7b62a296dd5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad3969c5-75de-45a6-a598-d7b62a296dd5-maas-api-openshift-ca\") pod \"ad3969c5-75de-45a6-a598-d7b62a296dd5\" (UID: \"ad3969c5-75de-45a6-a598-d7b62a296dd5\") " Oct 30 17:06:08.441896 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:08.441779 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad3969c5-75de-45a6-a598-d7b62a296dd5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad3969c5-75de-45a6-a598-d7b62a296dd5-maas-api-xks-ca\") pod \"ad3969c5-75de-45a6-a598-d7b62a296dd5\" (UID: \"ad3969c5-75de-45a6-a598-d7b62a296dd5\") " Oct 30 17:06:08.442047 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:08.442024 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad3969c5-75de-45a6-a598-d7b62a296dd5-maas-api-openshift-ca" pod "ad3969c5-75de-45a6-a598-d7b62a296dd5" (UID: "ad3969c5-75de-45a6-a598-d7b62a296dd5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:06:08.442117 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:08.442073 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad3969c5-75de-45a6-a598-d7b62a296dd5-kube-api-access-vx48x\" (UniqueName: \"kubernetes.io/projected/ad3969c5-75de-45a6-a598-d7b62a296dd5-kube-api-access-vx48x\") pod \"ad3969c5-75de-45a6-a598-d7b62a296dd5\" (UID: \"ad3969c5-75de-45a6-a598-d7b62a296dd5\") " Oct 30 17:06:08.442875 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:08.442848 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad3969c5-75de-45a6-a598-d7b62a296dd5-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:06:08.444278 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:08.444250 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad3969c5-75de-45a6-a598-d7b62a296dd5-kube-api-access-vx48x" pod "ad3969c5-75de-45a6-a598-d7b62a296dd5" (UID: "ad3969c5-75de-45a6-a598-d7b62a296dd5"). InnerVolumeSpecName "kube-api-access-vx48x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:06:08.444486 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:08.444464 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad3969c5-75de-45a6-a598-d7b62a296dd5-maas-api-xks-ca" pod "ad3969c5-75de-45a6-a598-d7b62a296dd5" (UID: "ad3969c5-75de-45a6-a598-d7b62a296dd5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:06:08.545997 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:08.545928 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vx48x\" (UniqueName: \"kubernetes.io/projected/ad3969c5-75de-45a6-a598-d7b62a296dd5-kube-api-access-vx48x\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:06:08.545997 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:08.545964 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad3969c5-75de-45a6-a598-d7b62a296dd5-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:06:08.920806 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:08.920766 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-b4f406/distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb"] Oct 30 17:06:08.921518 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:08.921496 4522 cpu_manager.go:414] "containerMap: removing container" podUID="ad3969c5-75de-45a6-a598-d7b62a296dd5" containerName="revoke-keys" Oct 30 17:06:08.921518 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:08.921515 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad3969c5-75de-45a6-a598-d7b62a296dd5" containerName="revoke-keys" Oct 30 17:06:08.921714 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:08.921677 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad3969c5-75de-45a6-a598-d7b62a296dd5" containerName="revoke-keys" Oct 30 17:06:08.925996 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:08.925969 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b4f406/distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb" Oct 30 17:06:08.948896 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:08.948865 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-b4f406/distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb"] Oct 30 17:06:08.952908 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:08.952875 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d86fded1-2e04-4d73-bb22-ea0d9a61b961-home\") pod \"distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb\" (UID: \"d86fded1-2e04-4d73-bb22-ea0d9a61b961\") " pod="e2e-models-e2e-worker-w2-b4f406/distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb" Oct 30 17:06:08.953056 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:08.953001 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d86fded1-2e04-4d73-bb22-ea0d9a61b961-model-cache\") pod \"distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb\" (UID: \"d86fded1-2e04-4d73-bb22-ea0d9a61b961\") " pod="e2e-models-e2e-worker-w2-b4f406/distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb" Oct 30 17:06:08.953056 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:08.953035 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d86fded1-2e04-4d73-bb22-ea0d9a61b961-tls-certs\") pod \"distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb\" (UID: \"d86fded1-2e04-4d73-bb22-ea0d9a61b961\") " pod="e2e-models-e2e-worker-w2-b4f406/distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb" Oct 30 17:06:08.953261 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:08.953241 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk9kp\" (UniqueName: \"kubernetes.io/projected/d86fded1-2e04-4d73-bb22-ea0d9a61b961-kube-api-access-zk9kp\") pod \"distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb\" (UID: \"d86fded1-2e04-4d73-bb22-ea0d9a61b961\") " pod="e2e-models-e2e-worker-w2-b4f406/distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb" Oct 30 17:06:08.953388 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:08.953368 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d86fded1-2e04-4d73-bb22-ea0d9a61b961-dshm\") pod \"distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb\" (UID: \"d86fded1-2e04-4d73-bb22-ea0d9a61b961\") " pod="e2e-models-e2e-worker-w2-b4f406/distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb" Oct 30 17:06:08.953458 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:08.953436 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d86fded1-2e04-4d73-bb22-ea0d9a61b961-tmp-dir\") pod \"distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb\" (UID: \"d86fded1-2e04-4d73-bb22-ea0d9a61b961\") " pod="e2e-models-e2e-worker-w2-b4f406/distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb" Oct 30 17:06:08.956532 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:08.956500 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-b4f406\"/\"distinct-2-w2-b4f406-kserve-self-signed-certs\"" Oct 30 17:06:09.055482 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:09.055434 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d86fded1-2e04-4d73-bb22-ea0d9a61b961-home\") pod \"distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb\" (UID: \"d86fded1-2e04-4d73-bb22-ea0d9a61b961\") " pod="e2e-models-e2e-worker-w2-b4f406/distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb" Oct 30 17:06:09.055694 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:09.055551 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d86fded1-2e04-4d73-bb22-ea0d9a61b961-model-cache\") pod \"distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb\" (UID: \"d86fded1-2e04-4d73-bb22-ea0d9a61b961\") " pod="e2e-models-e2e-worker-w2-b4f406/distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb" Oct 30 17:06:09.055694 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:09.055579 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d86fded1-2e04-4d73-bb22-ea0d9a61b961-tls-certs\") pod \"distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb\" (UID: \"d86fded1-2e04-4d73-bb22-ea0d9a61b961\") " pod="e2e-models-e2e-worker-w2-b4f406/distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb" Oct 30 17:06:09.055806 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:09.055763 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk9kp\" (UniqueName: \"kubernetes.io/projected/d86fded1-2e04-4d73-bb22-ea0d9a61b961-kube-api-access-zk9kp\") pod \"distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb\" (UID: \"d86fded1-2e04-4d73-bb22-ea0d9a61b961\") " pod="e2e-models-e2e-worker-w2-b4f406/distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb" Oct 30 17:06:09.055863 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:09.055833 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d86fded1-2e04-4d73-bb22-ea0d9a61b961-dshm\") pod \"distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb\" (UID: \"d86fded1-2e04-4d73-bb22-ea0d9a61b961\") " pod="e2e-models-e2e-worker-w2-b4f406/distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb" Oct 30 17:06:09.055863 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:09.055846 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d86fded1-2e04-4d73-bb22-ea0d9a61b961-home\") pod \"distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb\" (UID: \"d86fded1-2e04-4d73-bb22-ea0d9a61b961\") " pod="e2e-models-e2e-worker-w2-b4f406/distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb" Oct 30 17:06:09.055966 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:09.055880 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d86fded1-2e04-4d73-bb22-ea0d9a61b961-tmp-dir\") pod \"distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb\" (UID: \"d86fded1-2e04-4d73-bb22-ea0d9a61b961\") " pod="e2e-models-e2e-worker-w2-b4f406/distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb" Oct 30 17:06:09.055966 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:09.055882 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d86fded1-2e04-4d73-bb22-ea0d9a61b961-model-cache\") pod \"distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb\" (UID: \"d86fded1-2e04-4d73-bb22-ea0d9a61b961\") " pod="e2e-models-e2e-worker-w2-b4f406/distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb" Oct 30 17:06:09.056205 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:09.056183 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d86fded1-2e04-4d73-bb22-ea0d9a61b961-tmp-dir\") pod \"distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb\" (UID: \"d86fded1-2e04-4d73-bb22-ea0d9a61b961\") " pod="e2e-models-e2e-worker-w2-b4f406/distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb" Oct 30 17:06:09.058097 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:09.058075 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d86fded1-2e04-4d73-bb22-ea0d9a61b961-dshm\") pod \"distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb\" (UID: \"d86fded1-2e04-4d73-bb22-ea0d9a61b961\") " pod="e2e-models-e2e-worker-w2-b4f406/distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb" Oct 30 17:06:09.058206 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:09.058187 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d86fded1-2e04-4d73-bb22-ea0d9a61b961-tls-certs\") pod \"distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb\" (UID: \"d86fded1-2e04-4d73-bb22-ea0d9a61b961\") " pod="e2e-models-e2e-worker-w2-b4f406/distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb" Oct 30 17:06:09.086905 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:09.086867 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk9kp\" (UniqueName: \"kubernetes.io/projected/d86fded1-2e04-4d73-bb22-ea0d9a61b961-kube-api-access-zk9kp\") pod \"distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb\" (UID: \"d86fded1-2e04-4d73-bb22-ea0d9a61b961\") " pod="e2e-models-e2e-worker-w2-b4f406/distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb" Oct 30 17:06:09.147589 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:09.147560 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7eda4908-55hf8" Oct 30 17:06:09.147589 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:09.147580 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7eda4908-55hf8" event={"ID":"ad3969c5-75de-45a6-a598-d7b62a296dd5","Type":"ContainerDied","Data":"ddbccba1a4e5b9460a2d9d66ecc1aa44329b009f599114ccb1701ffff267af70"} Oct 30 17:06:09.147788 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:09.147625 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ddbccba1a4e5b9460a2d9d66ecc1aa44329b009f599114ccb1701ffff267af70" Oct 30 17:06:09.242846 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:09.242803 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b4f406/distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb" Oct 30 17:06:09.552343 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:06:09.552297 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd86fded1_2e04_4d73_bb22_ea0d9a61b961.slice/crio-cfe5071aafa8bdd008cf24ca8ec4ab0b8ea8212ce1675a5d16d2331af4e18222 WatchSource:0}: Error finding container cfe5071aafa8bdd008cf24ca8ec4ab0b8ea8212ce1675a5d16d2331af4e18222: Status 404 returned error can't find the container with id cfe5071aafa8bdd008cf24ca8ec4ab0b8ea8212ce1675a5d16d2331af4e18222 Oct 30 17:06:09.564678 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:09.564645 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-b4f406/distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb"] Oct 30 17:06:10.172946 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:10.172913 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b4f406/distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb" event={"ID":"d86fded1-2e04-4d73-bb22-ea0d9a61b961","Type":"ContainerStarted","Data":"d0c833e26aeb6db1b92a905fa9433641e73eee8d02a39465c5ba761a3f45401c"} Oct 30 17:06:10.173138 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:10.172956 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b4f406/distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb" event={"ID":"d86fded1-2e04-4d73-bb22-ea0d9a61b961","Type":"ContainerStarted","Data":"cfe5071aafa8bdd008cf24ca8ec4ab0b8ea8212ce1675a5d16d2331af4e18222"} Oct 30 17:06:10.221457 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:10.221405 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-b4f406/distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb" podStartSLOduration=2.221388555 podStartE2EDuration="2.221388555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:06:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:06:10.220602956 +0000 UTC m=+1144.221357858" watchObservedRunningTime="2026-10-30 17:06:10.221388555 +0000 UTC m=+1144.222143457" Oct 30 17:06:10.243353 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:10.243292 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b4f406/distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb" Oct 30 17:06:12.965286 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:06:12.965244 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:06:14.574711 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:14.574676 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7eda4908-59dccc9fbb-j26p6"] Oct 30 17:06:14.610129 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:14.610095 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7eda4908-7886d476c-zj8zq"] Oct 30 17:06:17.208564 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:17.208529 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2"] Oct 30 17:06:17.214760 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:17.214739 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2" Oct 30 17:06:17.269644 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:17.269615 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-952bc16e-openshift-default-dockercfg-w85kz\"" Oct 30 17:06:17.290953 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:17.290919 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2"] Oct 30 17:06:17.305062 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:17.305030 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlswc\" (UniqueName: \"kubernetes.io/projected/959b133a-58d9-4b6f-bece-8dfa089125dc-kube-api-access-qlswc\") pod \"e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2\" (UID: \"959b133a-58d9-4b6f-bece-8dfa089125dc\") " pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2" Oct 30 17:06:17.305062 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:17.305066 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/959b133a-58d9-4b6f-bece-8dfa089125dc-istiod-ca-cert\") pod \"e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2\" (UID: \"959b133a-58d9-4b6f-bece-8dfa089125dc\") " pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2" Oct 30 17:06:17.305324 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:17.305245 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/959b133a-58d9-4b6f-bece-8dfa089125dc-workload-socket\") pod \"e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2\" (UID: \"959b133a-58d9-4b6f-bece-8dfa089125dc\") " pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2" Oct 30 17:06:17.305324 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:17.305274 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/959b133a-58d9-4b6f-bece-8dfa089125dc-istio-data\") pod \"e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2\" (UID: \"959b133a-58d9-4b6f-bece-8dfa089125dc\") " pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2" Oct 30 17:06:17.305453 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:17.305370 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/959b133a-58d9-4b6f-bece-8dfa089125dc-istio-token\") pod \"e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2\" (UID: \"959b133a-58d9-4b6f-bece-8dfa089125dc\") " pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2" Oct 30 17:06:17.305527 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:17.305441 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/959b133a-58d9-4b6f-bece-8dfa089125dc-workload-certs\") pod \"e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2\" (UID: \"959b133a-58d9-4b6f-bece-8dfa089125dc\") " pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2" Oct 30 17:06:17.305527 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:17.305504 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/959b133a-58d9-4b6f-bece-8dfa089125dc-istio-envoy\") pod \"e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2\" (UID: \"959b133a-58d9-4b6f-bece-8dfa089125dc\") " pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2" Oct 30 17:06:17.305633 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:17.305538 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/959b133a-58d9-4b6f-bece-8dfa089125dc-credential-socket\") pod \"e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2\" (UID: \"959b133a-58d9-4b6f-bece-8dfa089125dc\") " pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2" Oct 30 17:06:17.305633 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:17.305577 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/959b133a-58d9-4b6f-bece-8dfa089125dc-istio-podinfo\") pod \"e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2\" (UID: \"959b133a-58d9-4b6f-bece-8dfa089125dc\") " pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2" Oct 30 17:06:17.407126 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:17.407087 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/959b133a-58d9-4b6f-bece-8dfa089125dc-workload-socket\") pod \"e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2\" (UID: \"959b133a-58d9-4b6f-bece-8dfa089125dc\") " pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2" Oct 30 17:06:17.407363 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:17.407146 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/959b133a-58d9-4b6f-bece-8dfa089125dc-istio-data\") pod \"e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2\" (UID: \"959b133a-58d9-4b6f-bece-8dfa089125dc\") " pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2" Oct 30 17:06:17.407363 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:17.407203 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/959b133a-58d9-4b6f-bece-8dfa089125dc-istio-token\") pod \"e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2\" (UID: \"959b133a-58d9-4b6f-bece-8dfa089125dc\") " pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2" Oct 30 17:06:17.407363 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:17.407226 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/959b133a-58d9-4b6f-bece-8dfa089125dc-workload-certs\") pod \"e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2\" (UID: \"959b133a-58d9-4b6f-bece-8dfa089125dc\") " pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2" Oct 30 17:06:17.407363 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:17.407263 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/959b133a-58d9-4b6f-bece-8dfa089125dc-istio-envoy\") pod \"e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2\" (UID: \"959b133a-58d9-4b6f-bece-8dfa089125dc\") " pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2" Oct 30 17:06:17.407363 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:17.407316 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/959b133a-58d9-4b6f-bece-8dfa089125dc-credential-socket\") pod \"e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2\" (UID: \"959b133a-58d9-4b6f-bece-8dfa089125dc\") " pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2" Oct 30 17:06:17.407644 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:17.407369 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/959b133a-58d9-4b6f-bece-8dfa089125dc-istio-podinfo\") pod \"e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2\" (UID: \"959b133a-58d9-4b6f-bece-8dfa089125dc\") " pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2" Oct 30 17:06:17.407644 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:17.407531 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlswc\" (UniqueName: \"kubernetes.io/projected/959b133a-58d9-4b6f-bece-8dfa089125dc-kube-api-access-qlswc\") pod \"e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2\" (UID: \"959b133a-58d9-4b6f-bece-8dfa089125dc\") " pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2" Oct 30 17:06:17.407644 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:17.407528 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/959b133a-58d9-4b6f-bece-8dfa089125dc-workload-socket\") pod \"e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2\" (UID: \"959b133a-58d9-4b6f-bece-8dfa089125dc\") " pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2" Oct 30 17:06:17.407644 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:17.407581 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/959b133a-58d9-4b6f-bece-8dfa089125dc-istio-data\") pod \"e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2\" (UID: \"959b133a-58d9-4b6f-bece-8dfa089125dc\") " pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2" Oct 30 17:06:17.407644 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:17.407569 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/959b133a-58d9-4b6f-bece-8dfa089125dc-istiod-ca-cert\") pod \"e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2\" (UID: \"959b133a-58d9-4b6f-bece-8dfa089125dc\") " pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2" Oct 30 17:06:17.407644 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:17.407633 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/959b133a-58d9-4b6f-bece-8dfa089125dc-credential-socket\") pod \"e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2\" (UID: \"959b133a-58d9-4b6f-bece-8dfa089125dc\") " pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2" Oct 30 17:06:17.408184 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:17.408163 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/959b133a-58d9-4b6f-bece-8dfa089125dc-workload-certs\") pod \"e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2\" (UID: \"959b133a-58d9-4b6f-bece-8dfa089125dc\") " pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2" Oct 30 17:06:17.408283 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:17.408266 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/959b133a-58d9-4b6f-bece-8dfa089125dc-istiod-ca-cert\") pod \"e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2\" (UID: \"959b133a-58d9-4b6f-bece-8dfa089125dc\") " pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2" Oct 30 17:06:17.409724 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:17.409695 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/959b133a-58d9-4b6f-bece-8dfa089125dc-istio-envoy\") pod \"e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2\" (UID: \"959b133a-58d9-4b6f-bece-8dfa089125dc\") " pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2" Oct 30 17:06:17.409960 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:17.409940 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/959b133a-58d9-4b6f-bece-8dfa089125dc-istio-podinfo\") pod \"e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2\" (UID: \"959b133a-58d9-4b6f-bece-8dfa089125dc\") " pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2" Oct 30 17:06:17.432788 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:17.432754 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/959b133a-58d9-4b6f-bece-8dfa089125dc-istio-token\") pod \"e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2\" (UID: \"959b133a-58d9-4b6f-bece-8dfa089125dc\") " pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2" Oct 30 17:06:17.433191 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:17.433174 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlswc\" (UniqueName: \"kubernetes.io/projected/959b133a-58d9-4b6f-bece-8dfa089125dc-kube-api-access-qlswc\") pod \"e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2\" (UID: \"959b133a-58d9-4b6f-bece-8dfa089125dc\") " pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2" Oct 30 17:06:17.529923 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:17.529840 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2" Oct 30 17:06:17.565786 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:17.565756 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-952bc16e-n9rtq"] Oct 30 17:06:17.571768 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:17.571745 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-952bc16e-n9rtq" Oct 30 17:06:17.589707 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:17.588554 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:06:17.589707 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:17.589571 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:06:17.591532 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:17.591161 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-952bc16e-n9rtq"] Oct 30 17:06:17.711844 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:17.711793 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1101d2a5-5794-4cd0-9cd7-93643444f030-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-952bc16e-n9rtq\" (UID: \"1101d2a5-5794-4cd0-9cd7-93643444f030\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-952bc16e-n9rtq" Oct 30 17:06:17.712014 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:17.711970 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbmgh\" (UniqueName: \"kubernetes.io/projected/1101d2a5-5794-4cd0-9cd7-93643444f030-kube-api-access-qbmgh\") pod \"maas-api-revoke-keys-e2e-ait-952bc16e-n9rtq\" (UID: \"1101d2a5-5794-4cd0-9cd7-93643444f030\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-952bc16e-n9rtq" Oct 30 17:06:17.712122 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:17.712102 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1101d2a5-5794-4cd0-9cd7-93643444f030-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-952bc16e-n9rtq\" (UID: \"1101d2a5-5794-4cd0-9cd7-93643444f030\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-952bc16e-n9rtq" Oct 30 17:06:17.814007 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:17.813923 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbmgh\" (UniqueName: \"kubernetes.io/projected/1101d2a5-5794-4cd0-9cd7-93643444f030-kube-api-access-qbmgh\") pod \"maas-api-revoke-keys-e2e-ait-952bc16e-n9rtq\" (UID: \"1101d2a5-5794-4cd0-9cd7-93643444f030\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-952bc16e-n9rtq" Oct 30 17:06:17.814206 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:17.814037 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1101d2a5-5794-4cd0-9cd7-93643444f030-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-952bc16e-n9rtq\" (UID: \"1101d2a5-5794-4cd0-9cd7-93643444f030\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-952bc16e-n9rtq" Oct 30 17:06:17.814206 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:17.814179 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1101d2a5-5794-4cd0-9cd7-93643444f030-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-952bc16e-n9rtq\" (UID: \"1101d2a5-5794-4cd0-9cd7-93643444f030\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-952bc16e-n9rtq" Oct 30 17:06:17.814951 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:17.814923 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1101d2a5-5794-4cd0-9cd7-93643444f030-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-952bc16e-n9rtq\" (UID: \"1101d2a5-5794-4cd0-9cd7-93643444f030\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-952bc16e-n9rtq" Oct 30 17:06:17.816803 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:17.816772 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1101d2a5-5794-4cd0-9cd7-93643444f030-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-952bc16e-n9rtq\" (UID: \"1101d2a5-5794-4cd0-9cd7-93643444f030\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-952bc16e-n9rtq" Oct 30 17:06:17.839971 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:17.839942 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbmgh\" (UniqueName: \"kubernetes.io/projected/1101d2a5-5794-4cd0-9cd7-93643444f030-kube-api-access-qbmgh\") pod \"maas-api-revoke-keys-e2e-ait-952bc16e-n9rtq\" (UID: \"1101d2a5-5794-4cd0-9cd7-93643444f030\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-952bc16e-n9rtq" Oct 30 17:06:17.877696 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:06:17.877667 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod959b133a_58d9_4b6f_bece_8dfa089125dc.slice/crio-1eba76310537eda16d24264a60640dc801df23236b7117fc2dbd943d83594819 WatchSource:0}: Error finding container 1eba76310537eda16d24264a60640dc801df23236b7117fc2dbd943d83594819: Status 404 returned error can't find the container with id 1eba76310537eda16d24264a60640dc801df23236b7117fc2dbd943d83594819 Oct 30 17:06:17.880615 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:17.880558 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:06:17.880672 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:17.880659 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:06:17.880708 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:17.880696 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:06:17.880749 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:17.880705 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2"] Oct 30 17:06:17.906366 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:17.906340 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-952bc16e-n9rtq" Oct 30 17:06:18.037114 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:18.037084 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdbb1f67-t7bps"] Oct 30 17:06:18.045815 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:18.045786 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdbb1f67-t7bps"] Oct 30 17:06:18.147195 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:18.147112 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcfdcc85-e3e7-4c06-94c8-c9d22a3862b7" path="/var/lib/kubelet/pods/fcfdcc85-e3e7-4c06-94c8-c9d22a3862b7/volumes" Oct 30 17:06:18.221351 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:06:18.221258 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1101d2a5_5794_4cd0_9cd7_93643444f030.slice/crio-23ad52d90495a2932d64f1ed4813ad6545b19980de93c2399dd1aad571a0870c WatchSource:0}: Error finding container 23ad52d90495a2932d64f1ed4813ad6545b19980de93c2399dd1aad571a0870c: Status 404 returned error can't find the container with id 23ad52d90495a2932d64f1ed4813ad6545b19980de93c2399dd1aad571a0870c Oct 30 17:06:18.223711 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:18.223688 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-952bc16e-n9rtq"] Oct 30 17:06:18.255559 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:18.255522 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-952bc16e-n9rtq" event={"ID":"1101d2a5-5794-4cd0-9cd7-93643444f030","Type":"ContainerStarted","Data":"23ad52d90495a2932d64f1ed4813ad6545b19980de93c2399dd1aad571a0870c"} Oct 30 17:06:18.257509 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:18.257483 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2" event={"ID":"959b133a-58d9-4b6f-bece-8dfa089125dc","Type":"ContainerStarted","Data":"7029ab243df10184d118d08f8142e41204ffa0cbb3537d7470b13baff6bf77de"} Oct 30 17:06:18.257641 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:18.257528 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2" event={"ID":"959b133a-58d9-4b6f-bece-8dfa089125dc","Type":"ContainerStarted","Data":"1eba76310537eda16d24264a60640dc801df23236b7117fc2dbd943d83594819"} Oct 30 17:06:18.307405 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:18.307354 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2" podStartSLOduration=1.307339746 podStartE2EDuration="1.307339746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:06:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:06:18.30588068 +0000 UTC m=+1152.306635603" watchObservedRunningTime="2026-10-30 17:06:18.307339746 +0000 UTC m=+1152.308094647" Oct 30 17:06:18.530272 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:18.530247 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2" Oct 30 17:06:18.531842 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:18.531820 4522 patch_prober.go:29] interesting pod/e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 30 17:06:18.531898 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:18.531865 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2" podUID="959b133a-58d9-4b6f-bece-8dfa089125dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 30 17:06:19.243911 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:19.243888 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-b4f406/distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb" Oct 30 17:06:19.256843 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:19.256809 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-b4f406/distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb" Oct 30 17:06:19.268063 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:19.268036 4522 generic.go:356] "Generic (PLEG): container finished" podID="1101d2a5-5794-4cd0-9cd7-93643444f030" containerID="524be0d6eaaf4bda7a7ee742c80b4e590e0e35f698b79a0770180be1c66987a1" exitCode=0 Oct 30 17:06:19.268204 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:19.268156 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-952bc16e-n9rtq" event={"ID":"1101d2a5-5794-4cd0-9cd7-93643444f030","Type":"ContainerDied","Data":"524be0d6eaaf4bda7a7ee742c80b4e590e0e35f698b79a0770180be1c66987a1"} Oct 30 17:06:19.279523 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:19.279502 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b4f406/distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb" Oct 30 17:06:19.530741 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:19.530663 4522 patch_prober.go:29] interesting pod/e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 30 17:06:19.530741 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:19.530714 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2" podUID="959b133a-58d9-4b6f-bece-8dfa089125dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 30 17:06:19.575999 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:19.575957 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-7eda4908-59dccc9fbb-j26p6" podUID="9aac958f-8ecb-4ac3-8c02-d00e448f74a5" containerName="payload-processing" containerID="cri-o://d34acdaad727fb05695cb566dfe4b81dea0937eaeac2a2cf3af2e089a7fe0484" gracePeriod=25 Oct 30 17:06:19.616814 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:19.614375 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-7eda4908-7886d476c-zj8zq" podUID="a9d0d135-98cd-40ca-aefd-9a13aafbbd68" containerName="payload-pre-processing" containerID="cri-o://d5a7ded6360a5e2a345446efa5bdeb747cf1909edaba7f6ebb5053ebab2c1002" gracePeriod=25 Oct 30 17:06:19.984659 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:19.984632 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7eda4908-59dccc9fbb-j26p6" Oct 30 17:06:20.050607 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:20.050576 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7eda4908-7886d476c-zj8zq" Oct 30 17:06:20.052874 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:20.052845 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9aac958f-8ecb-4ac3-8c02-d00e448f74a5-kube-api-access-grxwm\" (UniqueName: \"kubernetes.io/projected/9aac958f-8ecb-4ac3-8c02-d00e448f74a5-kube-api-access-grxwm\") pod \"9aac958f-8ecb-4ac3-8c02-d00e448f74a5\" (UID: \"9aac958f-8ecb-4ac3-8c02-d00e448f74a5\") " Oct 30 17:06:20.052995 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:20.052960 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9aac958f-8ecb-4ac3-8c02-d00e448f74a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9aac958f-8ecb-4ac3-8c02-d00e448f74a5-plugins-config-volume\") pod \"9aac958f-8ecb-4ac3-8c02-d00e448f74a5\" (UID: \"9aac958f-8ecb-4ac3-8c02-d00e448f74a5\") " Oct 30 17:06:20.053584 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:20.053552 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9aac958f-8ecb-4ac3-8c02-d00e448f74a5-plugins-config-volume" pod "9aac958f-8ecb-4ac3-8c02-d00e448f74a5" (UID: "9aac958f-8ecb-4ac3-8c02-d00e448f74a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:06:20.053986 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:20.053960 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9aac958f-8ecb-4ac3-8c02-d00e448f74a5-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:06:20.055892 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:20.055849 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aac958f-8ecb-4ac3-8c02-d00e448f74a5-kube-api-access-grxwm" pod "9aac958f-8ecb-4ac3-8c02-d00e448f74a5" (UID: "9aac958f-8ecb-4ac3-8c02-d00e448f74a5"). InnerVolumeSpecName "kube-api-access-grxwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:06:20.154585 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:20.154506 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9d0d135-98cd-40ca-aefd-9a13aafbbd68-kube-api-access-wltqb\" (UniqueName: \"kubernetes.io/projected/a9d0d135-98cd-40ca-aefd-9a13aafbbd68-kube-api-access-wltqb\") pod \"a9d0d135-98cd-40ca-aefd-9a13aafbbd68\" (UID: \"a9d0d135-98cd-40ca-aefd-9a13aafbbd68\") " Oct 30 17:06:20.154754 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:20.154609 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9d0d135-98cd-40ca-aefd-9a13aafbbd68-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9d0d135-98cd-40ca-aefd-9a13aafbbd68-plugins-config-volume\") pod \"a9d0d135-98cd-40ca-aefd-9a13aafbbd68\" (UID: \"a9d0d135-98cd-40ca-aefd-9a13aafbbd68\") " Oct 30 17:06:20.154996 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:20.154971 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9d0d135-98cd-40ca-aefd-9a13aafbbd68-plugins-config-volume" pod "a9d0d135-98cd-40ca-aefd-9a13aafbbd68" (UID: "a9d0d135-98cd-40ca-aefd-9a13aafbbd68"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:06:20.155613 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:20.155588 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grxwm\" (UniqueName: \"kubernetes.io/projected/9aac958f-8ecb-4ac3-8c02-d00e448f74a5-kube-api-access-grxwm\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:06:20.155723 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:20.155614 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9d0d135-98cd-40ca-aefd-9a13aafbbd68-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:06:20.156761 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:20.156736 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9d0d135-98cd-40ca-aefd-9a13aafbbd68-kube-api-access-wltqb" pod "a9d0d135-98cd-40ca-aefd-9a13aafbbd68" (UID: "a9d0d135-98cd-40ca-aefd-9a13aafbbd68"). InnerVolumeSpecName "kube-api-access-wltqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:06:20.256738 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:20.256705 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wltqb\" (UniqueName: \"kubernetes.io/projected/a9d0d135-98cd-40ca-aefd-9a13aafbbd68-kube-api-access-wltqb\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:06:20.278858 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:20.278829 4522 generic.go:356] "Generic (PLEG): container finished" podID="9aac958f-8ecb-4ac3-8c02-d00e448f74a5" containerID="d34acdaad727fb05695cb566dfe4b81dea0937eaeac2a2cf3af2e089a7fe0484" exitCode=0 Oct 30 17:06:20.279055 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:20.278890 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7eda4908-59dccc9fbb-j26p6" Oct 30 17:06:20.279055 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:20.278918 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7eda4908-59dccc9fbb-j26p6" event={"ID":"9aac958f-8ecb-4ac3-8c02-d00e448f74a5","Type":"ContainerDied","Data":"d34acdaad727fb05695cb566dfe4b81dea0937eaeac2a2cf3af2e089a7fe0484"} Oct 30 17:06:20.279055 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:20.278973 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7eda4908-59dccc9fbb-j26p6" event={"ID":"9aac958f-8ecb-4ac3-8c02-d00e448f74a5","Type":"ContainerDied","Data":"4037ba745474d61d56108d2d20c0158bc7267790c593b17bf37da9290f4e41ce"} Oct 30 17:06:20.279055 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:20.279004 4522 scope.go:122] "RemoveContainer" containerID="d34acdaad727fb05695cb566dfe4b81dea0937eaeac2a2cf3af2e089a7fe0484" Oct 30 17:06:20.280649 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:20.280625 4522 generic.go:356] "Generic (PLEG): container finished" podID="a9d0d135-98cd-40ca-aefd-9a13aafbbd68" containerID="d5a7ded6360a5e2a345446efa5bdeb747cf1909edaba7f6ebb5053ebab2c1002" exitCode=0 Oct 30 17:06:20.280778 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:20.280743 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7eda4908-7886d476c-zj8zq" event={"ID":"a9d0d135-98cd-40ca-aefd-9a13aafbbd68","Type":"ContainerDied","Data":"d5a7ded6360a5e2a345446efa5bdeb747cf1909edaba7f6ebb5053ebab2c1002"} Oct 30 17:06:20.280824 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:20.280785 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7eda4908-7886d476c-zj8zq" event={"ID":"a9d0d135-98cd-40ca-aefd-9a13aafbbd68","Type":"ContainerDied","Data":"2886e3cc329491a9707798de6afd3018bb3b9641a139ba58be4e8b577e555bdc"} Oct 30 17:06:20.280824 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:20.280816 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7eda4908-7886d476c-zj8zq" Oct 30 17:06:20.290941 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:20.290920 4522 scope.go:122] "RemoveContainer" containerID="d34acdaad727fb05695cb566dfe4b81dea0937eaeac2a2cf3af2e089a7fe0484" Oct 30 17:06:20.291250 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:06:20.291229 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d34acdaad727fb05695cb566dfe4b81dea0937eaeac2a2cf3af2e089a7fe0484\": container with ID starting with d34acdaad727fb05695cb566dfe4b81dea0937eaeac2a2cf3af2e089a7fe0484 not found: ID does not exist" containerID="d34acdaad727fb05695cb566dfe4b81dea0937eaeac2a2cf3af2e089a7fe0484" Oct 30 17:06:20.291386 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:20.291260 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d34acdaad727fb05695cb566dfe4b81dea0937eaeac2a2cf3af2e089a7fe0484"} err="failed to get container status \"d34acdaad727fb05695cb566dfe4b81dea0937eaeac2a2cf3af2e089a7fe0484\": rpc error: code = NotFound desc = could not find container \"d34acdaad727fb05695cb566dfe4b81dea0937eaeac2a2cf3af2e089a7fe0484\": container with ID starting with d34acdaad727fb05695cb566dfe4b81dea0937eaeac2a2cf3af2e089a7fe0484 not found: ID does not exist" Oct 30 17:06:20.291386 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:20.291284 4522 scope.go:122] "RemoveContainer" containerID="d5a7ded6360a5e2a345446efa5bdeb747cf1909edaba7f6ebb5053ebab2c1002" Oct 30 17:06:20.328130 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:20.328104 4522 scope.go:122] "RemoveContainer" containerID="d5a7ded6360a5e2a345446efa5bdeb747cf1909edaba7f6ebb5053ebab2c1002" Oct 30 17:06:20.328494 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:06:20.328466 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5a7ded6360a5e2a345446efa5bdeb747cf1909edaba7f6ebb5053ebab2c1002\": container with ID starting with d5a7ded6360a5e2a345446efa5bdeb747cf1909edaba7f6ebb5053ebab2c1002 not found: ID does not exist" containerID="d5a7ded6360a5e2a345446efa5bdeb747cf1909edaba7f6ebb5053ebab2c1002" Oct 30 17:06:20.328642 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:20.328496 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5a7ded6360a5e2a345446efa5bdeb747cf1909edaba7f6ebb5053ebab2c1002"} err="failed to get container status \"d5a7ded6360a5e2a345446efa5bdeb747cf1909edaba7f6ebb5053ebab2c1002\": rpc error: code = NotFound desc = could not find container \"d5a7ded6360a5e2a345446efa5bdeb747cf1909edaba7f6ebb5053ebab2c1002\": container with ID starting with d5a7ded6360a5e2a345446efa5bdeb747cf1909edaba7f6ebb5053ebab2c1002 not found: ID does not exist" Oct 30 17:06:20.355555 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:20.355520 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7eda4908-7886d476c-zj8zq"] Oct 30 17:06:20.371541 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:20.365157 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7eda4908-7886d476c-zj8zq"] Oct 30 17:06:20.442213 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:20.442135 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7eda4908-59dccc9fbb-j26p6"] Oct 30 17:06:20.450705 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:20.450678 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7eda4908-59dccc9fbb-j26p6"] Oct 30 17:06:20.481742 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:20.481710 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-952bc16e-n9rtq" Oct 30 17:06:20.530727 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:20.530687 4522 patch_prober.go:29] interesting pod/e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 30 17:06:20.530910 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:20.530765 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2" podUID="959b133a-58d9-4b6f-bece-8dfa089125dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 30 17:06:20.562862 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:20.562829 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1101d2a5-5794-4cd0-9cd7-93643444f030-kube-api-access-qbmgh\" (UniqueName: \"kubernetes.io/projected/1101d2a5-5794-4cd0-9cd7-93643444f030-kube-api-access-qbmgh\") pod \"1101d2a5-5794-4cd0-9cd7-93643444f030\" (UID: \"1101d2a5-5794-4cd0-9cd7-93643444f030\") " Oct 30 17:06:20.563037 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:20.562907 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1101d2a5-5794-4cd0-9cd7-93643444f030-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1101d2a5-5794-4cd0-9cd7-93643444f030-maas-api-openshift-ca\") pod \"1101d2a5-5794-4cd0-9cd7-93643444f030\" (UID: \"1101d2a5-5794-4cd0-9cd7-93643444f030\") " Oct 30 17:06:20.563037 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:20.563010 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1101d2a5-5794-4cd0-9cd7-93643444f030-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1101d2a5-5794-4cd0-9cd7-93643444f030-maas-api-xks-ca\") pod \"1101d2a5-5794-4cd0-9cd7-93643444f030\" (UID: \"1101d2a5-5794-4cd0-9cd7-93643444f030\") " Oct 30 17:06:20.563409 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:20.563375 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1101d2a5-5794-4cd0-9cd7-93643444f030-maas-api-openshift-ca" pod "1101d2a5-5794-4cd0-9cd7-93643444f030" (UID: "1101d2a5-5794-4cd0-9cd7-93643444f030"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:06:20.564265 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:20.564238 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1101d2a5-5794-4cd0-9cd7-93643444f030-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:06:20.565259 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:20.565229 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1101d2a5-5794-4cd0-9cd7-93643444f030-kube-api-access-qbmgh" pod "1101d2a5-5794-4cd0-9cd7-93643444f030" (UID: "1101d2a5-5794-4cd0-9cd7-93643444f030"). InnerVolumeSpecName "kube-api-access-qbmgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:06:20.565354 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:20.565237 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1101d2a5-5794-4cd0-9cd7-93643444f030-maas-api-xks-ca" pod "1101d2a5-5794-4cd0-9cd7-93643444f030" (UID: "1101d2a5-5794-4cd0-9cd7-93643444f030"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:06:20.666258 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:20.666223 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbmgh\" (UniqueName: \"kubernetes.io/projected/1101d2a5-5794-4cd0-9cd7-93643444f030-kube-api-access-qbmgh\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:06:20.666258 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:20.666251 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1101d2a5-5794-4cd0-9cd7-93643444f030-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:06:21.317834 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:21.317805 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-952bc16e-n9rtq" Oct 30 17:06:21.318281 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:21.317838 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-952bc16e-n9rtq" event={"ID":"1101d2a5-5794-4cd0-9cd7-93643444f030","Type":"ContainerDied","Data":"23ad52d90495a2932d64f1ed4813ad6545b19980de93c2399dd1aad571a0870c"} Oct 30 17:06:21.318281 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:21.317890 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23ad52d90495a2932d64f1ed4813ad6545b19980de93c2399dd1aad571a0870c" Oct 30 17:06:21.530720 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:21.530685 4522 patch_prober.go:29] interesting pod/e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 30 17:06:21.530923 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:21.530751 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2" podUID="959b133a-58d9-4b6f-bece-8dfa089125dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 30 17:06:22.147027 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:22.146993 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9aac958f-8ecb-4ac3-8c02-d00e448f74a5" path="/var/lib/kubelet/pods/9aac958f-8ecb-4ac3-8c02-d00e448f74a5/volumes" Oct 30 17:06:22.147543 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:22.147527 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9d0d135-98cd-40ca-aefd-9a13aafbbd68" path="/var/lib/kubelet/pods/a9d0d135-98cd-40ca-aefd-9a13aafbbd68/volumes" Oct 30 17:06:22.530886 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:22.530851 4522 patch_prober.go:29] interesting pod/e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 30 17:06:22.531290 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:22.530912 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2" podUID="959b133a-58d9-4b6f-bece-8dfa089125dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 30 17:06:23.036779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:23.036741 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02dc2587-j2jds"] Oct 30 17:06:23.049645 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:23.049610 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02dc2587-j2jds"] Oct 30 17:06:23.530873 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:23.530833 4522 patch_prober.go:29] interesting pod/e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 30 17:06:23.531057 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:23.530904 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2" podUID="959b133a-58d9-4b6f-bece-8dfa089125dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 30 17:06:23.553266 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:23.553233 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m"] Oct 30 17:06:23.553984 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:23.553955 4522 cpu_manager.go:414] "containerMap: removing container" podUID="1101d2a5-5794-4cd0-9cd7-93643444f030" containerName="revoke-keys" Oct 30 17:06:23.553984 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:23.553974 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1101d2a5-5794-4cd0-9cd7-93643444f030" containerName="revoke-keys" Oct 30 17:06:23.554181 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:23.554015 4522 cpu_manager.go:414] "containerMap: removing container" podUID="a9d0d135-98cd-40ca-aefd-9a13aafbbd68" containerName="payload-pre-processing" Oct 30 17:06:23.554181 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:23.554023 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9d0d135-98cd-40ca-aefd-9a13aafbbd68" containerName="payload-pre-processing" Oct 30 17:06:23.554181 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:23.554073 4522 cpu_manager.go:414] "containerMap: removing container" podUID="9aac958f-8ecb-4ac3-8c02-d00e448f74a5" containerName="payload-processing" Oct 30 17:06:23.554181 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:23.554081 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9aac958f-8ecb-4ac3-8c02-d00e448f74a5" containerName="payload-processing" Oct 30 17:06:23.554415 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:23.554252 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="9aac958f-8ecb-4ac3-8c02-d00e448f74a5" containerName="payload-processing" Oct 30 17:06:23.554415 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:23.554268 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="1101d2a5-5794-4cd0-9cd7-93643444f030" containerName="revoke-keys" Oct 30 17:06:23.554415 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:23.554282 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9d0d135-98cd-40ca-aefd-9a13aafbbd68" containerName="payload-pre-processing" Oct 30 17:06:23.559631 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:23.559610 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m" Oct 30 17:06:23.584856 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:23.584825 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m"] Oct 30 17:06:23.735244 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:23.735200 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da926b76-e144-4a03-bb81-67d2effff0a6-istio-podinfo\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m\" (UID: \"da926b76-e144-4a03-bb81-67d2effff0a6\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m" Oct 30 17:06:23.735533 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:23.735268 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da926b76-e144-4a03-bb81-67d2effff0a6-istio-data\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m\" (UID: \"da926b76-e144-4a03-bb81-67d2effff0a6\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m" Oct 30 17:06:23.735533 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:23.735403 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da926b76-e144-4a03-bb81-67d2effff0a6-credential-socket\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m\" (UID: \"da926b76-e144-4a03-bb81-67d2effff0a6\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m" Oct 30 17:06:23.735533 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:23.735487 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da926b76-e144-4a03-bb81-67d2effff0a6-istio-envoy\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m\" (UID: \"da926b76-e144-4a03-bb81-67d2effff0a6\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m" Oct 30 17:06:23.735740 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:23.735574 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4wb5\" (UniqueName: \"kubernetes.io/projected/da926b76-e144-4a03-bb81-67d2effff0a6-kube-api-access-p4wb5\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m\" (UID: \"da926b76-e144-4a03-bb81-67d2effff0a6\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m" Oct 30 17:06:23.735867 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:23.735838 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da926b76-e144-4a03-bb81-67d2effff0a6-istio-token\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m\" (UID: \"da926b76-e144-4a03-bb81-67d2effff0a6\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m" Oct 30 17:06:23.735956 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:23.735938 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da926b76-e144-4a03-bb81-67d2effff0a6-workload-socket\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m\" (UID: \"da926b76-e144-4a03-bb81-67d2effff0a6\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m" Oct 30 17:06:23.736016 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:23.736000 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da926b76-e144-4a03-bb81-67d2effff0a6-workload-certs\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m\" (UID: \"da926b76-e144-4a03-bb81-67d2effff0a6\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m" Oct 30 17:06:23.736096 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:23.736080 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da926b76-e144-4a03-bb81-67d2effff0a6-istiod-ca-cert\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m\" (UID: \"da926b76-e144-4a03-bb81-67d2effff0a6\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m" Oct 30 17:06:23.838467 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:23.838368 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da926b76-e144-4a03-bb81-67d2effff0a6-workload-socket\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m\" (UID: \"da926b76-e144-4a03-bb81-67d2effff0a6\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m" Oct 30 17:06:23.838467 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:23.838447 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da926b76-e144-4a03-bb81-67d2effff0a6-workload-certs\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m\" (UID: \"da926b76-e144-4a03-bb81-67d2effff0a6\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m" Oct 30 17:06:23.838710 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:23.838517 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da926b76-e144-4a03-bb81-67d2effff0a6-istiod-ca-cert\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m\" (UID: \"da926b76-e144-4a03-bb81-67d2effff0a6\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m" Oct 30 17:06:23.838710 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:23.838658 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da926b76-e144-4a03-bb81-67d2effff0a6-istio-podinfo\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m\" (UID: \"da926b76-e144-4a03-bb81-67d2effff0a6\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m" Oct 30 17:06:23.838710 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:23.838700 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da926b76-e144-4a03-bb81-67d2effff0a6-istio-data\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m\" (UID: \"da926b76-e144-4a03-bb81-67d2effff0a6\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m" Oct 30 17:06:23.838878 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:23.838860 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da926b76-e144-4a03-bb81-67d2effff0a6-credential-socket\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m\" (UID: \"da926b76-e144-4a03-bb81-67d2effff0a6\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m" Oct 30 17:06:23.838949 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:23.838871 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da926b76-e144-4a03-bb81-67d2effff0a6-workload-certs\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m\" (UID: \"da926b76-e144-4a03-bb81-67d2effff0a6\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m" Oct 30 17:06:23.838949 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:23.838861 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da926b76-e144-4a03-bb81-67d2effff0a6-workload-socket\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m\" (UID: \"da926b76-e144-4a03-bb81-67d2effff0a6\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m" Oct 30 17:06:23.838949 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:23.838894 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da926b76-e144-4a03-bb81-67d2effff0a6-istio-envoy\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m\" (UID: \"da926b76-e144-4a03-bb81-67d2effff0a6\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m" Oct 30 17:06:23.839090 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:23.838976 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4wb5\" (UniqueName: \"kubernetes.io/projected/da926b76-e144-4a03-bb81-67d2effff0a6-kube-api-access-p4wb5\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m\" (UID: \"da926b76-e144-4a03-bb81-67d2effff0a6\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m" Oct 30 17:06:23.839179 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:23.839158 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da926b76-e144-4a03-bb81-67d2effff0a6-istio-token\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m\" (UID: \"da926b76-e144-4a03-bb81-67d2effff0a6\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m" Oct 30 17:06:23.839290 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:23.839269 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da926b76-e144-4a03-bb81-67d2effff0a6-istiod-ca-cert\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m\" (UID: \"da926b76-e144-4a03-bb81-67d2effff0a6\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m" Oct 30 17:06:23.839526 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:23.839501 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da926b76-e144-4a03-bb81-67d2effff0a6-istio-data\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m\" (UID: \"da926b76-e144-4a03-bb81-67d2effff0a6\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m" Oct 30 17:06:23.839657 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:23.839632 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da926b76-e144-4a03-bb81-67d2effff0a6-credential-socket\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m\" (UID: \"da926b76-e144-4a03-bb81-67d2effff0a6\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m" Oct 30 17:06:23.841378 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:23.841346 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da926b76-e144-4a03-bb81-67d2effff0a6-istio-envoy\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m\" (UID: \"da926b76-e144-4a03-bb81-67d2effff0a6\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m" Oct 30 17:06:23.841667 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:23.841638 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da926b76-e144-4a03-bb81-67d2effff0a6-istio-podinfo\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m\" (UID: \"da926b76-e144-4a03-bb81-67d2effff0a6\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m" Oct 30 17:06:23.864147 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:23.864116 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4wb5\" (UniqueName: \"kubernetes.io/projected/da926b76-e144-4a03-bb81-67d2effff0a6-kube-api-access-p4wb5\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m\" (UID: \"da926b76-e144-4a03-bb81-67d2effff0a6\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m" Oct 30 17:06:23.864866 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:23.864837 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da926b76-e144-4a03-bb81-67d2effff0a6-istio-token\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m\" (UID: \"da926b76-e144-4a03-bb81-67d2effff0a6\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m" Oct 30 17:06:23.875220 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:23.875193 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m" Oct 30 17:06:24.146973 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:24.146892 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df27828a-9121-4255-88a5-1ea5f5066337" path="/var/lib/kubelet/pods/df27828a-9121-4255-88a5-1ea5f5066337/volumes" Oct 30 17:06:24.210368 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:06:24.210337 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda926b76_e144_4a03_bb81_67d2effff0a6.slice/crio-d8bc17c707aef38605fea8e2f91bef9539f70780f0b1e9138292d7e70cf2b14b WatchSource:0}: Error finding container d8bc17c707aef38605fea8e2f91bef9539f70780f0b1e9138292d7e70cf2b14b: Status 404 returned error can't find the container with id d8bc17c707aef38605fea8e2f91bef9539f70780f0b1e9138292d7e70cf2b14b Oct 30 17:06:24.212951 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:24.212919 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:06:24.213033 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:24.212983 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:06:24.213033 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:24.213012 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:06:24.214147 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:24.214118 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m"] Oct 30 17:06:24.354429 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:24.354396 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m" event={"ID":"da926b76-e144-4a03-bb81-67d2effff0a6","Type":"ContainerStarted","Data":"b10a9c1634fcda6bb2882eb293c4a023244cb460c922442454335a65a65d90c1"} Oct 30 17:06:24.354603 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:24.354456 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m" event={"ID":"da926b76-e144-4a03-bb81-67d2effff0a6","Type":"ContainerStarted","Data":"d8bc17c707aef38605fea8e2f91bef9539f70780f0b1e9138292d7e70cf2b14b"} Oct 30 17:06:24.411319 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:24.411197 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m" podStartSLOduration=1.411176621 podStartE2EDuration="1.411176621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:06:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:06:24.402069676 +0000 UTC m=+1158.402824590" watchObservedRunningTime="2026-10-30 17:06:24.411176621 +0000 UTC m=+1158.411931524" Oct 30 17:06:24.535887 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:24.535858 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2" Oct 30 17:06:24.536397 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:24.536325 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2" Oct 30 17:06:24.537128 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:24.537111 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2" Oct 30 17:06:24.875981 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:24.875950 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m" Oct 30 17:06:25.877427 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:25.877393 4522 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 17:06:25.877909 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:25.877467 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m" podUID="da926b76-e144-4a03-bb81-67d2effff0a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 17:06:26.876398 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:26.876360 4522 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 17:06:26.876621 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:26.876427 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m" podUID="da926b76-e144-4a03-bb81-67d2effff0a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 17:06:27.036487 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:27.036453 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6434aa3d-556sw"] Oct 30 17:06:27.052349 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:27.052288 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6434aa3d-556sw"] Oct 30 17:06:27.275512 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:27.275477 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eadf55b1-xznvl"] Oct 30 17:06:27.280250 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:27.280226 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eadf55b1-xznvl" Oct 30 17:06:27.318327 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:27.317298 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:06:27.318327 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:27.317516 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:06:27.433188 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:27.433149 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nljzk\" (UniqueName: \"kubernetes.io/projected/fd03a682-5ace-47b9-b539-9d0ddf3cbfdb-kube-api-access-nljzk\") pod \"maas-api-revoke-sub-eadf55b1-xznvl\" (UID: \"fd03a682-5ace-47b9-b539-9d0ddf3cbfdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eadf55b1-xznvl" Oct 30 17:06:27.464331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:27.464016 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd03a682-5ace-47b9-b539-9d0ddf3cbfdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eadf55b1-xznvl\" (UID: \"fd03a682-5ace-47b9-b539-9d0ddf3cbfdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eadf55b1-xznvl" Oct 30 17:06:27.464331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:27.464198 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd03a682-5ace-47b9-b539-9d0ddf3cbfdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eadf55b1-xznvl\" (UID: \"fd03a682-5ace-47b9-b539-9d0ddf3cbfdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eadf55b1-xznvl" Oct 30 17:06:27.466240 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:27.466213 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eadf55b1-xznvl"] Oct 30 17:06:27.565763 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:27.565668 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd03a682-5ace-47b9-b539-9d0ddf3cbfdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eadf55b1-xznvl\" (UID: \"fd03a682-5ace-47b9-b539-9d0ddf3cbfdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eadf55b1-xznvl" Oct 30 17:06:27.565920 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:27.565865 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd03a682-5ace-47b9-b539-9d0ddf3cbfdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eadf55b1-xznvl\" (UID: \"fd03a682-5ace-47b9-b539-9d0ddf3cbfdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eadf55b1-xznvl" Oct 30 17:06:27.565964 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:27.565949 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nljzk\" (UniqueName: \"kubernetes.io/projected/fd03a682-5ace-47b9-b539-9d0ddf3cbfdb-kube-api-access-nljzk\") pod \"maas-api-revoke-sub-eadf55b1-xznvl\" (UID: \"fd03a682-5ace-47b9-b539-9d0ddf3cbfdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eadf55b1-xznvl" Oct 30 17:06:27.566597 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:27.566574 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd03a682-5ace-47b9-b539-9d0ddf3cbfdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eadf55b1-xznvl\" (UID: \"fd03a682-5ace-47b9-b539-9d0ddf3cbfdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eadf55b1-xznvl" Oct 30 17:06:27.568941 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:27.568919 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd03a682-5ace-47b9-b539-9d0ddf3cbfdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eadf55b1-xznvl\" (UID: \"fd03a682-5ace-47b9-b539-9d0ddf3cbfdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eadf55b1-xznvl" Oct 30 17:06:27.590933 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:27.590895 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nljzk\" (UniqueName: \"kubernetes.io/projected/fd03a682-5ace-47b9-b539-9d0ddf3cbfdb-kube-api-access-nljzk\") pod \"maas-api-revoke-sub-eadf55b1-xznvl\" (UID: \"fd03a682-5ace-47b9-b539-9d0ddf3cbfdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eadf55b1-xznvl" Oct 30 17:06:27.633003 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:27.632972 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eadf55b1-xznvl" Oct 30 17:06:27.876332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:27.876219 4522 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 17:06:27.876332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:27.876281 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m" podUID="da926b76-e144-4a03-bb81-67d2effff0a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 17:06:28.077330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:28.077236 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eadf55b1-xznvl"] Oct 30 17:06:28.147564 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:28.147488 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa8d133a-3261-41e5-88f4-18fcaeb3ec46" path="/var/lib/kubelet/pods/fa8d133a-3261-41e5-88f4-18fcaeb3ec46/volumes" Oct 30 17:06:28.460333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:28.460285 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7eda4908-55hf8"] Oct 30 17:06:28.473730 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:28.473696 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7eda4908-55hf8"] Oct 30 17:06:28.487026 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:28.486977 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eadf55b1-xznvl" event={"ID":"fd03a682-5ace-47b9-b539-9d0ddf3cbfdb","Type":"ContainerStarted","Data":"4aa7be320fdaef8d6bd62b3b9b8345caee964fe7619f5d49436075c2a81be0df"} Oct 30 17:06:28.876382 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:28.876259 4522 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 17:06:28.876382 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:28.876344 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m" podUID="da926b76-e144-4a03-bb81-67d2effff0a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 17:06:28.923621 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:28.923588 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-952bc16e-5cdf6db644-hm4mt"] Oct 30 17:06:28.941601 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:28.940836 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-952bc16e-6865b69dc5-zqc5b"] Oct 30 17:06:29.528781 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:29.528750 4522 generic.go:356] "Generic (PLEG): container finished" podID="fd03a682-5ace-47b9-b539-9d0ddf3cbfdb" containerID="b4bde40ea0b334f350514a55a89c0c946e3f36eaf6a8ac209c37562c76092778" exitCode=0 Oct 30 17:06:29.529181 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:29.528837 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eadf55b1-xznvl" event={"ID":"fd03a682-5ace-47b9-b539-9d0ddf3cbfdb","Type":"ContainerDied","Data":"b4bde40ea0b334f350514a55a89c0c946e3f36eaf6a8ac209c37562c76092778"} Oct 30 17:06:29.879755 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:29.879717 4522 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 17:06:29.880047 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:29.880013 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m" podUID="da926b76-e144-4a03-bb81-67d2effff0a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 17:06:30.243290 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:30.243254 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad3969c5-75de-45a6-a598-d7b62a296dd5" path="/var/lib/kubelet/pods/ad3969c5-75de-45a6-a598-d7b62a296dd5/volumes" Oct 30 17:06:30.764883 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:30.764850 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eadf55b1-xznvl" Oct 30 17:06:30.834414 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:30.834327 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd03a682-5ace-47b9-b539-9d0ddf3cbfdb-kube-api-access-nljzk\" (UniqueName: \"kubernetes.io/projected/fd03a682-5ace-47b9-b539-9d0ddf3cbfdb-kube-api-access-nljzk\") pod \"fd03a682-5ace-47b9-b539-9d0ddf3cbfdb\" (UID: \"fd03a682-5ace-47b9-b539-9d0ddf3cbfdb\") " Oct 30 17:06:30.834667 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:30.834643 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd03a682-5ace-47b9-b539-9d0ddf3cbfdb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd03a682-5ace-47b9-b539-9d0ddf3cbfdb-maas-api-openshift-ca\") pod \"fd03a682-5ace-47b9-b539-9d0ddf3cbfdb\" (UID: \"fd03a682-5ace-47b9-b539-9d0ddf3cbfdb\") " Oct 30 17:06:30.834742 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:30.834710 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd03a682-5ace-47b9-b539-9d0ddf3cbfdb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd03a682-5ace-47b9-b539-9d0ddf3cbfdb-maas-api-xks-ca\") pod \"fd03a682-5ace-47b9-b539-9d0ddf3cbfdb\" (UID: \"fd03a682-5ace-47b9-b539-9d0ddf3cbfdb\") " Oct 30 17:06:30.835061 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:30.835034 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd03a682-5ace-47b9-b539-9d0ddf3cbfdb-maas-api-openshift-ca" pod "fd03a682-5ace-47b9-b539-9d0ddf3cbfdb" (UID: "fd03a682-5ace-47b9-b539-9d0ddf3cbfdb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:06:30.835538 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:30.835510 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd03a682-5ace-47b9-b539-9d0ddf3cbfdb-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:06:30.837413 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:30.837389 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd03a682-5ace-47b9-b539-9d0ddf3cbfdb-maas-api-xks-ca" pod "fd03a682-5ace-47b9-b539-9d0ddf3cbfdb" (UID: "fd03a682-5ace-47b9-b539-9d0ddf3cbfdb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:06:30.837488 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:30.837425 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd03a682-5ace-47b9-b539-9d0ddf3cbfdb-kube-api-access-nljzk" pod "fd03a682-5ace-47b9-b539-9d0ddf3cbfdb" (UID: "fd03a682-5ace-47b9-b539-9d0ddf3cbfdb"). InnerVolumeSpecName "kube-api-access-nljzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:06:30.876517 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:30.876479 4522 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 17:06:30.876830 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:30.876549 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m" podUID="da926b76-e144-4a03-bb81-67d2effff0a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 17:06:30.938022 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:30.937984 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd03a682-5ace-47b9-b539-9d0ddf3cbfdb-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:06:30.938022 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:30.938024 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nljzk\" (UniqueName: \"kubernetes.io/projected/fd03a682-5ace-47b9-b539-9d0ddf3cbfdb-kube-api-access-nljzk\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:06:31.574182 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:31.574142 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eadf55b1-xznvl" event={"ID":"fd03a682-5ace-47b9-b539-9d0ddf3cbfdb","Type":"ContainerDied","Data":"4aa7be320fdaef8d6bd62b3b9b8345caee964fe7619f5d49436075c2a81be0df"} Oct 30 17:06:31.574476 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:31.574457 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4aa7be320fdaef8d6bd62b3b9b8345caee964fe7619f5d49436075c2a81be0df" Oct 30 17:06:31.574669 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:31.574630 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eadf55b1-xznvl" Oct 30 17:06:32.036437 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:32.036395 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eea1ffaf-l78l7"] Oct 30 17:06:32.041666 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:32.041629 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eea1ffaf-l78l7"] Oct 30 17:06:32.073803 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:32.073770 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m" Oct 30 17:06:32.074746 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:32.074724 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m" Oct 30 17:06:32.075368 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:32.075347 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m" Oct 30 17:06:32.147371 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:32.147335 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b04afd1f-0a81-4e8f-bb65-1fb57a6df53d" path="/var/lib/kubelet/pods/b04afd1f-0a81-4e8f-bb65-1fb57a6df53d/volumes" Oct 30 17:06:33.924205 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:33.924160 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-952bc16e-5cdf6db644-hm4mt" podUID="77f5ee2d-25a4-486c-965a-324c473dd9b7" containerName="payload-processing" containerID="cri-o://7800ceadf8d9edb241e828e57b3eb76c8826c2e76df077a69939c4a3c40d6b2d" gracePeriod=25 Oct 30 17:06:33.942042 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:33.941994 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-952bc16e-6865b69dc5-zqc5b" podUID="64ddc647-b2ef-445a-a05a-96893865f100" containerName="payload-pre-processing" containerID="cri-o://bae4f1bf4fcc3682226d6dd85c5c1a1aa1138ec77ce216842d5ee14b16c8ed2e" gracePeriod=25 Oct 30 17:06:34.393221 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:34.391937 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-952bc16e-5cdf6db644-hm4mt" Oct 30 17:06:34.465535 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:34.465503 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-952bc16e-6865b69dc5-zqc5b" Oct 30 17:06:34.562151 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:34.562054 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64ddc647-b2ef-445a-a05a-96893865f100-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ddc647-b2ef-445a-a05a-96893865f100-plugins-config-volume\") pod \"64ddc647-b2ef-445a-a05a-96893865f100\" (UID: \"64ddc647-b2ef-445a-a05a-96893865f100\") " Oct 30 17:06:34.562370 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:34.562226 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77f5ee2d-25a4-486c-965a-324c473dd9b7-kube-api-access-qlpt7\" (UniqueName: \"kubernetes.io/projected/77f5ee2d-25a4-486c-965a-324c473dd9b7-kube-api-access-qlpt7\") pod \"77f5ee2d-25a4-486c-965a-324c473dd9b7\" (UID: \"77f5ee2d-25a4-486c-965a-324c473dd9b7\") " Oct 30 17:06:34.562503 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:34.562478 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64ddc647-b2ef-445a-a05a-96893865f100-plugins-config-volume" pod "64ddc647-b2ef-445a-a05a-96893865f100" (UID: "64ddc647-b2ef-445a-a05a-96893865f100"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:06:34.562598 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:34.562489 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77f5ee2d-25a4-486c-965a-324c473dd9b7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77f5ee2d-25a4-486c-965a-324c473dd9b7-plugins-config-volume\") pod \"77f5ee2d-25a4-486c-965a-324c473dd9b7\" (UID: \"77f5ee2d-25a4-486c-965a-324c473dd9b7\") " Oct 30 17:06:34.562598 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:34.562566 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64ddc647-b2ef-445a-a05a-96893865f100-kube-api-access-jrc9n\" (UniqueName: \"kubernetes.io/projected/64ddc647-b2ef-445a-a05a-96893865f100-kube-api-access-jrc9n\") pod \"64ddc647-b2ef-445a-a05a-96893865f100\" (UID: \"64ddc647-b2ef-445a-a05a-96893865f100\") " Oct 30 17:06:34.562846 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:34.562824 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77f5ee2d-25a4-486c-965a-324c473dd9b7-plugins-config-volume" pod "77f5ee2d-25a4-486c-965a-324c473dd9b7" (UID: "77f5ee2d-25a4-486c-965a-324c473dd9b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:06:34.563446 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:34.563423 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77f5ee2d-25a4-486c-965a-324c473dd9b7-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:06:34.563446 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:34.563446 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ddc647-b2ef-445a-a05a-96893865f100-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:06:34.564727 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:34.564700 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64ddc647-b2ef-445a-a05a-96893865f100-kube-api-access-jrc9n" pod "64ddc647-b2ef-445a-a05a-96893865f100" (UID: "64ddc647-b2ef-445a-a05a-96893865f100"). InnerVolumeSpecName "kube-api-access-jrc9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:06:34.564916 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:34.564902 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77f5ee2d-25a4-486c-965a-324c473dd9b7-kube-api-access-qlpt7" pod "77f5ee2d-25a4-486c-965a-324c473dd9b7" (UID: "77f5ee2d-25a4-486c-965a-324c473dd9b7"). InnerVolumeSpecName "kube-api-access-qlpt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:06:34.601527 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:34.601490 4522 generic.go:356] "Generic (PLEG): container finished" podID="77f5ee2d-25a4-486c-965a-324c473dd9b7" containerID="7800ceadf8d9edb241e828e57b3eb76c8826c2e76df077a69939c4a3c40d6b2d" exitCode=0 Oct 30 17:06:34.601705 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:34.601532 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-952bc16e-5cdf6db644-hm4mt" Oct 30 17:06:34.601705 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:34.601568 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-952bc16e-5cdf6db644-hm4mt" event={"ID":"77f5ee2d-25a4-486c-965a-324c473dd9b7","Type":"ContainerDied","Data":"7800ceadf8d9edb241e828e57b3eb76c8826c2e76df077a69939c4a3c40d6b2d"} Oct 30 17:06:34.601705 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:34.601628 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-952bc16e-5cdf6db644-hm4mt" event={"ID":"77f5ee2d-25a4-486c-965a-324c473dd9b7","Type":"ContainerDied","Data":"1fc84898be2776b368e26a36a390707254b47ff417efacdd4f0b87eece5e25ca"} Oct 30 17:06:34.601705 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:34.601658 4522 scope.go:122] "RemoveContainer" containerID="7800ceadf8d9edb241e828e57b3eb76c8826c2e76df077a69939c4a3c40d6b2d" Oct 30 17:06:34.603473 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:34.603443 4522 generic.go:356] "Generic (PLEG): container finished" podID="64ddc647-b2ef-445a-a05a-96893865f100" containerID="bae4f1bf4fcc3682226d6dd85c5c1a1aa1138ec77ce216842d5ee14b16c8ed2e" exitCode=0 Oct 30 17:06:34.603583 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:34.603491 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-952bc16e-6865b69dc5-zqc5b" Oct 30 17:06:34.603583 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:34.603527 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-952bc16e-6865b69dc5-zqc5b" event={"ID":"64ddc647-b2ef-445a-a05a-96893865f100","Type":"ContainerDied","Data":"bae4f1bf4fcc3682226d6dd85c5c1a1aa1138ec77ce216842d5ee14b16c8ed2e"} Oct 30 17:06:34.603675 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:34.603592 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-952bc16e-6865b69dc5-zqc5b" event={"ID":"64ddc647-b2ef-445a-a05a-96893865f100","Type":"ContainerDied","Data":"af573360279f9cc89b0ca1411ad7a76ed2324961226366a67822ec39e541e864"} Oct 30 17:06:34.613435 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:34.613419 4522 scope.go:122] "RemoveContainer" containerID="7800ceadf8d9edb241e828e57b3eb76c8826c2e76df077a69939c4a3c40d6b2d" Oct 30 17:06:34.613736 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:06:34.613711 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7800ceadf8d9edb241e828e57b3eb76c8826c2e76df077a69939c4a3c40d6b2d\": container with ID starting with 7800ceadf8d9edb241e828e57b3eb76c8826c2e76df077a69939c4a3c40d6b2d not found: ID does not exist" containerID="7800ceadf8d9edb241e828e57b3eb76c8826c2e76df077a69939c4a3c40d6b2d" Oct 30 17:06:34.613788 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:34.613748 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7800ceadf8d9edb241e828e57b3eb76c8826c2e76df077a69939c4a3c40d6b2d"} err="failed to get container status \"7800ceadf8d9edb241e828e57b3eb76c8826c2e76df077a69939c4a3c40d6b2d\": rpc error: code = NotFound desc = could not find container \"7800ceadf8d9edb241e828e57b3eb76c8826c2e76df077a69939c4a3c40d6b2d\": container with ID starting with 7800ceadf8d9edb241e828e57b3eb76c8826c2e76df077a69939c4a3c40d6b2d not found: ID does not exist" Oct 30 17:06:34.613788 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:34.613774 4522 scope.go:122] "RemoveContainer" containerID="bae4f1bf4fcc3682226d6dd85c5c1a1aa1138ec77ce216842d5ee14b16c8ed2e" Oct 30 17:06:34.625047 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:34.625028 4522 scope.go:122] "RemoveContainer" containerID="bae4f1bf4fcc3682226d6dd85c5c1a1aa1138ec77ce216842d5ee14b16c8ed2e" Oct 30 17:06:34.625412 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:06:34.625386 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bae4f1bf4fcc3682226d6dd85c5c1a1aa1138ec77ce216842d5ee14b16c8ed2e\": container with ID starting with bae4f1bf4fcc3682226d6dd85c5c1a1aa1138ec77ce216842d5ee14b16c8ed2e not found: ID does not exist" containerID="bae4f1bf4fcc3682226d6dd85c5c1a1aa1138ec77ce216842d5ee14b16c8ed2e" Oct 30 17:06:34.625516 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:34.625421 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bae4f1bf4fcc3682226d6dd85c5c1a1aa1138ec77ce216842d5ee14b16c8ed2e"} err="failed to get container status \"bae4f1bf4fcc3682226d6dd85c5c1a1aa1138ec77ce216842d5ee14b16c8ed2e\": rpc error: code = NotFound desc = could not find container \"bae4f1bf4fcc3682226d6dd85c5c1a1aa1138ec77ce216842d5ee14b16c8ed2e\": container with ID starting with bae4f1bf4fcc3682226d6dd85c5c1a1aa1138ec77ce216842d5ee14b16c8ed2e not found: ID does not exist" Oct 30 17:06:34.659876 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:34.659827 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-952bc16e-6865b69dc5-zqc5b"] Oct 30 17:06:34.667607 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:34.667556 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrc9n\" (UniqueName: \"kubernetes.io/projected/64ddc647-b2ef-445a-a05a-96893865f100-kube-api-access-jrc9n\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:06:34.668805 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:34.668670 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlpt7\" (UniqueName: \"kubernetes.io/projected/77f5ee2d-25a4-486c-965a-324c473dd9b7-kube-api-access-qlpt7\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:06:34.668805 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:34.668634 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-952bc16e-6865b69dc5-zqc5b"] Oct 30 17:06:34.701575 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:34.701506 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-952bc16e-5cdf6db644-hm4mt"] Oct 30 17:06:34.708010 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:34.707980 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-952bc16e-5cdf6db644-hm4mt"] Oct 30 17:06:34.776666 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:34.776635 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-952bc16e-n9rtq"] Oct 30 17:06:34.781727 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:34.781697 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-952bc16e-n9rtq"] Oct 30 17:06:34.913386 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:34.913289 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2"] Oct 30 17:06:34.913567 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:34.913544 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2" podUID="959b133a-58d9-4b6f-bece-8dfa089125dc" containerName="istio-proxy" containerID="cri-o://7029ab243df10184d118d08f8142e41204ffa0cbb3537d7470b13baff6bf77de" gracePeriod=30 Oct 30 17:06:36.148654 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:36.148615 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1101d2a5-5794-4cd0-9cd7-93643444f030" path="/var/lib/kubelet/pods/1101d2a5-5794-4cd0-9cd7-93643444f030/volumes" Oct 30 17:06:36.149317 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:36.149281 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64ddc647-b2ef-445a-a05a-96893865f100" path="/var/lib/kubelet/pods/64ddc647-b2ef-445a-a05a-96893865f100/volumes" Oct 30 17:06:36.149802 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:36.149778 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77f5ee2d-25a4-486c-965a-324c473dd9b7" path="/var/lib/kubelet/pods/77f5ee2d-25a4-486c-965a-324c473dd9b7/volumes" Oct 30 17:06:36.730788 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:36.730752 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1cdec0e0-fg5gg"] Oct 30 17:06:36.731496 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:36.731481 4522 cpu_manager.go:414] "containerMap: removing container" podUID="fd03a682-5ace-47b9-b539-9d0ddf3cbfdb" containerName="revoke-keys" Oct 30 17:06:36.731550 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:36.731497 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd03a682-5ace-47b9-b539-9d0ddf3cbfdb" containerName="revoke-keys" Oct 30 17:06:36.731550 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:36.731532 4522 cpu_manager.go:414] "containerMap: removing container" podUID="64ddc647-b2ef-445a-a05a-96893865f100" containerName="payload-pre-processing" Oct 30 17:06:36.731550 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:36.731537 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64ddc647-b2ef-445a-a05a-96893865f100" containerName="payload-pre-processing" Oct 30 17:06:36.731550 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:36.731544 4522 cpu_manager.go:414] "containerMap: removing container" podUID="77f5ee2d-25a4-486c-965a-324c473dd9b7" containerName="payload-processing" Oct 30 17:06:36.731550 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:36.731548 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77f5ee2d-25a4-486c-965a-324c473dd9b7" containerName="payload-processing" Oct 30 17:06:36.731709 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:36.731644 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="64ddc647-b2ef-445a-a05a-96893865f100" containerName="payload-pre-processing" Oct 30 17:06:36.731709 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:36.731652 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="77f5ee2d-25a4-486c-965a-324c473dd9b7" containerName="payload-processing" Oct 30 17:06:36.731709 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:36.731663 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd03a682-5ace-47b9-b539-9d0ddf3cbfdb" containerName="revoke-keys" Oct 30 17:06:36.736999 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:36.736979 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1cdec0e0-fg5gg" Oct 30 17:06:36.758938 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:36.758867 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1cdec0e0-fg5gg"] Oct 30 17:06:36.765798 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:36.765768 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:06:36.765946 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:36.765890 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:06:36.913150 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:36.913112 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1ecf5e7-ed7f-4587-932f-ebfc6aadcaba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1cdec0e0-fg5gg\" (UID: \"d1ecf5e7-ed7f-4587-932f-ebfc6aadcaba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1cdec0e0-fg5gg" Oct 30 17:06:36.913366 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:36.913316 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1ecf5e7-ed7f-4587-932f-ebfc6aadcaba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1cdec0e0-fg5gg\" (UID: \"d1ecf5e7-ed7f-4587-932f-ebfc6aadcaba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1cdec0e0-fg5gg" Oct 30 17:06:36.913451 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:36.913420 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm44k\" (UniqueName: \"kubernetes.io/projected/d1ecf5e7-ed7f-4587-932f-ebfc6aadcaba-kube-api-access-gm44k\") pod \"maas-api-revoke-keys-e2e-migrate-1cdec0e0-fg5gg\" (UID: \"d1ecf5e7-ed7f-4587-932f-ebfc6aadcaba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1cdec0e0-fg5gg" Oct 30 17:06:37.006430 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:37.004121 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4aba8b6b-7cd85f69d8-7skxc"] Oct 30 17:06:37.019769 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:37.018859 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1ecf5e7-ed7f-4587-932f-ebfc6aadcaba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1cdec0e0-fg5gg\" (UID: \"d1ecf5e7-ed7f-4587-932f-ebfc6aadcaba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1cdec0e0-fg5gg" Oct 30 17:06:37.019769 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:37.018999 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm44k\" (UniqueName: \"kubernetes.io/projected/d1ecf5e7-ed7f-4587-932f-ebfc6aadcaba-kube-api-access-gm44k\") pod \"maas-api-revoke-keys-e2e-migrate-1cdec0e0-fg5gg\" (UID: \"d1ecf5e7-ed7f-4587-932f-ebfc6aadcaba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1cdec0e0-fg5gg" Oct 30 17:06:37.019769 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:37.019142 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1ecf5e7-ed7f-4587-932f-ebfc6aadcaba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1cdec0e0-fg5gg\" (UID: \"d1ecf5e7-ed7f-4587-932f-ebfc6aadcaba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1cdec0e0-fg5gg" Oct 30 17:06:37.035564 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:37.035523 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1ecf5e7-ed7f-4587-932f-ebfc6aadcaba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1cdec0e0-fg5gg\" (UID: \"d1ecf5e7-ed7f-4587-932f-ebfc6aadcaba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1cdec0e0-fg5gg" Oct 30 17:06:37.037646 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:37.037601 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4aba8b6b-7cd85f69d8-7skxc" Oct 30 17:06:37.061855 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:37.061826 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-4aba8b6b\"" Oct 30 17:06:37.062281 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:37.062217 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-4aba8b6b-dockercfg-7tp6m\"" Oct 30 17:06:37.066410 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:37.066009 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1ecf5e7-ed7f-4587-932f-ebfc6aadcaba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1cdec0e0-fg5gg\" (UID: \"d1ecf5e7-ed7f-4587-932f-ebfc6aadcaba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1cdec0e0-fg5gg" Oct 30 17:06:37.067369 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:37.067347 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm44k\" (UniqueName: \"kubernetes.io/projected/d1ecf5e7-ed7f-4587-932f-ebfc6aadcaba-kube-api-access-gm44k\") pod \"maas-api-revoke-keys-e2e-migrate-1cdec0e0-fg5gg\" (UID: \"d1ecf5e7-ed7f-4587-932f-ebfc6aadcaba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1cdec0e0-fg5gg" Oct 30 17:06:37.102896 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:37.101497 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4aba8b6b-7cd85f69d8-7skxc"] Oct 30 17:06:37.121555 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:37.121520 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkbv5\" (UniqueName: \"kubernetes.io/projected/4f44bf3d-daea-4511-b385-a39d5ea7396c-kube-api-access-hkbv5\") pod \"payload-processing-e2e-derive-4aba8b6b-7cd85f69d8-7skxc\" (UID: \"4f44bf3d-daea-4511-b385-a39d5ea7396c\") " pod="openshift-ingress/payload-processing-e2e-derive-4aba8b6b-7cd85f69d8-7skxc" Oct 30 17:06:37.122115 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:37.122075 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f44bf3d-daea-4511-b385-a39d5ea7396c-plugins-config-volume\") pod \"payload-processing-e2e-derive-4aba8b6b-7cd85f69d8-7skxc\" (UID: \"4f44bf3d-daea-4511-b385-a39d5ea7396c\") " pod="openshift-ingress/payload-processing-e2e-derive-4aba8b6b-7cd85f69d8-7skxc" Oct 30 17:06:37.223537 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:37.223504 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkbv5\" (UniqueName: \"kubernetes.io/projected/4f44bf3d-daea-4511-b385-a39d5ea7396c-kube-api-access-hkbv5\") pod \"payload-processing-e2e-derive-4aba8b6b-7cd85f69d8-7skxc\" (UID: \"4f44bf3d-daea-4511-b385-a39d5ea7396c\") " pod="openshift-ingress/payload-processing-e2e-derive-4aba8b6b-7cd85f69d8-7skxc" Oct 30 17:06:37.224008 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:37.223663 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f44bf3d-daea-4511-b385-a39d5ea7396c-plugins-config-volume\") pod \"payload-processing-e2e-derive-4aba8b6b-7cd85f69d8-7skxc\" (UID: \"4f44bf3d-daea-4511-b385-a39d5ea7396c\") " pod="openshift-ingress/payload-processing-e2e-derive-4aba8b6b-7cd85f69d8-7skxc" Oct 30 17:06:37.224463 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:37.224443 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f44bf3d-daea-4511-b385-a39d5ea7396c-plugins-config-volume\") pod \"payload-processing-e2e-derive-4aba8b6b-7cd85f69d8-7skxc\" (UID: \"4f44bf3d-daea-4511-b385-a39d5ea7396c\") " pod="openshift-ingress/payload-processing-e2e-derive-4aba8b6b-7cd85f69d8-7skxc" Oct 30 17:06:37.248500 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:37.248460 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkbv5\" (UniqueName: \"kubernetes.io/projected/4f44bf3d-daea-4511-b385-a39d5ea7396c-kube-api-access-hkbv5\") pod \"payload-processing-e2e-derive-4aba8b6b-7cd85f69d8-7skxc\" (UID: \"4f44bf3d-daea-4511-b385-a39d5ea7396c\") " pod="openshift-ingress/payload-processing-e2e-derive-4aba8b6b-7cd85f69d8-7skxc" Oct 30 17:06:37.352173 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:37.352072 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1cdec0e0-fg5gg" Oct 30 17:06:37.365786 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:37.365764 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4aba8b6b-7cd85f69d8-7skxc" Oct 30 17:06:37.684659 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:37.684570 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1cdec0e0-fg5gg"] Oct 30 17:06:37.690676 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:06:37.690645 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1ecf5e7_ed7f_4587_932f_ebfc6aadcaba.slice/crio-1ae876fb89e1f5a7ce23b8dc71248a8d7c111e4dfb93796b442f69694a4dc771 WatchSource:0}: Error finding container 1ae876fb89e1f5a7ce23b8dc71248a8d7c111e4dfb93796b442f69694a4dc771: Status 404 returned error can't find the container with id 1ae876fb89e1f5a7ce23b8dc71248a8d7c111e4dfb93796b442f69694a4dc771 Oct 30 17:06:37.708085 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:37.708055 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4aba8b6b-7cd85f69d8-7skxc"] Oct 30 17:06:37.710602 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:06:37.710578 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f44bf3d_daea_4511_b385_a39d5ea7396c.slice/crio-3a4fb2850086a8e4faac30ad6d5734828f4b4006113247b1b9d907ac97fea688 WatchSource:0}: Error finding container 3a4fb2850086a8e4faac30ad6d5734828f4b4006113247b1b9d907ac97fea688: Status 404 returned error can't find the container with id 3a4fb2850086a8e4faac30ad6d5734828f4b4006113247b1b9d907ac97fea688 Oct 30 17:06:37.922996 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:37.922958 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11c7f0ab-m929r"] Oct 30 17:06:37.937994 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:37.937925 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c7f0ab-m929r" Oct 30 17:06:37.947139 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:37.947111 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11c7f0ab-m929r"] Oct 30 17:06:38.034787 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:38.034749 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15a438c9-99cvt"] Oct 30 17:06:38.039980 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:38.039950 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15a438c9-99cvt"] Oct 30 17:06:38.040805 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:38.040784 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db670098-c2e3-49bd-a107-8a9e72610845-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11c7f0ab-m929r\" (UID: \"db670098-c2e3-49bd-a107-8a9e72610845\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c7f0ab-m929r" Oct 30 17:06:38.040886 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:38.040868 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db670098-c2e3-49bd-a107-8a9e72610845-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11c7f0ab-m929r\" (UID: \"db670098-c2e3-49bd-a107-8a9e72610845\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c7f0ab-m929r" Oct 30 17:06:38.040973 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:38.040954 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c2ld\" (UniqueName: \"kubernetes.io/projected/db670098-c2e3-49bd-a107-8a9e72610845-kube-api-access-8c2ld\") pod \"maas-api-revoke-sub-11c7f0ab-m929r\" (UID: \"db670098-c2e3-49bd-a107-8a9e72610845\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c7f0ab-m929r" Oct 30 17:06:38.143096 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:38.143060 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db670098-c2e3-49bd-a107-8a9e72610845-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11c7f0ab-m929r\" (UID: \"db670098-c2e3-49bd-a107-8a9e72610845\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c7f0ab-m929r" Oct 30 17:06:38.143251 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:38.143170 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db670098-c2e3-49bd-a107-8a9e72610845-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11c7f0ab-m929r\" (UID: \"db670098-c2e3-49bd-a107-8a9e72610845\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c7f0ab-m929r" Oct 30 17:06:38.143251 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:38.143204 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c2ld\" (UniqueName: \"kubernetes.io/projected/db670098-c2e3-49bd-a107-8a9e72610845-kube-api-access-8c2ld\") pod \"maas-api-revoke-sub-11c7f0ab-m929r\" (UID: \"db670098-c2e3-49bd-a107-8a9e72610845\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c7f0ab-m929r" Oct 30 17:06:38.144038 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:38.144015 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db670098-c2e3-49bd-a107-8a9e72610845-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11c7f0ab-m929r\" (UID: \"db670098-c2e3-49bd-a107-8a9e72610845\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c7f0ab-m929r" Oct 30 17:06:38.145632 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:38.145605 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db670098-c2e3-49bd-a107-8a9e72610845-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11c7f0ab-m929r\" (UID: \"db670098-c2e3-49bd-a107-8a9e72610845\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c7f0ab-m929r" Oct 30 17:06:38.147736 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:38.147715 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9b46f5d-5403-4847-b498-bfae1b880d6a" path="/var/lib/kubelet/pods/a9b46f5d-5403-4847-b498-bfae1b880d6a/volumes" Oct 30 17:06:38.168369 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:38.168299 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c2ld\" (UniqueName: \"kubernetes.io/projected/db670098-c2e3-49bd-a107-8a9e72610845-kube-api-access-8c2ld\") pod \"maas-api-revoke-sub-11c7f0ab-m929r\" (UID: \"db670098-c2e3-49bd-a107-8a9e72610845\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c7f0ab-m929r" Oct 30 17:06:38.258493 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:38.258459 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c7f0ab-m929r" Oct 30 17:06:38.565571 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:06:38.565530 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb670098_c2e3_49bd_a107_8a9e72610845.slice/crio-735e5c2115cc87d228ef6b6f5e0be2d63ee8b674a6c1d72657312b84ab72faa5 WatchSource:0}: Error finding container 735e5c2115cc87d228ef6b6f5e0be2d63ee8b674a6c1d72657312b84ab72faa5: Status 404 returned error can't find the container with id 735e5c2115cc87d228ef6b6f5e0be2d63ee8b674a6c1d72657312b84ab72faa5 Oct 30 17:06:38.568770 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:38.568746 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11c7f0ab-m929r"] Oct 30 17:06:38.652010 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:38.651971 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1cdec0e0-fg5gg" event={"ID":"d1ecf5e7-ed7f-4587-932f-ebfc6aadcaba","Type":"ContainerStarted","Data":"5cfac171006fcc5c15d23fb5642ea21810ea480d4c6572b094edbdfee563c594"} Oct 30 17:06:38.652186 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:38.652031 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1cdec0e0-fg5gg" event={"ID":"d1ecf5e7-ed7f-4587-932f-ebfc6aadcaba","Type":"ContainerStarted","Data":"1ae876fb89e1f5a7ce23b8dc71248a8d7c111e4dfb93796b442f69694a4dc771"} Oct 30 17:06:38.653999 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:38.653965 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4aba8b6b-7cd85f69d8-7skxc" event={"ID":"4f44bf3d-daea-4511-b385-a39d5ea7396c","Type":"ContainerStarted","Data":"eeb62ad8089827ae8899445752302f02bb9ff6a5be1438d4c8511918d51e3065"} Oct 30 17:06:38.654130 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:38.654022 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4aba8b6b-7cd85f69d8-7skxc" event={"ID":"4f44bf3d-daea-4511-b385-a39d5ea7396c","Type":"ContainerStarted","Data":"3a4fb2850086a8e4faac30ad6d5734828f4b4006113247b1b9d907ac97fea688"} Oct 30 17:06:38.654130 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:38.654048 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-4aba8b6b-7cd85f69d8-7skxc" Oct 30 17:06:38.655653 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:38.655617 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c7f0ab-m929r" event={"ID":"db670098-c2e3-49bd-a107-8a9e72610845","Type":"ContainerStarted","Data":"735e5c2115cc87d228ef6b6f5e0be2d63ee8b674a6c1d72657312b84ab72faa5"} Oct 30 17:06:38.695146 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:38.695097 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1cdec0e0-fg5gg" podStartSLOduration=2.6950863739999997 podStartE2EDuration="2.695086374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:06:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:06:38.69398397 +0000 UTC m=+1172.694738871" watchObservedRunningTime="2026-10-30 17:06:38.695086374 +0000 UTC m=+1172.695841274" Oct 30 17:06:38.740870 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:38.740803 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-4aba8b6b-7cd85f69d8-7skxc" podStartSLOduration=2.740787127 podStartE2EDuration="2.740787127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:06:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:06:38.736217289 +0000 UTC m=+1172.736972218" watchObservedRunningTime="2026-10-30 17:06:38.740787127 +0000 UTC m=+1172.741542029" Oct 30 17:06:39.538941 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:39.538909 4522 patch_prober.go:29] interesting pod/e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:06:39.539317 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:39.538962 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2" podUID="959b133a-58d9-4b6f-bece-8dfa089125dc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:06:39.665439 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:39.665407 4522 generic.go:356] "Generic (PLEG): container finished" podID="db670098-c2e3-49bd-a107-8a9e72610845" containerID="c0b5faac0c01ceb47c78d6f3d72d9a9a222a2edc3bca5f5834424336402ce6bc" exitCode=0 Oct 30 17:06:39.665649 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:39.665494 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c7f0ab-m929r" event={"ID":"db670098-c2e3-49bd-a107-8a9e72610845","Type":"ContainerDied","Data":"c0b5faac0c01ceb47c78d6f3d72d9a9a222a2edc3bca5f5834424336402ce6bc"} Oct 30 17:06:39.668201 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:39.668169 4522 generic.go:356] "Generic (PLEG): container finished" podID="d1ecf5e7-ed7f-4587-932f-ebfc6aadcaba" containerID="5cfac171006fcc5c15d23fb5642ea21810ea480d4c6572b094edbdfee563c594" exitCode=0 Oct 30 17:06:39.668367 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:39.668213 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1cdec0e0-fg5gg" event={"ID":"d1ecf5e7-ed7f-4587-932f-ebfc6aadcaba","Type":"ContainerDied","Data":"5cfac171006fcc5c15d23fb5642ea21810ea480d4c6572b094edbdfee563c594"} Oct 30 17:06:40.202731 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:40.202704 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2" Oct 30 17:06:40.279318 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:40.279263 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/959b133a-58d9-4b6f-bece-8dfa089125dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/959b133a-58d9-4b6f-bece-8dfa089125dc-workload-certs\") pod \"959b133a-58d9-4b6f-bece-8dfa089125dc\" (UID: \"959b133a-58d9-4b6f-bece-8dfa089125dc\") " Oct 30 17:06:40.279509 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:40.279395 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/959b133a-58d9-4b6f-bece-8dfa089125dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/959b133a-58d9-4b6f-bece-8dfa089125dc-credential-socket\") pod \"959b133a-58d9-4b6f-bece-8dfa089125dc\" (UID: \"959b133a-58d9-4b6f-bece-8dfa089125dc\") " Oct 30 17:06:40.279578 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:40.279528 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/959b133a-58d9-4b6f-bece-8dfa089125dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/959b133a-58d9-4b6f-bece-8dfa089125dc-istio-data\") pod \"959b133a-58d9-4b6f-bece-8dfa089125dc\" (UID: \"959b133a-58d9-4b6f-bece-8dfa089125dc\") " Oct 30 17:06:40.279640 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:40.279576 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/959b133a-58d9-4b6f-bece-8dfa089125dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/959b133a-58d9-4b6f-bece-8dfa089125dc-istiod-ca-cert\") pod \"959b133a-58d9-4b6f-bece-8dfa089125dc\" (UID: \"959b133a-58d9-4b6f-bece-8dfa089125dc\") " Oct 30 17:06:40.279640 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:40.279618 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/959b133a-58d9-4b6f-bece-8dfa089125dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/959b133a-58d9-4b6f-bece-8dfa089125dc-istio-envoy\") pod \"959b133a-58d9-4b6f-bece-8dfa089125dc\" (UID: \"959b133a-58d9-4b6f-bece-8dfa089125dc\") " Oct 30 17:06:40.280158 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:40.279987 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/959b133a-58d9-4b6f-bece-8dfa089125dc-istio-data" pod "959b133a-58d9-4b6f-bece-8dfa089125dc" (UID: "959b133a-58d9-4b6f-bece-8dfa089125dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:06:40.280698 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:40.280677 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/959b133a-58d9-4b6f-bece-8dfa089125dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/959b133a-58d9-4b6f-bece-8dfa089125dc-istio-podinfo\") pod \"959b133a-58d9-4b6f-bece-8dfa089125dc\" (UID: \"959b133a-58d9-4b6f-bece-8dfa089125dc\") " Oct 30 17:06:40.280806 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:40.280726 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/959b133a-58d9-4b6f-bece-8dfa089125dc-istio-token\" (UniqueName: \"kubernetes.io/projected/959b133a-58d9-4b6f-bece-8dfa089125dc-istio-token\") pod \"959b133a-58d9-4b6f-bece-8dfa089125dc\" (UID: \"959b133a-58d9-4b6f-bece-8dfa089125dc\") " Oct 30 17:06:40.280948 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:40.280921 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/959b133a-58d9-4b6f-bece-8dfa089125dc-istiod-ca-cert" pod "959b133a-58d9-4b6f-bece-8dfa089125dc" (UID: "959b133a-58d9-4b6f-bece-8dfa089125dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:06:40.281007 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:40.280974 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/959b133a-58d9-4b6f-bece-8dfa089125dc-kube-api-access-qlswc\" (UniqueName: \"kubernetes.io/projected/959b133a-58d9-4b6f-bece-8dfa089125dc-kube-api-access-qlswc\") pod \"959b133a-58d9-4b6f-bece-8dfa089125dc\" (UID: \"959b133a-58d9-4b6f-bece-8dfa089125dc\") " Oct 30 17:06:40.281058 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:40.281030 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/959b133a-58d9-4b6f-bece-8dfa089125dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/959b133a-58d9-4b6f-bece-8dfa089125dc-workload-socket\") pod \"959b133a-58d9-4b6f-bece-8dfa089125dc\" (UID: \"959b133a-58d9-4b6f-bece-8dfa089125dc\") " Oct 30 17:06:40.281799 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:40.281775 4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/959b133a-58d9-4b6f-bece-8dfa089125dc-istio-data\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:06:40.281889 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:40.281805 4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/959b133a-58d9-4b6f-bece-8dfa089125dc-istiod-ca-cert\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:06:40.281943 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:40.281890 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/959b133a-58d9-4b6f-bece-8dfa089125dc-workload-socket" pod "959b133a-58d9-4b6f-bece-8dfa089125dc" (UID: "959b133a-58d9-4b6f-bece-8dfa089125dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:06:40.282393 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:40.282368 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/959b133a-58d9-4b6f-bece-8dfa089125dc-credential-socket" pod "959b133a-58d9-4b6f-bece-8dfa089125dc" (UID: "959b133a-58d9-4b6f-bece-8dfa089125dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:06:40.282485 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:40.282432 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/959b133a-58d9-4b6f-bece-8dfa089125dc-workload-certs" pod "959b133a-58d9-4b6f-bece-8dfa089125dc" (UID: "959b133a-58d9-4b6f-bece-8dfa089125dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:06:40.284170 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:40.284141 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/959b133a-58d9-4b6f-bece-8dfa089125dc-istio-token" pod "959b133a-58d9-4b6f-bece-8dfa089125dc" (UID: "959b133a-58d9-4b6f-bece-8dfa089125dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:06:40.284433 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:40.284397 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/959b133a-58d9-4b6f-bece-8dfa089125dc-kube-api-access-qlswc" pod "959b133a-58d9-4b6f-bece-8dfa089125dc" (UID: "959b133a-58d9-4b6f-bece-8dfa089125dc"). InnerVolumeSpecName "kube-api-access-qlswc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:06:40.285086 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:40.285041 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/959b133a-58d9-4b6f-bece-8dfa089125dc-istio-podinfo" pod "959b133a-58d9-4b6f-bece-8dfa089125dc" (UID: "959b133a-58d9-4b6f-bece-8dfa089125dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:06:40.286515 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:40.286485 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/959b133a-58d9-4b6f-bece-8dfa089125dc-istio-envoy" pod "959b133a-58d9-4b6f-bece-8dfa089125dc" (UID: "959b133a-58d9-4b6f-bece-8dfa089125dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:06:40.383811 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:40.383762 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlswc\" (UniqueName: \"kubernetes.io/projected/959b133a-58d9-4b6f-bece-8dfa089125dc-kube-api-access-qlswc\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:06:40.383811 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:40.383805 4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/959b133a-58d9-4b6f-bece-8dfa089125dc-workload-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:06:40.383811 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:40.383818 4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/959b133a-58d9-4b6f-bece-8dfa089125dc-workload-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:06:40.384060 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:40.383831 4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/959b133a-58d9-4b6f-bece-8dfa089125dc-credential-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:06:40.384060 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:40.383843 4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/959b133a-58d9-4b6f-bece-8dfa089125dc-istio-envoy\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:06:40.384060 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:40.383854 4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/959b133a-58d9-4b6f-bece-8dfa089125dc-istio-podinfo\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:06:40.384060 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:40.383864 4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/959b133a-58d9-4b6f-bece-8dfa089125dc-istio-token\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:06:40.684329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:40.684246 4522 generic.go:356] "Generic (PLEG): container finished" podID="959b133a-58d9-4b6f-bece-8dfa089125dc" containerID="7029ab243df10184d118d08f8142e41204ffa0cbb3537d7470b13baff6bf77de" exitCode=0 Oct 30 17:06:40.684832 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:40.684285 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2" event={"ID":"959b133a-58d9-4b6f-bece-8dfa089125dc","Type":"ContainerDied","Data":"7029ab243df10184d118d08f8142e41204ffa0cbb3537d7470b13baff6bf77de"} Oct 30 17:06:40.684832 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:40.684399 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2" event={"ID":"959b133a-58d9-4b6f-bece-8dfa089125dc","Type":"ContainerDied","Data":"1eba76310537eda16d24264a60640dc801df23236b7117fc2dbd943d83594819"} Oct 30 17:06:40.684832 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:40.684441 4522 scope.go:122] "RemoveContainer" containerID="7029ab243df10184d118d08f8142e41204ffa0cbb3537d7470b13baff6bf77de" Oct 30 17:06:40.689138 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:40.689114 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2" Oct 30 17:06:40.737635 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:40.737607 4522 scope.go:122] "RemoveContainer" containerID="7029ab243df10184d118d08f8142e41204ffa0cbb3537d7470b13baff6bf77de" Oct 30 17:06:40.738047 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:06:40.738018 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7029ab243df10184d118d08f8142e41204ffa0cbb3537d7470b13baff6bf77de\": container with ID starting with 7029ab243df10184d118d08f8142e41204ffa0cbb3537d7470b13baff6bf77de not found: ID does not exist" containerID="7029ab243df10184d118d08f8142e41204ffa0cbb3537d7470b13baff6bf77de" Oct 30 17:06:40.738126 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:40.738066 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7029ab243df10184d118d08f8142e41204ffa0cbb3537d7470b13baff6bf77de"} err="failed to get container status \"7029ab243df10184d118d08f8142e41204ffa0cbb3537d7470b13baff6bf77de\": rpc error: code = NotFound desc = could not find container \"7029ab243df10184d118d08f8142e41204ffa0cbb3537d7470b13baff6bf77de\": container with ID starting with 7029ab243df10184d118d08f8142e41204ffa0cbb3537d7470b13baff6bf77de not found: ID does not exist" Oct 30 17:06:40.859248 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:40.852662 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2"] Oct 30 17:06:40.909058 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:40.906171 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-k79b2"] Oct 30 17:06:40.961379 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:40.960695 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c7f0ab-m929r" Oct 30 17:06:41.009755 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:41.009729 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db670098-c2e3-49bd-a107-8a9e72610845-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db670098-c2e3-49bd-a107-8a9e72610845-maas-api-xks-ca\") pod \"db670098-c2e3-49bd-a107-8a9e72610845\" (UID: \"db670098-c2e3-49bd-a107-8a9e72610845\") " Oct 30 17:06:41.009906 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:41.009767 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db670098-c2e3-49bd-a107-8a9e72610845-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db670098-c2e3-49bd-a107-8a9e72610845-maas-api-openshift-ca\") pod \"db670098-c2e3-49bd-a107-8a9e72610845\" (UID: \"db670098-c2e3-49bd-a107-8a9e72610845\") " Oct 30 17:06:41.009980 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:41.009966 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db670098-c2e3-49bd-a107-8a9e72610845-kube-api-access-8c2ld\" (UniqueName: \"kubernetes.io/projected/db670098-c2e3-49bd-a107-8a9e72610845-kube-api-access-8c2ld\") pod \"db670098-c2e3-49bd-a107-8a9e72610845\" (UID: \"db670098-c2e3-49bd-a107-8a9e72610845\") " Oct 30 17:06:41.010488 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:41.010443 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db670098-c2e3-49bd-a107-8a9e72610845-maas-api-openshift-ca" pod "db670098-c2e3-49bd-a107-8a9e72610845" (UID: "db670098-c2e3-49bd-a107-8a9e72610845"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:06:41.010921 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:41.010896 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db670098-c2e3-49bd-a107-8a9e72610845-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:06:41.013259 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:41.013231 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db670098-c2e3-49bd-a107-8a9e72610845-kube-api-access-8c2ld" pod "db670098-c2e3-49bd-a107-8a9e72610845" (UID: "db670098-c2e3-49bd-a107-8a9e72610845"). InnerVolumeSpecName "kube-api-access-8c2ld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:06:41.022833 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:41.022467 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1cdec0e0-fg5gg" Oct 30 17:06:41.024046 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:41.024019 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db670098-c2e3-49bd-a107-8a9e72610845-maas-api-xks-ca" pod "db670098-c2e3-49bd-a107-8a9e72610845" (UID: "db670098-c2e3-49bd-a107-8a9e72610845"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:06:41.111839 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:41.111792 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1ecf5e7-ed7f-4587-932f-ebfc6aadcaba-kube-api-access-gm44k\" (UniqueName: \"kubernetes.io/projected/d1ecf5e7-ed7f-4587-932f-ebfc6aadcaba-kube-api-access-gm44k\") pod \"d1ecf5e7-ed7f-4587-932f-ebfc6aadcaba\" (UID: \"d1ecf5e7-ed7f-4587-932f-ebfc6aadcaba\") " Oct 30 17:06:41.112010 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:41.111989 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1ecf5e7-ed7f-4587-932f-ebfc6aadcaba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1ecf5e7-ed7f-4587-932f-ebfc6aadcaba-maas-api-openshift-ca\") pod \"d1ecf5e7-ed7f-4587-932f-ebfc6aadcaba\" (UID: \"d1ecf5e7-ed7f-4587-932f-ebfc6aadcaba\") " Oct 30 17:06:41.112139 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:41.112121 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1ecf5e7-ed7f-4587-932f-ebfc6aadcaba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1ecf5e7-ed7f-4587-932f-ebfc6aadcaba-maas-api-xks-ca\") pod \"d1ecf5e7-ed7f-4587-932f-ebfc6aadcaba\" (UID: \"d1ecf5e7-ed7f-4587-932f-ebfc6aadcaba\") " Oct 30 17:06:41.112595 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:41.112568 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1ecf5e7-ed7f-4587-932f-ebfc6aadcaba-maas-api-openshift-ca" pod "d1ecf5e7-ed7f-4587-932f-ebfc6aadcaba" (UID: "d1ecf5e7-ed7f-4587-932f-ebfc6aadcaba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:06:41.112936 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:41.112915 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1ecf5e7-ed7f-4587-932f-ebfc6aadcaba-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:06:41.113109 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:41.113092 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db670098-c2e3-49bd-a107-8a9e72610845-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:06:41.113214 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:41.113198 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c2ld\" (UniqueName: \"kubernetes.io/projected/db670098-c2e3-49bd-a107-8a9e72610845-kube-api-access-8c2ld\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:06:41.114680 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:41.114656 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1ecf5e7-ed7f-4587-932f-ebfc6aadcaba-kube-api-access-gm44k" pod "d1ecf5e7-ed7f-4587-932f-ebfc6aadcaba" (UID: "d1ecf5e7-ed7f-4587-932f-ebfc6aadcaba"). InnerVolumeSpecName "kube-api-access-gm44k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:06:41.115353 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:41.115289 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1ecf5e7-ed7f-4587-932f-ebfc6aadcaba-maas-api-xks-ca" pod "d1ecf5e7-ed7f-4587-932f-ebfc6aadcaba" (UID: "d1ecf5e7-ed7f-4587-932f-ebfc6aadcaba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:06:41.214587 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:41.214550 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gm44k\" (UniqueName: \"kubernetes.io/projected/d1ecf5e7-ed7f-4587-932f-ebfc6aadcaba-kube-api-access-gm44k\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:06:41.214587 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:41.214583 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1ecf5e7-ed7f-4587-932f-ebfc6aadcaba-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:06:41.699910 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:41.699822 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c7f0ab-m929r" Oct 30 17:06:41.699910 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:41.699830 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c7f0ab-m929r" event={"ID":"db670098-c2e3-49bd-a107-8a9e72610845","Type":"ContainerDied","Data":"735e5c2115cc87d228ef6b6f5e0be2d63ee8b674a6c1d72657312b84ab72faa5"} Oct 30 17:06:41.699910 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:41.699884 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="735e5c2115cc87d228ef6b6f5e0be2d63ee8b674a6c1d72657312b84ab72faa5" Oct 30 17:06:41.702190 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:41.701412 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1cdec0e0-fg5gg" event={"ID":"d1ecf5e7-ed7f-4587-932f-ebfc6aadcaba","Type":"ContainerDied","Data":"1ae876fb89e1f5a7ce23b8dc71248a8d7c111e4dfb93796b442f69694a4dc771"} Oct 30 17:06:41.702190 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:41.701455 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ae876fb89e1f5a7ce23b8dc71248a8d7c111e4dfb93796b442f69694a4dc771" Oct 30 17:06:41.702190 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:41.701755 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1cdec0e0-fg5gg" Oct 30 17:06:42.198242 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:42.198202 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="959b133a-58d9-4b6f-bece-8dfa089125dc" path="/var/lib/kubelet/pods/959b133a-58d9-4b6f-bece-8dfa089125dc/volumes" Oct 30 17:06:43.152330 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:06:43.151547 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:06:43.872021 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:43.871422 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" podUID="2f1e3bbb-ab1c-455d-a05d-d62f72c5d009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:06:43.988186 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:43.987728 4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-04248e/distinct-w1-04248e-kserve-5859d49965-kgq46" podUID="8dda5ac4-30dd-432e-81eb-cb6d39800628" containerName="main" probeResult="failure" output="Get \"https://10.134.0.108:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:06:43.993841 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:43.993422 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" podUID="75321c5d-8bf8-45c6-8811-81981fe3cad4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 17:06:44.062332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:44.061937 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-637f442f-h8ckw"] Oct 30 17:06:44.121324 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:44.107410 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" podUID="9bfbc6d3-91fd-41a7-9585-40a8f252f743" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:06:44.121324 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:44.107672 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/distinct-w1-04248e-kserve-5859d49965-kgq46" podUID="8dda5ac4-30dd-432e-81eb-cb6d39800628" containerName="main" probeResult="failure" output="Get \"https://10.134.0.108:8000/ready\": context deadline exceeded" Oct 30 17:06:44.121324 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:44.108847 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" podUID="0b832781-3d41-4496-aa86-e4d92792e590" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:06:44.133396 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:44.129986 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-637f442f-h8ckw"] Oct 30 17:06:44.251331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:44.251144 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c753be2-3052-46c0-844c-f7eebc0cbdcf" path="/var/lib/kubelet/pods/6c753be2-3052-46c0-844c-f7eebc0cbdcf/volumes" Oct 30 17:06:44.286332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:44.277286 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb" podUID="d86fded1-2e04-4d73-bb22-ea0d9a61b961" containerName="main" probeResult="failure" output="Get \"https://10.134.0.127:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:06:44.383331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:44.382405 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/distinct-w2-b4f406-kserve-5db4cc6969-7n846" podUID="b8810dbc-b769-46a4-90b9-72a68592bc1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:06:44.449388 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:44.448454 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" podUID="a3e5e732-2c67-4377-9755-e2178bf33772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:06:44.449584 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:44.449504 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" podUID="c9ad71b7-ad90-4c8c-bf27-0f10b23f1593" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:06:44.629555 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:44.629514 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" podUID="4b4ad752-e053-4cea-a3bc-fe2be6f92021" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Oct 30 17:06:44.659281 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:44.657670 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" podUID="8a6a85ad-c944-450f-b0bb-e3b64b0f56f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 17:06:44.659281 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:44.657982 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" podUID="43ca764d-ae0e-4858-9841-5bbd0ffdbb85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 17:06:44.794391 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:44.789557 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq" podUID="f18303fb-6d27-4ff1-9cc9-ed032e43d098" containerName="main" probeResult="failure" output="Get \"https://10.134.0.120:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:06:44.874432 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:44.874392 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" podUID="2f1e3bbb-ab1c-455d-a05d-d62f72c5d009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:06:44.875243 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:44.875217 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" Oct 30 17:06:45.031677 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:45.031630 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" podUID="4360e9dc-c001-4102-8af6-b55b692fcf2c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:06:45.033887 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:45.031936 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" podUID="75321c5d-8bf8-45c6-8811-81981fe3cad4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:06:45.034519 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:45.034498 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" Oct 30 17:06:45.118415 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:45.111615 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" podUID="0b832781-3d41-4496-aa86-e4d92792e590" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 17:06:45.118415 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:45.111974 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" podUID="9bfbc6d3-91fd-41a7-9585-40a8f252f743" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 30 17:06:45.118415 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:45.112286 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" Oct 30 17:06:45.118415 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:45.112365 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" Oct 30 17:06:45.269231 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:45.269169 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/distinct-w1-04248e-kserve-5859d49965-kgq46" podUID="8dda5ac4-30dd-432e-81eb-cb6d39800628" containerName="main" probeResult="failure" output="Get \"https://10.134.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:06:45.554927 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:45.538572 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04248e/distinct-w1-04248e-kserve-5859d49965-kgq46" Oct 30 17:06:45.554927 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:45.388681 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/distinct-w2-b4f406-kserve-5db4cc6969-7n846" podUID="b8810dbc-b769-46a4-90b9-72a68592bc1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:06:45.583328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:45.572653 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b4f406/distinct-w2-b4f406-kserve-5db4cc6969-7n846" Oct 30 17:06:45.583328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:45.299119 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb" podUID="d86fded1-2e04-4d73-bb22-ea0d9a61b961" containerName="main" probeResult="failure" output="Get \"https://10.134.0.127:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:06:45.583328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:45.582682 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b4f406/distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb" Oct 30 17:06:45.684290 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:45.680129 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" podUID="a3e5e732-2c67-4377-9755-e2178bf33772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 17:06:45.684290 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:45.680453 4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" podUID="43ca764d-ae0e-4858-9841-5bbd0ffdbb85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:06:45.684290 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:45.680545 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" podUID="4b4ad752-e053-4cea-a3bc-fe2be6f92021" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:06:45.684290 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:45.680826 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" Oct 30 17:06:45.684290 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:45.680903 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" Oct 30 17:06:45.684290 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:45.683849 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" podUID="43ca764d-ae0e-4858-9841-5bbd0ffdbb85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:06:45.776363 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:45.699138 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" Oct 30 17:06:45.776363 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:45.741502 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" podUID="c9ad71b7-ad90-4c8c-bf27-0f10b23f1593" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 17:06:45.776363 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:45.775172 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" Oct 30 17:06:45.776363 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:45.775252 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" Oct 30 17:06:45.776752 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:45.776476 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" podUID="8a6a85ad-c944-450f-b0bb-e3b64b0f56f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 17:06:45.780759 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:45.780716 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" Oct 30 17:06:45.786332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:45.784461 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b4f406/distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb" Oct 30 17:06:45.979772 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:45.979741 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" Oct 30 17:06:46.009860 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:46.001873 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" Oct 30 17:06:46.009860 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:46.003599 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" Oct 30 17:06:46.009860 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:46.006073 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq" podUID="f18303fb-6d27-4ff1-9cc9-ed032e43d098" containerName="main" probeResult="failure" output="Get \"https://10.134.0.120:8000/ready\": context deadline exceeded" Oct 30 17:06:46.030334 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:46.026235 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04248e/distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq" Oct 30 17:06:46.034596 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:46.034563 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" podUID="2f1e3bbb-ab1c-455d-a05d-d62f72c5d009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 17:06:46.037431 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:46.037405 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" Oct 30 17:06:46.051237 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:46.051211 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04248e/distinct-w1-04248e-kserve-5859d49965-kgq46" Oct 30 17:06:46.052737 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:46.052515 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" Oct 30 17:06:46.057753 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:46.055776 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04248e/distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq" Oct 30 17:06:46.072567 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:46.072547 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" Oct 30 17:06:46.075651 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:46.075610 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" Oct 30 17:06:46.086511 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:46.086478 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" Oct 30 17:06:46.089473 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:46.089449 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b4f406/distinct-w2-b4f406-kserve-5db4cc6969-7n846" Oct 30 17:06:46.887413 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:46.887377 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1cdec0e0-fb59dd5bb-g5xq4"] Oct 30 17:06:46.948183 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:46.948143 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-1cdec0e0-798d84688c-mtx4b"] Oct 30 17:06:49.676075 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:49.675910 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-4aba8b6b-7cd85f69d8-7skxc" Oct 30 17:06:51.888028 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:51.887976 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-1cdec0e0-fb59dd5bb-g5xq4" podUID="cf5eb021-2e73-46db-a0db-ab5dd73076e1" containerName="payload-processing" containerID="cri-o://57ea9f6405603940c16ca6f49d47e4e63f631c3520db909248211538b5218d61" gracePeriod=25 Oct 30 17:06:51.949446 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:51.949379 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1cdec0e0-798d84688c-mtx4b" podUID="780734db-39cc-4c52-bd39-efefc4ad7e72" containerName="payload-pre-processing" containerID="cri-o://c6fcbb8259641d6b40e6faceea46cd958b839e7cde4a18b01c7897d9feacbe38" gracePeriod=25 Oct 30 17:06:51.975032 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:51.975000 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efac793c-fcjzh"] Oct 30 17:06:51.976155 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:51.976131 4522 cpu_manager.go:414] "containerMap: removing container" podUID="db670098-c2e3-49bd-a107-8a9e72610845" containerName="revoke-keys" Oct 30 17:06:51.976202 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:51.976162 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db670098-c2e3-49bd-a107-8a9e72610845" containerName="revoke-keys" Oct 30 17:06:51.976202 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:51.976182 4522 cpu_manager.go:414] "containerMap: removing container" podUID="d1ecf5e7-ed7f-4587-932f-ebfc6aadcaba" containerName="revoke-keys" Oct 30 17:06:51.976202 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:51.976190 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1ecf5e7-ed7f-4587-932f-ebfc6aadcaba" containerName="revoke-keys" Oct 30 17:06:51.976299 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:51.976270 4522 cpu_manager.go:414] "containerMap: removing container" podUID="959b133a-58d9-4b6f-bece-8dfa089125dc" containerName="istio-proxy" Oct 30 17:06:51.976299 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:51.976279 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="959b133a-58d9-4b6f-bece-8dfa089125dc" containerName="istio-proxy" Oct 30 17:06:51.976519 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:51.976502 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="959b133a-58d9-4b6f-bece-8dfa089125dc" containerName="istio-proxy" Oct 30 17:06:51.976556 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:51.976534 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="db670098-c2e3-49bd-a107-8a9e72610845" containerName="revoke-keys" Oct 30 17:06:51.976556 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:51.976544 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1ecf5e7-ed7f-4587-932f-ebfc6aadcaba" containerName="revoke-keys" Oct 30 17:06:51.981436 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:51.981417 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efac793c-fcjzh" Oct 30 17:06:51.998757 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:51.998727 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efac793c-fcjzh"] Oct 30 17:06:52.006182 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:52.006152 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:06:52.006182 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:52.006146 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:06:52.034572 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:52.034534 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2a2c63d-154f-4950-95d6-55694fedd0c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efac793c-fcjzh\" (UID: \"a2a2c63d-154f-4950-95d6-55694fedd0c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efac793c-fcjzh" Oct 30 17:06:52.034738 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:52.034686 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtq72\" (UniqueName: \"kubernetes.io/projected/a2a2c63d-154f-4950-95d6-55694fedd0c0-kube-api-access-rtq72\") pod \"maas-api-revoke-sub-efac793c-fcjzh\" (UID: \"a2a2c63d-154f-4950-95d6-55694fedd0c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efac793c-fcjzh" Oct 30 17:06:52.034822 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:52.034791 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2a2c63d-154f-4950-95d6-55694fedd0c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efac793c-fcjzh\" (UID: \"a2a2c63d-154f-4950-95d6-55694fedd0c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efac793c-fcjzh" Oct 30 17:06:52.143399 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:52.136732 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2a2c63d-154f-4950-95d6-55694fedd0c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efac793c-fcjzh\" (UID: \"a2a2c63d-154f-4950-95d6-55694fedd0c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efac793c-fcjzh" Oct 30 17:06:52.143399 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:52.136843 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtq72\" (UniqueName: \"kubernetes.io/projected/a2a2c63d-154f-4950-95d6-55694fedd0c0-kube-api-access-rtq72\") pod \"maas-api-revoke-sub-efac793c-fcjzh\" (UID: \"a2a2c63d-154f-4950-95d6-55694fedd0c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efac793c-fcjzh" Oct 30 17:06:52.143399 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:52.136932 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2a2c63d-154f-4950-95d6-55694fedd0c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efac793c-fcjzh\" (UID: \"a2a2c63d-154f-4950-95d6-55694fedd0c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efac793c-fcjzh" Oct 30 17:06:52.146260 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:52.144971 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2a2c63d-154f-4950-95d6-55694fedd0c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efac793c-fcjzh\" (UID: \"a2a2c63d-154f-4950-95d6-55694fedd0c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efac793c-fcjzh" Oct 30 17:06:52.169006 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:52.168486 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtq72\" (UniqueName: \"kubernetes.io/projected/a2a2c63d-154f-4950-95d6-55694fedd0c0-kube-api-access-rtq72\") pod \"maas-api-revoke-sub-efac793c-fcjzh\" (UID: \"a2a2c63d-154f-4950-95d6-55694fedd0c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efac793c-fcjzh" Oct 30 17:06:52.173118 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:52.173048 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2a2c63d-154f-4950-95d6-55694fedd0c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efac793c-fcjzh\" (UID: \"a2a2c63d-154f-4950-95d6-55694fedd0c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efac793c-fcjzh" Oct 30 17:06:52.195134 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:52.194886 4522 generic.go:356] "Generic (PLEG): container finished" podID="780734db-39cc-4c52-bd39-efefc4ad7e72" containerID="c6fcbb8259641d6b40e6faceea46cd958b839e7cde4a18b01c7897d9feacbe38" exitCode=0 Oct 30 17:06:52.195134 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:52.195040 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1cdec0e0-798d84688c-mtx4b" event={"ID":"780734db-39cc-4c52-bd39-efefc4ad7e72","Type":"ContainerDied","Data":"c6fcbb8259641d6b40e6faceea46cd958b839e7cde4a18b01c7897d9feacbe38"} Oct 30 17:06:52.198082 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:52.197856 4522 generic.go:356] "Generic (PLEG): container finished" podID="cf5eb021-2e73-46db-a0db-ab5dd73076e1" containerID="57ea9f6405603940c16ca6f49d47e4e63f631c3520db909248211538b5218d61" exitCode=0 Oct 30 17:06:52.198082 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:52.198006 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-1cdec0e0-fb59dd5bb-g5xq4" event={"ID":"cf5eb021-2e73-46db-a0db-ab5dd73076e1","Type":"ContainerDied","Data":"57ea9f6405603940c16ca6f49d47e4e63f631c3520db909248211538b5218d61"} Oct 30 17:06:52.288623 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:52.278056 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-1cdec0e0-fb59dd5bb-g5xq4" Oct 30 17:06:52.321165 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:52.321139 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1cdec0e0-798d84688c-mtx4b" Oct 30 17:06:52.341837 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:52.341807 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf5eb021-2e73-46db-a0db-ab5dd73076e1-kube-api-access-fkj98\" (UniqueName: \"kubernetes.io/projected/cf5eb021-2e73-46db-a0db-ab5dd73076e1-kube-api-access-fkj98\") pod \"cf5eb021-2e73-46db-a0db-ab5dd73076e1\" (UID: \"cf5eb021-2e73-46db-a0db-ab5dd73076e1\") " Oct 30 17:06:52.342024 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:52.342006 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf5eb021-2e73-46db-a0db-ab5dd73076e1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf5eb021-2e73-46db-a0db-ab5dd73076e1-plugins-config-volume\") pod \"cf5eb021-2e73-46db-a0db-ab5dd73076e1\" (UID: \"cf5eb021-2e73-46db-a0db-ab5dd73076e1\") " Oct 30 17:06:52.342923 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:52.342881 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf5eb021-2e73-46db-a0db-ab5dd73076e1-plugins-config-volume" pod "cf5eb021-2e73-46db-a0db-ab5dd73076e1" (UID: "cf5eb021-2e73-46db-a0db-ab5dd73076e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:06:52.344947 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:52.344908 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf5eb021-2e73-46db-a0db-ab5dd73076e1-kube-api-access-fkj98" pod "cf5eb021-2e73-46db-a0db-ab5dd73076e1" (UID: "cf5eb021-2e73-46db-a0db-ab5dd73076e1"). InnerVolumeSpecName "kube-api-access-fkj98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:06:52.354004 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:52.353983 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efac793c-fcjzh" Oct 30 17:06:52.444237 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:52.443537 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/780734db-39cc-4c52-bd39-efefc4ad7e72-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/780734db-39cc-4c52-bd39-efefc4ad7e72-plugins-config-volume\") pod \"780734db-39cc-4c52-bd39-efefc4ad7e72\" (UID: \"780734db-39cc-4c52-bd39-efefc4ad7e72\") " Oct 30 17:06:52.444237 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:52.443790 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/780734db-39cc-4c52-bd39-efefc4ad7e72-kube-api-access-bjchd\" (UniqueName: \"kubernetes.io/projected/780734db-39cc-4c52-bd39-efefc4ad7e72-kube-api-access-bjchd\") pod \"780734db-39cc-4c52-bd39-efefc4ad7e72\" (UID: \"780734db-39cc-4c52-bd39-efefc4ad7e72\") " Oct 30 17:06:52.444846 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:52.444817 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkj98\" (UniqueName: \"kubernetes.io/projected/cf5eb021-2e73-46db-a0db-ab5dd73076e1-kube-api-access-fkj98\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:06:52.444962 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:52.444950 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf5eb021-2e73-46db-a0db-ab5dd73076e1-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:06:52.445758 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:52.445736 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/780734db-39cc-4c52-bd39-efefc4ad7e72-plugins-config-volume" pod "780734db-39cc-4c52-bd39-efefc4ad7e72" (UID: "780734db-39cc-4c52-bd39-efefc4ad7e72"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:06:52.453297 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:52.453257 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/780734db-39cc-4c52-bd39-efefc4ad7e72-kube-api-access-bjchd" pod "780734db-39cc-4c52-bd39-efefc4ad7e72" (UID: "780734db-39cc-4c52-bd39-efefc4ad7e72"). InnerVolumeSpecName "kube-api-access-bjchd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:06:52.546469 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:52.546436 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjchd\" (UniqueName: \"kubernetes.io/projected/780734db-39cc-4c52-bd39-efefc4ad7e72-kube-api-access-bjchd\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:06:52.546469 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:52.546468 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/780734db-39cc-4c52-bd39-efefc4ad7e72-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:06:52.795144 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:52.794369 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efac793c-fcjzh"] Oct 30 17:06:53.211518 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:53.211485 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1cdec0e0-798d84688c-mtx4b" event={"ID":"780734db-39cc-4c52-bd39-efefc4ad7e72","Type":"ContainerDied","Data":"8e0c46328abf4597a413c1f6584d88a2f2f8637e9ed6096608d829d116c551fe"} Oct 30 17:06:53.211518 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:53.211457 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1cdec0e0-798d84688c-mtx4b" Oct 30 17:06:53.212040 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:53.211556 4522 scope.go:122] "RemoveContainer" containerID="c6fcbb8259641d6b40e6faceea46cd958b839e7cde4a18b01c7897d9feacbe38" Oct 30 17:06:53.216235 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:53.214423 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-1cdec0e0-fb59dd5bb-g5xq4" event={"ID":"cf5eb021-2e73-46db-a0db-ab5dd73076e1","Type":"ContainerDied","Data":"0fe0a94ca0033a7438978dc533d9943daa8bb983a6780d5913a963fd810485ea"} Oct 30 17:06:53.216235 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:53.214550 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-1cdec0e0-fb59dd5bb-g5xq4" Oct 30 17:06:53.233762 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:53.233659 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efac793c-fcjzh" event={"ID":"a2a2c63d-154f-4950-95d6-55694fedd0c0","Type":"ContainerStarted","Data":"347601b30eae773e18854d80520631ee5491d1c4292c6aac2e2af08655055656"} Oct 30 17:06:53.240726 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:53.240699 4522 scope.go:122] "RemoveContainer" containerID="57ea9f6405603940c16ca6f49d47e4e63f631c3520db909248211538b5218d61" Oct 30 17:06:53.298586 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:53.298535 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efac793c-fcjzh" podStartSLOduration=2.298520232 podStartE2EDuration="2.298520232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:06:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:06:53.293364796 +0000 UTC m=+1187.294119698" watchObservedRunningTime="2026-10-30 17:06:53.298520232 +0000 UTC m=+1187.299275133" Oct 30 17:06:53.344965 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:53.344936 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-1cdec0e0-798d84688c-mtx4b"] Oct 30 17:06:53.350825 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:53.350800 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-1cdec0e0-798d84688c-mtx4b"] Oct 30 17:06:53.387281 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:53.387251 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1cdec0e0-fb59dd5bb-g5xq4"] Oct 30 17:06:53.393708 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:53.393683 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1cdec0e0-fb59dd5bb-g5xq4"] Oct 30 17:06:53.413068 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:53.413039 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1cdec0e0-fg5gg"] Oct 30 17:06:53.417867 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:53.417840 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1cdec0e0-fg5gg"] Oct 30 17:06:53.619426 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:53.617696 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh"] Oct 30 17:06:53.619426 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:53.617934 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh" podUID="8ce5ef44-9364-43ae-9c43-b46885be7b4f" containerName="istio-proxy" containerID="cri-o://6acc416bcf0088a8c8d2927f7a62519bf2217ea0fb732e585cd27dae98881a91" gracePeriod=30 Oct 30 17:06:53.647386 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:53.646812 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5"] Oct 30 17:06:53.647849 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:53.647825 4522 cpu_manager.go:414] "containerMap: removing container" podUID="780734db-39cc-4c52-bd39-efefc4ad7e72" containerName="payload-pre-processing" Oct 30 17:06:53.647940 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:53.647848 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="780734db-39cc-4c52-bd39-efefc4ad7e72" containerName="payload-pre-processing" Oct 30 17:06:53.647940 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:53.647868 4522 cpu_manager.go:414] "containerMap: removing container" podUID="cf5eb021-2e73-46db-a0db-ab5dd73076e1" containerName="payload-processing" Oct 30 17:06:53.647940 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:53.647876 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf5eb021-2e73-46db-a0db-ab5dd73076e1" containerName="payload-processing" Oct 30 17:06:53.648088 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:53.648069 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="780734db-39cc-4c52-bd39-efefc4ad7e72" containerName="payload-pre-processing" Oct 30 17:06:53.648088 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:53.648081 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf5eb021-2e73-46db-a0db-ab5dd73076e1" containerName="payload-processing" Oct 30 17:06:53.653967 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:53.653943 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5" Oct 30 17:06:53.676923 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:53.675838 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5"] Oct 30 17:06:53.843671 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:53.843607 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e1c3c74-139f-4299-97c4-4609e0b9e030-istio-data\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5\" (UID: \"6e1c3c74-139f-4299-97c4-4609e0b9e030\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5" Oct 30 17:06:53.844008 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:53.843975 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e1c3c74-139f-4299-97c4-4609e0b9e030-istio-token\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5\" (UID: \"6e1c3c74-139f-4299-97c4-4609e0b9e030\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5" Oct 30 17:06:53.845296 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:53.845236 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e1c3c74-139f-4299-97c4-4609e0b9e030-workload-certs\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5\" (UID: \"6e1c3c74-139f-4299-97c4-4609e0b9e030\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5" Oct 30 17:06:53.845615 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:53.845588 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e1c3c74-139f-4299-97c4-4609e0b9e030-istiod-ca-cert\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5\" (UID: \"6e1c3c74-139f-4299-97c4-4609e0b9e030\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5" Oct 30 17:06:53.845861 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:53.845833 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e1c3c74-139f-4299-97c4-4609e0b9e030-istio-podinfo\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5\" (UID: \"6e1c3c74-139f-4299-97c4-4609e0b9e030\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5" Oct 30 17:06:53.846031 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:53.846006 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e1c3c74-139f-4299-97c4-4609e0b9e030-workload-socket\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5\" (UID: \"6e1c3c74-139f-4299-97c4-4609e0b9e030\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5" Oct 30 17:06:53.846188 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:53.846164 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e1c3c74-139f-4299-97c4-4609e0b9e030-istio-envoy\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5\" (UID: \"6e1c3c74-139f-4299-97c4-4609e0b9e030\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5" Oct 30 17:06:53.846361 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:53.846333 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsb9w\" (UniqueName: \"kubernetes.io/projected/6e1c3c74-139f-4299-97c4-4609e0b9e030-kube-api-access-gsb9w\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5\" (UID: \"6e1c3c74-139f-4299-97c4-4609e0b9e030\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5" Oct 30 17:06:53.846632 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:53.846604 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e1c3c74-139f-4299-97c4-4609e0b9e030-credential-socket\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5\" (UID: \"6e1c3c74-139f-4299-97c4-4609e0b9e030\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5" Oct 30 17:06:53.948200 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:53.948152 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e1c3c74-139f-4299-97c4-4609e0b9e030-istiod-ca-cert\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5\" (UID: \"6e1c3c74-139f-4299-97c4-4609e0b9e030\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5" Oct 30 17:06:53.948491 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:53.948272 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e1c3c74-139f-4299-97c4-4609e0b9e030-istio-podinfo\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5\" (UID: \"6e1c3c74-139f-4299-97c4-4609e0b9e030\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5" Oct 30 17:06:53.948491 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:53.948333 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e1c3c74-139f-4299-97c4-4609e0b9e030-workload-socket\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5\" (UID: \"6e1c3c74-139f-4299-97c4-4609e0b9e030\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5" Oct 30 17:06:53.948491 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:53.948373 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e1c3c74-139f-4299-97c4-4609e0b9e030-istio-envoy\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5\" (UID: \"6e1c3c74-139f-4299-97c4-4609e0b9e030\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5" Oct 30 17:06:53.948491 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:53.948447 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsb9w\" (UniqueName: \"kubernetes.io/projected/6e1c3c74-139f-4299-97c4-4609e0b9e030-kube-api-access-gsb9w\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5\" (UID: \"6e1c3c74-139f-4299-97c4-4609e0b9e030\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5" Oct 30 17:06:53.948784 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:53.948599 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e1c3c74-139f-4299-97c4-4609e0b9e030-credential-socket\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5\" (UID: \"6e1c3c74-139f-4299-97c4-4609e0b9e030\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5" Oct 30 17:06:53.948784 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:53.948692 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e1c3c74-139f-4299-97c4-4609e0b9e030-istio-data\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5\" (UID: \"6e1c3c74-139f-4299-97c4-4609e0b9e030\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5" Oct 30 17:06:53.948784 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:53.948743 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e1c3c74-139f-4299-97c4-4609e0b9e030-istio-token\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5\" (UID: \"6e1c3c74-139f-4299-97c4-4609e0b9e030\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5" Oct 30 17:06:53.948957 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:53.948768 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e1c3c74-139f-4299-97c4-4609e0b9e030-workload-certs\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5\" (UID: \"6e1c3c74-139f-4299-97c4-4609e0b9e030\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5" Oct 30 17:06:53.949390 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:53.949147 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e1c3c74-139f-4299-97c4-4609e0b9e030-istiod-ca-cert\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5\" (UID: \"6e1c3c74-139f-4299-97c4-4609e0b9e030\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5" Oct 30 17:06:53.950330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:53.950281 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e1c3c74-139f-4299-97c4-4609e0b9e030-workload-certs\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5\" (UID: \"6e1c3c74-139f-4299-97c4-4609e0b9e030\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5" Oct 30 17:06:53.950500 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:53.950418 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e1c3c74-139f-4299-97c4-4609e0b9e030-credential-socket\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5\" (UID: \"6e1c3c74-139f-4299-97c4-4609e0b9e030\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5" Oct 30 17:06:53.950797 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:53.950774 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e1c3c74-139f-4299-97c4-4609e0b9e030-istio-data\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5\" (UID: \"6e1c3c74-139f-4299-97c4-4609e0b9e030\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5" Oct 30 17:06:53.950992 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:53.950968 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e1c3c74-139f-4299-97c4-4609e0b9e030-workload-socket\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5\" (UID: \"6e1c3c74-139f-4299-97c4-4609e0b9e030\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5" Oct 30 17:06:53.953239 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:53.953219 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e1c3c74-139f-4299-97c4-4609e0b9e030-istio-envoy\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5\" (UID: \"6e1c3c74-139f-4299-97c4-4609e0b9e030\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5" Oct 30 17:06:53.953427 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:53.953403 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e1c3c74-139f-4299-97c4-4609e0b9e030-istio-podinfo\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5\" (UID: \"6e1c3c74-139f-4299-97c4-4609e0b9e030\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5" Oct 30 17:06:53.975933 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:53.975902 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsb9w\" (UniqueName: \"kubernetes.io/projected/6e1c3c74-139f-4299-97c4-4609e0b9e030-kube-api-access-gsb9w\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5\" (UID: \"6e1c3c74-139f-4299-97c4-4609e0b9e030\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5" Oct 30 17:06:53.976476 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:53.976451 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e1c3c74-139f-4299-97c4-4609e0b9e030-istio-token\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5\" (UID: \"6e1c3c74-139f-4299-97c4-4609e0b9e030\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5" Oct 30 17:06:53.982451 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:53.982067 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh" Oct 30 17:06:54.050335 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:54.050290 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ce5ef44-9364-43ae-9c43-b46885be7b4f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce5ef44-9364-43ae-9c43-b46885be7b4f-credential-socket\") pod \"8ce5ef44-9364-43ae-9c43-b46885be7b4f\" (UID: \"8ce5ef44-9364-43ae-9c43-b46885be7b4f\") " Oct 30 17:06:54.050565 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:54.050547 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ce5ef44-9364-43ae-9c43-b46885be7b4f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ce5ef44-9364-43ae-9c43-b46885be7b4f-istio-podinfo\") pod \"8ce5ef44-9364-43ae-9c43-b46885be7b4f\" (UID: \"8ce5ef44-9364-43ae-9c43-b46885be7b4f\") " Oct 30 17:06:54.050664 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:54.050586 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ce5ef44-9364-43ae-9c43-b46885be7b4f-credential-socket" pod "8ce5ef44-9364-43ae-9c43-b46885be7b4f" (UID: "8ce5ef44-9364-43ae-9c43-b46885be7b4f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:06:54.050664 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:54.050642 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ce5ef44-9364-43ae-9c43-b46885be7b4f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ce5ef44-9364-43ae-9c43-b46885be7b4f-istio-data\") pod \"8ce5ef44-9364-43ae-9c43-b46885be7b4f\" (UID: \"8ce5ef44-9364-43ae-9c43-b46885be7b4f\") " Oct 30 17:06:54.050770 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:54.050700 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ce5ef44-9364-43ae-9c43-b46885be7b4f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ce5ef44-9364-43ae-9c43-b46885be7b4f-istio-envoy\") pod \"8ce5ef44-9364-43ae-9c43-b46885be7b4f\" (UID: \"8ce5ef44-9364-43ae-9c43-b46885be7b4f\") " Oct 30 17:06:54.050830 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:54.050779 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ce5ef44-9364-43ae-9c43-b46885be7b4f-kube-api-access-f6km9\" (UniqueName: \"kubernetes.io/projected/8ce5ef44-9364-43ae-9c43-b46885be7b4f-kube-api-access-f6km9\") pod \"8ce5ef44-9364-43ae-9c43-b46885be7b4f\" (UID: \"8ce5ef44-9364-43ae-9c43-b46885be7b4f\") " Oct 30 17:06:54.050830 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:54.050814 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ce5ef44-9364-43ae-9c43-b46885be7b4f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce5ef44-9364-43ae-9c43-b46885be7b4f-workload-socket\") pod \"8ce5ef44-9364-43ae-9c43-b46885be7b4f\" (UID: \"8ce5ef44-9364-43ae-9c43-b46885be7b4f\") " Oct 30 17:06:54.050946 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:54.050862 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ce5ef44-9364-43ae-9c43-b46885be7b4f-istio-token\" (UniqueName: \"kubernetes.io/projected/8ce5ef44-9364-43ae-9c43-b46885be7b4f-istio-token\") pod \"8ce5ef44-9364-43ae-9c43-b46885be7b4f\" (UID: \"8ce5ef44-9364-43ae-9c43-b46885be7b4f\") " Oct 30 17:06:54.050946 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:54.050920 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ce5ef44-9364-43ae-9c43-b46885be7b4f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ce5ef44-9364-43ae-9c43-b46885be7b4f-istiod-ca-cert\") pod \"8ce5ef44-9364-43ae-9c43-b46885be7b4f\" (UID: \"8ce5ef44-9364-43ae-9c43-b46885be7b4f\") " Oct 30 17:06:54.051057 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:54.050951 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ce5ef44-9364-43ae-9c43-b46885be7b4f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ce5ef44-9364-43ae-9c43-b46885be7b4f-workload-certs\") pod \"8ce5ef44-9364-43ae-9c43-b46885be7b4f\" (UID: \"8ce5ef44-9364-43ae-9c43-b46885be7b4f\") " Oct 30 17:06:54.051618 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:54.051349 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ce5ef44-9364-43ae-9c43-b46885be7b4f-workload-socket" pod "8ce5ef44-9364-43ae-9c43-b46885be7b4f" (UID: "8ce5ef44-9364-43ae-9c43-b46885be7b4f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:06:54.051618 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:54.051383 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ce5ef44-9364-43ae-9c43-b46885be7b4f-workload-certs" pod "8ce5ef44-9364-43ae-9c43-b46885be7b4f" (UID: "8ce5ef44-9364-43ae-9c43-b46885be7b4f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:06:54.052615 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:54.052589 4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce5ef44-9364-43ae-9c43-b46885be7b4f-credential-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:06:54.053406 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:54.053373 4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce5ef44-9364-43ae-9c43-b46885be7b4f-workload-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:06:54.053510 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:54.053411 4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ce5ef44-9364-43ae-9c43-b46885be7b4f-workload-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:06:54.053510 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:54.052991 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ce5ef44-9364-43ae-9c43-b46885be7b4f-istio-data" pod "8ce5ef44-9364-43ae-9c43-b46885be7b4f" (UID: "8ce5ef44-9364-43ae-9c43-b46885be7b4f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:06:54.053510 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:54.053123 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ce5ef44-9364-43ae-9c43-b46885be7b4f-istiod-ca-cert" pod "8ce5ef44-9364-43ae-9c43-b46885be7b4f" (UID: "8ce5ef44-9364-43ae-9c43-b46885be7b4f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:06:54.053510 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:54.053493 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ce5ef44-9364-43ae-9c43-b46885be7b4f-istio-envoy" pod "8ce5ef44-9364-43ae-9c43-b46885be7b4f" (UID: "8ce5ef44-9364-43ae-9c43-b46885be7b4f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:06:54.062756 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:54.062663 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ce5ef44-9364-43ae-9c43-b46885be7b4f-istio-token" pod "8ce5ef44-9364-43ae-9c43-b46885be7b4f" (UID: "8ce5ef44-9364-43ae-9c43-b46885be7b4f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:06:54.062756 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:54.062725 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ce5ef44-9364-43ae-9c43-b46885be7b4f-istio-podinfo" pod "8ce5ef44-9364-43ae-9c43-b46885be7b4f" (UID: "8ce5ef44-9364-43ae-9c43-b46885be7b4f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:06:54.068353 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:54.064451 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ce5ef44-9364-43ae-9c43-b46885be7b4f-kube-api-access-f6km9" pod "8ce5ef44-9364-43ae-9c43-b46885be7b4f" (UID: "8ce5ef44-9364-43ae-9c43-b46885be7b4f"). InnerVolumeSpecName "kube-api-access-f6km9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:06:54.147463 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:54.147432 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="780734db-39cc-4c52-bd39-efefc4ad7e72" path="/var/lib/kubelet/pods/780734db-39cc-4c52-bd39-efefc4ad7e72/volumes" Oct 30 17:06:54.148219 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:54.148195 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf5eb021-2e73-46db-a0db-ab5dd73076e1" path="/var/lib/kubelet/pods/cf5eb021-2e73-46db-a0db-ab5dd73076e1/volumes" Oct 30 17:06:54.148833 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:54.148810 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1ecf5e7-ed7f-4587-932f-ebfc6aadcaba" path="/var/lib/kubelet/pods/d1ecf5e7-ed7f-4587-932f-ebfc6aadcaba/volumes" Oct 30 17:06:54.155127 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:54.155102 4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ce5ef44-9364-43ae-9c43-b46885be7b4f-istio-envoy\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:06:54.155253 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:54.155130 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6km9\" (UniqueName: \"kubernetes.io/projected/8ce5ef44-9364-43ae-9c43-b46885be7b4f-kube-api-access-f6km9\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:06:54.155253 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:54.155147 4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ce5ef44-9364-43ae-9c43-b46885be7b4f-istio-token\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:06:54.155253 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:54.155161 4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ce5ef44-9364-43ae-9c43-b46885be7b4f-istiod-ca-cert\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:06:54.155253 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:54.155173 4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ce5ef44-9364-43ae-9c43-b46885be7b4f-istio-podinfo\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:06:54.155253 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:54.155185 4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ce5ef44-9364-43ae-9c43-b46885be7b4f-istio-data\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:06:54.248157 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:54.248126 4522 generic.go:356] "Generic (PLEG): container finished" podID="8ce5ef44-9364-43ae-9c43-b46885be7b4f" containerID="6acc416bcf0088a8c8d2927f7a62519bf2217ea0fb732e585cd27dae98881a91" exitCode=0 Oct 30 17:06:54.248738 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:54.248172 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh" Oct 30 17:06:54.248738 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:54.248204 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh" event={"ID":"8ce5ef44-9364-43ae-9c43-b46885be7b4f","Type":"ContainerDied","Data":"6acc416bcf0088a8c8d2927f7a62519bf2217ea0fb732e585cd27dae98881a91"} Oct 30 17:06:54.248738 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:54.248264 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh" event={"ID":"8ce5ef44-9364-43ae-9c43-b46885be7b4f","Type":"ContainerDied","Data":"4ecc31ea3691bef3d4162d62997cf45057ae35e4a84c77979423c4b5e0c85baa"} Oct 30 17:06:54.248738 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:54.248297 4522 scope.go:122] "RemoveContainer" containerID="6acc416bcf0088a8c8d2927f7a62519bf2217ea0fb732e585cd27dae98881a91" Oct 30 17:06:54.250493 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:54.250466 4522 generic.go:356] "Generic (PLEG): container finished" podID="a2a2c63d-154f-4950-95d6-55694fedd0c0" containerID="946f7c57908d8b1da989603bfbd684b6df04387bbfb4438fdc88492781057bdb" exitCode=0 Oct 30 17:06:54.250708 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:54.250496 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efac793c-fcjzh" event={"ID":"a2a2c63d-154f-4950-95d6-55694fedd0c0","Type":"ContainerDied","Data":"946f7c57908d8b1da989603bfbd684b6df04387bbfb4438fdc88492781057bdb"} Oct 30 17:06:54.260580 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:54.260559 4522 scope.go:122] "RemoveContainer" containerID="6acc416bcf0088a8c8d2927f7a62519bf2217ea0fb732e585cd27dae98881a91" Oct 30 17:06:54.260848 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:06:54.260828 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6acc416bcf0088a8c8d2927f7a62519bf2217ea0fb732e585cd27dae98881a91\": container with ID starting with 6acc416bcf0088a8c8d2927f7a62519bf2217ea0fb732e585cd27dae98881a91 not found: ID does not exist" containerID="6acc416bcf0088a8c8d2927f7a62519bf2217ea0fb732e585cd27dae98881a91" Oct 30 17:06:54.260910 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:54.260855 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6acc416bcf0088a8c8d2927f7a62519bf2217ea0fb732e585cd27dae98881a91"} err="failed to get container status \"6acc416bcf0088a8c8d2927f7a62519bf2217ea0fb732e585cd27dae98881a91\": rpc error: code = NotFound desc = could not find container \"6acc416bcf0088a8c8d2927f7a62519bf2217ea0fb732e585cd27dae98881a91\": container with ID starting with 6acc416bcf0088a8c8d2927f7a62519bf2217ea0fb732e585cd27dae98881a91 not found: ID does not exist" Oct 30 17:06:54.270904 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:54.270883 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5" Oct 30 17:06:54.313919 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:54.313884 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh"] Oct 30 17:06:54.330871 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:54.330843 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-1cdec0e0-gw-openshift-default-7968476969-v2qbh"] Oct 30 17:06:54.593028 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:06:54.592999 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e1c3c74_139f_4299_97c4_4609e0b9e030.slice/crio-809bd884c62ac8f3741c174e59ef033bbf03155b7e44a392e31e2b814a3bf5c1 WatchSource:0}: Error finding container 809bd884c62ac8f3741c174e59ef033bbf03155b7e44a392e31e2b814a3bf5c1: Status 404 returned error can't find the container with id 809bd884c62ac8f3741c174e59ef033bbf03155b7e44a392e31e2b814a3bf5c1 Oct 30 17:06:54.596389 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:54.596354 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:06:54.596521 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:54.596450 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:06:54.596521 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:54.596496 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:06:54.596632 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:54.596582 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5"] Oct 30 17:06:55.265602 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:55.265560 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5" event={"ID":"6e1c3c74-139f-4299-97c4-4609e0b9e030","Type":"ContainerStarted","Data":"a60d5b915fec64c6758b8ef572674b5753e55fc85993d77537df3276db0a3c87"} Oct 30 17:06:55.266195 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:55.265632 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5" event={"ID":"6e1c3c74-139f-4299-97c4-4609e0b9e030","Type":"ContainerStarted","Data":"809bd884c62ac8f3741c174e59ef033bbf03155b7e44a392e31e2b814a3bf5c1"} Oct 30 17:06:55.271654 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:55.271631 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5" Oct 30 17:06:55.273763 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:55.273738 4522 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 17:06:55.273882 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:55.273786 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5" podUID="6e1c3c74-139f-4299-97c4-4609e0b9e030" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 17:06:55.343231 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:55.343155 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5" podStartSLOduration=2.343133614 podStartE2EDuration="2.343133614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:06:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:06:55.318426234 +0000 UTC m=+1189.319181135" watchObservedRunningTime="2026-10-30 17:06:55.343133614 +0000 UTC m=+1189.343888516" Oct 30 17:06:55.582515 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:55.582085 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efac793c-fcjzh" Oct 30 17:06:55.719330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:55.718859 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2a2c63d-154f-4950-95d6-55694fedd0c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2a2c63d-154f-4950-95d6-55694fedd0c0-maas-api-xks-ca\") pod \"a2a2c63d-154f-4950-95d6-55694fedd0c0\" (UID: \"a2a2c63d-154f-4950-95d6-55694fedd0c0\") " Oct 30 17:06:55.719330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:55.719170 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2a2c63d-154f-4950-95d6-55694fedd0c0-kube-api-access-rtq72\" (UniqueName: \"kubernetes.io/projected/a2a2c63d-154f-4950-95d6-55694fedd0c0-kube-api-access-rtq72\") pod \"a2a2c63d-154f-4950-95d6-55694fedd0c0\" (UID: \"a2a2c63d-154f-4950-95d6-55694fedd0c0\") " Oct 30 17:06:55.719330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:55.719239 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2a2c63d-154f-4950-95d6-55694fedd0c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2a2c63d-154f-4950-95d6-55694fedd0c0-maas-api-openshift-ca\") pod \"a2a2c63d-154f-4950-95d6-55694fedd0c0\" (UID: \"a2a2c63d-154f-4950-95d6-55694fedd0c0\") " Oct 30 17:06:55.722329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:55.720179 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2a2c63d-154f-4950-95d6-55694fedd0c0-maas-api-openshift-ca" pod "a2a2c63d-154f-4950-95d6-55694fedd0c0" (UID: "a2a2c63d-154f-4950-95d6-55694fedd0c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:06:55.723244 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:55.723217 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2a2c63d-154f-4950-95d6-55694fedd0c0-kube-api-access-rtq72" pod "a2a2c63d-154f-4950-95d6-55694fedd0c0" (UID: "a2a2c63d-154f-4950-95d6-55694fedd0c0"). InnerVolumeSpecName "kube-api-access-rtq72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:06:55.727256 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:55.727233 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2a2c63d-154f-4950-95d6-55694fedd0c0-maas-api-xks-ca" pod "a2a2c63d-154f-4950-95d6-55694fedd0c0" (UID: "a2a2c63d-154f-4950-95d6-55694fedd0c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:06:55.822638 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:55.822560 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtq72\" (UniqueName: \"kubernetes.io/projected/a2a2c63d-154f-4950-95d6-55694fedd0c0-kube-api-access-rtq72\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:06:55.822638 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:55.822588 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2a2c63d-154f-4950-95d6-55694fedd0c0-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:06:55.822638 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:55.822598 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2a2c63d-154f-4950-95d6-55694fedd0c0-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:06:56.151341 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:56.151242 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ce5ef44-9364-43ae-9c43-b46885be7b4f" path="/var/lib/kubelet/pods/8ce5ef44-9364-43ae-9c43-b46885be7b4f/volumes" Oct 30 17:06:56.271960 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:56.271926 4522 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 17:06:56.272472 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:56.271976 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5" podUID="6e1c3c74-139f-4299-97c4-4609e0b9e030" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 17:06:56.283496 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:56.283454 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efac793c-fcjzh" Oct 30 17:06:56.283658 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:56.283503 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efac793c-fcjzh" event={"ID":"a2a2c63d-154f-4950-95d6-55694fedd0c0","Type":"ContainerDied","Data":"347601b30eae773e18854d80520631ee5491d1c4292c6aac2e2af08655055656"} Oct 30 17:06:56.283658 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:56.283546 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="347601b30eae773e18854d80520631ee5491d1c4292c6aac2e2af08655055656" Oct 30 17:06:57.283334 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:57.278879 4522 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 17:06:57.283334 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:57.278940 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5" podUID="6e1c3c74-139f-4299-97c4-4609e0b9e030" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 17:06:58.122597 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:58.122561 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3451d3b5-cm2tc"] Oct 30 17:06:58.123288 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:58.123272 4522 cpu_manager.go:414] "containerMap: removing container" podUID="a2a2c63d-154f-4950-95d6-55694fedd0c0" containerName="revoke-keys" Oct 30 17:06:58.123372 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:58.123289 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2a2c63d-154f-4950-95d6-55694fedd0c0" containerName="revoke-keys" Oct 30 17:06:58.123372 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:58.123326 4522 cpu_manager.go:414] "containerMap: removing container" podUID="8ce5ef44-9364-43ae-9c43-b46885be7b4f" containerName="istio-proxy" Oct 30 17:06:58.123372 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:58.123331 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ce5ef44-9364-43ae-9c43-b46885be7b4f" containerName="istio-proxy" Oct 30 17:06:58.123517 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:58.123455 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ce5ef44-9364-43ae-9c43-b46885be7b4f" containerName="istio-proxy" Oct 30 17:06:58.123517 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:58.123472 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2a2c63d-154f-4950-95d6-55694fedd0c0" containerName="revoke-keys" Oct 30 17:06:58.128164 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:58.128144 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3451d3b5-cm2tc" Oct 30 17:06:58.151738 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:58.151705 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3451d3b5-cm2tc"] Oct 30 17:06:58.157761 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:58.157735 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:06:58.157918 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:58.157840 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:06:58.271330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:58.271269 4522 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 17:06:58.271478 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:58.271380 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5" podUID="6e1c3c74-139f-4299-97c4-4609e0b9e030" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 17:06:58.283593 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:58.283544 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwts4\" (UniqueName: \"kubernetes.io/projected/e8950018-69d7-41f4-85ee-e56a6665f22a-kube-api-access-lwts4\") pod \"maas-api-revoke-sub-3451d3b5-cm2tc\" (UID: \"e8950018-69d7-41f4-85ee-e56a6665f22a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3451d3b5-cm2tc" Oct 30 17:06:58.284239 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:58.283684 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8950018-69d7-41f4-85ee-e56a6665f22a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3451d3b5-cm2tc\" (UID: \"e8950018-69d7-41f4-85ee-e56a6665f22a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3451d3b5-cm2tc" Oct 30 17:06:58.284239 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:58.284002 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8950018-69d7-41f4-85ee-e56a6665f22a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3451d3b5-cm2tc\" (UID: \"e8950018-69d7-41f4-85ee-e56a6665f22a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3451d3b5-cm2tc" Oct 30 17:06:58.385369 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:58.385243 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwts4\" (UniqueName: \"kubernetes.io/projected/e8950018-69d7-41f4-85ee-e56a6665f22a-kube-api-access-lwts4\") pod \"maas-api-revoke-sub-3451d3b5-cm2tc\" (UID: \"e8950018-69d7-41f4-85ee-e56a6665f22a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3451d3b5-cm2tc" Oct 30 17:06:58.385369 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:58.385343 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8950018-69d7-41f4-85ee-e56a6665f22a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3451d3b5-cm2tc\" (UID: \"e8950018-69d7-41f4-85ee-e56a6665f22a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3451d3b5-cm2tc" Oct 30 17:06:58.385622 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:58.385479 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8950018-69d7-41f4-85ee-e56a6665f22a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3451d3b5-cm2tc\" (UID: \"e8950018-69d7-41f4-85ee-e56a6665f22a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3451d3b5-cm2tc" Oct 30 17:06:58.386058 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:58.386029 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8950018-69d7-41f4-85ee-e56a6665f22a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3451d3b5-cm2tc\" (UID: \"e8950018-69d7-41f4-85ee-e56a6665f22a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3451d3b5-cm2tc" Oct 30 17:06:58.387843 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:58.387825 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8950018-69d7-41f4-85ee-e56a6665f22a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3451d3b5-cm2tc\" (UID: \"e8950018-69d7-41f4-85ee-e56a6665f22a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3451d3b5-cm2tc" Oct 30 17:06:58.418191 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:58.418158 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwts4\" (UniqueName: \"kubernetes.io/projected/e8950018-69d7-41f4-85ee-e56a6665f22a-kube-api-access-lwts4\") pod \"maas-api-revoke-sub-3451d3b5-cm2tc\" (UID: \"e8950018-69d7-41f4-85ee-e56a6665f22a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3451d3b5-cm2tc" Oct 30 17:06:58.444220 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:58.444193 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3451d3b5-cm2tc" Oct 30 17:06:58.784969 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:06:58.784931 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8950018_69d7_41f4_85ee_e56a6665f22a.slice/crio-52fec7b5b97b94ebf8f951e1b032cac2c68737b61a5ac0ae109795636f574f92 WatchSource:0}: Error finding container 52fec7b5b97b94ebf8f951e1b032cac2c68737b61a5ac0ae109795636f574f92: Status 404 returned error can't find the container with id 52fec7b5b97b94ebf8f951e1b032cac2c68737b61a5ac0ae109795636f574f92 Oct 30 17:06:58.787264 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:58.787240 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3451d3b5-cm2tc"] Oct 30 17:06:58.874916 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:58.874887 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-11f480-5d57fc56dc-w48fx"] Oct 30 17:06:58.881131 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:58.881110 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-11f480-5d57fc56dc-w48fx" Oct 30 17:06:58.924415 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:58.924377 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-11f480\"" Oct 30 17:06:58.925110 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:58.925086 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-11f480-dockercfg-ppnnh\"" Oct 30 17:06:58.929255 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:58.929229 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-11f480-5d57fc56dc-w48fx"] Oct 30 17:06:58.996827 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:58.996556 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-11f480-59c5945bbd-fjf2v"] Oct 30 17:06:59.022530 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:59.022495 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e520acad-555a-429f-be9a-928ef8749774-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-11f480-5d57fc56dc-w48fx\" (UID: \"e520acad-555a-429f-be9a-928ef8749774\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-11f480-5d57fc56dc-w48fx" Oct 30 17:06:59.052380 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:59.022699 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-11f480-59c5945bbd-fjf2v" Oct 30 17:06:59.053017 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:59.052973 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74p2n\" (UniqueName: \"kubernetes.io/projected/e520acad-555a-429f-be9a-928ef8749774-kube-api-access-74p2n\") pod \"payload-pre-processing-e2e-shared-a-w2-11f480-5d57fc56dc-w48fx\" (UID: \"e520acad-555a-429f-be9a-928ef8749774\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-11f480-5d57fc56dc-w48fx" Oct 30 17:06:59.152255 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:59.152224 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-11f480-59c5945bbd-fjf2v"] Oct 30 17:06:59.154975 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:59.154949 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74p2n\" (UniqueName: \"kubernetes.io/projected/e520acad-555a-429f-be9a-928ef8749774-kube-api-access-74p2n\") pod \"payload-pre-processing-e2e-shared-a-w2-11f480-5d57fc56dc-w48fx\" (UID: \"e520acad-555a-429f-be9a-928ef8749774\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-11f480-5d57fc56dc-w48fx" Oct 30 17:06:59.155632 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:59.155608 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vf8m\" (UniqueName: \"kubernetes.io/projected/b4c4560b-846b-40b9-b143-bab12720112a-kube-api-access-7vf8m\") pod \"payload-processing-e2e-shared-a-w2-11f480-59c5945bbd-fjf2v\" (UID: \"b4c4560b-846b-40b9-b143-bab12720112a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-11f480-59c5945bbd-fjf2v" Oct 30 17:06:59.157536 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:59.157510 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4c4560b-846b-40b9-b143-bab12720112a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-11f480-59c5945bbd-fjf2v\" (UID: \"b4c4560b-846b-40b9-b143-bab12720112a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-11f480-59c5945bbd-fjf2v" Oct 30 17:06:59.157787 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:59.157770 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e520acad-555a-429f-be9a-928ef8749774-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-11f480-5d57fc56dc-w48fx\" (UID: \"e520acad-555a-429f-be9a-928ef8749774\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-11f480-5d57fc56dc-w48fx" Oct 30 17:06:59.159371 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:59.159330 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e520acad-555a-429f-be9a-928ef8749774-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-11f480-5d57fc56dc-w48fx\" (UID: \"e520acad-555a-429f-be9a-928ef8749774\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-11f480-5d57fc56dc-w48fx" Oct 30 17:06:59.183980 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:59.183944 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74p2n\" (UniqueName: \"kubernetes.io/projected/e520acad-555a-429f-be9a-928ef8749774-kube-api-access-74p2n\") pod \"payload-pre-processing-e2e-shared-a-w2-11f480-5d57fc56dc-w48fx\" (UID: \"e520acad-555a-429f-be9a-928ef8749774\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-11f480-5d57fc56dc-w48fx" Oct 30 17:06:59.228658 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:59.228620 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-11f480-5d57fc56dc-w48fx" Oct 30 17:06:59.259417 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:59.259372 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vf8m\" (UniqueName: \"kubernetes.io/projected/b4c4560b-846b-40b9-b143-bab12720112a-kube-api-access-7vf8m\") pod \"payload-processing-e2e-shared-a-w2-11f480-59c5945bbd-fjf2v\" (UID: \"b4c4560b-846b-40b9-b143-bab12720112a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-11f480-59c5945bbd-fjf2v" Oct 30 17:06:59.259576 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:59.259487 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4c4560b-846b-40b9-b143-bab12720112a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-11f480-59c5945bbd-fjf2v\" (UID: \"b4c4560b-846b-40b9-b143-bab12720112a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-11f480-59c5945bbd-fjf2v" Oct 30 17:06:59.260761 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:59.260733 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4c4560b-846b-40b9-b143-bab12720112a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-11f480-59c5945bbd-fjf2v\" (UID: \"b4c4560b-846b-40b9-b143-bab12720112a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-11f480-59c5945bbd-fjf2v" Oct 30 17:06:59.271882 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:59.271769 4522 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 17:06:59.271882 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:59.271827 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5" podUID="6e1c3c74-139f-4299-97c4-4609e0b9e030" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 17:06:59.300076 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:59.290823 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vf8m\" (UniqueName: \"kubernetes.io/projected/b4c4560b-846b-40b9-b143-bab12720112a-kube-api-access-7vf8m\") pod \"payload-processing-e2e-shared-a-w2-11f480-59c5945bbd-fjf2v\" (UID: \"b4c4560b-846b-40b9-b143-bab12720112a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-11f480-59c5945bbd-fjf2v" Oct 30 17:06:59.345539 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:59.345496 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3451d3b5-cm2tc" event={"ID":"e8950018-69d7-41f4-85ee-e56a6665f22a","Type":"ContainerStarted","Data":"52fec7b5b97b94ebf8f951e1b032cac2c68737b61a5ac0ae109795636f574f92"} Oct 30 17:06:59.485923 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:59.485681 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-11f480-59c5945bbd-fjf2v" Oct 30 17:06:59.630096 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:59.630066 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-11f480-5d57fc56dc-w48fx"] Oct 30 17:06:59.633475 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:06:59.631738 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode520acad_555a_429f_be9a_928ef8749774.slice/crio-007d3176417e7216a2485ef7c2c9591e8cdec4dc3ad07d050b6b6326efdd1118 WatchSource:0}: Error finding container 007d3176417e7216a2485ef7c2c9591e8cdec4dc3ad07d050b6b6326efdd1118: Status 404 returned error can't find the container with id 007d3176417e7216a2485ef7c2c9591e8cdec4dc3ad07d050b6b6326efdd1118 Oct 30 17:06:59.827016 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:06:59.826986 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4c4560b_846b_40b9_b143_bab12720112a.slice/crio-da1ba77907466f7ccd5df0b6d542085eabf5c57fd1787ef177a112362a96b229 WatchSource:0}: Error finding container da1ba77907466f7ccd5df0b6d542085eabf5c57fd1787ef177a112362a96b229: Status 404 returned error can't find the container with id da1ba77907466f7ccd5df0b6d542085eabf5c57fd1787ef177a112362a96b229 Oct 30 17:06:59.829920 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:06:59.829894 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-11f480-59c5945bbd-fjf2v"] Oct 30 17:07:00.272057 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:00.272020 4522 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 17:07:00.272212 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:00.272088 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5" podUID="6e1c3c74-139f-4299-97c4-4609e0b9e030" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 17:07:00.355256 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:00.355214 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-11f480-5d57fc56dc-w48fx" event={"ID":"e520acad-555a-429f-be9a-928ef8749774","Type":"ContainerStarted","Data":"fd8ee145cc9b2eaf530cdb66f081a5a655fab467b846bf95d651d07b85033bf1"} Oct 30 17:07:00.355705 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:00.355287 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-11f480-5d57fc56dc-w48fx" Oct 30 17:07:00.355705 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:00.355323 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-11f480-5d57fc56dc-w48fx" event={"ID":"e520acad-555a-429f-be9a-928ef8749774","Type":"ContainerStarted","Data":"007d3176417e7216a2485ef7c2c9591e8cdec4dc3ad07d050b6b6326efdd1118"} Oct 30 17:07:00.357042 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:00.357009 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-11f480-59c5945bbd-fjf2v" event={"ID":"b4c4560b-846b-40b9-b143-bab12720112a","Type":"ContainerStarted","Data":"c7be5c1c043be5aa62343ed71512a0259d409caef31a9a6b2bf1a7c3d7d7e015"} Oct 30 17:07:00.357190 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:00.357060 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-11f480-59c5945bbd-fjf2v" event={"ID":"b4c4560b-846b-40b9-b143-bab12720112a","Type":"ContainerStarted","Data":"da1ba77907466f7ccd5df0b6d542085eabf5c57fd1787ef177a112362a96b229"} Oct 30 17:07:00.357190 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:00.357088 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-11f480-59c5945bbd-fjf2v" Oct 30 17:07:00.358851 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:00.358829 4522 generic.go:356] "Generic (PLEG): container finished" podID="e8950018-69d7-41f4-85ee-e56a6665f22a" containerID="10129b75a3b759ac5a0f505882112e1359ffcbcf6ed1159dce7e4dd5b824f666" exitCode=0 Oct 30 17:07:00.358936 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:00.358888 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3451d3b5-cm2tc" event={"ID":"e8950018-69d7-41f4-85ee-e56a6665f22a","Type":"ContainerDied","Data":"10129b75a3b759ac5a0f505882112e1359ffcbcf6ed1159dce7e4dd5b824f666"} Oct 30 17:07:00.424367 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:00.424290 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-11f480-5d57fc56dc-w48fx" podStartSLOduration=2.424272175 podStartE2EDuration="2.424272175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:06:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:07:00.415338273 +0000 UTC m=+1194.416093203" watchObservedRunningTime="2026-10-30 17:07:00.424272175 +0000 UTC m=+1194.425027079" Oct 30 17:07:00.464036 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:00.463982 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-11f480-59c5945bbd-fjf2v" podStartSLOduration=2.463968358 podStartE2EDuration="2.463968358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:06:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:07:00.461503181 +0000 UTC m=+1194.462258081" watchObservedRunningTime="2026-10-30 17:07:00.463968358 +0000 UTC m=+1194.464723311" Oct 30 17:07:01.271934 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:01.271900 4522 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 17:07:01.272128 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:01.271954 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5" podUID="6e1c3c74-139f-4299-97c4-4609e0b9e030" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 17:07:01.546283 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:01.546258 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3451d3b5-cm2tc" Oct 30 17:07:01.613481 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:01.613445 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8950018-69d7-41f4-85ee-e56a6665f22a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8950018-69d7-41f4-85ee-e56a6665f22a-maas-api-xks-ca\") pod \"e8950018-69d7-41f4-85ee-e56a6665f22a\" (UID: \"e8950018-69d7-41f4-85ee-e56a6665f22a\") " Oct 30 17:07:01.613676 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:01.613623 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8950018-69d7-41f4-85ee-e56a6665f22a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8950018-69d7-41f4-85ee-e56a6665f22a-maas-api-openshift-ca\") pod \"e8950018-69d7-41f4-85ee-e56a6665f22a\" (UID: \"e8950018-69d7-41f4-85ee-e56a6665f22a\") " Oct 30 17:07:01.613676 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:01.613667 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8950018-69d7-41f4-85ee-e56a6665f22a-kube-api-access-lwts4\" (UniqueName: \"kubernetes.io/projected/e8950018-69d7-41f4-85ee-e56a6665f22a-kube-api-access-lwts4\") pod \"e8950018-69d7-41f4-85ee-e56a6665f22a\" (UID: \"e8950018-69d7-41f4-85ee-e56a6665f22a\") " Oct 30 17:07:01.614220 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:01.614191 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8950018-69d7-41f4-85ee-e56a6665f22a-maas-api-openshift-ca" pod "e8950018-69d7-41f4-85ee-e56a6665f22a" (UID: "e8950018-69d7-41f4-85ee-e56a6665f22a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:07:01.615686 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:01.615658 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8950018-69d7-41f4-85ee-e56a6665f22a-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:07:01.616804 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:01.616777 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8950018-69d7-41f4-85ee-e56a6665f22a-maas-api-xks-ca" pod "e8950018-69d7-41f4-85ee-e56a6665f22a" (UID: "e8950018-69d7-41f4-85ee-e56a6665f22a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:07:01.617104 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:01.617075 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8950018-69d7-41f4-85ee-e56a6665f22a-kube-api-access-lwts4" pod "e8950018-69d7-41f4-85ee-e56a6665f22a" (UID: "e8950018-69d7-41f4-85ee-e56a6665f22a"). InnerVolumeSpecName "kube-api-access-lwts4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:07:01.717292 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:01.717263 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8950018-69d7-41f4-85ee-e56a6665f22a-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:07:01.717292 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:01.717290 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwts4\" (UniqueName: \"kubernetes.io/projected/e8950018-69d7-41f4-85ee-e56a6665f22a-kube-api-access-lwts4\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:07:02.382083 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:02.382023 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3451d3b5-cm2tc" Oct 30 17:07:02.382083 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:02.382070 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3451d3b5-cm2tc" event={"ID":"e8950018-69d7-41f4-85ee-e56a6665f22a","Type":"ContainerDied","Data":"52fec7b5b97b94ebf8f951e1b032cac2c68737b61a5ac0ae109795636f574f92"} Oct 30 17:07:02.382485 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:02.382157 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52fec7b5b97b94ebf8f951e1b032cac2c68737b61a5ac0ae109795636f574f92" Oct 30 17:07:02.403325 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:02.403280 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5" Oct 30 17:07:02.404285 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:02.404257 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5" Oct 30 17:07:02.405228 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:02.405205 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5" Oct 30 17:07:03.816915 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:03.816880 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4"] Oct 30 17:07:03.817971 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:03.817946 4522 cpu_manager.go:414] "containerMap: removing container" podUID="e8950018-69d7-41f4-85ee-e56a6665f22a" containerName="revoke-keys" Oct 30 17:07:03.817971 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:03.817969 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8950018-69d7-41f4-85ee-e56a6665f22a" containerName="revoke-keys" Oct 30 17:07:03.818232 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:03.818217 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8950018-69d7-41f4-85ee-e56a6665f22a" containerName="revoke-keys" Oct 30 17:07:03.822507 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:03.822485 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4" Oct 30 17:07:03.850458 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:03.850427 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4"] Oct 30 17:07:03.857996 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:03.857971 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-5d8947-openshift-default-dockercfg-kfm5z\"" Oct 30 17:07:03.954789 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:03.954749 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef9c4eec-8676-4a8c-9913-da321bb96baf-istio-data\") pod \"e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4\" (UID: \"ef9c4eec-8676-4a8c-9913-da321bb96baf\") " pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4" Oct 30 17:07:03.954983 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:03.954801 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef9c4eec-8676-4a8c-9913-da321bb96baf-istio-podinfo\") pod \"e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4\" (UID: \"ef9c4eec-8676-4a8c-9913-da321bb96baf\") " pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4" Oct 30 17:07:03.954983 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:03.954926 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef9c4eec-8676-4a8c-9913-da321bb96baf-istio-envoy\") pod \"e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4\" (UID: \"ef9c4eec-8676-4a8c-9913-da321bb96baf\") " pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4" Oct 30 17:07:03.954983 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:03.954974 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d5qs\" (UniqueName: \"kubernetes.io/projected/ef9c4eec-8676-4a8c-9913-da321bb96baf-kube-api-access-5d5qs\") pod \"e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4\" (UID: \"ef9c4eec-8676-4a8c-9913-da321bb96baf\") " pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4" Oct 30 17:07:03.955151 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:03.955100 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef9c4eec-8676-4a8c-9913-da321bb96baf-credential-socket\") pod \"e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4\" (UID: \"ef9c4eec-8676-4a8c-9913-da321bb96baf\") " pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4" Oct 30 17:07:03.955214 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:03.955194 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef9c4eec-8676-4a8c-9913-da321bb96baf-istio-token\") pod \"e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4\" (UID: \"ef9c4eec-8676-4a8c-9913-da321bb96baf\") " pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4" Oct 30 17:07:03.955247 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:03.955227 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef9c4eec-8676-4a8c-9913-da321bb96baf-istiod-ca-cert\") pod \"e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4\" (UID: \"ef9c4eec-8676-4a8c-9913-da321bb96baf\") " pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4" Oct 30 17:07:03.955387 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:03.955369 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef9c4eec-8676-4a8c-9913-da321bb96baf-workload-certs\") pod \"e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4\" (UID: \"ef9c4eec-8676-4a8c-9913-da321bb96baf\") " pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4" Oct 30 17:07:03.955507 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:03.955491 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef9c4eec-8676-4a8c-9913-da321bb96baf-workload-socket\") pod \"e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4\" (UID: \"ef9c4eec-8676-4a8c-9913-da321bb96baf\") " pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4" Oct 30 17:07:04.065318 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:04.065144 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef9c4eec-8676-4a8c-9913-da321bb96baf-workload-socket\") pod \"e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4\" (UID: \"ef9c4eec-8676-4a8c-9913-da321bb96baf\") " pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4" Oct 30 17:07:04.065523 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:04.065408 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef9c4eec-8676-4a8c-9913-da321bb96baf-istio-data\") pod \"e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4\" (UID: \"ef9c4eec-8676-4a8c-9913-da321bb96baf\") " pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4" Oct 30 17:07:04.065523 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:04.065448 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef9c4eec-8676-4a8c-9913-da321bb96baf-istio-podinfo\") pod \"e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4\" (UID: \"ef9c4eec-8676-4a8c-9913-da321bb96baf\") " pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4" Oct 30 17:07:04.065523 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:04.065493 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef9c4eec-8676-4a8c-9913-da321bb96baf-istio-envoy\") pod \"e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4\" (UID: \"ef9c4eec-8676-4a8c-9913-da321bb96baf\") " pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4" Oct 30 17:07:04.065713 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:04.065535 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d5qs\" (UniqueName: \"kubernetes.io/projected/ef9c4eec-8676-4a8c-9913-da321bb96baf-kube-api-access-5d5qs\") pod \"e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4\" (UID: \"ef9c4eec-8676-4a8c-9913-da321bb96baf\") " pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4" Oct 30 17:07:04.065713 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:04.065617 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef9c4eec-8676-4a8c-9913-da321bb96baf-credential-socket\") pod \"e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4\" (UID: \"ef9c4eec-8676-4a8c-9913-da321bb96baf\") " pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4" Oct 30 17:07:04.065713 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:04.065704 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef9c4eec-8676-4a8c-9913-da321bb96baf-istio-token\") pod \"e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4\" (UID: \"ef9c4eec-8676-4a8c-9913-da321bb96baf\") " pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4" Oct 30 17:07:04.065877 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:04.065728 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef9c4eec-8676-4a8c-9913-da321bb96baf-istiod-ca-cert\") pod \"e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4\" (UID: \"ef9c4eec-8676-4a8c-9913-da321bb96baf\") " pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4" Oct 30 17:07:04.065877 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:04.065847 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef9c4eec-8676-4a8c-9913-da321bb96baf-workload-certs\") pod \"e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4\" (UID: \"ef9c4eec-8676-4a8c-9913-da321bb96baf\") " pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4" Oct 30 17:07:04.069508 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:04.066463 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef9c4eec-8676-4a8c-9913-da321bb96baf-workload-certs\") pod \"e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4\" (UID: \"ef9c4eec-8676-4a8c-9913-da321bb96baf\") " pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4" Oct 30 17:07:04.069508 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:04.066728 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef9c4eec-8676-4a8c-9913-da321bb96baf-workload-socket\") pod \"e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4\" (UID: \"ef9c4eec-8676-4a8c-9913-da321bb96baf\") " pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4" Oct 30 17:07:04.069508 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:04.066959 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef9c4eec-8676-4a8c-9913-da321bb96baf-istio-data\") pod \"e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4\" (UID: \"ef9c4eec-8676-4a8c-9913-da321bb96baf\") " pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4" Oct 30 17:07:04.069508 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:04.068271 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef9c4eec-8676-4a8c-9913-da321bb96baf-credential-socket\") pod \"e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4\" (UID: \"ef9c4eec-8676-4a8c-9913-da321bb96baf\") " pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4" Oct 30 17:07:04.089333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:04.080339 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef9c4eec-8676-4a8c-9913-da321bb96baf-istiod-ca-cert\") pod \"e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4\" (UID: \"ef9c4eec-8676-4a8c-9913-da321bb96baf\") " pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4" Oct 30 17:07:04.145377 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:04.133367 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef9c4eec-8676-4a8c-9913-da321bb96baf-istio-envoy\") pod \"e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4\" (UID: \"ef9c4eec-8676-4a8c-9913-da321bb96baf\") " pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4" Oct 30 17:07:04.173550 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:04.160687 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef9c4eec-8676-4a8c-9913-da321bb96baf-istio-podinfo\") pod \"e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4\" (UID: \"ef9c4eec-8676-4a8c-9913-da321bb96baf\") " pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4" Oct 30 17:07:04.173550 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:04.171014 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d5qs\" (UniqueName: \"kubernetes.io/projected/ef9c4eec-8676-4a8c-9913-da321bb96baf-kube-api-access-5d5qs\") pod \"e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4\" (UID: \"ef9c4eec-8676-4a8c-9913-da321bb96baf\") " pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4" Oct 30 17:07:04.177708 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:04.177677 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef9c4eec-8676-4a8c-9913-da321bb96baf-istio-token\") pod \"e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4\" (UID: \"ef9c4eec-8676-4a8c-9913-da321bb96baf\") " pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4" Oct 30 17:07:04.444356 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:04.444252 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4" Oct 30 17:07:04.780523 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:07:04.780495 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef9c4eec_8676_4a8c_9913_da321bb96baf.slice/crio-0d83b7929ebd4e864989a5da912380ea87ec62337ad7c69b20c6927d19b4e04e WatchSource:0}: Error finding container 0d83b7929ebd4e864989a5da912380ea87ec62337ad7c69b20c6927d19b4e04e: Status 404 returned error can't find the container with id 0d83b7929ebd4e864989a5da912380ea87ec62337ad7c69b20c6927d19b4e04e Oct 30 17:07:04.783886 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:04.783848 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:07:04.783953 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:04.783915 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4"] Oct 30 17:07:04.783953 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:04.783932 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:07:04.784029 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:04.783974 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:07:05.422707 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:05.422669 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4" event={"ID":"ef9c4eec-8676-4a8c-9913-da321bb96baf","Type":"ContainerStarted","Data":"adf01b0a01c87f846fad0f93969483a10b706c4c156414f8d5a068c9d364f732"} Oct 30 17:07:05.423099 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:05.422740 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4" event={"ID":"ef9c4eec-8676-4a8c-9913-da321bb96baf","Type":"ContainerStarted","Data":"0d83b7929ebd4e864989a5da912380ea87ec62337ad7c69b20c6927d19b4e04e"} Oct 30 17:07:05.445315 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:05.445286 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4" Oct 30 17:07:05.451664 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:05.451641 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4" Oct 30 17:07:05.495230 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:05.495175 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4" podStartSLOduration=2.495161763 podStartE2EDuration="2.495161763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:07:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:07:05.490372258 +0000 UTC m=+1199.491127198" watchObservedRunningTime="2026-10-30 17:07:05.495161763 +0000 UTC m=+1199.495916663" Oct 30 17:07:06.153814 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:06.153783 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9v2l"] Oct 30 17:07:06.158766 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:06.158744 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9v2l" Oct 30 17:07:06.178382 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:06.178343 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 17:07:06.180959 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:06.180937 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 17:07:06.181184 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:06.180960 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 17:07:06.185383 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:06.185362 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9v2l"] Oct 30 17:07:06.278983 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:06.278955 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-xjmfv\"" Oct 30 17:07:06.308808 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:06.308776 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g9v2l\" (UID: \"6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9v2l" Oct 30 17:07:06.308983 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:06.308824 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g9v2l\" (UID: \"6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9v2l" Oct 30 17:07:06.308983 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:06.308842 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g9v2l\" (UID: \"6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9v2l" Oct 30 17:07:06.308983 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:06.308975 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7bfx\" (UniqueName: \"kubernetes.io/projected/6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a-kube-api-access-d7bfx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g9v2l\" (UID: \"6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9v2l" Oct 30 17:07:06.309081 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:06.309052 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g9v2l\" (UID: \"6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9v2l" Oct 30 17:07:06.309206 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:06.309187 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g9v2l\" (UID: \"6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9v2l" Oct 30 17:07:06.309294 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:06.309279 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g9v2l\" (UID: \"6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9v2l" Oct 30 17:07:06.413945 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:06.413858 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g9v2l\" (UID: \"6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9v2l" Oct 30 17:07:06.413945 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:06.413937 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g9v2l\" (UID: \"6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9v2l" Oct 30 17:07:06.414183 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:06.413969 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g9v2l\" (UID: \"6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9v2l" Oct 30 17:07:06.414183 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:06.414028 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7bfx\" (UniqueName: \"kubernetes.io/projected/6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a-kube-api-access-d7bfx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g9v2l\" (UID: \"6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9v2l" Oct 30 17:07:06.414183 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:06.414116 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g9v2l\" (UID: \"6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9v2l" Oct 30 17:07:06.414367 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:06.414218 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g9v2l\" (UID: \"6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9v2l" Oct 30 17:07:06.414367 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:06.414326 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g9v2l\" (UID: \"6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9v2l" Oct 30 17:07:06.414940 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:06.414907 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g9v2l\" (UID: \"6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9v2l" Oct 30 17:07:06.416747 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:06.416702 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g9v2l\" (UID: \"6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9v2l" Oct 30 17:07:06.416838 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:06.416767 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g9v2l\" (UID: \"6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9v2l" Oct 30 17:07:06.416906 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:06.416882 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g9v2l\" (UID: \"6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9v2l" Oct 30 17:07:06.417103 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:06.417078 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g9v2l\" (UID: \"6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9v2l" Oct 30 17:07:06.433317 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:06.433281 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4" Oct 30 17:07:06.434510 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:06.434486 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4" Oct 30 17:07:06.440011 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:06.439989 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g9v2l\" (UID: \"6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9v2l" Oct 30 17:07:06.440209 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:06.440191 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7bfx\" (UniqueName: \"kubernetes.io/projected/6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a-kube-api-access-d7bfx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g9v2l\" (UID: \"6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9v2l" Oct 30 17:07:06.473266 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:06.473243 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9v2l" Oct 30 17:07:06.484006 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:06.483980 4522 scope.go:122] "RemoveContainer" containerID="2d91be15d4729f16224665ad75cf7dae38bafc99fc2a46f5729dcfa2963f490c" Oct 30 17:07:06.526502 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:06.526475 4522 scope.go:122] "RemoveContainer" containerID="4b2f11f60141d2b33b2301113c9b0db2c18bb1ddec1ce60fa4983407751fee99" Oct 30 17:07:06.543194 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:06.543158 4522 scope.go:122] "RemoveContainer" containerID="ecce0806f2ee29c940e91b1ee9b06b3c328972b799f85b1737e38b3dc21645d1" Oct 30 17:07:06.561746 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:06.561443 4522 scope.go:122] "RemoveContainer" containerID="88474834a289b245a637490c9e041f5fb68c42ab96eb7c5b99b1cea2a873057c" Oct 30 17:07:06.818428 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:07:06.818390 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ae5f037_50a6_4b57_b59e_b5bcbfb99f3a.slice/crio-a5a20873ab3dc1319765403e58ab55e07da380043a91a92386b564c71230aca3 WatchSource:0}: Error finding container a5a20873ab3dc1319765403e58ab55e07da380043a91a92386b564c71230aca3: Status 404 returned error can't find the container with id a5a20873ab3dc1319765403e58ab55e07da380043a91a92386b564c71230aca3 Oct 30 17:07:06.821993 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:06.821971 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9v2l"] Oct 30 17:07:07.443059 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:07.443026 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9v2l" event={"ID":"6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a","Type":"ContainerStarted","Data":"a5a20873ab3dc1319765403e58ab55e07da380043a91a92386b564c71230aca3"} Oct 30 17:07:09.353445 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:09.353413 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:07:09.353740 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:09.353475 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:07:09.463791 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:09.463760 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9v2l" event={"ID":"6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a","Type":"ContainerStarted","Data":"efd44db602cacb6307d931fa4263aae0a699d6c56b6691862282811d4c651389"} Oct 30 17:07:09.463929 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:09.463830 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9v2l" Oct 30 17:07:09.547956 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:09.547904 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9v2l" podStartSLOduration=1.016271009 podStartE2EDuration="3.547888016s" totalImagesPullingTime="2.531617007s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:07:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:07:09.528719194 +0000 UTC m=+1203.529474095" watchObservedRunningTime="2026-10-30 17:07:09.547888016 +0000 UTC m=+1203.548642918" Oct 30 17:07:10.474077 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:10.474038 4522 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-g9v2l container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:07:10.474566 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:10.474100 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9v2l" podUID="6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:07:11.370764 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:11.370734 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-11f480-5d57fc56dc-w48fx" Oct 30 17:07:11.370912 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:11.370843 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-11f480-59c5945bbd-fjf2v" Oct 30 17:07:12.952737 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:07:12.952704 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:07:13.473472 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:13.473439 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9v2l" Oct 30 17:07:18.206333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:18.138419 4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-04248e/distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq" podUID="f18303fb-6d27-4ff1-9cc9-ed032e43d098" containerName="main" probeResult="failure" output="Get \"https://10.134.0.120:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:07:18.230336 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:18.222473 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq" podUID="f18303fb-6d27-4ff1-9cc9-ed032e43d098" containerName="main" probeResult="failure" output="Get \"https://10.134.0.120:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:07:18.466461 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:18.465040 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" podUID="c9ad71b7-ad90-4c8c-bf27-0f10b23f1593" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 17:07:18.540552 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:18.540473 4522 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:07:18.540779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:18.540581 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5" podUID="6e1c3c74-139f-4299-97c4-4609e0b9e030" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:07:18.633892 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:18.633835 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" podUID="4b4ad752-e053-4cea-a3bc-fe2be6f92021" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:07:18.800596 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:18.800486 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" podUID="a3e5e732-2c67-4377-9755-e2178bf33772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 17:07:18.801264 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:18.801232 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb" podUID="d86fded1-2e04-4d73-bb22-ea0d9a61b961" containerName="main" probeResult="failure" output="Get \"https://10.134.0.127:8000/ready\": context deadline exceeded" Oct 30 17:07:18.987385 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:18.975074 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" podUID="4360e9dc-c001-4102-8af6-b55b692fcf2c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:07:19.023331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:19.019159 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" podUID="9bfbc6d3-91fd-41a7-9585-40a8f252f743" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:07:19.073403 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:19.054427 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" podUID="0b832781-3d41-4496-aa86-e4d92792e590" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:07:19.073403 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:19.054822 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/distinct-w1-04248e-kserve-5859d49965-kgq46" podUID="8dda5ac4-30dd-432e-81eb-cb6d39800628" containerName="main" probeResult="failure" output="Get \"https://10.134.0.108:8000/ready\": context deadline exceeded" Oct 30 17:07:19.092332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:19.091123 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" podUID="75321c5d-8bf8-45c6-8811-81981fe3cad4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 17:07:19.097488 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:19.097437 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" podUID="8a6a85ad-c944-450f-b0bb-e3b64b0f56f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 17:07:19.098111 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:19.098081 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" podUID="2f1e3bbb-ab1c-455d-a05d-d62f72c5d009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 17:07:19.098567 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:19.098543 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" podUID="43ca764d-ae0e-4858-9841-5bbd0ffdbb85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 17:07:19.099015 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:19.098719 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/distinct-w2-b4f406-kserve-5db4cc6969-7n846" podUID="b8810dbc-b769-46a4-90b9-72a68592bc1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:07:19.238330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:19.237949 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq" podUID="f18303fb-6d27-4ff1-9cc9-ed032e43d098" containerName="main" probeResult="failure" output="Get \"https://10.134.0.120:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:07:19.238888 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:19.238868 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04248e/distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq" Oct 30 17:07:19.331705 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:19.327116 4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" podUID="c9ad71b7-ad90-4c8c-bf27-0f10b23f1593" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:07:19.492507 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:19.492464 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" podUID="c9ad71b7-ad90-4c8c-bf27-0f10b23f1593" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 17:07:19.493117 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:19.493092 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" Oct 30 17:07:19.838332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:19.804991 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" podUID="4b4ad752-e053-4cea-a3bc-fe2be6f92021" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:07:19.897331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:19.883853 4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" podUID="8a6a85ad-c944-450f-b0bb-e3b64b0f56f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:07:19.920535 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:19.920506 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" Oct 30 17:07:19.957330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:19.948336 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb" podUID="d86fded1-2e04-4d73-bb22-ea0d9a61b961" containerName="main" probeResult="failure" output="Get \"https://10.134.0.127:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:07:19.962563 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:19.962523 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" podUID="a3e5e732-2c67-4377-9755-e2178bf33772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 17:07:19.991062 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:19.991024 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b4f406/distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb" Oct 30 17:07:19.991350 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:19.991330 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" Oct 30 17:07:19.997537 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:19.997508 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" podUID="4360e9dc-c001-4102-8af6-b55b692fcf2c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Oct 30 17:07:19.997910 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:19.997889 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" Oct 30 17:07:20.027563 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:20.027512 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" podUID="9bfbc6d3-91fd-41a7-9585-40a8f252f743" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:07:20.029580 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:20.029525 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" Oct 30 17:07:20.045081 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:20.044598 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" Oct 30 17:07:20.071212 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:20.068464 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/distinct-w1-04248e-kserve-5859d49965-kgq46" podUID="8dda5ac4-30dd-432e-81eb-cb6d39800628" containerName="main" probeResult="failure" output="Get \"https://10.134.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:07:20.071212 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:20.068772 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" podUID="0b832781-3d41-4496-aa86-e4d92792e590" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 17:07:20.071212 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:20.069356 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04248e/distinct-w1-04248e-kserve-5859d49965-kgq46" Oct 30 17:07:20.071212 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:20.069418 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" Oct 30 17:07:20.149388 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:20.105717 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" podUID="43ca764d-ae0e-4858-9841-5bbd0ffdbb85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:07:20.174746 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:20.106825 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/distinct-w2-b4f406-kserve-5db4cc6969-7n846" podUID="b8810dbc-b769-46a4-90b9-72a68592bc1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:07:20.201456 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:20.107596 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" podUID="2f1e3bbb-ab1c-455d-a05d-d62f72c5d009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 17:07:20.203879 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:20.107625 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" podUID="75321c5d-8bf8-45c6-8811-81981fe3cad4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 17:07:20.205279 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:20.109119 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" podUID="8a6a85ad-c944-450f-b0bb-e3b64b0f56f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 17:07:20.250619 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:20.249629 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq" podUID="f18303fb-6d27-4ff1-9cc9-ed032e43d098" containerName="main" probeResult="failure" output="Get \"https://10.134.0.120:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:07:20.364334 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:20.352652 4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b4f406/distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb" podUID="d86fded1-2e04-4d73-bb22-ea0d9a61b961" containerName="main" probeResult="failure" output="Get \"https://10.134.0.127:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:07:20.513331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:20.511240 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" podUID="c9ad71b7-ad90-4c8c-bf27-0f10b23f1593" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 17:07:20.538331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:20.517055 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" Oct 30 17:07:20.538331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:20.517121 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b4f406/distinct-w2-b4f406-kserve-5db4cc6969-7n846" Oct 30 17:07:20.538331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:20.517160 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" Oct 30 17:07:20.538331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:20.517193 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" Oct 30 17:07:20.538331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:20.517226 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" Oct 30 17:07:20.538331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:20.517256 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04248e/distinct-w1-04248e-kserve-5859d49965-kgq46" Oct 30 17:07:20.831380 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:20.812377 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" Oct 30 17:07:20.831380 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:20.817680 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" Oct 30 17:07:20.898359 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:20.898257 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b4f406/distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb" Oct 30 17:07:21.005581 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:21.003431 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" podUID="4360e9dc-c001-4102-8af6-b55b692fcf2c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:07:21.084414 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:21.084281 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" Oct 30 17:07:21.085417 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:21.085375 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" podUID="0b832781-3d41-4496-aa86-e4d92792e590" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:07:21.086206 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:21.086029 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" podUID="9bfbc6d3-91fd-41a7-9585-40a8f252f743" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 30 17:07:21.086958 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:21.086943 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" Oct 30 17:07:21.119047 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:21.118971 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" Oct 30 17:07:21.180332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:21.175507 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/distinct-w2-b4f406-kserve-5db4cc6969-7n846" podUID="b8810dbc-b769-46a4-90b9-72a68592bc1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:07:21.186335 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:21.184848 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04248e/distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq" Oct 30 17:07:21.188870 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:21.188847 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" Oct 30 17:07:21.308080 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:21.299131 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" Oct 30 17:07:21.343623 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:21.343545 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b4f406/distinct-w2-b4f406-kserve-5db4cc6969-7n846" Oct 30 17:07:21.408341 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:21.395340 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" Oct 30 17:07:21.571763 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:21.571728 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43815402-p2dnc"] Oct 30 17:07:21.581754 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:21.581727 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43815402-p2dnc"] Oct 30 17:07:21.584733 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:21.584713 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" Oct 30 17:07:22.155964 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:22.155930 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb92d902-69e6-4f48-946e-035115b4559a" path="/var/lib/kubelet/pods/bb92d902-69e6-4f48-946e-035115b4559a/volumes" Oct 30 17:07:25.059329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:25.052419 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4aba8b6b-7cd85f69d8-7skxc"] Oct 30 17:07:30.053811 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:30.053746 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-4aba8b6b-7cd85f69d8-7skxc" podUID="4f44bf3d-daea-4511-b385-a39d5ea7396c" containerName="payload-processing" containerID="cri-o://eeb62ad8089827ae8899445752302f02bb9ff6a5be1438d4c8511918d51e3065" gracePeriod=25 Oct 30 17:07:30.258975 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:30.258943 4522 generic.go:356] "Generic (PLEG): container finished" podID="4f44bf3d-daea-4511-b385-a39d5ea7396c" containerID="eeb62ad8089827ae8899445752302f02bb9ff6a5be1438d4c8511918d51e3065" exitCode=0 Oct 30 17:07:30.259117 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:30.259008 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4aba8b6b-7cd85f69d8-7skxc" event={"ID":"4f44bf3d-daea-4511-b385-a39d5ea7396c","Type":"ContainerDied","Data":"eeb62ad8089827ae8899445752302f02bb9ff6a5be1438d4c8511918d51e3065"} Oct 30 17:07:30.351838 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:30.351812 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4aba8b6b-7cd85f69d8-7skxc" Oct 30 17:07:30.390574 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:30.390543 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f44bf3d-daea-4511-b385-a39d5ea7396c-kube-api-access-hkbv5\" (UniqueName: \"kubernetes.io/projected/4f44bf3d-daea-4511-b385-a39d5ea7396c-kube-api-access-hkbv5\") pod \"4f44bf3d-daea-4511-b385-a39d5ea7396c\" (UID: \"4f44bf3d-daea-4511-b385-a39d5ea7396c\") " Oct 30 17:07:30.390714 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:30.390610 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f44bf3d-daea-4511-b385-a39d5ea7396c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f44bf3d-daea-4511-b385-a39d5ea7396c-plugins-config-volume\") pod \"4f44bf3d-daea-4511-b385-a39d5ea7396c\" (UID: \"4f44bf3d-daea-4511-b385-a39d5ea7396c\") " Oct 30 17:07:30.390980 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:30.390958 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f44bf3d-daea-4511-b385-a39d5ea7396c-plugins-config-volume" pod "4f44bf3d-daea-4511-b385-a39d5ea7396c" (UID: "4f44bf3d-daea-4511-b385-a39d5ea7396c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:07:30.391486 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:30.391467 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f44bf3d-daea-4511-b385-a39d5ea7396c-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:07:30.392740 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:30.392721 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f44bf3d-daea-4511-b385-a39d5ea7396c-kube-api-access-hkbv5" pod "4f44bf3d-daea-4511-b385-a39d5ea7396c" (UID: "4f44bf3d-daea-4511-b385-a39d5ea7396c"). InnerVolumeSpecName "kube-api-access-hkbv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:07:30.493062 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:30.493030 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkbv5\" (UniqueName: \"kubernetes.io/projected/4f44bf3d-daea-4511-b385-a39d5ea7396c-kube-api-access-hkbv5\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:07:31.269220 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:31.269190 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4aba8b6b-7cd85f69d8-7skxc" Oct 30 17:07:31.269220 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:31.269194 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4aba8b6b-7cd85f69d8-7skxc" event={"ID":"4f44bf3d-daea-4511-b385-a39d5ea7396c","Type":"ContainerDied","Data":"3a4fb2850086a8e4faac30ad6d5734828f4b4006113247b1b9d907ac97fea688"} Oct 30 17:07:31.269774 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:31.269270 4522 scope.go:122] "RemoveContainer" containerID="eeb62ad8089827ae8899445752302f02bb9ff6a5be1438d4c8511918d51e3065" Oct 30 17:07:31.333081 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:31.333045 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4aba8b6b-7cd85f69d8-7skxc"] Oct 30 17:07:31.339354 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:31.339320 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4aba8b6b-7cd85f69d8-7skxc"] Oct 30 17:07:32.146675 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:32.146641 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f44bf3d-daea-4511-b385-a39d5ea7396c" path="/var/lib/kubelet/pods/4f44bf3d-daea-4511-b385-a39d5ea7396c/volumes" Oct 30 17:07:42.948459 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:07:42.948423 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:07:50.460633 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:50.455162 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-6d4f99-59dc8b84b6-z9qh6"] Oct 30 17:07:50.460633 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:50.456220 4522 cpu_manager.go:414] "containerMap: removing container" podUID="4f44bf3d-daea-4511-b385-a39d5ea7396c" containerName="payload-processing" Oct 30 17:07:50.460633 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:50.456238 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f44bf3d-daea-4511-b385-a39d5ea7396c" containerName="payload-processing" Oct 30 17:07:50.460633 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:50.456627 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f44bf3d-daea-4511-b385-a39d5ea7396c" containerName="payload-processing" Oct 30 17:07:50.462521 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:50.462415 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-6d4f99-59dc8b84b6-z9qh6"] Oct 30 17:07:50.465569 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:50.465542 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6d4f99-59dc8b84b6-z9qh6" Oct 30 17:07:50.502982 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:50.500087 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-6d4f99\"" Oct 30 17:07:50.521987 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:50.521955 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-6d4f99-dockercfg-jtbbp\"" Oct 30 17:07:50.652272 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:50.652238 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49b34c5e-95a9-43e7-8eb4-0034619e1733-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-6d4f99-59dc8b84b6-z9qh6\" (UID: \"49b34c5e-95a9-43e7-8eb4-0034619e1733\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6d4f99-59dc8b84b6-z9qh6" Oct 30 17:07:50.652559 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:50.652541 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9895\" (UniqueName: \"kubernetes.io/projected/49b34c5e-95a9-43e7-8eb4-0034619e1733-kube-api-access-d9895\") pod \"payload-processing-e2e-shared-b-w2-6d4f99-59dc8b84b6-z9qh6\" (UID: \"49b34c5e-95a9-43e7-8eb4-0034619e1733\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6d4f99-59dc8b84b6-z9qh6" Oct 30 17:07:50.754281 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:50.754182 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49b34c5e-95a9-43e7-8eb4-0034619e1733-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-6d4f99-59dc8b84b6-z9qh6\" (UID: \"49b34c5e-95a9-43e7-8eb4-0034619e1733\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6d4f99-59dc8b84b6-z9qh6" Oct 30 17:07:50.754467 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:50.754418 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9895\" (UniqueName: \"kubernetes.io/projected/49b34c5e-95a9-43e7-8eb4-0034619e1733-kube-api-access-d9895\") pod \"payload-processing-e2e-shared-b-w2-6d4f99-59dc8b84b6-z9qh6\" (UID: \"49b34c5e-95a9-43e7-8eb4-0034619e1733\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6d4f99-59dc8b84b6-z9qh6" Oct 30 17:07:50.755079 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:50.755046 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49b34c5e-95a9-43e7-8eb4-0034619e1733-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-6d4f99-59dc8b84b6-z9qh6\" (UID: \"49b34c5e-95a9-43e7-8eb4-0034619e1733\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6d4f99-59dc8b84b6-z9qh6" Oct 30 17:07:50.781350 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:50.781291 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9895\" (UniqueName: \"kubernetes.io/projected/49b34c5e-95a9-43e7-8eb4-0034619e1733-kube-api-access-d9895\") pod \"payload-processing-e2e-shared-b-w2-6d4f99-59dc8b84b6-z9qh6\" (UID: \"49b34c5e-95a9-43e7-8eb4-0034619e1733\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6d4f99-59dc8b84b6-z9qh6" Oct 30 17:07:50.803343 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:50.803279 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6d4f99-59dc8b84b6-z9qh6" Oct 30 17:07:51.127760 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:07:51.127726 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49b34c5e_95a9_43e7_8eb4_0034619e1733.slice/crio-aa2b6b4ce4ea389cd23264626ca86470fb90791baaa3000bd799893bb0caa1c4 WatchSource:0}: Error finding container aa2b6b4ce4ea389cd23264626ca86470fb90791baaa3000bd799893bb0caa1c4: Status 404 returned error can't find the container with id aa2b6b4ce4ea389cd23264626ca86470fb90791baaa3000bd799893bb0caa1c4 Oct 30 17:07:51.130575 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:51.130551 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-6d4f99-59dc8b84b6-z9qh6"] Oct 30 17:07:51.526005 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:51.525964 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6d4f99-59dc8b84b6-z9qh6" event={"ID":"49b34c5e-95a9-43e7-8eb4-0034619e1733","Type":"ContainerStarted","Data":"475908fe8b9912a214bf22fe965d27ced77dc9aacc8f21d968bf92781fb34081"} Oct 30 17:07:51.526452 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:51.526033 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6d4f99-59dc8b84b6-z9qh6" event={"ID":"49b34c5e-95a9-43e7-8eb4-0034619e1733","Type":"ContainerStarted","Data":"aa2b6b4ce4ea389cd23264626ca86470fb90791baaa3000bd799893bb0caa1c4"} Oct 30 17:07:51.526452 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:51.526060 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6d4f99-59dc8b84b6-z9qh6" Oct 30 17:07:51.576334 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:51.575426 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6d4f99-59dc8b84b6-z9qh6" podStartSLOduration=1.5754067950000001 podStartE2EDuration="1.575406795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:07:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:07:51.571828442 +0000 UTC m=+1245.572583372" watchObservedRunningTime="2026-10-30 17:07:51.575406795 +0000 UTC m=+1245.576161697" Oct 30 17:07:54.087361 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:54.086393 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" podUID="c9ad71b7-ad90-4c8c-bf27-0f10b23f1593" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 17:07:54.168330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:54.147456 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb" podUID="d86fded1-2e04-4d73-bb22-ea0d9a61b961" containerName="main" probeResult="failure" output="Get \"https://10.134.0.127:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:07:54.238156 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:54.237438 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq" podUID="f18303fb-6d27-4ff1-9cc9-ed032e43d098" containerName="main" probeResult="failure" output="Get \"https://10.134.0.120:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:07:54.349523 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:54.349411 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/distinct-w2-b4f406-kserve-5db4cc6969-7n846" podUID="b8810dbc-b769-46a4-90b9-72a68592bc1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:07:54.417330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:54.413596 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" podUID="4360e9dc-c001-4102-8af6-b55b692fcf2c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:07:54.460658 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:54.460615 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" podUID="2f1e3bbb-ab1c-455d-a05d-d62f72c5d009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 17:07:54.827477 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:54.827424 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" podUID="43ca764d-ae0e-4858-9841-5bbd0ffdbb85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:07:54.828138 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:54.828109 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" podUID="a3e5e732-2c67-4377-9755-e2178bf33772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 17:07:55.093099 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:55.092995 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" podUID="0b832781-3d41-4496-aa86-e4d92792e590" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:07:55.094866 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:55.094762 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" podUID="c9ad71b7-ad90-4c8c-bf27-0f10b23f1593" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:07:55.095817 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:55.095792 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" Oct 30 17:07:55.146330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:55.142696 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" podUID="75321c5d-8bf8-45c6-8811-81981fe3cad4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:07:55.221420 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:55.217787 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb" podUID="d86fded1-2e04-4d73-bb22-ea0d9a61b961" containerName="main" probeResult="failure" output="Get \"https://10.134.0.127:8000/ready\": context deadline exceeded" Oct 30 17:07:55.221420 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:55.218189 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b4f406/distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb" Oct 30 17:07:55.266353 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:55.264227 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq" podUID="f18303fb-6d27-4ff1-9cc9-ed032e43d098" containerName="main" probeResult="failure" output="Get \"https://10.134.0.120:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:07:55.266353 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:55.264772 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04248e/distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq" Oct 30 17:07:55.426206 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:55.426102 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" podUID="4360e9dc-c001-4102-8af6-b55b692fcf2c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:07:55.468613 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:55.429265 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/distinct-w2-b4f406-kserve-5db4cc6969-7n846" podUID="b8810dbc-b769-46a4-90b9-72a68592bc1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:07:55.468858 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:55.468810 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" Oct 30 17:07:55.468936 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:55.468929 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b4f406/distinct-w2-b4f406-kserve-5db4cc6969-7n846" Oct 30 17:07:55.491167 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:55.491137 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" Oct 30 17:07:55.539968 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:55.533832 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b4f406/distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb" Oct 30 17:07:55.539968 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:55.534242 4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" podUID="43ca764d-ae0e-4858-9841-5bbd0ffdbb85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:07:55.539968 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:55.538068 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04248e/distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq" Oct 30 17:07:55.546173 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:55.546084 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" Oct 30 17:07:55.553817 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:55.553796 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b4f406/distinct-w2-b4f406-kserve-5db4cc6969-7n846" Oct 30 17:07:57.037063 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:57.037027 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9810290f-2lnpx"] Oct 30 17:07:57.051569 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:57.051535 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9810290f-2lnpx"] Oct 30 17:07:58.150357 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:07:58.150323 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16d26c33-9ea4-406c-b1d5-0cb773380404" path="/var/lib/kubelet/pods/16d26c33-9ea4-406c-b1d5-0cb773380404/volumes" Oct 30 17:08:01.705162 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:01.705133 4522 generic.go:356] "Generic (PLEG): container finished" podID="ef9c4eec-8676-4a8c-9913-da321bb96baf" containerID="adf01b0a01c87f846fad0f93969483a10b706c4c156414f8d5a068c9d364f732" exitCode=0 Oct 30 17:08:01.705624 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:01.705201 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4" event={"ID":"ef9c4eec-8676-4a8c-9913-da321bb96baf","Type":"ContainerDied","Data":"adf01b0a01c87f846fad0f93969483a10b706c4c156414f8d5a068c9d364f732"} Oct 30 17:08:01.705797 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:01.705782 4522 scope.go:122] "RemoveContainer" containerID="adf01b0a01c87f846fad0f93969483a10b706c4c156414f8d5a068c9d364f732" Oct 30 17:08:01.707780 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:01.707631 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:08:01.707780 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:01.707710 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:08:01.707780 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:01.707763 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:08:02.562170 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:02.562141 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6d4f99-59dc8b84b6-z9qh6" Oct 30 17:08:02.736543 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:02.736504 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4" event={"ID":"ef9c4eec-8676-4a8c-9913-da321bb96baf","Type":"ContainerStarted","Data":"de6f92a0ac08af360214a01412eea84e8ea5fa6c0c56e2880e309b1ea89643e9"} Oct 30 17:08:03.445618 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:03.445584 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4" Oct 30 17:08:03.447464 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:03.447430 4522 patch_prober.go:29] interesting pod/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 17:08:03.447693 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:03.447664 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4" podUID="ef9c4eec-8676-4a8c-9913-da321bb96baf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 17:08:04.444744 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:04.444712 4522 patch_prober.go:29] interesting pod/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 17:08:04.445174 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:04.444780 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4" podUID="ef9c4eec-8676-4a8c-9913-da321bb96baf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 17:08:05.221781 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:05.221745 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a43843-86c7c5b7-dxhbc"] Oct 30 17:08:05.225723 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:05.225702 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a43843-86c7c5b7-dxhbc" Oct 30 17:08:05.243924 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:05.243892 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a43843-86c7c5b7-dxhbc"] Oct 30 17:08:05.248795 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:05.248768 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-a43843\"" Oct 30 17:08:05.248988 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:05.248770 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-a43843-dockercfg-bqdqs\"" Oct 30 17:08:05.350855 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:05.350821 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f396ca5e-93c9-4dd4-a03e-f9150e784873-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-a43843-86c7c5b7-dxhbc\" (UID: \"f396ca5e-93c9-4dd4-a03e-f9150e784873\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a43843-86c7c5b7-dxhbc" Oct 30 17:08:05.351083 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:05.351062 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbt7h\" (UniqueName: \"kubernetes.io/projected/f396ca5e-93c9-4dd4-a03e-f9150e784873-kube-api-access-xbt7h\") pod \"payload-pre-processing-e2e-shared-b-w1-a43843-86c7c5b7-dxhbc\" (UID: \"f396ca5e-93c9-4dd4-a03e-f9150e784873\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a43843-86c7c5b7-dxhbc" Oct 30 17:08:05.471533 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:05.471493 4522 patch_prober.go:29] interesting pod/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 17:08:05.472186 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:05.472105 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4" podUID="ef9c4eec-8676-4a8c-9913-da321bb96baf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 17:08:05.481515 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:05.481479 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f396ca5e-93c9-4dd4-a03e-f9150e784873-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-a43843-86c7c5b7-dxhbc\" (UID: \"f396ca5e-93c9-4dd4-a03e-f9150e784873\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a43843-86c7c5b7-dxhbc" Oct 30 17:08:05.481839 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:05.481769 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbt7h\" (UniqueName: \"kubernetes.io/projected/f396ca5e-93c9-4dd4-a03e-f9150e784873-kube-api-access-xbt7h\") pod \"payload-pre-processing-e2e-shared-b-w1-a43843-86c7c5b7-dxhbc\" (UID: \"f396ca5e-93c9-4dd4-a03e-f9150e784873\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a43843-86c7c5b7-dxhbc" Oct 30 17:08:05.482464 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:05.482357 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f396ca5e-93c9-4dd4-a03e-f9150e784873-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-a43843-86c7c5b7-dxhbc\" (UID: \"f396ca5e-93c9-4dd4-a03e-f9150e784873\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a43843-86c7c5b7-dxhbc" Oct 30 17:08:05.528993 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:05.528949 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbt7h\" (UniqueName: \"kubernetes.io/projected/f396ca5e-93c9-4dd4-a03e-f9150e784873-kube-api-access-xbt7h\") pod \"payload-pre-processing-e2e-shared-b-w1-a43843-86c7c5b7-dxhbc\" (UID: \"f396ca5e-93c9-4dd4-a03e-f9150e784873\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a43843-86c7c5b7-dxhbc" Oct 30 17:08:05.547869 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:05.547843 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a43843-86c7c5b7-dxhbc" Oct 30 17:08:06.126988 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:06.126953 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a43843-86c7c5b7-dxhbc"] Oct 30 17:08:06.433230 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:06.433145 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4" Oct 30 17:08:06.444616 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:06.444587 4522 patch_prober.go:29] interesting pod/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 17:08:06.444779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:06.444636 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4" podUID="ef9c4eec-8676-4a8c-9913-da321bb96baf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 17:08:06.689443 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:06.689364 4522 scope.go:122] "RemoveContainer" containerID="0acead728e9f4a6068a06c6ab55a84934c877d74ac67897c4af0a6d9ebfa362f" Oct 30 17:08:06.722666 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:06.722641 4522 scope.go:122] "RemoveContainer" containerID="3738f941f6a7d993244e8310aed350aa1b56bad3c22c48e1c269c1ad173331d4" Oct 30 17:08:06.733331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:06.733278 4522 scope.go:122] "RemoveContainer" containerID="810c545e1a8dbf8222b4aa8dfcce30e268c8e9da13283002f80ecb2025afe5b3" Oct 30 17:08:06.745263 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:06.745243 4522 scope.go:122] "RemoveContainer" containerID="1ece4e9e30d17b7b59483e7da57ee98c6ea168764ed07c547f5bd107513d4096" Oct 30 17:08:06.754869 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:06.754850 4522 scope.go:122] "RemoveContainer" containerID="cb73fc8da8a19694d9f513b2a7bbb9f9db30548966647dccf31ee17387d58747" Oct 30 17:08:06.764742 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:06.764721 4522 scope.go:122] "RemoveContainer" containerID="78a8ea388c8d5d0ea1da0c59f040b9c5c2d0a006cb3c78620391a7db2d691a40" Oct 30 17:08:06.884676 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:06.884639 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a43843-86c7c5b7-dxhbc" event={"ID":"f396ca5e-93c9-4dd4-a03e-f9150e784873","Type":"ContainerStarted","Data":"2ea2424f211d1167717197695a1bb59f56f24a1a98c0efb798094c67585c15c2"} Oct 30 17:08:06.884832 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:06.884699 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a43843-86c7c5b7-dxhbc" event={"ID":"f396ca5e-93c9-4dd4-a03e-f9150e784873","Type":"ContainerStarted","Data":"c31757f918cf270e392802503bfa327f85befb74b8743bb31870c7e05a2c4bf2"} Oct 30 17:08:06.884832 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:06.884764 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a43843-86c7c5b7-dxhbc" Oct 30 17:08:06.942225 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:06.942111 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a43843-86c7c5b7-dxhbc" podStartSLOduration=1.942095328 podStartE2EDuration="1.942095328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:08:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:08:06.937975133 +0000 UTC m=+1260.938730034" watchObservedRunningTime="2026-10-30 17:08:06.942095328 +0000 UTC m=+1260.942850229" Oct 30 17:08:07.445414 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:07.445376 4522 patch_prober.go:29] interesting pod/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 17:08:07.445608 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:07.445440 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4" podUID="ef9c4eec-8676-4a8c-9913-da321bb96baf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 17:08:08.450835 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:08.450804 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4" Oct 30 17:08:08.451977 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:08.451953 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4" Oct 30 17:08:09.037012 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:09.036981 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65f9f450-lxclp"] Oct 30 17:08:09.048970 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:09.048926 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65f9f450-lxclp"] Oct 30 17:08:10.146894 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:10.146857 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81aedff4-a52f-4d6c-b0be-6905f9703a30" path="/var/lib/kubelet/pods/81aedff4-a52f-4d6c-b0be-6905f9703a30/volumes" Oct 30 17:08:12.970554 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:08:12.970517 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:08:14.035235 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:14.035191 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b48042d6-nbn57"] Oct 30 17:08:14.048636 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:14.048607 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b48042d6-nbn57"] Oct 30 17:08:14.151073 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:14.151033 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e6051e6-4d38-43a6-a8ae-83a82263ac3f" path="/var/lib/kubelet/pods/6e6051e6-4d38-43a6-a8ae-83a82263ac3f/volumes" Oct 30 17:08:17.930913 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:17.930885 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a43843-86c7c5b7-dxhbc" Oct 30 17:08:23.040374 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:23.040333 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30d09eab-w7nmx"] Oct 30 17:08:23.049572 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:23.049546 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30d09eab-w7nmx"] Oct 30 17:08:24.149557 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:24.149515 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d232f5be-69ed-4150-908c-b5a13b78e223" path="/var/lib/kubelet/pods/d232f5be-69ed-4150-908c-b5a13b78e223/volumes" Oct 30 17:08:29.036229 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:29.036193 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6edf9460-x8x96"] Oct 30 17:08:29.048481 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:29.048455 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6edf9460-x8x96"] Oct 30 17:08:30.147422 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:30.147386 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63b543bd-6567-43b7-8f6b-138ae297ddaa" path="/var/lib/kubelet/pods/63b543bd-6567-43b7-8f6b-138ae297ddaa/volumes" Oct 30 17:08:39.111809 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:39.111775 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12388952-l92fj"] Oct 30 17:08:39.116595 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:39.116572 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12388952-l92fj" Oct 30 17:08:39.135986 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:39.135960 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12388952-l92fj"] Oct 30 17:08:39.143446 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:39.143417 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:08:39.143626 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:39.143464 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:08:39.184380 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:39.184349 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t24sk\" (UniqueName: \"kubernetes.io/projected/bb824f5f-750a-41f1-ba75-51de3cfa276c-kube-api-access-t24sk\") pod \"maas-api-revoke-sub-12388952-l92fj\" (UID: \"bb824f5f-750a-41f1-ba75-51de3cfa276c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12388952-l92fj" Oct 30 17:08:39.184708 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:39.184683 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb824f5f-750a-41f1-ba75-51de3cfa276c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12388952-l92fj\" (UID: \"bb824f5f-750a-41f1-ba75-51de3cfa276c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12388952-l92fj" Oct 30 17:08:39.184772 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:39.184732 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb824f5f-750a-41f1-ba75-51de3cfa276c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12388952-l92fj\" (UID: \"bb824f5f-750a-41f1-ba75-51de3cfa276c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12388952-l92fj" Oct 30 17:08:39.286114 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:39.286066 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t24sk\" (UniqueName: \"kubernetes.io/projected/bb824f5f-750a-41f1-ba75-51de3cfa276c-kube-api-access-t24sk\") pod \"maas-api-revoke-sub-12388952-l92fj\" (UID: \"bb824f5f-750a-41f1-ba75-51de3cfa276c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12388952-l92fj" Oct 30 17:08:39.286296 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:39.286259 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb824f5f-750a-41f1-ba75-51de3cfa276c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12388952-l92fj\" (UID: \"bb824f5f-750a-41f1-ba75-51de3cfa276c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12388952-l92fj" Oct 30 17:08:39.286296 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:39.286282 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb824f5f-750a-41f1-ba75-51de3cfa276c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12388952-l92fj\" (UID: \"bb824f5f-750a-41f1-ba75-51de3cfa276c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12388952-l92fj" Oct 30 17:08:39.287023 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:39.286990 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb824f5f-750a-41f1-ba75-51de3cfa276c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12388952-l92fj\" (UID: \"bb824f5f-750a-41f1-ba75-51de3cfa276c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12388952-l92fj" Oct 30 17:08:39.288720 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:39.288690 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb824f5f-750a-41f1-ba75-51de3cfa276c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12388952-l92fj\" (UID: \"bb824f5f-750a-41f1-ba75-51de3cfa276c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12388952-l92fj" Oct 30 17:08:39.314201 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:39.314175 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t24sk\" (UniqueName: \"kubernetes.io/projected/bb824f5f-750a-41f1-ba75-51de3cfa276c-kube-api-access-t24sk\") pod \"maas-api-revoke-sub-12388952-l92fj\" (UID: \"bb824f5f-750a-41f1-ba75-51de3cfa276c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12388952-l92fj" Oct 30 17:08:39.432224 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:39.432136 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12388952-l92fj" Oct 30 17:08:39.740638 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:08:39.740601 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb824f5f_750a_41f1_ba75_51de3cfa276c.slice/crio-ebe02bfbe42e7b02d1b28e92a627f2787c6d4a5704e894c8c83d7ef74e6cf8f1 WatchSource:0}: Error finding container ebe02bfbe42e7b02d1b28e92a627f2787c6d4a5704e894c8c83d7ef74e6cf8f1: Status 404 returned error can't find the container with id ebe02bfbe42e7b02d1b28e92a627f2787c6d4a5704e894c8c83d7ef74e6cf8f1 Oct 30 17:08:39.743282 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:39.743256 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12388952-l92fj"] Oct 30 17:08:40.226134 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:40.226096 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12388952-l92fj" event={"ID":"bb824f5f-750a-41f1-ba75-51de3cfa276c","Type":"ContainerStarted","Data":"955e9113a96933e27a89161b0ddae216db049944af6d8c48c6066a8f63b968a4"} Oct 30 17:08:40.226561 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:40.226149 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12388952-l92fj" event={"ID":"bb824f5f-750a-41f1-ba75-51de3cfa276c","Type":"ContainerStarted","Data":"ebe02bfbe42e7b02d1b28e92a627f2787c6d4a5704e894c8c83d7ef74e6cf8f1"} Oct 30 17:08:40.273833 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:40.273779 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12388952-l92fj" podStartSLOduration=1.273765472 podStartE2EDuration="1.273765472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:08:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:08:40.269208342 +0000 UTC m=+1294.269963242" watchObservedRunningTime="2026-10-30 17:08:40.273765472 +0000 UTC m=+1294.274520411" Oct 30 17:08:41.237115 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:41.237081 4522 generic.go:356] "Generic (PLEG): container finished" podID="bb824f5f-750a-41f1-ba75-51de3cfa276c" containerID="955e9113a96933e27a89161b0ddae216db049944af6d8c48c6066a8f63b968a4" exitCode=0 Oct 30 17:08:41.237530 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:41.237162 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12388952-l92fj" event={"ID":"bb824f5f-750a-41f1-ba75-51de3cfa276c","Type":"ContainerDied","Data":"955e9113a96933e27a89161b0ddae216db049944af6d8c48c6066a8f63b968a4"} Oct 30 17:08:42.405541 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:42.405514 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12388952-l92fj" Oct 30 17:08:42.538342 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:42.538240 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb824f5f-750a-41f1-ba75-51de3cfa276c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb824f5f-750a-41f1-ba75-51de3cfa276c-maas-api-xks-ca\") pod \"bb824f5f-750a-41f1-ba75-51de3cfa276c\" (UID: \"bb824f5f-750a-41f1-ba75-51de3cfa276c\") " Oct 30 17:08:42.538342 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:42.538273 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb824f5f-750a-41f1-ba75-51de3cfa276c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb824f5f-750a-41f1-ba75-51de3cfa276c-maas-api-openshift-ca\") pod \"bb824f5f-750a-41f1-ba75-51de3cfa276c\" (UID: \"bb824f5f-750a-41f1-ba75-51de3cfa276c\") " Oct 30 17:08:42.538589 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:42.538378 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb824f5f-750a-41f1-ba75-51de3cfa276c-kube-api-access-t24sk\" (UniqueName: \"kubernetes.io/projected/bb824f5f-750a-41f1-ba75-51de3cfa276c-kube-api-access-t24sk\") pod \"bb824f5f-750a-41f1-ba75-51de3cfa276c\" (UID: \"bb824f5f-750a-41f1-ba75-51de3cfa276c\") " Oct 30 17:08:42.538772 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:42.538741 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb824f5f-750a-41f1-ba75-51de3cfa276c-maas-api-openshift-ca" pod "bb824f5f-750a-41f1-ba75-51de3cfa276c" (UID: "bb824f5f-750a-41f1-ba75-51de3cfa276c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:08:42.539268 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:42.539248 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb824f5f-750a-41f1-ba75-51de3cfa276c-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:08:42.541631 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:42.541604 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb824f5f-750a-41f1-ba75-51de3cfa276c-kube-api-access-t24sk" pod "bb824f5f-750a-41f1-ba75-51de3cfa276c" (UID: "bb824f5f-750a-41f1-ba75-51de3cfa276c"). InnerVolumeSpecName "kube-api-access-t24sk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:08:42.541800 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:42.541752 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb824f5f-750a-41f1-ba75-51de3cfa276c-maas-api-xks-ca" pod "bb824f5f-750a-41f1-ba75-51de3cfa276c" (UID: "bb824f5f-750a-41f1-ba75-51de3cfa276c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:08:42.641337 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:42.641271 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb824f5f-750a-41f1-ba75-51de3cfa276c-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:08:42.641337 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:42.641331 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t24sk\" (UniqueName: \"kubernetes.io/projected/bb824f5f-750a-41f1-ba75-51de3cfa276c-kube-api-access-t24sk\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:08:42.955821 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:08:42.955784 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:08:43.257928 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:43.257823 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12388952-l92fj" event={"ID":"bb824f5f-750a-41f1-ba75-51de3cfa276c","Type":"ContainerDied","Data":"ebe02bfbe42e7b02d1b28e92a627f2787c6d4a5704e894c8c83d7ef74e6cf8f1"} Oct 30 17:08:43.257928 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:43.257887 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebe02bfbe42e7b02d1b28e92a627f2787c6d4a5704e894c8c83d7ef74e6cf8f1" Oct 30 17:08:43.257928 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:43.257842 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12388952-l92fj" Oct 30 17:08:44.034315 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:44.034281 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e802cc01-qxkv9"] Oct 30 17:08:44.047734 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:44.047708 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e802cc01-qxkv9"] Oct 30 17:08:44.147015 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:44.146985 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89112b19-c140-40c2-8bd3-474c37608706" path="/var/lib/kubelet/pods/89112b19-c140-40c2-8bd3-474c37608706/volumes" Oct 30 17:08:46.032041 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:46.032012 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr"] Oct 30 17:08:46.032787 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:46.032770 4522 cpu_manager.go:414] "containerMap: removing container" podUID="bb824f5f-750a-41f1-ba75-51de3cfa276c" containerName="revoke-keys" Oct 30 17:08:46.032837 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:46.032789 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb824f5f-750a-41f1-ba75-51de3cfa276c" containerName="revoke-keys" Oct 30 17:08:46.032947 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:46.032936 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb824f5f-750a-41f1-ba75-51de3cfa276c" containerName="revoke-keys" Oct 30 17:08:46.037439 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:46.037413 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr" Oct 30 17:08:46.073022 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:46.072989 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr"] Oct 30 17:08:46.077571 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:46.077539 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-7d1e5c-openshift-default-dockercfg-wqq76\"" Oct 30 17:08:46.097955 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:46.097920 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/073c8bab-6c64-4fa5-8075-80f71c994845-credential-socket\") pod \"e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr\" (UID: \"073c8bab-6c64-4fa5-8075-80f71c994845\") " pod="openshift-ingress/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr" Oct 30 17:08:46.098119 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:46.098005 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h8zd\" (UniqueName: \"kubernetes.io/projected/073c8bab-6c64-4fa5-8075-80f71c994845-kube-api-access-6h8zd\") pod \"e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr\" (UID: \"073c8bab-6c64-4fa5-8075-80f71c994845\") " pod="openshift-ingress/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr" Oct 30 17:08:46.098290 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:46.098271 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/073c8bab-6c64-4fa5-8075-80f71c994845-workload-certs\") pod \"e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr\" (UID: \"073c8bab-6c64-4fa5-8075-80f71c994845\") " pod="openshift-ingress/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr" Oct 30 17:08:46.098713 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:46.098682 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/073c8bab-6c64-4fa5-8075-80f71c994845-istio-envoy\") pod \"e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr\" (UID: \"073c8bab-6c64-4fa5-8075-80f71c994845\") " pod="openshift-ingress/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr" Oct 30 17:08:46.098824 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:46.098806 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/073c8bab-6c64-4fa5-8075-80f71c994845-istio-podinfo\") pod \"e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr\" (UID: \"073c8bab-6c64-4fa5-8075-80f71c994845\") " pod="openshift-ingress/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr" Oct 30 17:08:46.098929 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:46.098913 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/073c8bab-6c64-4fa5-8075-80f71c994845-istio-data\") pod \"e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr\" (UID: \"073c8bab-6c64-4fa5-8075-80f71c994845\") " pod="openshift-ingress/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr" Oct 30 17:08:46.099007 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:46.098973 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/073c8bab-6c64-4fa5-8075-80f71c994845-istio-token\") pod \"e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr\" (UID: \"073c8bab-6c64-4fa5-8075-80f71c994845\") " pod="openshift-ingress/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr" Oct 30 17:08:46.099120 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:46.099105 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/073c8bab-6c64-4fa5-8075-80f71c994845-istiod-ca-cert\") pod \"e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr\" (UID: \"073c8bab-6c64-4fa5-8075-80f71c994845\") " pod="openshift-ingress/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr" Oct 30 17:08:46.099191 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:46.099138 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/073c8bab-6c64-4fa5-8075-80f71c994845-workload-socket\") pod \"e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr\" (UID: \"073c8bab-6c64-4fa5-8075-80f71c994845\") " pod="openshift-ingress/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr" Oct 30 17:08:46.328617 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:46.328511 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/073c8bab-6c64-4fa5-8075-80f71c994845-credential-socket\") pod \"e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr\" (UID: \"073c8bab-6c64-4fa5-8075-80f71c994845\") " pod="openshift-ingress/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr" Oct 30 17:08:46.328889 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:46.328867 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h8zd\" (UniqueName: \"kubernetes.io/projected/073c8bab-6c64-4fa5-8075-80f71c994845-kube-api-access-6h8zd\") pod \"e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr\" (UID: \"073c8bab-6c64-4fa5-8075-80f71c994845\") " pod="openshift-ingress/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr" Oct 30 17:08:46.329084 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:46.329066 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/073c8bab-6c64-4fa5-8075-80f71c994845-workload-certs\") pod \"e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr\" (UID: \"073c8bab-6c64-4fa5-8075-80f71c994845\") " pod="openshift-ingress/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr" Oct 30 17:08:46.329249 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:46.329234 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/073c8bab-6c64-4fa5-8075-80f71c994845-istio-envoy\") pod \"e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr\" (UID: \"073c8bab-6c64-4fa5-8075-80f71c994845\") " pod="openshift-ingress/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr" Oct 30 17:08:46.348087 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:46.348052 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/073c8bab-6c64-4fa5-8075-80f71c994845-workload-certs\") pod \"e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr\" (UID: \"073c8bab-6c64-4fa5-8075-80f71c994845\") " pod="openshift-ingress/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr" Oct 30 17:08:46.359491 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:46.329475 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/073c8bab-6c64-4fa5-8075-80f71c994845-istio-podinfo\") pod \"e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr\" (UID: \"073c8bab-6c64-4fa5-8075-80f71c994845\") " pod="openshift-ingress/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr" Oct 30 17:08:46.359891 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:46.359852 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/073c8bab-6c64-4fa5-8075-80f71c994845-istio-data\") pod \"e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr\" (UID: \"073c8bab-6c64-4fa5-8075-80f71c994845\") " pod="openshift-ingress/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr" Oct 30 17:08:46.360060 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:46.360044 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/073c8bab-6c64-4fa5-8075-80f71c994845-istio-token\") pod \"e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr\" (UID: \"073c8bab-6c64-4fa5-8075-80f71c994845\") " pod="openshift-ingress/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr" Oct 30 17:08:46.360326 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:46.360286 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/073c8bab-6c64-4fa5-8075-80f71c994845-istiod-ca-cert\") pod \"e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr\" (UID: \"073c8bab-6c64-4fa5-8075-80f71c994845\") " pod="openshift-ingress/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr" Oct 30 17:08:46.360487 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:46.360472 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/073c8bab-6c64-4fa5-8075-80f71c994845-workload-socket\") pod \"e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr\" (UID: \"073c8bab-6c64-4fa5-8075-80f71c994845\") " pod="openshift-ingress/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr" Oct 30 17:08:46.361225 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:46.361187 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/073c8bab-6c64-4fa5-8075-80f71c994845-workload-socket\") pod \"e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr\" (UID: \"073c8bab-6c64-4fa5-8075-80f71c994845\") " pod="openshift-ingress/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr" Oct 30 17:08:46.361704 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:46.361683 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/073c8bab-6c64-4fa5-8075-80f71c994845-istio-data\") pod \"e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr\" (UID: \"073c8bab-6c64-4fa5-8075-80f71c994845\") " pod="openshift-ingress/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr" Oct 30 17:08:46.363111 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:46.363077 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/073c8bab-6c64-4fa5-8075-80f71c994845-istiod-ca-cert\") pod \"e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr\" (UID: \"073c8bab-6c64-4fa5-8075-80f71c994845\") " pod="openshift-ingress/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr" Oct 30 17:08:46.371210 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:46.370686 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/073c8bab-6c64-4fa5-8075-80f71c994845-credential-socket\") pod \"e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr\" (UID: \"073c8bab-6c64-4fa5-8075-80f71c994845\") " pod="openshift-ingress/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr" Oct 30 17:08:46.374410 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:46.374383 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/073c8bab-6c64-4fa5-8075-80f71c994845-istio-envoy\") pod \"e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr\" (UID: \"073c8bab-6c64-4fa5-8075-80f71c994845\") " pod="openshift-ingress/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr" Oct 30 17:08:46.374666 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:46.374638 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/073c8bab-6c64-4fa5-8075-80f71c994845-istio-podinfo\") pod \"e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr\" (UID: \"073c8bab-6c64-4fa5-8075-80f71c994845\") " pod="openshift-ingress/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr" Oct 30 17:08:46.384136 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:46.384100 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h8zd\" (UniqueName: \"kubernetes.io/projected/073c8bab-6c64-4fa5-8075-80f71c994845-kube-api-access-6h8zd\") pod \"e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr\" (UID: \"073c8bab-6c64-4fa5-8075-80f71c994845\") " pod="openshift-ingress/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr" Oct 30 17:08:46.385933 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:46.385911 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/073c8bab-6c64-4fa5-8075-80f71c994845-istio-token\") pod \"e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr\" (UID: \"073c8bab-6c64-4fa5-8075-80f71c994845\") " pod="openshift-ingress/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr" Oct 30 17:08:46.661646 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:46.661563 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr" Oct 30 17:08:47.003097 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:08:47.003066 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod073c8bab_6c64_4fa5_8075_80f71c994845.slice/crio-8693cda88f5e5e9b795721ff2b56f8091097adf4e1d139f67aec82c4e7cebf6e WatchSource:0}: Error finding container 8693cda88f5e5e9b795721ff2b56f8091097adf4e1d139f67aec82c4e7cebf6e: Status 404 returned error can't find the container with id 8693cda88f5e5e9b795721ff2b56f8091097adf4e1d139f67aec82c4e7cebf6e Oct 30 17:08:47.006222 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:47.006200 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr"] Oct 30 17:08:47.006420 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:47.006391 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:08:47.006470 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:47.006455 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:08:47.006509 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:47.006486 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:08:47.387324 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:47.387236 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr" event={"ID":"073c8bab-6c64-4fa5-8075-80f71c994845","Type":"ContainerStarted","Data":"fa24750506a1ec43eac352c1a2660077859f7a98e44268cf9240cc97136bb8d7"} Oct 30 17:08:47.387324 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:47.387295 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr" event={"ID":"073c8bab-6c64-4fa5-8075-80f71c994845","Type":"ContainerStarted","Data":"8693cda88f5e5e9b795721ff2b56f8091097adf4e1d139f67aec82c4e7cebf6e"} Oct 30 17:08:47.451621 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:47.451567 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr" podStartSLOduration=2.451552829 podStartE2EDuration="2.451552829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:08:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:08:47.450977331 +0000 UTC m=+1301.451732233" watchObservedRunningTime="2026-10-30 17:08:47.451552829 +0000 UTC m=+1301.452307730" Oct 30 17:08:47.662588 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:47.662511 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr" Oct 30 17:08:48.668479 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:48.668448 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr" Oct 30 17:08:49.035349 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:49.035296 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-205ecb34-t9bsr"] Oct 30 17:08:49.040753 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:49.040727 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-205ecb34-t9bsr"] Oct 30 17:08:49.405413 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:49.405334 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr" Oct 30 17:08:49.406434 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:49.406415 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr" Oct 30 17:08:50.147159 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:50.147127 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54ff10e5-bdc9-4077-89eb-de38fe313bd3" path="/var/lib/kubelet/pods/54ff10e5-bdc9-4077-89eb-de38fe313bd3/volumes" Oct 30 17:08:51.852207 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:51.852175 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7d1e5c-79c87484bc-6dx5v"] Oct 30 17:08:51.857131 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:51.857110 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d1e5c-79c87484bc-6dx5v" Oct 30 17:08:51.880065 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:51.880038 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7d1e5c-79c87484bc-6dx5v"] Oct 30 17:08:51.887854 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:51.887828 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-7d1e5c\"" Oct 30 17:08:51.888126 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:51.887971 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-7d1e5c-dockercfg-swss4\"" Oct 30 17:08:51.930761 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:51.930716 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7d1e5c-547b5cccb8-r8865"] Oct 30 17:08:52.016331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:51.996565 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7d1e5c-547b5cccb8-r8865" Oct 30 17:08:52.061589 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:52.016656 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7x8f\" (UniqueName: \"kubernetes.io/projected/d40e76c4-53d4-4ad0-8952-46a6bac79333-kube-api-access-d7x8f\") pod \"payload-pre-processing-e2e-api-a-7d1e5c-79c87484bc-6dx5v\" (UID: \"d40e76c4-53d4-4ad0-8952-46a6bac79333\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d1e5c-79c87484bc-6dx5v" Oct 30 17:08:52.061589 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:52.016777 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d40e76c4-53d4-4ad0-8952-46a6bac79333-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-7d1e5c-79c87484bc-6dx5v\" (UID: \"d40e76c4-53d4-4ad0-8952-46a6bac79333\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d1e5c-79c87484bc-6dx5v" Oct 30 17:08:52.149653 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:52.122648 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee88e1ad-1e39-49b4-8a3e-70455749bbe6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-7d1e5c-547b5cccb8-r8865\" (UID: \"ee88e1ad-1e39-49b4-8a3e-70455749bbe6\") " pod="openshift-ingress/payload-processing-e2e-api-a-7d1e5c-547b5cccb8-r8865" Oct 30 17:08:52.149653 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:52.122702 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d40e76c4-53d4-4ad0-8952-46a6bac79333-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-7d1e5c-79c87484bc-6dx5v\" (UID: \"d40e76c4-53d4-4ad0-8952-46a6bac79333\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d1e5c-79c87484bc-6dx5v" Oct 30 17:08:52.149653 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:52.122842 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x72w\" (UniqueName: \"kubernetes.io/projected/ee88e1ad-1e39-49b4-8a3e-70455749bbe6-kube-api-access-2x72w\") pod \"payload-processing-e2e-api-a-7d1e5c-547b5cccb8-r8865\" (UID: \"ee88e1ad-1e39-49b4-8a3e-70455749bbe6\") " pod="openshift-ingress/payload-processing-e2e-api-a-7d1e5c-547b5cccb8-r8865" Oct 30 17:08:52.149653 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:52.123063 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7x8f\" (UniqueName: \"kubernetes.io/projected/d40e76c4-53d4-4ad0-8952-46a6bac79333-kube-api-access-d7x8f\") pod \"payload-pre-processing-e2e-api-a-7d1e5c-79c87484bc-6dx5v\" (UID: \"d40e76c4-53d4-4ad0-8952-46a6bac79333\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d1e5c-79c87484bc-6dx5v" Oct 30 17:08:52.149653 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:52.127226 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d40e76c4-53d4-4ad0-8952-46a6bac79333-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-7d1e5c-79c87484bc-6dx5v\" (UID: \"d40e76c4-53d4-4ad0-8952-46a6bac79333\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d1e5c-79c87484bc-6dx5v" Oct 30 17:08:52.184127 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:52.184094 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7x8f\" (UniqueName: \"kubernetes.io/projected/d40e76c4-53d4-4ad0-8952-46a6bac79333-kube-api-access-d7x8f\") pod \"payload-pre-processing-e2e-api-a-7d1e5c-79c87484bc-6dx5v\" (UID: \"d40e76c4-53d4-4ad0-8952-46a6bac79333\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d1e5c-79c87484bc-6dx5v" Oct 30 17:08:52.186672 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:52.186644 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d1e5c-79c87484bc-6dx5v" Oct 30 17:08:52.215489 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:52.214560 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7d1e5c-547b5cccb8-r8865"] Oct 30 17:08:52.225237 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:52.225199 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee88e1ad-1e39-49b4-8a3e-70455749bbe6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-7d1e5c-547b5cccb8-r8865\" (UID: \"ee88e1ad-1e39-49b4-8a3e-70455749bbe6\") " pod="openshift-ingress/payload-processing-e2e-api-a-7d1e5c-547b5cccb8-r8865" Oct 30 17:08:52.225456 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:52.225428 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x72w\" (UniqueName: \"kubernetes.io/projected/ee88e1ad-1e39-49b4-8a3e-70455749bbe6-kube-api-access-2x72w\") pod \"payload-processing-e2e-api-a-7d1e5c-547b5cccb8-r8865\" (UID: \"ee88e1ad-1e39-49b4-8a3e-70455749bbe6\") " pod="openshift-ingress/payload-processing-e2e-api-a-7d1e5c-547b5cccb8-r8865" Oct 30 17:08:52.226128 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:52.226106 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee88e1ad-1e39-49b4-8a3e-70455749bbe6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-7d1e5c-547b5cccb8-r8865\" (UID: \"ee88e1ad-1e39-49b4-8a3e-70455749bbe6\") " pod="openshift-ingress/payload-processing-e2e-api-a-7d1e5c-547b5cccb8-r8865" Oct 30 17:08:52.253788 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:52.253748 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x72w\" (UniqueName: \"kubernetes.io/projected/ee88e1ad-1e39-49b4-8a3e-70455749bbe6-kube-api-access-2x72w\") pod \"payload-processing-e2e-api-a-7d1e5c-547b5cccb8-r8865\" (UID: \"ee88e1ad-1e39-49b4-8a3e-70455749bbe6\") " pod="openshift-ingress/payload-processing-e2e-api-a-7d1e5c-547b5cccb8-r8865" Oct 30 17:08:52.431206 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:52.431112 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7d1e5c-547b5cccb8-r8865" Oct 30 17:08:52.545229 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:52.544578 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7d1e5c-79c87484bc-6dx5v"] Oct 30 17:08:52.742613 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:08:52.742581 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee88e1ad_1e39_49b4_8a3e_70455749bbe6.slice/crio-951d20ca5c1bed7e6f3e0e666bdf10df466157b358f13ea9dd573b392b3fbcd6 WatchSource:0}: Error finding container 951d20ca5c1bed7e6f3e0e666bdf10df466157b358f13ea9dd573b392b3fbcd6: Status 404 returned error can't find the container with id 951d20ca5c1bed7e6f3e0e666bdf10df466157b358f13ea9dd573b392b3fbcd6 Oct 30 17:08:52.745851 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:52.745822 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7d1e5c-547b5cccb8-r8865"] Oct 30 17:08:53.442549 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:53.442509 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7d1e5c-547b5cccb8-r8865" event={"ID":"ee88e1ad-1e39-49b4-8a3e-70455749bbe6","Type":"ContainerStarted","Data":"ef0bf74cb880993cfa2ba7f0854c7ffbf3807baede26a4208b4980c050f4e001"} Oct 30 17:08:53.443070 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:53.442581 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-7d1e5c-547b5cccb8-r8865" Oct 30 17:08:53.443070 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:53.442605 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7d1e5c-547b5cccb8-r8865" event={"ID":"ee88e1ad-1e39-49b4-8a3e-70455749bbe6","Type":"ContainerStarted","Data":"951d20ca5c1bed7e6f3e0e666bdf10df466157b358f13ea9dd573b392b3fbcd6"} Oct 30 17:08:53.444349 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:53.444320 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d1e5c-79c87484bc-6dx5v" event={"ID":"d40e76c4-53d4-4ad0-8952-46a6bac79333","Type":"ContainerStarted","Data":"cd6d80ca8b79a32b4feaa6590c4957e302ef0709b0a25a16539f54025cc668e4"} Oct 30 17:08:53.444493 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:53.444371 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d1e5c-79c87484bc-6dx5v" event={"ID":"d40e76c4-53d4-4ad0-8952-46a6bac79333","Type":"ContainerStarted","Data":"6f662a2b61f768afae5b9811f248a978bb29e2d5f0d046dccc8cd27c1387a5c1"} Oct 30 17:08:53.444493 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:53.444397 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d1e5c-79c87484bc-6dx5v" Oct 30 17:08:53.491505 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:53.491454 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-7d1e5c-547b5cccb8-r8865" podStartSLOduration=2.491440723 podStartE2EDuration="2.491440723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:08:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:08:53.487536157 +0000 UTC m=+1307.488291060" watchObservedRunningTime="2026-10-30 17:08:53.491440723 +0000 UTC m=+1307.492195624" Oct 30 17:08:53.535853 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:08:53.535797 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d1e5c-79c87484bc-6dx5v" podStartSLOduration=2.535784417 podStartE2EDuration="2.535784417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:08:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:08:53.533872314 +0000 UTC m=+1307.534627216" watchObservedRunningTime="2026-10-30 17:08:53.535784417 +0000 UTC m=+1307.536539318" Oct 30 17:09:01.045787 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:01.045714 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f6793c9-kd4rj"] Oct 30 17:09:01.063851 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:01.063815 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f6793c9-kd4rj"] Oct 30 17:09:02.154859 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:02.154817 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b55e2d8-0ec6-44c4-ab7d-5e2a74aa8096" path="/var/lib/kubelet/pods/1b55e2d8-0ec6-44c4-ab7d-5e2a74aa8096/volumes" Oct 30 17:09:04.471613 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:04.471574 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d1e5c-79c87484bc-6dx5v" Oct 30 17:09:04.472595 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:04.472570 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-7d1e5c-547b5cccb8-r8865" Oct 30 17:09:05.034368 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:05.034334 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43acf099-5pc5b"] Oct 30 17:09:05.039446 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:05.039419 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43acf099-5pc5b"] Oct 30 17:09:06.147459 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:06.147428 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08f2d7b7-67d4-4ffa-a773-bd28a9befe03" path="/var/lib/kubelet/pods/08f2d7b7-67d4-4ffa-a773-bd28a9befe03/volumes" Oct 30 17:09:06.856876 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:06.856841 4522 scope.go:122] "RemoveContainer" containerID="63b79ea2c7d60912e2d1dcd3f716cf20cbcd064422ce781e10b35319d01257eb" Oct 30 17:09:06.874532 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:06.874506 4522 scope.go:122] "RemoveContainer" containerID="e15e04735a83948ae21f8c772527b25a72708264c120a6dc358ca3e7b41bebf0" Oct 30 17:09:06.890506 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:06.890479 4522 scope.go:122] "RemoveContainer" containerID="7b6554b0b238d9af1218131b038b154e6a0ae4ac4675e31cea18c539e0c1431f" Oct 30 17:09:08.366333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:08.366292 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-409387-6d8f8cdb55-qm6nw"] Oct 30 17:09:08.372734 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:08.372716 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-409387-6d8f8cdb55-qm6nw" Oct 30 17:09:08.397169 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:08.397138 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-409387-6d8f8cdb55-qm6nw"] Oct 30 17:09:08.400063 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:08.399968 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-409387-dockercfg-bmnql\"" Oct 30 17:09:08.400063 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:08.400030 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-409387\"" Oct 30 17:09:08.472327 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:08.457049 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l89z\" (UniqueName: \"kubernetes.io/projected/e5551ac9-3a13-4b7f-9110-b93923d75d26-kube-api-access-9l89z\") pod \"payload-pre-processing-e2e-api-a-409387-6d8f8cdb55-qm6nw\" (UID: \"e5551ac9-3a13-4b7f-9110-b93923d75d26\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-409387-6d8f8cdb55-qm6nw" Oct 30 17:09:08.472327 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:08.457396 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5551ac9-3a13-4b7f-9110-b93923d75d26-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-409387-6d8f8cdb55-qm6nw\" (UID: \"e5551ac9-3a13-4b7f-9110-b93923d75d26\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-409387-6d8f8cdb55-qm6nw" Oct 30 17:09:08.655592 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:08.609950 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5551ac9-3a13-4b7f-9110-b93923d75d26-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-409387-6d8f8cdb55-qm6nw\" (UID: \"e5551ac9-3a13-4b7f-9110-b93923d75d26\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-409387-6d8f8cdb55-qm6nw" Oct 30 17:09:08.689331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:08.656123 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l89z\" (UniqueName: \"kubernetes.io/projected/e5551ac9-3a13-4b7f-9110-b93923d75d26-kube-api-access-9l89z\") pod \"payload-pre-processing-e2e-api-a-409387-6d8f8cdb55-qm6nw\" (UID: \"e5551ac9-3a13-4b7f-9110-b93923d75d26\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-409387-6d8f8cdb55-qm6nw" Oct 30 17:09:08.700420 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:08.614240 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5551ac9-3a13-4b7f-9110-b93923d75d26-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-409387-6d8f8cdb55-qm6nw\" (UID: \"e5551ac9-3a13-4b7f-9110-b93923d75d26\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-409387-6d8f8cdb55-qm6nw" Oct 30 17:09:08.763629 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:08.763591 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-409387-6487549584-9fc97"] Oct 30 17:09:08.769247 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:08.769213 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l89z\" (UniqueName: \"kubernetes.io/projected/e5551ac9-3a13-4b7f-9110-b93923d75d26-kube-api-access-9l89z\") pod \"payload-pre-processing-e2e-api-a-409387-6d8f8cdb55-qm6nw\" (UID: \"e5551ac9-3a13-4b7f-9110-b93923d75d26\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-409387-6d8f8cdb55-qm6nw" Oct 30 17:09:08.769940 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:08.769924 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-409387-6487549584-9fc97" Oct 30 17:09:08.770984 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:08.770954 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-409387-6487549584-9fc97"] Oct 30 17:09:08.904421 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:08.904362 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmqdj\" (UniqueName: \"kubernetes.io/projected/d3cd810a-a584-4693-abbc-9146035416f3-kube-api-access-dmqdj\") pod \"payload-processing-e2e-api-a-409387-6487549584-9fc97\" (UID: \"d3cd810a-a584-4693-abbc-9146035416f3\") " pod="openshift-ingress/payload-processing-e2e-api-a-409387-6487549584-9fc97" Oct 30 17:09:08.904608 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:08.904567 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3cd810a-a584-4693-abbc-9146035416f3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-409387-6487549584-9fc97\" (UID: \"d3cd810a-a584-4693-abbc-9146035416f3\") " pod="openshift-ingress/payload-processing-e2e-api-a-409387-6487549584-9fc97" Oct 30 17:09:08.996908 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:08.996872 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-409387-6d8f8cdb55-qm6nw" Oct 30 17:09:09.006563 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:09.006534 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3cd810a-a584-4693-abbc-9146035416f3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-409387-6487549584-9fc97\" (UID: \"d3cd810a-a584-4693-abbc-9146035416f3\") " pod="openshift-ingress/payload-processing-e2e-api-a-409387-6487549584-9fc97" Oct 30 17:09:09.006904 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:09.006885 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmqdj\" (UniqueName: \"kubernetes.io/projected/d3cd810a-a584-4693-abbc-9146035416f3-kube-api-access-dmqdj\") pod \"payload-processing-e2e-api-a-409387-6487549584-9fc97\" (UID: \"d3cd810a-a584-4693-abbc-9146035416f3\") " pod="openshift-ingress/payload-processing-e2e-api-a-409387-6487549584-9fc97" Oct 30 17:09:09.007492 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:09.007467 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3cd810a-a584-4693-abbc-9146035416f3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-409387-6487549584-9fc97\" (UID: \"d3cd810a-a584-4693-abbc-9146035416f3\") " pod="openshift-ingress/payload-processing-e2e-api-a-409387-6487549584-9fc97" Oct 30 17:09:09.034754 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:09.034713 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmqdj\" (UniqueName: \"kubernetes.io/projected/d3cd810a-a584-4693-abbc-9146035416f3-kube-api-access-dmqdj\") pod \"payload-processing-e2e-api-a-409387-6487549584-9fc97\" (UID: \"d3cd810a-a584-4693-abbc-9146035416f3\") " pod="openshift-ingress/payload-processing-e2e-api-a-409387-6487549584-9fc97" Oct 30 17:09:09.125699 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:09.117627 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-409387-6487549584-9fc97" Oct 30 17:09:09.345858 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:09.344902 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-409387-6d8f8cdb55-qm6nw"] Oct 30 17:09:09.351232 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:09:09.351183 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5551ac9_3a13_4b7f_9110_b93923d75d26.slice/crio-de46503e09d85f80c5311b1d4f2304936b268eae30a01a3e4a0e0058675f532e WatchSource:0}: Error finding container de46503e09d85f80c5311b1d4f2304936b268eae30a01a3e4a0e0058675f532e: Status 404 returned error can't find the container with id de46503e09d85f80c5311b1d4f2304936b268eae30a01a3e4a0e0058675f532e Oct 30 17:09:09.626751 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:09.626706 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-409387-6487549584-9fc97"] Oct 30 17:09:09.762475 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:09.762438 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-409387-6487549584-9fc97" event={"ID":"d3cd810a-a584-4693-abbc-9146035416f3","Type":"ContainerStarted","Data":"030f89508093cb320ad3e8bc90f04d649015523d93506e83151ed5f0ea1adafe"} Oct 30 17:09:09.763095 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:09.762499 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-409387-6487549584-9fc97" Oct 30 17:09:09.763095 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:09.762523 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-409387-6487549584-9fc97" event={"ID":"d3cd810a-a584-4693-abbc-9146035416f3","Type":"ContainerStarted","Data":"fc6d6e85dae3c1ad031f6f84244fc8d2ff8237529083f3e51cde0a181808a3c6"} Oct 30 17:09:09.764204 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:09.764175 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-409387-6d8f8cdb55-qm6nw" event={"ID":"e5551ac9-3a13-4b7f-9110-b93923d75d26","Type":"ContainerStarted","Data":"73465da99ae00b79839e4ac5c64cd1628309e80dfb012a7ca969b5b001b67461"} Oct 30 17:09:09.764357 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:09.764217 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-409387-6d8f8cdb55-qm6nw" event={"ID":"e5551ac9-3a13-4b7f-9110-b93923d75d26","Type":"ContainerStarted","Data":"de46503e09d85f80c5311b1d4f2304936b268eae30a01a3e4a0e0058675f532e"} Oct 30 17:09:09.764357 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:09.764248 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-409387-6d8f8cdb55-qm6nw" Oct 30 17:09:09.812527 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:09.812397 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-409387-6487549584-9fc97" podStartSLOduration=1.8123754619999999 podStartE2EDuration="1.812375462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:09:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:09:09.805547147 +0000 UTC m=+1323.806302050" watchObservedRunningTime="2026-10-30 17:09:09.812375462 +0000 UTC m=+1323.813130391" Oct 30 17:09:09.858431 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:09.858331 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-409387-6d8f8cdb55-qm6nw" podStartSLOduration=1.858317262 podStartE2EDuration="1.858317262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:09:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:09:09.855159322 +0000 UTC m=+1323.855914239" watchObservedRunningTime="2026-10-30 17:09:09.858317262 +0000 UTC m=+1323.859072154" Oct 30 17:09:12.949592 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:09:12.949556 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:09:13.034557 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:13.034524 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b299bf6b-7kj9h"] Oct 30 17:09:13.048130 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:13.048101 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b299bf6b-7kj9h"] Oct 30 17:09:14.146975 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:14.146943 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9b1a1bc-e0d2-4325-8090-9857d12b8046" path="/var/lib/kubelet/pods/b9b1a1bc-e0d2-4325-8090-9857d12b8046/volumes" Oct 30 17:09:15.846211 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:15.846175 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-728ad7-7b4857f67-lf57w"] Oct 30 17:09:15.852007 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:15.851981 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-728ad7-7b4857f67-lf57w" Oct 30 17:09:15.872841 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:15.872815 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-728ad7-7b4857f67-lf57w"] Oct 30 17:09:15.880394 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:15.880369 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-728ad7\"" Oct 30 17:09:15.880550 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:15.880502 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-728ad7-dockercfg-c4tzt\"" Oct 30 17:09:15.923799 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:15.923763 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-728ad7-66cb7bbbd6-dwrgr"] Oct 30 17:09:15.928815 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:15.928790 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-728ad7-66cb7bbbd6-dwrgr" Oct 30 17:09:15.956011 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:15.954232 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxmpr\" (UniqueName: \"kubernetes.io/projected/58921afe-0009-4d33-b402-0c708e26fe57-kube-api-access-cxmpr\") pod \"payload-pre-processing-e2e-inf-a-728ad7-7b4857f67-lf57w\" (UID: \"58921afe-0009-4d33-b402-0c708e26fe57\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-728ad7-7b4857f67-lf57w" Oct 30 17:09:15.956011 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:15.954536 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttcj7\" (UniqueName: \"kubernetes.io/projected/3fe580bc-2cda-4b54-8b89-61ec5b147ab8-kube-api-access-ttcj7\") pod \"payload-processing-e2e-inf-a-728ad7-66cb7bbbd6-dwrgr\" (UID: \"3fe580bc-2cda-4b54-8b89-61ec5b147ab8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-728ad7-66cb7bbbd6-dwrgr" Oct 30 17:09:15.956011 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:15.954608 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fe580bc-2cda-4b54-8b89-61ec5b147ab8-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-728ad7-66cb7bbbd6-dwrgr\" (UID: \"3fe580bc-2cda-4b54-8b89-61ec5b147ab8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-728ad7-66cb7bbbd6-dwrgr" Oct 30 17:09:15.956011 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:15.954634 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58921afe-0009-4d33-b402-0c708e26fe57-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-728ad7-7b4857f67-lf57w\" (UID: \"58921afe-0009-4d33-b402-0c708e26fe57\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-728ad7-7b4857f67-lf57w" Oct 30 17:09:16.120797 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:16.116261 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxmpr\" (UniqueName: \"kubernetes.io/projected/58921afe-0009-4d33-b402-0c708e26fe57-kube-api-access-cxmpr\") pod \"payload-pre-processing-e2e-inf-a-728ad7-7b4857f67-lf57w\" (UID: \"58921afe-0009-4d33-b402-0c708e26fe57\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-728ad7-7b4857f67-lf57w" Oct 30 17:09:16.120797 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:16.116551 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttcj7\" (UniqueName: \"kubernetes.io/projected/3fe580bc-2cda-4b54-8b89-61ec5b147ab8-kube-api-access-ttcj7\") pod \"payload-processing-e2e-inf-a-728ad7-66cb7bbbd6-dwrgr\" (UID: \"3fe580bc-2cda-4b54-8b89-61ec5b147ab8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-728ad7-66cb7bbbd6-dwrgr" Oct 30 17:09:16.120797 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:16.116619 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fe580bc-2cda-4b54-8b89-61ec5b147ab8-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-728ad7-66cb7bbbd6-dwrgr\" (UID: \"3fe580bc-2cda-4b54-8b89-61ec5b147ab8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-728ad7-66cb7bbbd6-dwrgr" Oct 30 17:09:16.120797 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:16.116645 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58921afe-0009-4d33-b402-0c708e26fe57-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-728ad7-7b4857f67-lf57w\" (UID: \"58921afe-0009-4d33-b402-0c708e26fe57\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-728ad7-7b4857f67-lf57w" Oct 30 17:09:16.120797 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:16.117767 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58921afe-0009-4d33-b402-0c708e26fe57-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-728ad7-7b4857f67-lf57w\" (UID: \"58921afe-0009-4d33-b402-0c708e26fe57\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-728ad7-7b4857f67-lf57w" Oct 30 17:09:16.120797 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:16.118993 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fe580bc-2cda-4b54-8b89-61ec5b147ab8-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-728ad7-66cb7bbbd6-dwrgr\" (UID: \"3fe580bc-2cda-4b54-8b89-61ec5b147ab8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-728ad7-66cb7bbbd6-dwrgr" Oct 30 17:09:16.129334 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:16.121989 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-728ad7-66cb7bbbd6-dwrgr"] Oct 30 17:09:16.154373 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:16.154266 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttcj7\" (UniqueName: \"kubernetes.io/projected/3fe580bc-2cda-4b54-8b89-61ec5b147ab8-kube-api-access-ttcj7\") pod \"payload-processing-e2e-inf-a-728ad7-66cb7bbbd6-dwrgr\" (UID: \"3fe580bc-2cda-4b54-8b89-61ec5b147ab8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-728ad7-66cb7bbbd6-dwrgr" Oct 30 17:09:16.156716 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:16.156685 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxmpr\" (UniqueName: \"kubernetes.io/projected/58921afe-0009-4d33-b402-0c708e26fe57-kube-api-access-cxmpr\") pod \"payload-pre-processing-e2e-inf-a-728ad7-7b4857f67-lf57w\" (UID: \"58921afe-0009-4d33-b402-0c708e26fe57\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-728ad7-7b4857f67-lf57w" Oct 30 17:09:16.167341 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:16.167287 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-728ad7-7b4857f67-lf57w" Oct 30 17:09:16.379144 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:16.379055 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-728ad7-66cb7bbbd6-dwrgr" Oct 30 17:09:16.509348 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:09:16.509272 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58921afe_0009_4d33_b402_0c708e26fe57.slice/crio-ab12214f76d770e16a77ce6afb111923f1e1df0353e3f80f97bbbb6e0aac9af9 WatchSource:0}: Error finding container ab12214f76d770e16a77ce6afb111923f1e1df0353e3f80f97bbbb6e0aac9af9: Status 404 returned error can't find the container with id ab12214f76d770e16a77ce6afb111923f1e1df0353e3f80f97bbbb6e0aac9af9 Oct 30 17:09:16.518386 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:16.518363 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-728ad7-7b4857f67-lf57w"] Oct 30 17:09:16.699906 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:09:16.699876 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fe580bc_2cda_4b54_8b89_61ec5b147ab8.slice/crio-a0f6d71cfc99bd693ae7d72e349a606d45522a28f7646fcf37ad016a4204b7b2 WatchSource:0}: Error finding container a0f6d71cfc99bd693ae7d72e349a606d45522a28f7646fcf37ad016a4204b7b2: Status 404 returned error can't find the container with id a0f6d71cfc99bd693ae7d72e349a606d45522a28f7646fcf37ad016a4204b7b2 Oct 30 17:09:16.705212 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:16.705187 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-728ad7-66cb7bbbd6-dwrgr"] Oct 30 17:09:16.825858 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:16.825824 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-728ad7-66cb7bbbd6-dwrgr" event={"ID":"3fe580bc-2cda-4b54-8b89-61ec5b147ab8","Type":"ContainerStarted","Data":"b0f57b65d2cf0f86cf533b56d7049e9f9231f5ce7d8c4733857010950cc7ffce"} Oct 30 17:09:16.826103 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:16.825883 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-728ad7-66cb7bbbd6-dwrgr" event={"ID":"3fe580bc-2cda-4b54-8b89-61ec5b147ab8","Type":"ContainerStarted","Data":"a0f6d71cfc99bd693ae7d72e349a606d45522a28f7646fcf37ad016a4204b7b2"} Oct 30 17:09:16.826103 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:16.825915 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-728ad7-66cb7bbbd6-dwrgr" Oct 30 17:09:16.827617 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:16.827593 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-728ad7-7b4857f67-lf57w" event={"ID":"58921afe-0009-4d33-b402-0c708e26fe57","Type":"ContainerStarted","Data":"8948cacb23212173411a211fac0d68f5028f574dfb06f4d1034d8865b52fc210"} Oct 30 17:09:16.827736 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:16.827631 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-728ad7-7b4857f67-lf57w" event={"ID":"58921afe-0009-4d33-b402-0c708e26fe57","Type":"ContainerStarted","Data":"ab12214f76d770e16a77ce6afb111923f1e1df0353e3f80f97bbbb6e0aac9af9"} Oct 30 17:09:16.827736 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:16.827666 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-728ad7-7b4857f67-lf57w" Oct 30 17:09:16.872167 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:16.872110 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-728ad7-66cb7bbbd6-dwrgr" podStartSLOduration=1.8720947749999999 podStartE2EDuration="1.872094775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:09:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:09:16.869086859 +0000 UTC m=+1330.869841761" watchObservedRunningTime="2026-10-30 17:09:16.872094775 +0000 UTC m=+1330.872849683" Oct 30 17:09:16.915001 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:16.914954 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-728ad7-7b4857f67-lf57w" podStartSLOduration=1.914941089 podStartE2EDuration="1.914941089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:09:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:09:16.912161471 +0000 UTC m=+1330.912916373" watchObservedRunningTime="2026-10-30 17:09:16.914941089 +0000 UTC m=+1330.915696019" Oct 30 17:09:20.776426 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:20.776379 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-409387-6487549584-9fc97" Oct 30 17:09:20.776811 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:20.776455 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-409387-6d8f8cdb55-qm6nw" Oct 30 17:09:21.034199 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:21.034107 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1fa217-jhcsb"] Oct 30 17:09:21.039829 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:21.039796 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1fa217-jhcsb"] Oct 30 17:09:22.147826 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:22.147791 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42542b00-e51a-4f16-8987-26f02d7c4f85" path="/var/lib/kubelet/pods/42542b00-e51a-4f16-8987-26f02d7c4f85/volumes" Oct 30 17:09:27.840744 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:27.840709 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-728ad7-7b4857f67-lf57w" Oct 30 17:09:27.841236 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:27.840786 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-728ad7-66cb7bbbd6-dwrgr" Oct 30 17:09:28.034885 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:28.034849 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fd22682-dvzrn"] Oct 30 17:09:28.043338 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:28.043287 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fd22682-dvzrn"] Oct 30 17:09:28.147147 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:28.147066 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e15c46d-5a43-4f41-8645-43bdb8ed2845" path="/var/lib/kubelet/pods/2e15c46d-5a43-4f41-8645-43bdb8ed2845/volumes" Oct 30 17:09:36.035961 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:36.035928 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1322290-9cgh5"] Oct 30 17:09:36.049292 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:36.049262 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1322290-9cgh5"] Oct 30 17:09:36.152589 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:36.152553 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ab92463-8644-45d5-a8dc-dde087cc8f4e" path="/var/lib/kubelet/pods/1ab92463-8644-45d5-a8dc-dde087cc8f4e/volumes" Oct 30 17:09:42.965196 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:09:42.965155 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:09:44.034416 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:44.034379 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fc76a9f-c9rfl"] Oct 30 17:09:44.048046 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:44.048015 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fc76a9f-c9rfl"] Oct 30 17:09:44.147927 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:44.147898 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf80b052-02c6-43a3-87f5-6fd7c59a7c41" path="/var/lib/kubelet/pods/cf80b052-02c6-43a3-87f5-6fd7c59a7c41/volumes" Oct 30 17:09:47.035791 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:47.035751 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad0ce664-jlzrk"] Oct 30 17:09:47.049142 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:47.049114 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad0ce664-jlzrk"] Oct 30 17:09:48.147781 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:48.147741 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28e852d6-7b4b-41c6-9c45-437db3f4beca" path="/var/lib/kubelet/pods/28e852d6-7b4b-41c6-9c45-437db3f4beca/volumes" Oct 30 17:09:49.035194 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:49.035158 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-500d6aa3-klfgm"] Oct 30 17:09:49.049479 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:49.049450 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-500d6aa3-klfgm"] Oct 30 17:09:50.148165 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:50.148126 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d9871ea-340a-4768-8640-9980529335ef" path="/var/lib/kubelet/pods/7d9871ea-340a-4768-8640-9980529335ef/volumes" Oct 30 17:09:55.034937 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:55.034902 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e52a6c23-kxgkf"] Oct 30 17:09:55.049370 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:55.049338 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e52a6c23-kxgkf"] Oct 30 17:09:55.422375 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:55.422269 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ebeb6e1-bptwm"] Oct 30 17:09:55.428851 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:55.428830 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ebeb6e1-bptwm" Oct 30 17:09:55.445116 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:55.445089 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ebeb6e1-bptwm"] Oct 30 17:09:55.465375 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:55.465349 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:09:55.465611 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:55.465581 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:09:55.581277 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:55.581241 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c48e028-1439-430b-ae8e-ec12b9dce540-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-7ebeb6e1-bptwm\" (UID: \"7c48e028-1439-430b-ae8e-ec12b9dce540\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ebeb6e1-bptwm" Oct 30 17:09:55.581599 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:55.581487 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v82r\" (UniqueName: \"kubernetes.io/projected/7c48e028-1439-430b-ae8e-ec12b9dce540-kube-api-access-9v82r\") pod \"maas-api-revoke-keys-e2e-mt-7ebeb6e1-bptwm\" (UID: \"7c48e028-1439-430b-ae8e-ec12b9dce540\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ebeb6e1-bptwm" Oct 30 17:09:55.581698 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:55.581673 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c48e028-1439-430b-ae8e-ec12b9dce540-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-7ebeb6e1-bptwm\" (UID: \"7c48e028-1439-430b-ae8e-ec12b9dce540\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ebeb6e1-bptwm" Oct 30 17:09:55.683104 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:55.683012 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c48e028-1439-430b-ae8e-ec12b9dce540-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-7ebeb6e1-bptwm\" (UID: \"7c48e028-1439-430b-ae8e-ec12b9dce540\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ebeb6e1-bptwm" Oct 30 17:09:55.683275 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:55.683229 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c48e028-1439-430b-ae8e-ec12b9dce540-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-7ebeb6e1-bptwm\" (UID: \"7c48e028-1439-430b-ae8e-ec12b9dce540\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ebeb6e1-bptwm" Oct 30 17:09:55.683371 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:55.683292 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v82r\" (UniqueName: \"kubernetes.io/projected/7c48e028-1439-430b-ae8e-ec12b9dce540-kube-api-access-9v82r\") pod \"maas-api-revoke-keys-e2e-mt-7ebeb6e1-bptwm\" (UID: \"7c48e028-1439-430b-ae8e-ec12b9dce540\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ebeb6e1-bptwm" Oct 30 17:09:55.684014 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:55.683986 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c48e028-1439-430b-ae8e-ec12b9dce540-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-7ebeb6e1-bptwm\" (UID: \"7c48e028-1439-430b-ae8e-ec12b9dce540\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ebeb6e1-bptwm" Oct 30 17:09:55.685631 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:55.685611 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c48e028-1439-430b-ae8e-ec12b9dce540-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-7ebeb6e1-bptwm\" (UID: \"7c48e028-1439-430b-ae8e-ec12b9dce540\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ebeb6e1-bptwm" Oct 30 17:09:55.723146 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:55.723113 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v82r\" (UniqueName: \"kubernetes.io/projected/7c48e028-1439-430b-ae8e-ec12b9dce540-kube-api-access-9v82r\") pod \"maas-api-revoke-keys-e2e-mt-7ebeb6e1-bptwm\" (UID: \"7c48e028-1439-430b-ae8e-ec12b9dce540\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ebeb6e1-bptwm" Oct 30 17:09:55.767110 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:55.767079 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ebeb6e1-bptwm" Oct 30 17:09:56.036085 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:56.036053 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff01347a-plj8r"] Oct 30 17:09:56.040741 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:56.040710 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff01347a-plj8r"] Oct 30 17:09:56.123119 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:09:56.122891 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c48e028_1439_430b_ae8e_ec12b9dce540.slice/crio-c029220028c7148d3f7c8f2adb26ec537073ec327e7240803b8eeb0b75ae63ca WatchSource:0}: Error finding container c029220028c7148d3f7c8f2adb26ec537073ec327e7240803b8eeb0b75ae63ca: Status 404 returned error can't find the container with id c029220028c7148d3f7c8f2adb26ec537073ec327e7240803b8eeb0b75ae63ca Oct 30 17:09:56.124268 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:56.124243 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ebeb6e1-bptwm"] Oct 30 17:09:56.149668 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:56.149643 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5e2c1aa-a72b-4fdf-ad50-6714890937b1" path="/var/lib/kubelet/pods/b5e2c1aa-a72b-4fdf-ad50-6714890937b1/volumes" Oct 30 17:09:56.150156 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:56.150140 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5803ca4-1b00-4cf8-a466-77c4314d10fc" path="/var/lib/kubelet/pods/e5803ca4-1b00-4cf8-a466-77c4314d10fc/volumes" Oct 30 17:09:56.442850 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:56.442812 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ebeb6e1-bptwm" event={"ID":"7c48e028-1439-430b-ae8e-ec12b9dce540","Type":"ContainerStarted","Data":"c029220028c7148d3f7c8f2adb26ec537073ec327e7240803b8eeb0b75ae63ca"} Oct 30 17:09:57.435973 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:57.435941 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m"] Oct 30 17:09:57.440666 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:57.440641 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m" Oct 30 17:09:57.474322 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:57.471623 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-362bbf-openshift-default-dockercfg-x9svv\"" Oct 30 17:09:57.474322 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:57.472241 4522 generic.go:356] "Generic (PLEG): container finished" podID="7c48e028-1439-430b-ae8e-ec12b9dce540" containerID="6fd76ad60df611663e7eb95993ac17a802e7373ad61d70f0d3ce959b1d6afc22" exitCode=0 Oct 30 17:09:57.474322 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:57.472447 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ebeb6e1-bptwm" event={"ID":"7c48e028-1439-430b-ae8e-ec12b9dce540","Type":"ContainerDied","Data":"6fd76ad60df611663e7eb95993ac17a802e7373ad61d70f0d3ce959b1d6afc22"} Oct 30 17:09:57.522230 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:57.522172 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m"] Oct 30 17:09:57.629397 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:57.629000 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d683a0f-9de0-46dc-9772-d0915b0cb985-workload-socket\") pod \"e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m\" (UID: \"4d683a0f-9de0-46dc-9772-d0915b0cb985\") " pod="openshift-ingress/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m" Oct 30 17:09:57.629397 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:57.629090 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d683a0f-9de0-46dc-9772-d0915b0cb985-istio-token\") pod \"e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m\" (UID: \"4d683a0f-9de0-46dc-9772-d0915b0cb985\") " pod="openshift-ingress/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m" Oct 30 17:09:57.629397 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:57.629134 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d683a0f-9de0-46dc-9772-d0915b0cb985-workload-certs\") pod \"e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m\" (UID: \"4d683a0f-9de0-46dc-9772-d0915b0cb985\") " pod="openshift-ingress/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m" Oct 30 17:09:57.629397 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:57.629175 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d683a0f-9de0-46dc-9772-d0915b0cb985-credential-socket\") pod \"e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m\" (UID: \"4d683a0f-9de0-46dc-9772-d0915b0cb985\") " pod="openshift-ingress/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m" Oct 30 17:09:57.629397 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:57.629283 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d683a0f-9de0-46dc-9772-d0915b0cb985-istio-envoy\") pod \"e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m\" (UID: \"4d683a0f-9de0-46dc-9772-d0915b0cb985\") " pod="openshift-ingress/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m" Oct 30 17:09:57.741342 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:57.734540 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d683a0f-9de0-46dc-9772-d0915b0cb985-istiod-ca-cert\") pod \"e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m\" (UID: \"4d683a0f-9de0-46dc-9772-d0915b0cb985\") " pod="openshift-ingress/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m" Oct 30 17:09:57.741342 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:57.734740 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d683a0f-9de0-46dc-9772-d0915b0cb985-istio-data\") pod \"e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m\" (UID: \"4d683a0f-9de0-46dc-9772-d0915b0cb985\") " pod="openshift-ingress/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m" Oct 30 17:09:57.741342 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:57.734952 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvsr9\" (UniqueName: \"kubernetes.io/projected/4d683a0f-9de0-46dc-9772-d0915b0cb985-kube-api-access-pvsr9\") pod \"e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m\" (UID: \"4d683a0f-9de0-46dc-9772-d0915b0cb985\") " pod="openshift-ingress/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m" Oct 30 17:09:57.741342 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:57.735116 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d683a0f-9de0-46dc-9772-d0915b0cb985-istio-podinfo\") pod \"e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m\" (UID: \"4d683a0f-9de0-46dc-9772-d0915b0cb985\") " pod="openshift-ingress/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m" Oct 30 17:09:57.856331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:57.847003 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d683a0f-9de0-46dc-9772-d0915b0cb985-istio-data\") pod \"e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m\" (UID: \"4d683a0f-9de0-46dc-9772-d0915b0cb985\") " pod="openshift-ingress/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m" Oct 30 17:09:57.856331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:57.847155 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvsr9\" (UniqueName: \"kubernetes.io/projected/4d683a0f-9de0-46dc-9772-d0915b0cb985-kube-api-access-pvsr9\") pod \"e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m\" (UID: \"4d683a0f-9de0-46dc-9772-d0915b0cb985\") " pod="openshift-ingress/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m" Oct 30 17:09:57.856331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:57.847258 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d683a0f-9de0-46dc-9772-d0915b0cb985-istio-podinfo\") pod \"e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m\" (UID: \"4d683a0f-9de0-46dc-9772-d0915b0cb985\") " pod="openshift-ingress/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m" Oct 30 17:09:57.856331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:57.847328 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d683a0f-9de0-46dc-9772-d0915b0cb985-workload-socket\") pod \"e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m\" (UID: \"4d683a0f-9de0-46dc-9772-d0915b0cb985\") " pod="openshift-ingress/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m" Oct 30 17:09:57.856331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:57.847398 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d683a0f-9de0-46dc-9772-d0915b0cb985-istio-token\") pod \"e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m\" (UID: \"4d683a0f-9de0-46dc-9772-d0915b0cb985\") " pod="openshift-ingress/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m" Oct 30 17:09:57.856331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:57.847439 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d683a0f-9de0-46dc-9772-d0915b0cb985-workload-certs\") pod \"e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m\" (UID: \"4d683a0f-9de0-46dc-9772-d0915b0cb985\") " pod="openshift-ingress/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m" Oct 30 17:09:57.856331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:57.847480 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d683a0f-9de0-46dc-9772-d0915b0cb985-credential-socket\") pod \"e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m\" (UID: \"4d683a0f-9de0-46dc-9772-d0915b0cb985\") " pod="openshift-ingress/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m" Oct 30 17:09:57.856331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:57.847595 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d683a0f-9de0-46dc-9772-d0915b0cb985-istio-envoy\") pod \"e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m\" (UID: \"4d683a0f-9de0-46dc-9772-d0915b0cb985\") " pod="openshift-ingress/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m" Oct 30 17:09:57.856331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:57.847626 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d683a0f-9de0-46dc-9772-d0915b0cb985-istiod-ca-cert\") pod \"e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m\" (UID: \"4d683a0f-9de0-46dc-9772-d0915b0cb985\") " pod="openshift-ingress/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m" Oct 30 17:09:57.856331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:57.848614 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d683a0f-9de0-46dc-9772-d0915b0cb985-istiod-ca-cert\") pod \"e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m\" (UID: \"4d683a0f-9de0-46dc-9772-d0915b0cb985\") " pod="openshift-ingress/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m" Oct 30 17:09:57.856331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:57.848905 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d683a0f-9de0-46dc-9772-d0915b0cb985-istio-data\") pod \"e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m\" (UID: \"4d683a0f-9de0-46dc-9772-d0915b0cb985\") " pod="openshift-ingress/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m" Oct 30 17:09:57.856331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:57.852326 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d683a0f-9de0-46dc-9772-d0915b0cb985-workload-certs\") pod \"e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m\" (UID: \"4d683a0f-9de0-46dc-9772-d0915b0cb985\") " pod="openshift-ingress/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m" Oct 30 17:09:57.856331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:57.852624 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d683a0f-9de0-46dc-9772-d0915b0cb985-workload-socket\") pod \"e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m\" (UID: \"4d683a0f-9de0-46dc-9772-d0915b0cb985\") " pod="openshift-ingress/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m" Oct 30 17:09:57.856331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:57.853159 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d683a0f-9de0-46dc-9772-d0915b0cb985-credential-socket\") pod \"e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m\" (UID: \"4d683a0f-9de0-46dc-9772-d0915b0cb985\") " pod="openshift-ingress/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m" Oct 30 17:09:57.866379 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:57.864543 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d683a0f-9de0-46dc-9772-d0915b0cb985-istio-envoy\") pod \"e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m\" (UID: \"4d683a0f-9de0-46dc-9772-d0915b0cb985\") " pod="openshift-ingress/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m" Oct 30 17:09:57.880922 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:57.880843 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d683a0f-9de0-46dc-9772-d0915b0cb985-istio-podinfo\") pod \"e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m\" (UID: \"4d683a0f-9de0-46dc-9772-d0915b0cb985\") " pod="openshift-ingress/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m" Oct 30 17:09:57.886128 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:57.886100 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d683a0f-9de0-46dc-9772-d0915b0cb985-istio-token\") pod \"e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m\" (UID: \"4d683a0f-9de0-46dc-9772-d0915b0cb985\") " pod="openshift-ingress/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m" Oct 30 17:09:57.886935 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:57.886911 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvsr9\" (UniqueName: \"kubernetes.io/projected/4d683a0f-9de0-46dc-9772-d0915b0cb985-kube-api-access-pvsr9\") pod \"e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m\" (UID: \"4d683a0f-9de0-46dc-9772-d0915b0cb985\") " pod="openshift-ingress/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m" Oct 30 17:09:58.072807 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:58.072766 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m" Oct 30 17:09:58.484335 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:58.484296 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m"] Oct 30 17:09:58.486454 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:09:58.486424 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d683a0f_9de0_46dc_9772_d0915b0cb985.slice/crio-71b12b9d9a1d05168da6f3cb56c4dc052ec1d898e0aa2c1506b194408c3df704 WatchSource:0}: Error finding container 71b12b9d9a1d05168da6f3cb56c4dc052ec1d898e0aa2c1506b194408c3df704: Status 404 returned error can't find the container with id 71b12b9d9a1d05168da6f3cb56c4dc052ec1d898e0aa2c1506b194408c3df704 Oct 30 17:09:58.488623 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:58.488588 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:09:58.488695 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:58.488659 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:09:58.488695 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:58.488690 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:09:58.741404 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:58.740092 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ebeb6e1-bptwm" Oct 30 17:09:58.766319 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:58.766271 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c48e028-1439-430b-ae8e-ec12b9dce540-kube-api-access-9v82r\" (UniqueName: \"kubernetes.io/projected/7c48e028-1439-430b-ae8e-ec12b9dce540-kube-api-access-9v82r\") pod \"7c48e028-1439-430b-ae8e-ec12b9dce540\" (UID: \"7c48e028-1439-430b-ae8e-ec12b9dce540\") " Oct 30 17:09:58.766824 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:58.766496 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c48e028-1439-430b-ae8e-ec12b9dce540-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c48e028-1439-430b-ae8e-ec12b9dce540-maas-api-openshift-ca\") pod \"7c48e028-1439-430b-ae8e-ec12b9dce540\" (UID: \"7c48e028-1439-430b-ae8e-ec12b9dce540\") " Oct 30 17:09:58.766824 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:58.766543 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c48e028-1439-430b-ae8e-ec12b9dce540-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c48e028-1439-430b-ae8e-ec12b9dce540-maas-api-xks-ca\") pod \"7c48e028-1439-430b-ae8e-ec12b9dce540\" (UID: \"7c48e028-1439-430b-ae8e-ec12b9dce540\") " Oct 30 17:09:58.767022 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:58.766940 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c48e028-1439-430b-ae8e-ec12b9dce540-maas-api-openshift-ca" pod "7c48e028-1439-430b-ae8e-ec12b9dce540" (UID: "7c48e028-1439-430b-ae8e-ec12b9dce540"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:09:58.767454 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:58.767431 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c48e028-1439-430b-ae8e-ec12b9dce540-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:09:58.768702 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:58.768683 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c48e028-1439-430b-ae8e-ec12b9dce540-maas-api-xks-ca" pod "7c48e028-1439-430b-ae8e-ec12b9dce540" (UID: "7c48e028-1439-430b-ae8e-ec12b9dce540"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:09:58.768849 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:58.768829 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c48e028-1439-430b-ae8e-ec12b9dce540-kube-api-access-9v82r" pod "7c48e028-1439-430b-ae8e-ec12b9dce540" (UID: "7c48e028-1439-430b-ae8e-ec12b9dce540"). InnerVolumeSpecName "kube-api-access-9v82r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:09:58.873917 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:58.873213 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c48e028-1439-430b-ae8e-ec12b9dce540-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:09:58.873917 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:58.873249 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9v82r\" (UniqueName: \"kubernetes.io/projected/7c48e028-1439-430b-ae8e-ec12b9dce540-kube-api-access-9v82r\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:09:59.505332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:59.503256 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ebeb6e1-bptwm" event={"ID":"7c48e028-1439-430b-ae8e-ec12b9dce540","Type":"ContainerDied","Data":"c029220028c7148d3f7c8f2adb26ec537073ec327e7240803b8eeb0b75ae63ca"} Oct 30 17:09:59.505332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:59.503345 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c029220028c7148d3f7c8f2adb26ec537073ec327e7240803b8eeb0b75ae63ca" Oct 30 17:09:59.505332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:59.503426 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ebeb6e1-bptwm" Oct 30 17:09:59.520670 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:59.520636 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m" event={"ID":"4d683a0f-9de0-46dc-9772-d0915b0cb985","Type":"ContainerStarted","Data":"35d32d97fd6f7931d29ae91feb87af869106f6d3cf72391bf7f35b37dbd5e81a"} Oct 30 17:09:59.520860 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:59.520690 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m" event={"ID":"4d683a0f-9de0-46dc-9772-d0915b0cb985","Type":"ContainerStarted","Data":"71b12b9d9a1d05168da6f3cb56c4dc052ec1d898e0aa2c1506b194408c3df704"} Oct 30 17:09:59.587233 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:09:59.587186 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m" podStartSLOduration=2.587172946 podStartE2EDuration="2.587172946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:09:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:09:59.586378548 +0000 UTC m=+1373.587133445" watchObservedRunningTime="2026-10-30 17:09:59.587172946 +0000 UTC m=+1373.587927927" Oct 30 17:10:00.073567 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:00.073532 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m" Oct 30 17:10:00.079801 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:00.079778 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m" Oct 30 17:10:00.529432 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:00.529396 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m" Oct 30 17:10:00.530426 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:00.530408 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m" Oct 30 17:10:03.035454 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:03.035417 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d4057dd-j9pc9"] Oct 30 17:10:03.049765 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:03.049728 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d4057dd-j9pc9"] Oct 30 17:10:03.962185 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:03.962150 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-362bbf-85765567b7-kqfrd"] Oct 30 17:10:03.963038 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:03.963019 4522 cpu_manager.go:414] "containerMap: removing container" podUID="7c48e028-1439-430b-ae8e-ec12b9dce540" containerName="revoke-keys" Oct 30 17:10:03.963099 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:03.963041 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c48e028-1439-430b-ae8e-ec12b9dce540" containerName="revoke-keys" Oct 30 17:10:03.963191 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:03.963181 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c48e028-1439-430b-ae8e-ec12b9dce540" containerName="revoke-keys" Oct 30 17:10:03.967179 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:03.967160 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-362bbf-85765567b7-kqfrd" Oct 30 17:10:03.987774 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:03.987743 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-362bbf-85765567b7-kqfrd"] Oct 30 17:10:03.996817 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:03.996762 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-362bbf\"" Oct 30 17:10:03.997039 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:03.996860 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-362bbf-dockercfg-bvqdd\"" Oct 30 17:10:04.066896 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:04.066850 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8da017e0-da57-4687-96cc-505f32038b61-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-362bbf-85765567b7-kqfrd\" (UID: \"8da017e0-da57-4687-96cc-505f32038b61\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-362bbf-85765567b7-kqfrd" Oct 30 17:10:04.074281 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:04.074247 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbxcm\" (UniqueName: \"kubernetes.io/projected/8da017e0-da57-4687-96cc-505f32038b61-kube-api-access-hbxcm\") pod \"payload-pre-processing-e2e-api-b-362bbf-85765567b7-kqfrd\" (UID: \"8da017e0-da57-4687-96cc-505f32038b61\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-362bbf-85765567b7-kqfrd" Oct 30 17:10:04.262105 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:04.241903 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8da017e0-da57-4687-96cc-505f32038b61-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-362bbf-85765567b7-kqfrd\" (UID: \"8da017e0-da57-4687-96cc-505f32038b61\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-362bbf-85765567b7-kqfrd" Oct 30 17:10:04.262866 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:04.262832 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbxcm\" (UniqueName: \"kubernetes.io/projected/8da017e0-da57-4687-96cc-505f32038b61-kube-api-access-hbxcm\") pod \"payload-pre-processing-e2e-api-b-362bbf-85765567b7-kqfrd\" (UID: \"8da017e0-da57-4687-96cc-505f32038b61\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-362bbf-85765567b7-kqfrd" Oct 30 17:10:04.263635 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:04.247336 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8da017e0-da57-4687-96cc-505f32038b61-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-362bbf-85765567b7-kqfrd\" (UID: \"8da017e0-da57-4687-96cc-505f32038b61\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-362bbf-85765567b7-kqfrd" Oct 30 17:10:04.291263 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:04.291234 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbxcm\" (UniqueName: \"kubernetes.io/projected/8da017e0-da57-4687-96cc-505f32038b61-kube-api-access-hbxcm\") pod \"payload-pre-processing-e2e-api-b-362bbf-85765567b7-kqfrd\" (UID: \"8da017e0-da57-4687-96cc-505f32038b61\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-362bbf-85765567b7-kqfrd" Oct 30 17:10:04.291997 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:04.291969 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b05eaa8-4779-420d-8add-1c9bb962d90a" path="/var/lib/kubelet/pods/4b05eaa8-4779-420d-8add-1c9bb962d90a/volumes" Oct 30 17:10:04.407710 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:04.407674 4522 patch_prober.go:29] interesting pod/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 17:10:04.407890 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:04.407738 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr" podUID="073c8bab-6c64-4fa5-8075-80f71c994845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 17:10:04.571580 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:04.571495 4522 generic.go:356] "Generic (PLEG): container finished" podID="073c8bab-6c64-4fa5-8075-80f71c994845" containerID="fa24750506a1ec43eac352c1a2660077859f7a98e44268cf9240cc97136bb8d7" exitCode=0 Oct 30 17:10:04.571732 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:04.571575 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr" event={"ID":"073c8bab-6c64-4fa5-8075-80f71c994845","Type":"ContainerDied","Data":"fa24750506a1ec43eac352c1a2660077859f7a98e44268cf9240cc97136bb8d7"} Oct 30 17:10:04.572000 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:04.571981 4522 scope.go:122] "RemoveContainer" containerID="fa24750506a1ec43eac352c1a2660077859f7a98e44268cf9240cc97136bb8d7" Oct 30 17:10:04.573769 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:04.573745 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:10:04.573843 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:04.573810 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:10:04.573885 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:04.573859 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:10:04.582051 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:04.582030 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-362bbf-85765567b7-kqfrd" Oct 30 17:10:04.903145 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:10:04.903106 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8da017e0_da57_4687_96cc_505f32038b61.slice/crio-9fad12ccf903cc01702cf2f4d26084fc52ff608fec0f60ee9abdd7ec135c1924 WatchSource:0}: Error finding container 9fad12ccf903cc01702cf2f4d26084fc52ff608fec0f60ee9abdd7ec135c1924: Status 404 returned error can't find the container with id 9fad12ccf903cc01702cf2f4d26084fc52ff608fec0f60ee9abdd7ec135c1924 Oct 30 17:10:04.906941 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:04.906911 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-362bbf-85765567b7-kqfrd"] Oct 30 17:10:05.583573 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:05.583534 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-362bbf-85765567b7-kqfrd" event={"ID":"8da017e0-da57-4687-96cc-505f32038b61","Type":"ContainerStarted","Data":"9d232138584436b3b41176d2e7988d2e2e60dc5b6477e722eaa8bc1792d215f0"} Oct 30 17:10:05.584040 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:05.583592 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-362bbf-85765567b7-kqfrd" event={"ID":"8da017e0-da57-4687-96cc-505f32038b61","Type":"ContainerStarted","Data":"9fad12ccf903cc01702cf2f4d26084fc52ff608fec0f60ee9abdd7ec135c1924"} Oct 30 17:10:05.584040 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:05.583614 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-362bbf-85765567b7-kqfrd" Oct 30 17:10:05.586081 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:05.586052 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr" event={"ID":"073c8bab-6c64-4fa5-8075-80f71c994845","Type":"ContainerStarted","Data":"6edf92bc06a4e6acf907fb52c588cd0d0f7b6cd017647bb93529b310fc8f7952"} Oct 30 17:10:05.648356 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:05.643787 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-362bbf-85765567b7-kqfrd" podStartSLOduration=2.643767578 podStartE2EDuration="2.643767578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:10:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:10:05.634751379 +0000 UTC m=+1379.635506280" watchObservedRunningTime="2026-10-30 17:10:05.643767578 +0000 UTC m=+1379.644522480" Oct 30 17:10:05.662564 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:05.662531 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr" Oct 30 17:10:05.662815 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:05.662781 4522 patch_prober.go:29] interesting pod/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 17:10:05.662932 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:05.662832 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr" podUID="073c8bab-6c64-4fa5-8075-80f71c994845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 17:10:06.662919 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:06.662889 4522 patch_prober.go:29] interesting pod/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 17:10:06.663348 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:06.662937 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr" podUID="073c8bab-6c64-4fa5-8075-80f71c994845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 17:10:06.935918 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:06.935820 4522 scope.go:122] "RemoveContainer" containerID="9c93624faf76105aa4f7314a9be7276cc4df64933f8a67f3de93068248489b29" Oct 30 17:10:06.947195 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:06.947171 4522 scope.go:122] "RemoveContainer" containerID="7960ee5f6ea9dccaae9b475f800c4409b021763be44246b67b29564704e4715f" Oct 30 17:10:06.957885 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:06.957861 4522 scope.go:122] "RemoveContainer" containerID="1c5908797b81a32f971ba1ae8d082b8cf8e8acca8c8fa35e5542a6c2cc354b26" Oct 30 17:10:06.971548 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:06.971524 4522 scope.go:122] "RemoveContainer" containerID="baf1aae4838cd5706963456dd3f421484c27d50c1f3b6ffb656a8ed073efbc3b" Oct 30 17:10:06.982186 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:06.982166 4522 scope.go:122] "RemoveContainer" containerID="5097e7baa61fb493fe09b367f6e74424fd7fb91f31cddbd8ccc33ae14756d127" Oct 30 17:10:06.993905 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:06.993768 4522 scope.go:122] "RemoveContainer" containerID="019384f535cac6f7f822049e1373ba022c0db5515e23ffa8110f7d0862425da0" Oct 30 17:10:07.011623 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:07.011455 4522 scope.go:122] "RemoveContainer" containerID="abd24e3c6e73ec455d869be2d32619c6b0f0be4dcc234aa49bfdd0d9715b1018" Oct 30 17:10:07.662261 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:07.662225 4522 patch_prober.go:29] interesting pod/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 17:10:07.662513 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:07.662277 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr" podUID="073c8bab-6c64-4fa5-8075-80f71c994845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 17:10:08.662685 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:08.662649 4522 patch_prober.go:29] interesting pod/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 17:10:08.674143 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:08.674101 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr" podUID="073c8bab-6c64-4fa5-8075-80f71c994845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 17:10:09.662466 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:09.662429 4522 patch_prober.go:29] interesting pod/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 17:10:09.662684 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:09.662502 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr" podUID="073c8bab-6c64-4fa5-8075-80f71c994845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 17:10:09.955865 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:09.955832 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c425a3e3-8jg79"] Oct 30 17:10:09.962811 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:09.962787 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c425a3e3-8jg79" Oct 30 17:10:09.982544 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:09.982517 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c425a3e3-8jg79"] Oct 30 17:10:09.990023 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:09.989997 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:10:09.990145 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:09.990127 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:10:10.123677 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:10.123643 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8b3d01a-b9b2-4f94-aa60-e24c45aa4a14-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-c425a3e3-8jg79\" (UID: \"e8b3d01a-b9b2-4f94-aa60-e24c45aa4a14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c425a3e3-8jg79" Oct 30 17:10:10.123677 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:10.123690 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8zzz\" (UniqueName: \"kubernetes.io/projected/e8b3d01a-b9b2-4f94-aa60-e24c45aa4a14-kube-api-access-k8zzz\") pod \"maas-api-revoke-keys-e2e-mt-c425a3e3-8jg79\" (UID: \"e8b3d01a-b9b2-4f94-aa60-e24c45aa4a14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c425a3e3-8jg79" Oct 30 17:10:10.123969 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:10.123945 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8b3d01a-b9b2-4f94-aa60-e24c45aa4a14-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-c425a3e3-8jg79\" (UID: \"e8b3d01a-b9b2-4f94-aa60-e24c45aa4a14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c425a3e3-8jg79" Oct 30 17:10:10.225953 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:10.225865 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8b3d01a-b9b2-4f94-aa60-e24c45aa4a14-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-c425a3e3-8jg79\" (UID: \"e8b3d01a-b9b2-4f94-aa60-e24c45aa4a14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c425a3e3-8jg79" Oct 30 17:10:10.226145 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:10.226123 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8b3d01a-b9b2-4f94-aa60-e24c45aa4a14-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-c425a3e3-8jg79\" (UID: \"e8b3d01a-b9b2-4f94-aa60-e24c45aa4a14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c425a3e3-8jg79" Oct 30 17:10:10.226220 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:10.226199 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8zzz\" (UniqueName: \"kubernetes.io/projected/e8b3d01a-b9b2-4f94-aa60-e24c45aa4a14-kube-api-access-k8zzz\") pod \"maas-api-revoke-keys-e2e-mt-c425a3e3-8jg79\" (UID: \"e8b3d01a-b9b2-4f94-aa60-e24c45aa4a14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c425a3e3-8jg79" Oct 30 17:10:10.226742 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:10.226713 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8b3d01a-b9b2-4f94-aa60-e24c45aa4a14-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-c425a3e3-8jg79\" (UID: \"e8b3d01a-b9b2-4f94-aa60-e24c45aa4a14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c425a3e3-8jg79" Oct 30 17:10:10.228558 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:10.228538 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8b3d01a-b9b2-4f94-aa60-e24c45aa4a14-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-c425a3e3-8jg79\" (UID: \"e8b3d01a-b9b2-4f94-aa60-e24c45aa4a14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c425a3e3-8jg79" Oct 30 17:10:10.251929 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:10.251899 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8zzz\" (UniqueName: \"kubernetes.io/projected/e8b3d01a-b9b2-4f94-aa60-e24c45aa4a14-kube-api-access-k8zzz\") pod \"maas-api-revoke-keys-e2e-mt-c425a3e3-8jg79\" (UID: \"e8b3d01a-b9b2-4f94-aa60-e24c45aa4a14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c425a3e3-8jg79" Oct 30 17:10:10.277732 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:10.277703 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c425a3e3-8jg79" Oct 30 17:10:10.624736 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:10.624705 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c425a3e3-8jg79"] Oct 30 17:10:10.625302 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:10:10.625273 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8b3d01a_b9b2_4f94_aa60_e24c45aa4a14.slice/crio-eaa2ae903cc393c222473be98557ca9088c4d9ecd964078f0ea4d3e3151fd914 WatchSource:0}: Error finding container eaa2ae903cc393c222473be98557ca9088c4d9ecd964078f0ea4d3e3151fd914: Status 404 returned error can't find the container with id eaa2ae903cc393c222473be98557ca9088c4d9ecd964078f0ea4d3e3151fd914 Oct 30 17:10:10.663177 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:10.663148 4522 patch_prober.go:29] interesting pod/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 17:10:10.663294 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:10.663206 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr" podUID="073c8bab-6c64-4fa5-8075-80f71c994845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 17:10:10.722764 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:10.722724 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c425a3e3-8jg79" event={"ID":"e8b3d01a-b9b2-4f94-aa60-e24c45aa4a14","Type":"ContainerStarted","Data":"eaa2ae903cc393c222473be98557ca9088c4d9ecd964078f0ea4d3e3151fd914"} Oct 30 17:10:11.666956 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:11.666926 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr" Oct 30 17:10:11.667461 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:11.667332 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr" Oct 30 17:10:11.668043 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:11.668023 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr" Oct 30 17:10:11.733188 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:11.733155 4522 generic.go:356] "Generic (PLEG): container finished" podID="e8b3d01a-b9b2-4f94-aa60-e24c45aa4a14" containerID="602aeed98b61acf04200090738ece7e608656c04e428e159c582b6075dce6eb1" exitCode=0 Oct 30 17:10:11.733393 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:11.733238 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c425a3e3-8jg79" event={"ID":"e8b3d01a-b9b2-4f94-aa60-e24c45aa4a14","Type":"ContainerDied","Data":"602aeed98b61acf04200090738ece7e608656c04e428e159c582b6075dce6eb1"} Oct 30 17:10:12.050945 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:12.050907 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e98948d5-wz288"] Oct 30 17:10:12.062392 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:12.062357 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e98948d5-wz288"] Oct 30 17:10:12.147524 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:12.147491 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81dd4cb6-82cd-4ba7-832d-fd43c4432044" path="/var/lib/kubelet/pods/81dd4cb6-82cd-4ba7-832d-fd43c4432044/volumes" Oct 30 17:10:12.968856 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:10:12.968822 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:10:12.990931 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:12.990900 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c425a3e3-8jg79" Oct 30 17:10:13.093149 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:13.093053 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8b3d01a-b9b2-4f94-aa60-e24c45aa4a14-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8b3d01a-b9b2-4f94-aa60-e24c45aa4a14-maas-api-openshift-ca\") pod \"e8b3d01a-b9b2-4f94-aa60-e24c45aa4a14\" (UID: \"e8b3d01a-b9b2-4f94-aa60-e24c45aa4a14\") " Oct 30 17:10:13.093149 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:13.093106 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8b3d01a-b9b2-4f94-aa60-e24c45aa4a14-kube-api-access-k8zzz\" (UniqueName: \"kubernetes.io/projected/e8b3d01a-b9b2-4f94-aa60-e24c45aa4a14-kube-api-access-k8zzz\") pod \"e8b3d01a-b9b2-4f94-aa60-e24c45aa4a14\" (UID: \"e8b3d01a-b9b2-4f94-aa60-e24c45aa4a14\") " Oct 30 17:10:13.093486 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:13.093464 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8b3d01a-b9b2-4f94-aa60-e24c45aa4a14-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8b3d01a-b9b2-4f94-aa60-e24c45aa4a14-maas-api-xks-ca\") pod \"e8b3d01a-b9b2-4f94-aa60-e24c45aa4a14\" (UID: \"e8b3d01a-b9b2-4f94-aa60-e24c45aa4a14\") " Oct 30 17:10:13.094021 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:13.093988 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8b3d01a-b9b2-4f94-aa60-e24c45aa4a14-maas-api-openshift-ca" pod "e8b3d01a-b9b2-4f94-aa60-e24c45aa4a14" (UID: "e8b3d01a-b9b2-4f94-aa60-e24c45aa4a14"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:10:13.094556 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:13.094529 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8b3d01a-b9b2-4f94-aa60-e24c45aa4a14-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:10:13.096278 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:13.096054 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8b3d01a-b9b2-4f94-aa60-e24c45aa4a14-kube-api-access-k8zzz" pod "e8b3d01a-b9b2-4f94-aa60-e24c45aa4a14" (UID: "e8b3d01a-b9b2-4f94-aa60-e24c45aa4a14"). InnerVolumeSpecName "kube-api-access-k8zzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:10:13.097538 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:13.097503 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8b3d01a-b9b2-4f94-aa60-e24c45aa4a14-maas-api-xks-ca" pod "e8b3d01a-b9b2-4f94-aa60-e24c45aa4a14" (UID: "e8b3d01a-b9b2-4f94-aa60-e24c45aa4a14"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:10:13.196004 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:13.195957 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8b3d01a-b9b2-4f94-aa60-e24c45aa4a14-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:10:13.196004 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:13.195989 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8zzz\" (UniqueName: \"kubernetes.io/projected/e8b3d01a-b9b2-4f94-aa60-e24c45aa4a14-kube-api-access-k8zzz\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:10:13.776717 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:13.773692 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c425a3e3-8jg79" event={"ID":"e8b3d01a-b9b2-4f94-aa60-e24c45aa4a14","Type":"ContainerDied","Data":"eaa2ae903cc393c222473be98557ca9088c4d9ecd964078f0ea4d3e3151fd914"} Oct 30 17:10:13.776717 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:13.773793 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eaa2ae903cc393c222473be98557ca9088c4d9ecd964078f0ea4d3e3151fd914" Oct 30 17:10:13.776717 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:13.773883 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c425a3e3-8jg79" Oct 30 17:10:16.620200 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:16.620166 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-362bbf-85765567b7-kqfrd" Oct 30 17:10:18.556771 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:18.556736 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd5f560f-ltrhc"] Oct 30 17:10:18.557625 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:18.557608 4522 cpu_manager.go:414] "containerMap: removing container" podUID="e8b3d01a-b9b2-4f94-aa60-e24c45aa4a14" containerName="revoke-keys" Oct 30 17:10:18.557690 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:18.557626 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8b3d01a-b9b2-4f94-aa60-e24c45aa4a14" containerName="revoke-keys" Oct 30 17:10:18.557771 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:18.557762 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8b3d01a-b9b2-4f94-aa60-e24c45aa4a14" containerName="revoke-keys" Oct 30 17:10:18.562021 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:18.562001 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd5f560f-ltrhc" Oct 30 17:10:18.580452 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:18.580425 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd5f560f-ltrhc"] Oct 30 17:10:18.621104 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:18.621074 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:10:18.621324 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:18.621134 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:10:18.707071 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:18.707032 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwbx4\" (UniqueName: \"kubernetes.io/projected/1e4272a1-630d-41c7-bd48-bc0c58bd54c4-kube-api-access-fwbx4\") pod \"maas-api-revoke-sub-bd5f560f-ltrhc\" (UID: \"1e4272a1-630d-41c7-bd48-bc0c58bd54c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd5f560f-ltrhc" Oct 30 17:10:18.707400 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:18.707377 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e4272a1-630d-41c7-bd48-bc0c58bd54c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd5f560f-ltrhc\" (UID: \"1e4272a1-630d-41c7-bd48-bc0c58bd54c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd5f560f-ltrhc" Oct 30 17:10:18.707460 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:18.707412 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e4272a1-630d-41c7-bd48-bc0c58bd54c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd5f560f-ltrhc\" (UID: \"1e4272a1-630d-41c7-bd48-bc0c58bd54c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd5f560f-ltrhc" Oct 30 17:10:18.808792 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:18.808709 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e4272a1-630d-41c7-bd48-bc0c58bd54c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd5f560f-ltrhc\" (UID: \"1e4272a1-630d-41c7-bd48-bc0c58bd54c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd5f560f-ltrhc" Oct 30 17:10:18.808792 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:18.808748 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e4272a1-630d-41c7-bd48-bc0c58bd54c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd5f560f-ltrhc\" (UID: \"1e4272a1-630d-41c7-bd48-bc0c58bd54c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd5f560f-ltrhc" Oct 30 17:10:18.808988 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:18.808823 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwbx4\" (UniqueName: \"kubernetes.io/projected/1e4272a1-630d-41c7-bd48-bc0c58bd54c4-kube-api-access-fwbx4\") pod \"maas-api-revoke-sub-bd5f560f-ltrhc\" (UID: \"1e4272a1-630d-41c7-bd48-bc0c58bd54c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd5f560f-ltrhc" Oct 30 17:10:18.809515 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:18.809485 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e4272a1-630d-41c7-bd48-bc0c58bd54c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd5f560f-ltrhc\" (UID: \"1e4272a1-630d-41c7-bd48-bc0c58bd54c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd5f560f-ltrhc" Oct 30 17:10:18.811136 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:18.811116 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e4272a1-630d-41c7-bd48-bc0c58bd54c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd5f560f-ltrhc\" (UID: \"1e4272a1-630d-41c7-bd48-bc0c58bd54c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd5f560f-ltrhc" Oct 30 17:10:18.835589 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:18.835556 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwbx4\" (UniqueName: \"kubernetes.io/projected/1e4272a1-630d-41c7-bd48-bc0c58bd54c4-kube-api-access-fwbx4\") pod \"maas-api-revoke-sub-bd5f560f-ltrhc\" (UID: \"1e4272a1-630d-41c7-bd48-bc0c58bd54c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd5f560f-ltrhc" Oct 30 17:10:18.877154 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:18.877121 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd5f560f-ltrhc" Oct 30 17:10:19.035766 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:19.035728 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43e9434f-nzkjb"] Oct 30 17:10:19.040455 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:19.040422 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43e9434f-nzkjb"] Oct 30 17:10:19.194636 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:10:19.194602 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e4272a1_630d_41c7_bd48_bc0c58bd54c4.slice/crio-c3c44f4665fe1f8dc4984115783042213871bc3d8d4a59ff8ab87a655978fc1e WatchSource:0}: Error finding container c3c44f4665fe1f8dc4984115783042213871bc3d8d4a59ff8ab87a655978fc1e: Status 404 returned error can't find the container with id c3c44f4665fe1f8dc4984115783042213871bc3d8d4a59ff8ab87a655978fc1e Oct 30 17:10:19.198844 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:19.198820 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd5f560f-ltrhc"] Oct 30 17:10:19.841509 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:19.841475 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd5f560f-ltrhc" event={"ID":"1e4272a1-630d-41c7-bd48-bc0c58bd54c4","Type":"ContainerStarted","Data":"c96540258c0cfd6aa00452fd8e8b2577dc5b7f5cccbe23146422a6272051a866"} Oct 30 17:10:19.841877 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:19.841528 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd5f560f-ltrhc" event={"ID":"1e4272a1-630d-41c7-bd48-bc0c58bd54c4","Type":"ContainerStarted","Data":"c3c44f4665fe1f8dc4984115783042213871bc3d8d4a59ff8ab87a655978fc1e"} Oct 30 17:10:19.891447 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:19.891333 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd5f560f-ltrhc" podStartSLOduration=1.8912931560000001 podStartE2EDuration="1.891293156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:10:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:10:19.887235479 +0000 UTC m=+1393.887990380" watchObservedRunningTime="2026-10-30 17:10:19.891293156 +0000 UTC m=+1393.892048058" Oct 30 17:10:20.147011 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:20.146932 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bfc83a8-a240-4f65-8034-6dc828a8c14e" path="/var/lib/kubelet/pods/2bfc83a8-a240-4f65-8034-6dc828a8c14e/volumes" Oct 30 17:10:20.356529 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:20.356485 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql"] Oct 30 17:10:20.361718 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:20.361694 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql" Oct 30 17:10:20.389013 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:20.388687 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql"] Oct 30 17:10:20.391824 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:20.391794 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-8681b5-openshift-default-dockercfg-pd5qg\"" Oct 30 17:10:20.437953 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:20.437858 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/827251e7-f88d-4ed1-81bb-ce4960304aed-istiod-ca-cert\") pod \"e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql\" (UID: \"827251e7-f88d-4ed1-81bb-ce4960304aed\") " pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql" Oct 30 17:10:20.438122 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:20.438021 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/827251e7-f88d-4ed1-81bb-ce4960304aed-istio-data\") pod \"e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql\" (UID: \"827251e7-f88d-4ed1-81bb-ce4960304aed\") " pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql" Oct 30 17:10:20.438247 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:20.438223 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/827251e7-f88d-4ed1-81bb-ce4960304aed-workload-socket\") pod \"e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql\" (UID: \"827251e7-f88d-4ed1-81bb-ce4960304aed\") " pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql" Oct 30 17:10:20.438356 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:20.438288 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24qrb\" (UniqueName: \"kubernetes.io/projected/827251e7-f88d-4ed1-81bb-ce4960304aed-kube-api-access-24qrb\") pod \"e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql\" (UID: \"827251e7-f88d-4ed1-81bb-ce4960304aed\") " pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql" Oct 30 17:10:20.438431 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:20.438365 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/827251e7-f88d-4ed1-81bb-ce4960304aed-workload-certs\") pod \"e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql\" (UID: \"827251e7-f88d-4ed1-81bb-ce4960304aed\") " pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql" Oct 30 17:10:20.438573 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:20.438552 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/827251e7-f88d-4ed1-81bb-ce4960304aed-credential-socket\") pod \"e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql\" (UID: \"827251e7-f88d-4ed1-81bb-ce4960304aed\") " pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql" Oct 30 17:10:20.438627 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:20.438595 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/827251e7-f88d-4ed1-81bb-ce4960304aed-istio-envoy\") pod \"e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql\" (UID: \"827251e7-f88d-4ed1-81bb-ce4960304aed\") " pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql" Oct 30 17:10:20.438705 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:20.438688 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/827251e7-f88d-4ed1-81bb-ce4960304aed-istio-podinfo\") pod \"e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql\" (UID: \"827251e7-f88d-4ed1-81bb-ce4960304aed\") " pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql" Oct 30 17:10:20.438774 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:20.438759 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/827251e7-f88d-4ed1-81bb-ce4960304aed-istio-token\") pod \"e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql\" (UID: \"827251e7-f88d-4ed1-81bb-ce4960304aed\") " pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql" Oct 30 17:10:20.539863 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:20.539831 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/827251e7-f88d-4ed1-81bb-ce4960304aed-istio-token\") pod \"e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql\" (UID: \"827251e7-f88d-4ed1-81bb-ce4960304aed\") " pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql" Oct 30 17:10:20.540053 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:20.539923 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/827251e7-f88d-4ed1-81bb-ce4960304aed-istiod-ca-cert\") pod \"e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql\" (UID: \"827251e7-f88d-4ed1-81bb-ce4960304aed\") " pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql" Oct 30 17:10:20.540053 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:20.539982 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/827251e7-f88d-4ed1-81bb-ce4960304aed-istio-data\") pod \"e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql\" (UID: \"827251e7-f88d-4ed1-81bb-ce4960304aed\") " pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql" Oct 30 17:10:20.540186 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:20.540091 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/827251e7-f88d-4ed1-81bb-ce4960304aed-workload-socket\") pod \"e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql\" (UID: \"827251e7-f88d-4ed1-81bb-ce4960304aed\") " pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql" Oct 30 17:10:20.540186 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:20.540126 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24qrb\" (UniqueName: \"kubernetes.io/projected/827251e7-f88d-4ed1-81bb-ce4960304aed-kube-api-access-24qrb\") pod \"e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql\" (UID: \"827251e7-f88d-4ed1-81bb-ce4960304aed\") " pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql" Oct 30 17:10:20.540186 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:20.540152 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/827251e7-f88d-4ed1-81bb-ce4960304aed-workload-certs\") pod \"e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql\" (UID: \"827251e7-f88d-4ed1-81bb-ce4960304aed\") " pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql" Oct 30 17:10:20.540436 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:20.540279 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/827251e7-f88d-4ed1-81bb-ce4960304aed-credential-socket\") pod \"e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql\" (UID: \"827251e7-f88d-4ed1-81bb-ce4960304aed\") " pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql" Oct 30 17:10:20.540436 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:20.540328 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/827251e7-f88d-4ed1-81bb-ce4960304aed-istio-envoy\") pod \"e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql\" (UID: \"827251e7-f88d-4ed1-81bb-ce4960304aed\") " pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql" Oct 30 17:10:20.540436 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:20.540373 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/827251e7-f88d-4ed1-81bb-ce4960304aed-istio-podinfo\") pod \"e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql\" (UID: \"827251e7-f88d-4ed1-81bb-ce4960304aed\") " pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql" Oct 30 17:10:20.540758 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:20.540727 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/827251e7-f88d-4ed1-81bb-ce4960304aed-istio-data\") pod \"e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql\" (UID: \"827251e7-f88d-4ed1-81bb-ce4960304aed\") " pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql" Oct 30 17:10:20.540882 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:20.540771 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/827251e7-f88d-4ed1-81bb-ce4960304aed-workload-socket\") pod \"e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql\" (UID: \"827251e7-f88d-4ed1-81bb-ce4960304aed\") " pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql" Oct 30 17:10:20.540882 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:20.540817 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/827251e7-f88d-4ed1-81bb-ce4960304aed-workload-certs\") pod \"e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql\" (UID: \"827251e7-f88d-4ed1-81bb-ce4960304aed\") " pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql" Oct 30 17:10:20.540882 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:20.540825 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/827251e7-f88d-4ed1-81bb-ce4960304aed-istiod-ca-cert\") pod \"e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql\" (UID: \"827251e7-f88d-4ed1-81bb-ce4960304aed\") " pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql" Oct 30 17:10:20.541075 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:20.541054 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/827251e7-f88d-4ed1-81bb-ce4960304aed-credential-socket\") pod \"e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql\" (UID: \"827251e7-f88d-4ed1-81bb-ce4960304aed\") " pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql" Oct 30 17:10:20.542747 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:20.542714 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/827251e7-f88d-4ed1-81bb-ce4960304aed-istio-envoy\") pod \"e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql\" (UID: \"827251e7-f88d-4ed1-81bb-ce4960304aed\") " pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql" Oct 30 17:10:20.543342 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:20.543297 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/827251e7-f88d-4ed1-81bb-ce4960304aed-istio-podinfo\") pod \"e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql\" (UID: \"827251e7-f88d-4ed1-81bb-ce4960304aed\") " pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql" Oct 30 17:10:20.567419 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:20.567385 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24qrb\" (UniqueName: \"kubernetes.io/projected/827251e7-f88d-4ed1-81bb-ce4960304aed-kube-api-access-24qrb\") pod \"e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql\" (UID: \"827251e7-f88d-4ed1-81bb-ce4960304aed\") " pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql" Oct 30 17:10:20.570286 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:20.570257 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/827251e7-f88d-4ed1-81bb-ce4960304aed-istio-token\") pod \"e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql\" (UID: \"827251e7-f88d-4ed1-81bb-ce4960304aed\") " pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql" Oct 30 17:10:20.685371 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:20.683982 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql" Oct 30 17:10:20.773453 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:20.773416 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c4c2e4-6f66b94d58-6xb5n"] Oct 30 17:10:20.816584 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:20.816555 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c4c2e4-6f66b94d58-6xb5n" Oct 30 17:10:20.863519 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:20.839280 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c4c2e4-6f66b94d58-6xb5n"] Oct 30 17:10:20.863519 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:20.840339 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-c4c2e4\"" Oct 30 17:10:20.863519 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:20.840502 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-c4c2e4-dockercfg-l7vzs\"" Oct 30 17:10:20.863519 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:20.859518 4522 generic.go:356] "Generic (PLEG): container finished" podID="1e4272a1-630d-41c7-bd48-bc0c58bd54c4" containerID="c96540258c0cfd6aa00452fd8e8b2577dc5b7f5cccbe23146422a6272051a866" exitCode=0 Oct 30 17:10:20.863519 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:20.859777 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd5f560f-ltrhc" event={"ID":"1e4272a1-630d-41c7-bd48-bc0c58bd54c4","Type":"ContainerDied","Data":"c96540258c0cfd6aa00452fd8e8b2577dc5b7f5cccbe23146422a6272051a866"} Oct 30 17:10:20.962556 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:20.962516 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44mkv\" (UniqueName: \"kubernetes.io/projected/bff503f8-b6d2-41d4-a8c4-883a3afa38a1-kube-api-access-44mkv\") pod \"payload-processing-e2e-api-b-c4c2e4-6f66b94d58-6xb5n\" (UID: \"bff503f8-b6d2-41d4-a8c4-883a3afa38a1\") " pod="openshift-ingress/payload-processing-e2e-api-b-c4c2e4-6f66b94d58-6xb5n" Oct 30 17:10:20.963505 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:20.963475 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bff503f8-b6d2-41d4-a8c4-883a3afa38a1-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c4c2e4-6f66b94d58-6xb5n\" (UID: \"bff503f8-b6d2-41d4-a8c4-883a3afa38a1\") " pod="openshift-ingress/payload-processing-e2e-api-b-c4c2e4-6f66b94d58-6xb5n" Oct 30 17:10:21.072103 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:21.072002 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44mkv\" (UniqueName: \"kubernetes.io/projected/bff503f8-b6d2-41d4-a8c4-883a3afa38a1-kube-api-access-44mkv\") pod \"payload-processing-e2e-api-b-c4c2e4-6f66b94d58-6xb5n\" (UID: \"bff503f8-b6d2-41d4-a8c4-883a3afa38a1\") " pod="openshift-ingress/payload-processing-e2e-api-b-c4c2e4-6f66b94d58-6xb5n" Oct 30 17:10:21.072475 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:21.072452 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bff503f8-b6d2-41d4-a8c4-883a3afa38a1-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c4c2e4-6f66b94d58-6xb5n\" (UID: \"bff503f8-b6d2-41d4-a8c4-883a3afa38a1\") " pod="openshift-ingress/payload-processing-e2e-api-b-c4c2e4-6f66b94d58-6xb5n" Oct 30 17:10:21.073632 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:21.073601 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bff503f8-b6d2-41d4-a8c4-883a3afa38a1-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c4c2e4-6f66b94d58-6xb5n\" (UID: \"bff503f8-b6d2-41d4-a8c4-883a3afa38a1\") " pod="openshift-ingress/payload-processing-e2e-api-b-c4c2e4-6f66b94d58-6xb5n" Oct 30 17:10:21.100557 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:21.100529 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44mkv\" (UniqueName: \"kubernetes.io/projected/bff503f8-b6d2-41d4-a8c4-883a3afa38a1-kube-api-access-44mkv\") pod \"payload-processing-e2e-api-b-c4c2e4-6f66b94d58-6xb5n\" (UID: \"bff503f8-b6d2-41d4-a8c4-883a3afa38a1\") " pod="openshift-ingress/payload-processing-e2e-api-b-c4c2e4-6f66b94d58-6xb5n" Oct 30 17:10:21.182767 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:10:21.182692 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod827251e7_f88d_4ed1_81bb_ce4960304aed.slice/crio-7f0d7cc1a6cb2fafa04bfa2a265e64d43512492f4db50f38f5a461fb1a50c740 WatchSource:0}: Error finding container 7f0d7cc1a6cb2fafa04bfa2a265e64d43512492f4db50f38f5a461fb1a50c740: Status 404 returned error can't find the container with id 7f0d7cc1a6cb2fafa04bfa2a265e64d43512492f4db50f38f5a461fb1a50c740 Oct 30 17:10:21.185208 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:21.185165 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql"] Oct 30 17:10:21.186054 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:21.186015 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:10:21.186168 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:21.186087 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:10:21.186168 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:21.186123 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:10:21.202287 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:21.202258 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c4c2e4-6f66b94d58-6xb5n" Oct 30 17:10:21.548414 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:21.548370 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c4c2e4-6f66b94d58-6xb5n"] Oct 30 17:10:21.548929 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:10:21.548904 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbff503f8_b6d2_41d4_a8c4_883a3afa38a1.slice/crio-f5d6054415017f2bf4129df4479ed32e5ec57d974d67a467d73f9b2cf3b2e296 WatchSource:0}: Error finding container f5d6054415017f2bf4129df4479ed32e5ec57d974d67a467d73f9b2cf3b2e296: Status 404 returned error can't find the container with id f5d6054415017f2bf4129df4479ed32e5ec57d974d67a467d73f9b2cf3b2e296 Oct 30 17:10:21.875040 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:21.874999 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c4c2e4-6f66b94d58-6xb5n" event={"ID":"bff503f8-b6d2-41d4-a8c4-883a3afa38a1","Type":"ContainerStarted","Data":"b54bde885f10a4861270a34e8860cf90aa8ec9105bb946239bf9a86a13ffaad3"} Oct 30 17:10:21.875545 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:21.875069 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-c4c2e4-6f66b94d58-6xb5n" Oct 30 17:10:21.875545 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:21.875093 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c4c2e4-6f66b94d58-6xb5n" event={"ID":"bff503f8-b6d2-41d4-a8c4-883a3afa38a1","Type":"ContainerStarted","Data":"f5d6054415017f2bf4129df4479ed32e5ec57d974d67a467d73f9b2cf3b2e296"} Oct 30 17:10:21.878807 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:21.878777 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql" event={"ID":"827251e7-f88d-4ed1-81bb-ce4960304aed","Type":"ContainerStarted","Data":"b686e611a6d354837289bab219e846e56d5bfc25c70979e7d7e2df9e7faf90a1"} Oct 30 17:10:21.878996 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:21.878976 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql" event={"ID":"827251e7-f88d-4ed1-81bb-ce4960304aed","Type":"ContainerStarted","Data":"7f0d7cc1a6cb2fafa04bfa2a265e64d43512492f4db50f38f5a461fb1a50c740"} Oct 30 17:10:21.928334 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:21.927936 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-c4c2e4-6f66b94d58-6xb5n" podStartSLOduration=1.927916561 podStartE2EDuration="1.927916561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:10:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:10:21.926706888 +0000 UTC m=+1395.927461807" watchObservedRunningTime="2026-10-30 17:10:21.927916561 +0000 UTC m=+1395.928671462" Oct 30 17:10:22.013388 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:22.010749 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql" podStartSLOduration=2.010710497 podStartE2EDuration="2.010710497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:10:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:10:22.007582396 +0000 UTC m=+1396.008337323" watchObservedRunningTime="2026-10-30 17:10:22.010710497 +0000 UTC m=+1396.011465400" Oct 30 17:10:22.178126 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:22.178096 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd5f560f-ltrhc" Oct 30 17:10:22.255764 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:22.255730 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e4272a1-630d-41c7-bd48-bc0c58bd54c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e4272a1-630d-41c7-bd48-bc0c58bd54c4-maas-api-openshift-ca\") pod \"1e4272a1-630d-41c7-bd48-bc0c58bd54c4\" (UID: \"1e4272a1-630d-41c7-bd48-bc0c58bd54c4\") " Oct 30 17:10:22.255955 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:22.255812 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e4272a1-630d-41c7-bd48-bc0c58bd54c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e4272a1-630d-41c7-bd48-bc0c58bd54c4-maas-api-xks-ca\") pod \"1e4272a1-630d-41c7-bd48-bc0c58bd54c4\" (UID: \"1e4272a1-630d-41c7-bd48-bc0c58bd54c4\") " Oct 30 17:10:22.256100 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:22.256070 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e4272a1-630d-41c7-bd48-bc0c58bd54c4-kube-api-access-fwbx4\" (UniqueName: \"kubernetes.io/projected/1e4272a1-630d-41c7-bd48-bc0c58bd54c4-kube-api-access-fwbx4\") pod \"1e4272a1-630d-41c7-bd48-bc0c58bd54c4\" (UID: \"1e4272a1-630d-41c7-bd48-bc0c58bd54c4\") " Oct 30 17:10:22.256265 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:22.256235 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e4272a1-630d-41c7-bd48-bc0c58bd54c4-maas-api-openshift-ca" pod "1e4272a1-630d-41c7-bd48-bc0c58bd54c4" (UID: "1e4272a1-630d-41c7-bd48-bc0c58bd54c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:10:22.256975 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:22.256951 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e4272a1-630d-41c7-bd48-bc0c58bd54c4-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:10:22.258410 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:22.258381 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e4272a1-630d-41c7-bd48-bc0c58bd54c4-kube-api-access-fwbx4" pod "1e4272a1-630d-41c7-bd48-bc0c58bd54c4" (UID: "1e4272a1-630d-41c7-bd48-bc0c58bd54c4"). InnerVolumeSpecName "kube-api-access-fwbx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:10:22.258849 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:22.258815 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e4272a1-630d-41c7-bd48-bc0c58bd54c4-maas-api-xks-ca" pod "1e4272a1-630d-41c7-bd48-bc0c58bd54c4" (UID: "1e4272a1-630d-41c7-bd48-bc0c58bd54c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:10:22.358692 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:22.358594 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e4272a1-630d-41c7-bd48-bc0c58bd54c4-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:10:22.358692 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:22.358636 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwbx4\" (UniqueName: \"kubernetes.io/projected/1e4272a1-630d-41c7-bd48-bc0c58bd54c4-kube-api-access-fwbx4\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:10:22.685831 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:22.685720 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql" Oct 30 17:10:22.696214 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:22.696177 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql" Oct 30 17:10:22.891251 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:22.891223 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd5f560f-ltrhc" Oct 30 17:10:22.891768 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:22.891270 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd5f560f-ltrhc" event={"ID":"1e4272a1-630d-41c7-bd48-bc0c58bd54c4","Type":"ContainerDied","Data":"c3c44f4665fe1f8dc4984115783042213871bc3d8d4a59ff8ab87a655978fc1e"} Oct 30 17:10:22.891768 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:22.891371 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3c44f4665fe1f8dc4984115783042213871bc3d8d4a59ff8ab87a655978fc1e" Oct 30 17:10:22.891913 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:22.891840 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql" Oct 30 17:10:22.892832 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:22.892812 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql" Oct 30 17:10:26.528758 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:26.528713 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8681b5-d7469f799-jsz5b"] Oct 30 17:10:26.529549 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:26.529535 4522 cpu_manager.go:414] "containerMap: removing container" podUID="1e4272a1-630d-41c7-bd48-bc0c58bd54c4" containerName="revoke-keys" Oct 30 17:10:26.529599 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:26.529551 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e4272a1-630d-41c7-bd48-bc0c58bd54c4" containerName="revoke-keys" Oct 30 17:10:26.529705 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:26.529696 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e4272a1-630d-41c7-bd48-bc0c58bd54c4" containerName="revoke-keys" Oct 30 17:10:26.533577 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:26.533559 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8681b5-d7469f799-jsz5b" Oct 30 17:10:26.555179 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:26.555147 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8681b5-d7469f799-jsz5b"] Oct 30 17:10:26.567332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:26.562358 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-8681b5\"" Oct 30 17:10:26.567332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:26.562500 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-8681b5-dockercfg-cttbq\"" Oct 30 17:10:26.649329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:26.648869 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-8681b5-7d9676c6d6-khtxf"] Oct 30 17:10:26.666329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:26.653536 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bptb4\" (UniqueName: \"kubernetes.io/projected/af8374c3-9f76-4183-9eee-d781479514a0-kube-api-access-bptb4\") pod \"payload-pre-processing-e2e-shared-a-w3-8681b5-d7469f799-jsz5b\" (UID: \"af8374c3-9f76-4183-9eee-d781479514a0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8681b5-d7469f799-jsz5b" Oct 30 17:10:26.666329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:26.653784 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af8374c3-9f76-4183-9eee-d781479514a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-8681b5-d7469f799-jsz5b\" (UID: \"af8374c3-9f76-4183-9eee-d781479514a0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8681b5-d7469f799-jsz5b" Oct 30 17:10:26.679629 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:26.675978 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8681b5-7d9676c6d6-khtxf" Oct 30 17:10:26.700752 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:26.700714 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-8681b5-7d9676c6d6-khtxf"] Oct 30 17:10:26.758734 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:26.758686 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bptb4\" (UniqueName: \"kubernetes.io/projected/af8374c3-9f76-4183-9eee-d781479514a0-kube-api-access-bptb4\") pod \"payload-pre-processing-e2e-shared-a-w3-8681b5-d7469f799-jsz5b\" (UID: \"af8374c3-9f76-4183-9eee-d781479514a0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8681b5-d7469f799-jsz5b" Oct 30 17:10:26.759713 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:26.759661 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af8374c3-9f76-4183-9eee-d781479514a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-8681b5-d7469f799-jsz5b\" (UID: \"af8374c3-9f76-4183-9eee-d781479514a0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8681b5-d7469f799-jsz5b" Oct 30 17:10:26.770116 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:26.760884 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af8374c3-9f76-4183-9eee-d781479514a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-8681b5-d7469f799-jsz5b\" (UID: \"af8374c3-9f76-4183-9eee-d781479514a0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8681b5-d7469f799-jsz5b" Oct 30 17:10:26.816471 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:26.816290 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bptb4\" (UniqueName: \"kubernetes.io/projected/af8374c3-9f76-4183-9eee-d781479514a0-kube-api-access-bptb4\") pod \"payload-pre-processing-e2e-shared-a-w3-8681b5-d7469f799-jsz5b\" (UID: \"af8374c3-9f76-4183-9eee-d781479514a0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8681b5-d7469f799-jsz5b" Oct 30 17:10:26.850270 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:26.850235 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8681b5-d7469f799-jsz5b" Oct 30 17:10:26.869386 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:26.869320 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac7e71fc-8c27-4f6f-b9f2-bf5fbd115267-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-8681b5-7d9676c6d6-khtxf\" (UID: \"ac7e71fc-8c27-4f6f-b9f2-bf5fbd115267\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8681b5-7d9676c6d6-khtxf" Oct 30 17:10:26.869386 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:26.869384 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptvk6\" (UniqueName: \"kubernetes.io/projected/ac7e71fc-8c27-4f6f-b9f2-bf5fbd115267-kube-api-access-ptvk6\") pod \"payload-processing-e2e-shared-a-w3-8681b5-7d9676c6d6-khtxf\" (UID: \"ac7e71fc-8c27-4f6f-b9f2-bf5fbd115267\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8681b5-7d9676c6d6-khtxf" Oct 30 17:10:26.980998 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:26.971911 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac7e71fc-8c27-4f6f-b9f2-bf5fbd115267-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-8681b5-7d9676c6d6-khtxf\" (UID: \"ac7e71fc-8c27-4f6f-b9f2-bf5fbd115267\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8681b5-7d9676c6d6-khtxf" Oct 30 17:10:26.980998 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:26.972010 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptvk6\" (UniqueName: \"kubernetes.io/projected/ac7e71fc-8c27-4f6f-b9f2-bf5fbd115267-kube-api-access-ptvk6\") pod \"payload-processing-e2e-shared-a-w3-8681b5-7d9676c6d6-khtxf\" (UID: \"ac7e71fc-8c27-4f6f-b9f2-bf5fbd115267\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8681b5-7d9676c6d6-khtxf" Oct 30 17:10:26.980998 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:26.973887 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac7e71fc-8c27-4f6f-b9f2-bf5fbd115267-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-8681b5-7d9676c6d6-khtxf\" (UID: \"ac7e71fc-8c27-4f6f-b9f2-bf5fbd115267\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8681b5-7d9676c6d6-khtxf" Oct 30 17:10:27.012492 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:27.009936 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptvk6\" (UniqueName: \"kubernetes.io/projected/ac7e71fc-8c27-4f6f-b9f2-bf5fbd115267-kube-api-access-ptvk6\") pod \"payload-processing-e2e-shared-a-w3-8681b5-7d9676c6d6-khtxf\" (UID: \"ac7e71fc-8c27-4f6f-b9f2-bf5fbd115267\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8681b5-7d9676c6d6-khtxf" Oct 30 17:10:27.185349 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:10:27.185319 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf8374c3_9f76_4183_9eee_d781479514a0.slice/crio-82f488457990c0fd284f63f9b8ca792b4d3cb8a7165dca2d13f8c95831a3fa1f WatchSource:0}: Error finding container 82f488457990c0fd284f63f9b8ca792b4d3cb8a7165dca2d13f8c95831a3fa1f: Status 404 returned error can't find the container with id 82f488457990c0fd284f63f9b8ca792b4d3cb8a7165dca2d13f8c95831a3fa1f Oct 30 17:10:27.189906 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:27.189878 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8681b5-d7469f799-jsz5b"] Oct 30 17:10:27.303470 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:27.303441 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8681b5-7d9676c6d6-khtxf" Oct 30 17:10:27.613011 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:10:27.612975 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac7e71fc_8c27_4f6f_b9f2_bf5fbd115267.slice/crio-d743e7dc83295ff7e4e83da602efb2550e241a2750a35fb5a4dbbd78e8985892 WatchSource:0}: Error finding container d743e7dc83295ff7e4e83da602efb2550e241a2750a35fb5a4dbbd78e8985892: Status 404 returned error can't find the container with id d743e7dc83295ff7e4e83da602efb2550e241a2750a35fb5a4dbbd78e8985892 Oct 30 17:10:27.615970 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:27.615941 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-8681b5-7d9676c6d6-khtxf"] Oct 30 17:10:27.965256 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:27.965223 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8681b5-7d9676c6d6-khtxf" event={"ID":"ac7e71fc-8c27-4f6f-b9f2-bf5fbd115267","Type":"ContainerStarted","Data":"a81c11b95b85566b3fa66e25ef6d1cbd67a8cdce505f8cd739ecaebd95d43108"} Oct 30 17:10:27.965488 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:27.965276 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8681b5-7d9676c6d6-khtxf" event={"ID":"ac7e71fc-8c27-4f6f-b9f2-bf5fbd115267","Type":"ContainerStarted","Data":"d743e7dc83295ff7e4e83da602efb2550e241a2750a35fb5a4dbbd78e8985892"} Oct 30 17:10:27.965488 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:27.965342 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8681b5-7d9676c6d6-khtxf" Oct 30 17:10:27.967341 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:27.967292 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8681b5-d7469f799-jsz5b" event={"ID":"af8374c3-9f76-4183-9eee-d781479514a0","Type":"ContainerStarted","Data":"cf41024ad80e3b478243d0c04b796986fe8976a6568f6c7ea4f1b1df9eb68ecd"} Oct 30 17:10:27.967461 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:27.967352 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8681b5-d7469f799-jsz5b" event={"ID":"af8374c3-9f76-4183-9eee-d781479514a0","Type":"ContainerStarted","Data":"82f488457990c0fd284f63f9b8ca792b4d3cb8a7165dca2d13f8c95831a3fa1f"} Oct 30 17:10:27.967461 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:27.967381 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8681b5-d7469f799-jsz5b" Oct 30 17:10:28.012293 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:28.012239 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8681b5-7d9676c6d6-khtxf" podStartSLOduration=2.012226964 podStartE2EDuration="2.012226964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:10:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:10:28.01077018 +0000 UTC m=+1402.011525080" watchObservedRunningTime="2026-10-30 17:10:28.012226964 +0000 UTC m=+1402.012981864" Oct 30 17:10:28.061371 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:28.061299 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8681b5-d7469f799-jsz5b" podStartSLOduration=2.061272109 podStartE2EDuration="2.061272109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:10:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:10:28.05671998 +0000 UTC m=+1402.057474882" watchObservedRunningTime="2026-10-30 17:10:28.061272109 +0000 UTC m=+1402.062027010" Oct 30 17:10:31.039096 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:31.039060 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc707697-hbvdm"] Oct 30 17:10:31.051763 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:31.051734 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc707697-hbvdm"] Oct 30 17:10:32.148046 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:32.148012 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="677fd30b-2b0d-42ff-8351-8d6e801adf5f" path="/var/lib/kubelet/pods/677fd30b-2b0d-42ff-8351-8d6e801adf5f/volumes" Oct 30 17:10:32.894763 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:32.894731 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-c4c2e4-6f66b94d58-6xb5n" Oct 30 17:10:38.981000 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:38.980969 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8681b5-7d9676c6d6-khtxf" Oct 30 17:10:38.981522 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:38.981046 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8681b5-d7469f799-jsz5b" Oct 30 17:10:39.875275 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:39.875244 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c425a3e3-8jg79"] Oct 30 17:10:39.889650 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:39.889622 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c425a3e3-8jg79"] Oct 30 17:10:40.148894 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:40.148812 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8b3d01a-b9b2-4f94-aa60-e24c45aa4a14" path="/var/lib/kubelet/pods/e8b3d01a-b9b2-4f94-aa60-e24c45aa4a14/volumes" Oct 30 17:10:42.950360 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:10:42.950323 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:10:51.291034 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:51.290972 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06a99010-dwbd7"] Oct 30 17:10:51.302674 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:51.302634 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a99010-dwbd7" Oct 30 17:10:51.307097 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:51.307069 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06a99010-dwbd7"] Oct 30 17:10:51.318558 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:51.318526 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:10:51.318751 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:51.318588 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:10:51.389851 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:51.389811 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59fadb0c-7100-4e9d-91a5-959c6e747228-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06a99010-dwbd7\" (UID: \"59fadb0c-7100-4e9d-91a5-959c6e747228\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a99010-dwbd7" Oct 30 17:10:51.390218 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:51.390190 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qxsk\" (UniqueName: \"kubernetes.io/projected/59fadb0c-7100-4e9d-91a5-959c6e747228-kube-api-access-8qxsk\") pod \"maas-api-revoke-sub-06a99010-dwbd7\" (UID: \"59fadb0c-7100-4e9d-91a5-959c6e747228\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a99010-dwbd7" Oct 30 17:10:51.390754 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:51.390728 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59fadb0c-7100-4e9d-91a5-959c6e747228-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06a99010-dwbd7\" (UID: \"59fadb0c-7100-4e9d-91a5-959c6e747228\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a99010-dwbd7" Oct 30 17:10:51.492550 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:51.492499 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59fadb0c-7100-4e9d-91a5-959c6e747228-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06a99010-dwbd7\" (UID: \"59fadb0c-7100-4e9d-91a5-959c6e747228\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a99010-dwbd7" Oct 30 17:10:51.492744 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:51.492638 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59fadb0c-7100-4e9d-91a5-959c6e747228-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06a99010-dwbd7\" (UID: \"59fadb0c-7100-4e9d-91a5-959c6e747228\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a99010-dwbd7" Oct 30 17:10:51.492788 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:51.492745 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qxsk\" (UniqueName: \"kubernetes.io/projected/59fadb0c-7100-4e9d-91a5-959c6e747228-kube-api-access-8qxsk\") pod \"maas-api-revoke-sub-06a99010-dwbd7\" (UID: \"59fadb0c-7100-4e9d-91a5-959c6e747228\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a99010-dwbd7" Oct 30 17:10:51.493515 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:51.493483 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59fadb0c-7100-4e9d-91a5-959c6e747228-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06a99010-dwbd7\" (UID: \"59fadb0c-7100-4e9d-91a5-959c6e747228\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a99010-dwbd7" Oct 30 17:10:51.495192 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:51.495170 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59fadb0c-7100-4e9d-91a5-959c6e747228-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06a99010-dwbd7\" (UID: \"59fadb0c-7100-4e9d-91a5-959c6e747228\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a99010-dwbd7" Oct 30 17:10:51.517555 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:51.517525 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qxsk\" (UniqueName: \"kubernetes.io/projected/59fadb0c-7100-4e9d-91a5-959c6e747228-kube-api-access-8qxsk\") pod \"maas-api-revoke-sub-06a99010-dwbd7\" (UID: \"59fadb0c-7100-4e9d-91a5-959c6e747228\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a99010-dwbd7" Oct 30 17:10:51.630159 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:51.630072 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a99010-dwbd7" Oct 30 17:10:51.940454 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:10:51.940425 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59fadb0c_7100_4e9d_91a5_959c6e747228.slice/crio-16c02c840f762f7f2b1278dab2fc011f85b433a1f6c6ebc2848a142d83f7c348 WatchSource:0}: Error finding container 16c02c840f762f7f2b1278dab2fc011f85b433a1f6c6ebc2848a142d83f7c348: Status 404 returned error can't find the container with id 16c02c840f762f7f2b1278dab2fc011f85b433a1f6c6ebc2848a142d83f7c348 Oct 30 17:10:51.943395 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:51.943373 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06a99010-dwbd7"] Oct 30 17:10:52.321994 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:52.321951 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a99010-dwbd7" event={"ID":"59fadb0c-7100-4e9d-91a5-959c6e747228","Type":"ContainerStarted","Data":"41456b0436842af2976abe5a579dbb7f43c0d600fe5649cc9e0aa17ab6a710f0"} Oct 30 17:10:52.322469 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:52.322011 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a99010-dwbd7" event={"ID":"59fadb0c-7100-4e9d-91a5-959c6e747228","Type":"ContainerStarted","Data":"16c02c840f762f7f2b1278dab2fc011f85b433a1f6c6ebc2848a142d83f7c348"} Oct 30 17:10:52.368074 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:52.368022 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a99010-dwbd7" podStartSLOduration=1.368006828 podStartE2EDuration="1.368006828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:10:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:10:52.363456985 +0000 UTC m=+1426.364211906" watchObservedRunningTime="2026-10-30 17:10:52.368006828 +0000 UTC m=+1426.368761730" Oct 30 17:10:53.335967 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:53.335932 4522 generic.go:356] "Generic (PLEG): container finished" podID="59fadb0c-7100-4e9d-91a5-959c6e747228" containerID="41456b0436842af2976abe5a579dbb7f43c0d600fe5649cc9e0aa17ab6a710f0" exitCode=0 Oct 30 17:10:53.336377 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:53.336019 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a99010-dwbd7" event={"ID":"59fadb0c-7100-4e9d-91a5-959c6e747228","Type":"ContainerDied","Data":"41456b0436842af2976abe5a579dbb7f43c0d600fe5649cc9e0aa17ab6a710f0"} Oct 30 17:10:54.637685 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:54.637617 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a99010-dwbd7" Oct 30 17:10:54.770249 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:54.770218 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59fadb0c-7100-4e9d-91a5-959c6e747228-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59fadb0c-7100-4e9d-91a5-959c6e747228-maas-api-openshift-ca\") pod \"59fadb0c-7100-4e9d-91a5-959c6e747228\" (UID: \"59fadb0c-7100-4e9d-91a5-959c6e747228\") " Oct 30 17:10:54.770461 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:54.770408 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59fadb0c-7100-4e9d-91a5-959c6e747228-kube-api-access-8qxsk\" (UniqueName: \"kubernetes.io/projected/59fadb0c-7100-4e9d-91a5-959c6e747228-kube-api-access-8qxsk\") pod \"59fadb0c-7100-4e9d-91a5-959c6e747228\" (UID: \"59fadb0c-7100-4e9d-91a5-959c6e747228\") " Oct 30 17:10:54.770461 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:54.770452 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59fadb0c-7100-4e9d-91a5-959c6e747228-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59fadb0c-7100-4e9d-91a5-959c6e747228-maas-api-xks-ca\") pod \"59fadb0c-7100-4e9d-91a5-959c6e747228\" (UID: \"59fadb0c-7100-4e9d-91a5-959c6e747228\") " Oct 30 17:10:54.770654 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:54.770630 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59fadb0c-7100-4e9d-91a5-959c6e747228-maas-api-openshift-ca" pod "59fadb0c-7100-4e9d-91a5-959c6e747228" (UID: "59fadb0c-7100-4e9d-91a5-959c6e747228"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:10:54.771073 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:54.771051 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59fadb0c-7100-4e9d-91a5-959c6e747228-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:10:54.772614 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:54.772590 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59fadb0c-7100-4e9d-91a5-959c6e747228-maas-api-xks-ca" pod "59fadb0c-7100-4e9d-91a5-959c6e747228" (UID: "59fadb0c-7100-4e9d-91a5-959c6e747228"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:10:54.772698 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:54.772645 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59fadb0c-7100-4e9d-91a5-959c6e747228-kube-api-access-8qxsk" pod "59fadb0c-7100-4e9d-91a5-959c6e747228" (UID: "59fadb0c-7100-4e9d-91a5-959c6e747228"). InnerVolumeSpecName "kube-api-access-8qxsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:10:54.873058 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:54.873022 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qxsk\" (UniqueName: \"kubernetes.io/projected/59fadb0c-7100-4e9d-91a5-959c6e747228-kube-api-access-8qxsk\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:10:54.873058 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:54.873054 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59fadb0c-7100-4e9d-91a5-959c6e747228-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:10:55.375680 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:55.375655 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a99010-dwbd7" Oct 30 17:10:55.375883 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:55.375650 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a99010-dwbd7" event={"ID":"59fadb0c-7100-4e9d-91a5-959c6e747228","Type":"ContainerDied","Data":"16c02c840f762f7f2b1278dab2fc011f85b433a1f6c6ebc2848a142d83f7c348"} Oct 30 17:10:55.375883 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:10:55.375778 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16c02c840f762f7f2b1278dab2fc011f85b433a1f6c6ebc2848a142d83f7c348" Oct 30 17:11:02.785023 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:02.784908 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ebeb6e1-bptwm"] Oct 30 17:11:02.797981 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:02.797942 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ebeb6e1-bptwm"] Oct 30 17:11:04.147494 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:04.147462 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c48e028-1439-430b-ae8e-ec12b9dce540" path="/var/lib/kubelet/pods/7c48e028-1439-430b-ae8e-ec12b9dce540/volumes" Oct 30 17:11:07.082955 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:07.082922 4522 scope.go:122] "RemoveContainer" containerID="b955334b78e41bc6f8553b9870e1f0c61ce2840485b3c8742d3dbf766e1c6dd0" Oct 30 17:11:07.096830 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:07.096805 4522 scope.go:122] "RemoveContainer" containerID="a8c6583a4fe72c4667be38f28da2f2e9e374792ab0882f60db71285d78931f14" Oct 30 17:11:07.119656 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:07.119507 4522 scope.go:122] "RemoveContainer" containerID="acc2bbce2d157acaca9c817854f37aa397f0cc943d172ae5dc677fdc848f2566" Oct 30 17:11:07.191923 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:07.191878 4522 scope.go:122] "RemoveContainer" containerID="51eea853c3ed5b0566a65665a435f912cb68ccde063ae2fd3d6b30a19e2243b5" Oct 30 17:11:07.215578 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:07.211774 4522 scope.go:122] "RemoveContainer" containerID="6ba783db16a28d987c813cc7365c685286eea5908c8e352ed861e0effbe646be" Oct 30 17:11:07.235873 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:07.235718 4522 scope.go:122] "RemoveContainer" containerID="40f15f9eaabc54cdc360c3d14031fc4d99ead13d7e0d2967aa89f45e2202f19b" Oct 30 17:11:07.274273 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:07.273862 4522 scope.go:122] "RemoveContainer" containerID="0a944c591ac537cd870e1944f6a7114fc3b367115234c514807ba57c93f77a7a" Oct 30 17:11:07.321259 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:07.320692 4522 scope.go:122] "RemoveContainer" containerID="4a51773c7fa60abee216fe1723f125470f4c5fd3c2e84ccabf9870a3067daf0c" Oct 30 17:11:07.353675 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:07.353568 4522 scope.go:122] "RemoveContainer" containerID="2923fd24d9c676cbe5acec5373d13b1b088485143583cd36ec3d2b85de8aeb70" Oct 30 17:11:07.369782 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:07.369736 4522 scope.go:122] "RemoveContainer" containerID="d68ddbd3391a428c3d9ff3ad458209be35ee5a438b325b63c88b8430c5d6c808" Oct 30 17:11:09.000523 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:09.000489 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz"] Oct 30 17:11:09.001061 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:09.000725 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz" podUID="b2010977-7978-4ee0-9b32-3449329c0b4d" containerName="istio-proxy" containerID="cri-o://46a704c49374a5a30045b654592d5236cd5dfd6d0886689f0268884bd1e84237" gracePeriod=30 Oct 30 17:11:09.005944 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:09.005915 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp"] Oct 30 17:11:09.006153 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:09.006125 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp" podUID="9eb030f0-e0a2-446e-b35f-4c497d9a3aa2" containerName="istio-proxy" containerID="cri-o://a87317b6fc68451491a9fbed0fc18be64d8efac699dc6db5620732df065e2be6" gracePeriod=30 Oct 30 17:11:09.011553 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:09.011530 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf"] Oct 30 17:11:09.011720 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:09.011699 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf" podUID="b590c8f6-9d91-49bb-8e0f-5e65eb24e631" containerName="istio-proxy" containerID="cri-o://7eeccabc24f491c40eb12f9abf02055c5ffc5daf41c72cf1ae7e75df2f088204" gracePeriod=30 Oct 30 17:11:09.015935 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:09.015911 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn"] Oct 30 17:11:09.016104 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:09.016085 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn" podUID="015c4531-2766-4522-acf0-0427f1ba0a3d" containerName="istio-proxy" containerID="cri-o://ec16c40a6b42782747d2a94269022619f5fb1a13bafcac50046ba80bf2766aa9" gracePeriod=30 Oct 30 17:11:12.951036 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:11:12.951001 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:11:14.368280 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.368249 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz" Oct 30 17:11:14.373011 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.372983 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf" Oct 30 17:11:14.463845 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.463738 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp" Oct 30 17:11:14.468671 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.468596 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn" Oct 30 17:11:14.487535 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.487506 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/015c4531-2766-4522-acf0-0427f1ba0a3d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/015c4531-2766-4522-acf0-0427f1ba0a3d-istio-podinfo\") pod \"015c4531-2766-4522-acf0-0427f1ba0a3d\" (UID: \"015c4531-2766-4522-acf0-0427f1ba0a3d\") " Oct 30 17:11:14.487721 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.487541 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/015c4531-2766-4522-acf0-0427f1ba0a3d-kube-api-access-zs4wp\" (UniqueName: \"kubernetes.io/projected/015c4531-2766-4522-acf0-0427f1ba0a3d-kube-api-access-zs4wp\") pod \"015c4531-2766-4522-acf0-0427f1ba0a3d\" (UID: \"015c4531-2766-4522-acf0-0427f1ba0a3d\") " Oct 30 17:11:14.487721 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.487586 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2-istio-podinfo\") pod \"9eb030f0-e0a2-446e-b35f-4c497d9a3aa2\" (UID: \"9eb030f0-e0a2-446e-b35f-4c497d9a3aa2\") " Oct 30 17:11:14.487839 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.487703 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/015c4531-2766-4522-acf0-0427f1ba0a3d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/015c4531-2766-4522-acf0-0427f1ba0a3d-istio-envoy\") pod \"015c4531-2766-4522-acf0-0427f1ba0a3d\" (UID: \"015c4531-2766-4522-acf0-0427f1ba0a3d\") " Oct 30 17:11:14.487839 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.487764 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2010977-7978-4ee0-9b32-3449329c0b4d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2010977-7978-4ee0-9b32-3449329c0b4d-credential-socket\") pod \"b2010977-7978-4ee0-9b32-3449329c0b4d\" (UID: \"b2010977-7978-4ee0-9b32-3449329c0b4d\") " Oct 30 17:11:14.487839 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.487800 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b590c8f6-9d91-49bb-8e0f-5e65eb24e631-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b590c8f6-9d91-49bb-8e0f-5e65eb24e631-istiod-ca-cert\") pod \"b590c8f6-9d91-49bb-8e0f-5e65eb24e631\" (UID: \"b590c8f6-9d91-49bb-8e0f-5e65eb24e631\") " Oct 30 17:11:14.487991 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.487836 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2010977-7978-4ee0-9b32-3449329c0b4d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2010977-7978-4ee0-9b32-3449329c0b4d-istio-envoy\") pod \"b2010977-7978-4ee0-9b32-3449329c0b4d\" (UID: \"b2010977-7978-4ee0-9b32-3449329c0b4d\") " Oct 30 17:11:14.487991 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.487923 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2-workload-certs\") pod \"9eb030f0-e0a2-446e-b35f-4c497d9a3aa2\" (UID: \"9eb030f0-e0a2-446e-b35f-4c497d9a3aa2\") " Oct 30 17:11:14.487991 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.487953 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2010977-7978-4ee0-9b32-3449329c0b4d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2010977-7978-4ee0-9b32-3449329c0b4d-istio-data\") pod \"b2010977-7978-4ee0-9b32-3449329c0b4d\" (UID: \"b2010977-7978-4ee0-9b32-3449329c0b4d\") " Oct 30 17:11:14.487991 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.487983 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b590c8f6-9d91-49bb-8e0f-5e65eb24e631-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b590c8f6-9d91-49bb-8e0f-5e65eb24e631-workload-socket\") pod \"b590c8f6-9d91-49bb-8e0f-5e65eb24e631\" (UID: \"b590c8f6-9d91-49bb-8e0f-5e65eb24e631\") " Oct 30 17:11:14.488187 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.488031 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b590c8f6-9d91-49bb-8e0f-5e65eb24e631-kube-api-access-6nzsc\" (UniqueName: \"kubernetes.io/projected/b590c8f6-9d91-49bb-8e0f-5e65eb24e631-kube-api-access-6nzsc\") pod \"b590c8f6-9d91-49bb-8e0f-5e65eb24e631\" (UID: \"b590c8f6-9d91-49bb-8e0f-5e65eb24e631\") " Oct 30 17:11:14.488187 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.488058 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2010977-7978-4ee0-9b32-3449329c0b4d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2010977-7978-4ee0-9b32-3449329c0b4d-workload-certs\") pod \"b2010977-7978-4ee0-9b32-3449329c0b4d\" (UID: \"b2010977-7978-4ee0-9b32-3449329c0b4d\") " Oct 30 17:11:14.488187 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.488108 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2-credential-socket\") pod \"9eb030f0-e0a2-446e-b35f-4c497d9a3aa2\" (UID: \"9eb030f0-e0a2-446e-b35f-4c497d9a3aa2\") " Oct 30 17:11:14.488187 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.488136 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/015c4531-2766-4522-acf0-0427f1ba0a3d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/015c4531-2766-4522-acf0-0427f1ba0a3d-workload-certs\") pod \"015c4531-2766-4522-acf0-0427f1ba0a3d\" (UID: \"015c4531-2766-4522-acf0-0427f1ba0a3d\") " Oct 30 17:11:14.488187 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.488170 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/015c4531-2766-4522-acf0-0427f1ba0a3d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/015c4531-2766-4522-acf0-0427f1ba0a3d-istiod-ca-cert\") pod \"015c4531-2766-4522-acf0-0427f1ba0a3d\" (UID: \"015c4531-2766-4522-acf0-0427f1ba0a3d\") " Oct 30 17:11:14.488450 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.488220 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b590c8f6-9d91-49bb-8e0f-5e65eb24e631-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b590c8f6-9d91-49bb-8e0f-5e65eb24e631-istio-envoy\") pod \"b590c8f6-9d91-49bb-8e0f-5e65eb24e631\" (UID: \"b590c8f6-9d91-49bb-8e0f-5e65eb24e631\") " Oct 30 17:11:14.488450 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.488258 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b590c8f6-9d91-49bb-8e0f-5e65eb24e631-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b590c8f6-9d91-49bb-8e0f-5e65eb24e631-istio-data\") pod \"b590c8f6-9d91-49bb-8e0f-5e65eb24e631\" (UID: \"b590c8f6-9d91-49bb-8e0f-5e65eb24e631\") " Oct 30 17:11:14.488450 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.488360 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/015c4531-2766-4522-acf0-0427f1ba0a3d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/015c4531-2766-4522-acf0-0427f1ba0a3d-credential-socket\") pod \"015c4531-2766-4522-acf0-0427f1ba0a3d\" (UID: \"015c4531-2766-4522-acf0-0427f1ba0a3d\") " Oct 30 17:11:14.488450 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.488414 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2-workload-certs" pod "9eb030f0-e0a2-446e-b35f-4c497d9a3aa2" (UID: "9eb030f0-e0a2-446e-b35f-4c497d9a3aa2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:14.488450 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.488431 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b590c8f6-9d91-49bb-8e0f-5e65eb24e631-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b590c8f6-9d91-49bb-8e0f-5e65eb24e631-istio-podinfo\") pod \"b590c8f6-9d91-49bb-8e0f-5e65eb24e631\" (UID: \"b590c8f6-9d91-49bb-8e0f-5e65eb24e631\") " Oct 30 17:11:14.488703 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.488479 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b590c8f6-9d91-49bb-8e0f-5e65eb24e631-istio-token\" (UniqueName: \"kubernetes.io/projected/b590c8f6-9d91-49bb-8e0f-5e65eb24e631-istio-token\") pod \"b590c8f6-9d91-49bb-8e0f-5e65eb24e631\" (UID: \"b590c8f6-9d91-49bb-8e0f-5e65eb24e631\") " Oct 30 17:11:14.488703 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.488525 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/015c4531-2766-4522-acf0-0427f1ba0a3d-istio-token\" (UniqueName: \"kubernetes.io/projected/015c4531-2766-4522-acf0-0427f1ba0a3d-istio-token\") pod \"015c4531-2766-4522-acf0-0427f1ba0a3d\" (UID: \"015c4531-2766-4522-acf0-0427f1ba0a3d\") " Oct 30 17:11:14.488703 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.488563 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/015c4531-2766-4522-acf0-0427f1ba0a3d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/015c4531-2766-4522-acf0-0427f1ba0a3d-workload-socket\") pod \"015c4531-2766-4522-acf0-0427f1ba0a3d\" (UID: \"015c4531-2766-4522-acf0-0427f1ba0a3d\") " Oct 30 17:11:14.488703 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.488598 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2-istiod-ca-cert\") pod \"9eb030f0-e0a2-446e-b35f-4c497d9a3aa2\" (UID: \"9eb030f0-e0a2-446e-b35f-4c497d9a3aa2\") " Oct 30 17:11:14.488703 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.488617 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b590c8f6-9d91-49bb-8e0f-5e65eb24e631-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b590c8f6-9d91-49bb-8e0f-5e65eb24e631-credential-socket\") pod \"b590c8f6-9d91-49bb-8e0f-5e65eb24e631\" (UID: \"b590c8f6-9d91-49bb-8e0f-5e65eb24e631\") " Oct 30 17:11:14.488703 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.488637 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2-workload-socket\") pod \"9eb030f0-e0a2-446e-b35f-4c497d9a3aa2\" (UID: \"9eb030f0-e0a2-446e-b35f-4c497d9a3aa2\") " Oct 30 17:11:14.488703 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.488658 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2-istio-token\" (UniqueName: \"kubernetes.io/projected/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2-istio-token\") pod \"9eb030f0-e0a2-446e-b35f-4c497d9a3aa2\" (UID: \"9eb030f0-e0a2-446e-b35f-4c497d9a3aa2\") " Oct 30 17:11:14.488703 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.488682 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/015c4531-2766-4522-acf0-0427f1ba0a3d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/015c4531-2766-4522-acf0-0427f1ba0a3d-istio-data\") pod \"015c4531-2766-4522-acf0-0427f1ba0a3d\" (UID: \"015c4531-2766-4522-acf0-0427f1ba0a3d\") " Oct 30 17:11:14.488703 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.488701 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2-istio-envoy\") pod \"9eb030f0-e0a2-446e-b35f-4c497d9a3aa2\" (UID: \"9eb030f0-e0a2-446e-b35f-4c497d9a3aa2\") " Oct 30 17:11:14.489190 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.488795 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2010977-7978-4ee0-9b32-3449329c0b4d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2010977-7978-4ee0-9b32-3449329c0b4d-workload-socket\") pod \"b2010977-7978-4ee0-9b32-3449329c0b4d\" (UID: \"b2010977-7978-4ee0-9b32-3449329c0b4d\") " Oct 30 17:11:14.489190 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.488829 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2-istio-data\") pod \"9eb030f0-e0a2-446e-b35f-4c497d9a3aa2\" (UID: \"9eb030f0-e0a2-446e-b35f-4c497d9a3aa2\") " Oct 30 17:11:14.489190 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.488867 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2010977-7978-4ee0-9b32-3449329c0b4d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2010977-7978-4ee0-9b32-3449329c0b4d-istiod-ca-cert\") pod \"b2010977-7978-4ee0-9b32-3449329c0b4d\" (UID: \"b2010977-7978-4ee0-9b32-3449329c0b4d\") " Oct 30 17:11:14.489190 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.488934 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2-kube-api-access-4fdz6\" (UniqueName: \"kubernetes.io/projected/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2-kube-api-access-4fdz6\") pod \"9eb030f0-e0a2-446e-b35f-4c497d9a3aa2\" (UID: \"9eb030f0-e0a2-446e-b35f-4c497d9a3aa2\") " Oct 30 17:11:14.489190 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.488970 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2010977-7978-4ee0-9b32-3449329c0b4d-kube-api-access-vx5x2\" (UniqueName: \"kubernetes.io/projected/b2010977-7978-4ee0-9b32-3449329c0b4d-kube-api-access-vx5x2\") pod \"b2010977-7978-4ee0-9b32-3449329c0b4d\" (UID: \"b2010977-7978-4ee0-9b32-3449329c0b4d\") " Oct 30 17:11:14.489190 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.489085 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b590c8f6-9d91-49bb-8e0f-5e65eb24e631-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b590c8f6-9d91-49bb-8e0f-5e65eb24e631-workload-certs\") pod \"b590c8f6-9d91-49bb-8e0f-5e65eb24e631\" (UID: \"b590c8f6-9d91-49bb-8e0f-5e65eb24e631\") " Oct 30 17:11:14.489190 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.489123 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b2010977-7978-4ee0-9b32-3449329c0b4d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2010977-7978-4ee0-9b32-3449329c0b4d-istio-podinfo\") pod \"b2010977-7978-4ee0-9b32-3449329c0b4d\" (UID: \"b2010977-7978-4ee0-9b32-3449329c0b4d\") " Oct 30 17:11:14.489542 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.489208 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2010977-7978-4ee0-9b32-3449329c0b4d-istio-token\" (UniqueName: \"kubernetes.io/projected/b2010977-7978-4ee0-9b32-3449329c0b4d-istio-token\") pod \"b2010977-7978-4ee0-9b32-3449329c0b4d\" (UID: \"b2010977-7978-4ee0-9b32-3449329c0b4d\") " Oct 30 17:11:14.489904 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.489816 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/015c4531-2766-4522-acf0-0427f1ba0a3d-workload-socket" pod "015c4531-2766-4522-acf0-0427f1ba0a3d" (UID: "015c4531-2766-4522-acf0-0427f1ba0a3d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:14.490004 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.489956 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2010977-7978-4ee0-9b32-3449329c0b4d-credential-socket" pod "b2010977-7978-4ee0-9b32-3449329c0b4d" (UID: "b2010977-7978-4ee0-9b32-3449329c0b4d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:14.490004 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.489981 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b590c8f6-9d91-49bb-8e0f-5e65eb24e631-istiod-ca-cert" pod "b590c8f6-9d91-49bb-8e0f-5e65eb24e631" (UID: "b590c8f6-9d91-49bb-8e0f-5e65eb24e631"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:11:14.490262 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.490233 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2010977-7978-4ee0-9b32-3449329c0b4d-istio-data" pod "b2010977-7978-4ee0-9b32-3449329c0b4d" (UID: "b2010977-7978-4ee0-9b32-3449329c0b4d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:14.490439 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.490329 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2-istiod-ca-cert" pod "9eb030f0-e0a2-446e-b35f-4c497d9a3aa2" (UID: "9eb030f0-e0a2-446e-b35f-4c497d9a3aa2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:11:14.490502 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.490456 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b590c8f6-9d91-49bb-8e0f-5e65eb24e631-workload-socket" pod "b590c8f6-9d91-49bb-8e0f-5e65eb24e631" (UID: "b590c8f6-9d91-49bb-8e0f-5e65eb24e631"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:14.490636 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.490620 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b590c8f6-9d91-49bb-8e0f-5e65eb24e631-istio-data" pod "b590c8f6-9d91-49bb-8e0f-5e65eb24e631" (UID: "b590c8f6-9d91-49bb-8e0f-5e65eb24e631"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:14.490783 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.490757 4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/015c4531-2766-4522-acf0-0427f1ba0a3d-workload-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:14.490783 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.490777 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/015c4531-2766-4522-acf0-0427f1ba0a3d-credential-socket" pod "015c4531-2766-4522-acf0-0427f1ba0a3d" (UID: "015c4531-2766-4522-acf0-0427f1ba0a3d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:14.490948 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.490792 4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2-istiod-ca-cert\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:14.490948 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.490808 4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2010977-7978-4ee0-9b32-3449329c0b4d-credential-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:14.490948 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.490822 4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b590c8f6-9d91-49bb-8e0f-5e65eb24e631-istiod-ca-cert\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:14.490948 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.490836 4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2-workload-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:14.490948 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.490848 4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2010977-7978-4ee0-9b32-3449329c0b4d-istio-data\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:14.490948 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.490861 4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b590c8f6-9d91-49bb-8e0f-5e65eb24e631-workload-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:14.491261 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.491060 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b590c8f6-9d91-49bb-8e0f-5e65eb24e631-credential-socket" pod "b590c8f6-9d91-49bb-8e0f-5e65eb24e631" (UID: "b590c8f6-9d91-49bb-8e0f-5e65eb24e631"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:14.491261 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.491242 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2-workload-socket" pod "9eb030f0-e0a2-446e-b35f-4c497d9a3aa2" (UID: "9eb030f0-e0a2-446e-b35f-4c497d9a3aa2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:14.492743 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.492712 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2010977-7978-4ee0-9b32-3449329c0b4d-istio-envoy" pod "b2010977-7978-4ee0-9b32-3449329c0b4d" (UID: "b2010977-7978-4ee0-9b32-3449329c0b4d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:14.493129 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.493108 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2010977-7978-4ee0-9b32-3449329c0b4d-istiod-ca-cert" pod "b2010977-7978-4ee0-9b32-3449329c0b4d" (UID: "b2010977-7978-4ee0-9b32-3449329c0b4d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:11:14.493192 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.493150 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b590c8f6-9d91-49bb-8e0f-5e65eb24e631-istio-podinfo" pod "b590c8f6-9d91-49bb-8e0f-5e65eb24e631" (UID: "b590c8f6-9d91-49bb-8e0f-5e65eb24e631"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:11:14.493369 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.493298 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b590c8f6-9d91-49bb-8e0f-5e65eb24e631-istio-envoy" pod "b590c8f6-9d91-49bb-8e0f-5e65eb24e631" (UID: "b590c8f6-9d91-49bb-8e0f-5e65eb24e631"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:14.493456 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.493395 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/015c4531-2766-4522-acf0-0427f1ba0a3d-istio-data" pod "015c4531-2766-4522-acf0-0427f1ba0a3d" (UID: "015c4531-2766-4522-acf0-0427f1ba0a3d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:14.493928 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.493906 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2010977-7978-4ee0-9b32-3449329c0b4d-workload-socket" pod "b2010977-7978-4ee0-9b32-3449329c0b4d" (UID: "b2010977-7978-4ee0-9b32-3449329c0b4d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:14.494125 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.494102 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2-credential-socket" pod "9eb030f0-e0a2-446e-b35f-4c497d9a3aa2" (UID: "9eb030f0-e0a2-446e-b35f-4c497d9a3aa2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:14.494395 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.494374 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2010977-7978-4ee0-9b32-3449329c0b4d-workload-certs" pod "b2010977-7978-4ee0-9b32-3449329c0b4d" (UID: "b2010977-7978-4ee0-9b32-3449329c0b4d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:14.494627 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.494609 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/015c4531-2766-4522-acf0-0427f1ba0a3d-workload-certs" pod "015c4531-2766-4522-acf0-0427f1ba0a3d" (UID: "015c4531-2766-4522-acf0-0427f1ba0a3d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:14.494817 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.494799 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2-istio-data" pod "9eb030f0-e0a2-446e-b35f-4c497d9a3aa2" (UID: "9eb030f0-e0a2-446e-b35f-4c497d9a3aa2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:14.495039 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.495022 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b590c8f6-9d91-49bb-8e0f-5e65eb24e631-workload-certs" pod "b590c8f6-9d91-49bb-8e0f-5e65eb24e631" (UID: "b590c8f6-9d91-49bb-8e0f-5e65eb24e631"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:14.495354 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.495329 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2-istio-token" pod "9eb030f0-e0a2-446e-b35f-4c497d9a3aa2" (UID: "9eb030f0-e0a2-446e-b35f-4c497d9a3aa2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:11:14.496014 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.495709 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2-istio-envoy" pod "9eb030f0-e0a2-446e-b35f-4c497d9a3aa2" (UID: "9eb030f0-e0a2-446e-b35f-4c497d9a3aa2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:14.496014 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.495866 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b590c8f6-9d91-49bb-8e0f-5e65eb24e631-istio-token" pod "b590c8f6-9d91-49bb-8e0f-5e65eb24e631" (UID: "b590c8f6-9d91-49bb-8e0f-5e65eb24e631"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:11:14.496291 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.496260 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b590c8f6-9d91-49bb-8e0f-5e65eb24e631-kube-api-access-6nzsc" pod "b590c8f6-9d91-49bb-8e0f-5e65eb24e631" (UID: "b590c8f6-9d91-49bb-8e0f-5e65eb24e631"). InnerVolumeSpecName "kube-api-access-6nzsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:11:14.498281 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.498249 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2010977-7978-4ee0-9b32-3449329c0b4d-istio-podinfo" pod "b2010977-7978-4ee0-9b32-3449329c0b4d" (UID: "b2010977-7978-4ee0-9b32-3449329c0b4d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:11:14.502066 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.502000 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2010977-7978-4ee0-9b32-3449329c0b4d-istio-token" pod "b2010977-7978-4ee0-9b32-3449329c0b4d" (UID: "b2010977-7978-4ee0-9b32-3449329c0b4d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:11:14.502163 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.502108 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/015c4531-2766-4522-acf0-0427f1ba0a3d-kube-api-access-zs4wp" pod "015c4531-2766-4522-acf0-0427f1ba0a3d" (UID: "015c4531-2766-4522-acf0-0427f1ba0a3d"). InnerVolumeSpecName "kube-api-access-zs4wp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:11:14.504081 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.502823 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/015c4531-2766-4522-acf0-0427f1ba0a3d-istiod-ca-cert" pod "015c4531-2766-4522-acf0-0427f1ba0a3d" (UID: "015c4531-2766-4522-acf0-0427f1ba0a3d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:11:14.504081 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.503766 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/015c4531-2766-4522-acf0-0427f1ba0a3d-istio-token" pod "015c4531-2766-4522-acf0-0427f1ba0a3d" (UID: "015c4531-2766-4522-acf0-0427f1ba0a3d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:11:14.505792 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.505769 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2010977-7978-4ee0-9b32-3449329c0b4d-kube-api-access-vx5x2" pod "b2010977-7978-4ee0-9b32-3449329c0b4d" (UID: "b2010977-7978-4ee0-9b32-3449329c0b4d"). InnerVolumeSpecName "kube-api-access-vx5x2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:11:14.508813 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.508779 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/015c4531-2766-4522-acf0-0427f1ba0a3d-istio-podinfo" pod "015c4531-2766-4522-acf0-0427f1ba0a3d" (UID: "015c4531-2766-4522-acf0-0427f1ba0a3d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:11:14.508916 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.508818 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2-istio-podinfo" pod "9eb030f0-e0a2-446e-b35f-4c497d9a3aa2" (UID: "9eb030f0-e0a2-446e-b35f-4c497d9a3aa2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:11:14.509080 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.509050 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2-kube-api-access-4fdz6" pod "9eb030f0-e0a2-446e-b35f-4c497d9a3aa2" (UID: "9eb030f0-e0a2-446e-b35f-4c497d9a3aa2"). InnerVolumeSpecName "kube-api-access-4fdz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:11:14.510844 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.510820 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/015c4531-2766-4522-acf0-0427f1ba0a3d-istio-envoy" pod "015c4531-2766-4522-acf0-0427f1ba0a3d" (UID: "015c4531-2766-4522-acf0-0427f1ba0a3d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:14.593718 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.593629 4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2-istio-data\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:14.593718 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.593668 4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2010977-7978-4ee0-9b32-3449329c0b4d-istiod-ca-cert\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:14.593718 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.593684 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fdz6\" (UniqueName: \"kubernetes.io/projected/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2-kube-api-access-4fdz6\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:14.593718 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.593709 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vx5x2\" (UniqueName: \"kubernetes.io/projected/b2010977-7978-4ee0-9b32-3449329c0b4d-kube-api-access-vx5x2\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:14.594036 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.593725 4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b590c8f6-9d91-49bb-8e0f-5e65eb24e631-workload-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:14.594036 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.593738 4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2010977-7978-4ee0-9b32-3449329c0b4d-istio-podinfo\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:14.594036 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.593751 4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2010977-7978-4ee0-9b32-3449329c0b4d-istio-token\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:14.594036 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.593765 4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/015c4531-2766-4522-acf0-0427f1ba0a3d-istio-podinfo\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:14.594036 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.593778 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zs4wp\" (UniqueName: \"kubernetes.io/projected/015c4531-2766-4522-acf0-0427f1ba0a3d-kube-api-access-zs4wp\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:14.594036 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.593791 4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2-istio-podinfo\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:14.594036 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.593804 4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/015c4531-2766-4522-acf0-0427f1ba0a3d-istio-envoy\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:14.594036 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.593818 4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2010977-7978-4ee0-9b32-3449329c0b4d-istio-envoy\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:14.594036 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.593831 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nzsc\" (UniqueName: \"kubernetes.io/projected/b590c8f6-9d91-49bb-8e0f-5e65eb24e631-kube-api-access-6nzsc\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:14.594036 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.593843 4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2010977-7978-4ee0-9b32-3449329c0b4d-workload-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:14.594036 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.593856 4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2-credential-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:14.594036 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.593868 4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/015c4531-2766-4522-acf0-0427f1ba0a3d-workload-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:14.594036 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.593881 4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/015c4531-2766-4522-acf0-0427f1ba0a3d-istiod-ca-cert\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:14.594036 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.593894 4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b590c8f6-9d91-49bb-8e0f-5e65eb24e631-istio-envoy\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:14.594036 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.593905 4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b590c8f6-9d91-49bb-8e0f-5e65eb24e631-istio-data\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:14.594036 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.593917 4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/015c4531-2766-4522-acf0-0427f1ba0a3d-credential-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:14.594036 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.593930 4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b590c8f6-9d91-49bb-8e0f-5e65eb24e631-istio-podinfo\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:14.594036 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.593943 4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b590c8f6-9d91-49bb-8e0f-5e65eb24e631-istio-token\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:14.594036 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.593957 4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/015c4531-2766-4522-acf0-0427f1ba0a3d-istio-token\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:14.594036 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.593971 4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b590c8f6-9d91-49bb-8e0f-5e65eb24e631-credential-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:14.594036 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.593984 4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2-workload-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:14.594036 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.593997 4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2-istio-token\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:14.594036 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.594010 4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/015c4531-2766-4522-acf0-0427f1ba0a3d-istio-data\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:14.594036 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.594021 4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2-istio-envoy\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:14.594036 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.594034 4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2010977-7978-4ee0-9b32-3449329c0b4d-workload-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:14.636655 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.636620 4522 generic.go:356] "Generic (PLEG): container finished" podID="015c4531-2766-4522-acf0-0427f1ba0a3d" containerID="ec16c40a6b42782747d2a94269022619f5fb1a13bafcac50046ba80bf2766aa9" exitCode=0 Oct 30 17:11:14.636858 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.636688 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn" Oct 30 17:11:14.636858 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.636719 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn" event={"ID":"015c4531-2766-4522-acf0-0427f1ba0a3d","Type":"ContainerDied","Data":"ec16c40a6b42782747d2a94269022619f5fb1a13bafcac50046ba80bf2766aa9"} Oct 30 17:11:14.637370 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.637351 4522 scope.go:122] "RemoveContainer" containerID="ec16c40a6b42782747d2a94269022619f5fb1a13bafcac50046ba80bf2766aa9" Oct 30 17:11:14.638413 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.638382 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn" event={"ID":"015c4531-2766-4522-acf0-0427f1ba0a3d","Type":"ContainerDied","Data":"860191f8246c8d4324f1b4adcbb2ca96b58444dd8335b21940ba49e89fa09f9b"} Oct 30 17:11:14.640737 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.640715 4522 generic.go:356] "Generic (PLEG): container finished" podID="9eb030f0-e0a2-446e-b35f-4c497d9a3aa2" containerID="a87317b6fc68451491a9fbed0fc18be64d8efac699dc6db5620732df065e2be6" exitCode=0 Oct 30 17:11:14.640843 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.640769 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp" Oct 30 17:11:14.640921 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.640861 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp" event={"ID":"9eb030f0-e0a2-446e-b35f-4c497d9a3aa2","Type":"ContainerDied","Data":"a87317b6fc68451491a9fbed0fc18be64d8efac699dc6db5620732df065e2be6"} Oct 30 17:11:14.640921 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.640904 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp" event={"ID":"9eb030f0-e0a2-446e-b35f-4c497d9a3aa2","Type":"ContainerDied","Data":"c50002850adb34f86d769e15c0dd24a169908ad128be77f2b2ad7fa6708cbc62"} Oct 30 17:11:14.643339 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.643298 4522 generic.go:356] "Generic (PLEG): container finished" podID="b590c8f6-9d91-49bb-8e0f-5e65eb24e631" containerID="7eeccabc24f491c40eb12f9abf02055c5ffc5daf41c72cf1ae7e75df2f088204" exitCode=0 Oct 30 17:11:14.643471 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.643452 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf" event={"ID":"b590c8f6-9d91-49bb-8e0f-5e65eb24e631","Type":"ContainerDied","Data":"7eeccabc24f491c40eb12f9abf02055c5ffc5daf41c72cf1ae7e75df2f088204"} Oct 30 17:11:14.643544 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.643498 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf" event={"ID":"b590c8f6-9d91-49bb-8e0f-5e65eb24e631","Type":"ContainerDied","Data":"0419c901e7364525b1adff66cb8e77a30b904632c755cebe76966148b7cf5224"} Oct 30 17:11:14.643600 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.643577 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf" Oct 30 17:11:14.668680 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.666587 4522 scope.go:122] "RemoveContainer" containerID="ec16c40a6b42782747d2a94269022619f5fb1a13bafcac50046ba80bf2766aa9" Oct 30 17:11:14.668680 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.666772 4522 generic.go:356] "Generic (PLEG): container finished" podID="b2010977-7978-4ee0-9b32-3449329c0b4d" containerID="46a704c49374a5a30045b654592d5236cd5dfd6d0886689f0268884bd1e84237" exitCode=0 Oct 30 17:11:14.668680 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.666902 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz" event={"ID":"b2010977-7978-4ee0-9b32-3449329c0b4d","Type":"ContainerDied","Data":"46a704c49374a5a30045b654592d5236cd5dfd6d0886689f0268884bd1e84237"} Oct 30 17:11:14.668680 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.666975 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz" event={"ID":"b2010977-7978-4ee0-9b32-3449329c0b4d","Type":"ContainerDied","Data":"990cda7f69c389d59b9174db2972f6f792f7f6a9bbf9bb65ed19c6b46f74a8c4"} Oct 30 17:11:14.668680 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.667076 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz" Oct 30 17:11:14.669472 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:11:14.669439 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec16c40a6b42782747d2a94269022619f5fb1a13bafcac50046ba80bf2766aa9\": container with ID starting with ec16c40a6b42782747d2a94269022619f5fb1a13bafcac50046ba80bf2766aa9 not found: ID does not exist" containerID="ec16c40a6b42782747d2a94269022619f5fb1a13bafcac50046ba80bf2766aa9" Oct 30 17:11:14.669603 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.669581 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec16c40a6b42782747d2a94269022619f5fb1a13bafcac50046ba80bf2766aa9"} err="failed to get container status \"ec16c40a6b42782747d2a94269022619f5fb1a13bafcac50046ba80bf2766aa9\": rpc error: code = NotFound desc = could not find container \"ec16c40a6b42782747d2a94269022619f5fb1a13bafcac50046ba80bf2766aa9\": container with ID starting with ec16c40a6b42782747d2a94269022619f5fb1a13bafcac50046ba80bf2766aa9 not found: ID does not exist" Oct 30 17:11:14.669692 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.669681 4522 scope.go:122] "RemoveContainer" containerID="a87317b6fc68451491a9fbed0fc18be64d8efac699dc6db5620732df065e2be6" Oct 30 17:11:14.685980 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.685960 4522 scope.go:122] "RemoveContainer" containerID="a87317b6fc68451491a9fbed0fc18be64d8efac699dc6db5620732df065e2be6" Oct 30 17:11:14.686273 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:11:14.686254 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a87317b6fc68451491a9fbed0fc18be64d8efac699dc6db5620732df065e2be6\": container with ID starting with a87317b6fc68451491a9fbed0fc18be64d8efac699dc6db5620732df065e2be6 not found: ID does not exist" containerID="a87317b6fc68451491a9fbed0fc18be64d8efac699dc6db5620732df065e2be6" Oct 30 17:11:14.687263 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.686284 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a87317b6fc68451491a9fbed0fc18be64d8efac699dc6db5620732df065e2be6"} err="failed to get container status \"a87317b6fc68451491a9fbed0fc18be64d8efac699dc6db5620732df065e2be6\": rpc error: code = NotFound desc = could not find container \"a87317b6fc68451491a9fbed0fc18be64d8efac699dc6db5620732df065e2be6\": container with ID starting with a87317b6fc68451491a9fbed0fc18be64d8efac699dc6db5620732df065e2be6 not found: ID does not exist" Oct 30 17:11:14.687263 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.686477 4522 scope.go:122] "RemoveContainer" containerID="7eeccabc24f491c40eb12f9abf02055c5ffc5daf41c72cf1ae7e75df2f088204" Oct 30 17:11:14.699490 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.699472 4522 scope.go:122] "RemoveContainer" containerID="29549c3eb9f4e617a2608e54db8fa9e328f8a3843ab700753cb96f333c593ef8" Oct 30 17:11:14.709677 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.709656 4522 scope.go:122] "RemoveContainer" containerID="7eeccabc24f491c40eb12f9abf02055c5ffc5daf41c72cf1ae7e75df2f088204" Oct 30 17:11:14.710005 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:11:14.709975 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7eeccabc24f491c40eb12f9abf02055c5ffc5daf41c72cf1ae7e75df2f088204\": container with ID starting with 7eeccabc24f491c40eb12f9abf02055c5ffc5daf41c72cf1ae7e75df2f088204 not found: ID does not exist" containerID="7eeccabc24f491c40eb12f9abf02055c5ffc5daf41c72cf1ae7e75df2f088204" Oct 30 17:11:14.710106 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.710015 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7eeccabc24f491c40eb12f9abf02055c5ffc5daf41c72cf1ae7e75df2f088204"} err="failed to get container status \"7eeccabc24f491c40eb12f9abf02055c5ffc5daf41c72cf1ae7e75df2f088204\": rpc error: code = NotFound desc = could not find container \"7eeccabc24f491c40eb12f9abf02055c5ffc5daf41c72cf1ae7e75df2f088204\": container with ID starting with 7eeccabc24f491c40eb12f9abf02055c5ffc5daf41c72cf1ae7e75df2f088204 not found: ID does not exist" Oct 30 17:11:14.710106 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.710040 4522 scope.go:122] "RemoveContainer" containerID="29549c3eb9f4e617a2608e54db8fa9e328f8a3843ab700753cb96f333c593ef8" Oct 30 17:11:14.710420 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:11:14.710399 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29549c3eb9f4e617a2608e54db8fa9e328f8a3843ab700753cb96f333c593ef8\": container with ID starting with 29549c3eb9f4e617a2608e54db8fa9e328f8a3843ab700753cb96f333c593ef8 not found: ID does not exist" containerID="29549c3eb9f4e617a2608e54db8fa9e328f8a3843ab700753cb96f333c593ef8" Oct 30 17:11:14.710532 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.710424 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29549c3eb9f4e617a2608e54db8fa9e328f8a3843ab700753cb96f333c593ef8"} err="failed to get container status \"29549c3eb9f4e617a2608e54db8fa9e328f8a3843ab700753cb96f333c593ef8\": rpc error: code = NotFound desc = could not find container \"29549c3eb9f4e617a2608e54db8fa9e328f8a3843ab700753cb96f333c593ef8\": container with ID starting with 29549c3eb9f4e617a2608e54db8fa9e328f8a3843ab700753cb96f333c593ef8 not found: ID does not exist" Oct 30 17:11:14.710532 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.710441 4522 scope.go:122] "RemoveContainer" containerID="46a704c49374a5a30045b654592d5236cd5dfd6d0886689f0268884bd1e84237" Oct 30 17:11:14.726837 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.726807 4522 scope.go:122] "RemoveContainer" containerID="46a704c49374a5a30045b654592d5236cd5dfd6d0886689f0268884bd1e84237" Oct 30 17:11:14.727454 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:11:14.727400 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46a704c49374a5a30045b654592d5236cd5dfd6d0886689f0268884bd1e84237\": container with ID starting with 46a704c49374a5a30045b654592d5236cd5dfd6d0886689f0268884bd1e84237 not found: ID does not exist" containerID="46a704c49374a5a30045b654592d5236cd5dfd6d0886689f0268884bd1e84237" Oct 30 17:11:14.727692 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.727464 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46a704c49374a5a30045b654592d5236cd5dfd6d0886689f0268884bd1e84237"} err="failed to get container status \"46a704c49374a5a30045b654592d5236cd5dfd6d0886689f0268884bd1e84237\": rpc error: code = NotFound desc = could not find container \"46a704c49374a5a30045b654592d5236cd5dfd6d0886689f0268884bd1e84237\": container with ID starting with 46a704c49374a5a30045b654592d5236cd5dfd6d0886689f0268884bd1e84237 not found: ID does not exist" Oct 30 17:11:14.756213 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.756177 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp"] Oct 30 17:11:14.759042 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.759015 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-dxcsp"] Oct 30 17:11:14.784405 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.784371 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf"] Oct 30 17:11:14.791047 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.791013 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-7npzf"] Oct 30 17:11:14.831529 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.831500 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz"] Oct 30 17:11:14.836820 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.836791 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-sxtpz"] Oct 30 17:11:14.877483 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.877404 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn"] Oct 30 17:11:14.882374 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:14.882344 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-v5zzn"] Oct 30 17:11:16.148800 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:16.148769 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="015c4531-2766-4522-acf0-0427f1ba0a3d" path="/var/lib/kubelet/pods/015c4531-2766-4522-acf0-0427f1ba0a3d/volumes" Oct 30 17:11:16.149275 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:16.149261 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9eb030f0-e0a2-446e-b35f-4c497d9a3aa2" path="/var/lib/kubelet/pods/9eb030f0-e0a2-446e-b35f-4c497d9a3aa2/volumes" Oct 30 17:11:16.149736 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:16.149722 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2010977-7978-4ee0-9b32-3449329c0b4d" path="/var/lib/kubelet/pods/b2010977-7978-4ee0-9b32-3449329c0b4d/volumes" Oct 30 17:11:16.150654 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:16.150633 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b590c8f6-9d91-49bb-8e0f-5e65eb24e631" path="/var/lib/kubelet/pods/b590c8f6-9d91-49bb-8e0f-5e65eb24e631/volumes" Oct 30 17:11:20.478333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:20.476816 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7d1e5c-sl2nh"] Oct 30 17:11:20.478333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:20.478067 4522 cpu_manager.go:414] "containerMap: removing container" podUID="b590c8f6-9d91-49bb-8e0f-5e65eb24e631" containerName="istio-proxy" Oct 30 17:11:20.478333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:20.478087 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b590c8f6-9d91-49bb-8e0f-5e65eb24e631" containerName="istio-proxy" Oct 30 17:11:20.478333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:20.478114 4522 cpu_manager.go:414] "containerMap: removing container" podUID="b2010977-7978-4ee0-9b32-3449329c0b4d" containerName="istio-proxy" Oct 30 17:11:20.478333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:20.478123 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2010977-7978-4ee0-9b32-3449329c0b4d" containerName="istio-proxy" Oct 30 17:11:20.478333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:20.478140 4522 cpu_manager.go:414] "containerMap: removing container" podUID="59fadb0c-7100-4e9d-91a5-959c6e747228" containerName="revoke-keys" Oct 30 17:11:20.478333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:20.478151 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59fadb0c-7100-4e9d-91a5-959c6e747228" containerName="revoke-keys" Oct 30 17:11:20.478333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:20.478172 4522 cpu_manager.go:414] "containerMap: removing container" podUID="b590c8f6-9d91-49bb-8e0f-5e65eb24e631" containerName="istio-proxy" Oct 30 17:11:20.478333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:20.478180 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b590c8f6-9d91-49bb-8e0f-5e65eb24e631" containerName="istio-proxy" Oct 30 17:11:20.478333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:20.478210 4522 cpu_manager.go:414] "containerMap: removing container" podUID="9eb030f0-e0a2-446e-b35f-4c497d9a3aa2" containerName="istio-proxy" Oct 30 17:11:20.478333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:20.478219 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9eb030f0-e0a2-446e-b35f-4c497d9a3aa2" containerName="istio-proxy" Oct 30 17:11:20.478333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:20.478234 4522 cpu_manager.go:414] "containerMap: removing container" podUID="015c4531-2766-4522-acf0-0427f1ba0a3d" containerName="istio-proxy" Oct 30 17:11:20.478333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:20.478241 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="015c4531-2766-4522-acf0-0427f1ba0a3d" containerName="istio-proxy" Oct 30 17:11:20.510336 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:20.509127 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2010977-7978-4ee0-9b32-3449329c0b4d" containerName="istio-proxy" Oct 30 17:11:20.510336 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:20.509162 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="59fadb0c-7100-4e9d-91a5-959c6e747228" containerName="revoke-keys" Oct 30 17:11:20.510336 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:20.509181 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="b590c8f6-9d91-49bb-8e0f-5e65eb24e631" containerName="istio-proxy" Oct 30 17:11:20.510336 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:20.509199 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="9eb030f0-e0a2-446e-b35f-4c497d9a3aa2" containerName="istio-proxy" Oct 30 17:11:20.510336 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:20.509208 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="015c4531-2766-4522-acf0-0427f1ba0a3d" containerName="istio-proxy" Oct 30 17:11:20.577656 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:20.547858 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7d1e5c-sl2nh" Oct 30 17:11:20.620340 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:20.594923 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4089af4a-de4d-453b-8e73-3a447e197309-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7d1e5c-sl2nh\" (UID: \"4089af4a-de4d-453b-8e73-3a447e197309\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7d1e5c-sl2nh" Oct 30 17:11:20.872401 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:20.803461 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4089af4a-de4d-453b-8e73-3a447e197309-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7d1e5c-sl2nh\" (UID: \"4089af4a-de4d-453b-8e73-3a447e197309\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7d1e5c-sl2nh" Oct 30 17:11:20.872401 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:20.803546 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqz87\" (UniqueName: \"kubernetes.io/projected/4089af4a-de4d-453b-8e73-3a447e197309-kube-api-access-cqz87\") pod \"maas-api-revoke-keys-e2e-api-a-7d1e5c-sl2nh\" (UID: \"4089af4a-de4d-453b-8e73-3a447e197309\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7d1e5c-sl2nh" Oct 30 17:11:20.872401 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:20.621094 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7d1e5c-sl2nh"] Oct 30 17:11:20.872401 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:20.621000 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:11:20.872401 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:20.621051 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:11:20.989661 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:20.937967 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4089af4a-de4d-453b-8e73-3a447e197309-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7d1e5c-sl2nh\" (UID: \"4089af4a-de4d-453b-8e73-3a447e197309\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7d1e5c-sl2nh" Oct 30 17:11:20.989661 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:20.938035 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4089af4a-de4d-453b-8e73-3a447e197309-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7d1e5c-sl2nh\" (UID: \"4089af4a-de4d-453b-8e73-3a447e197309\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7d1e5c-sl2nh" Oct 30 17:11:20.989661 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:20.938081 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqz87\" (UniqueName: \"kubernetes.io/projected/4089af4a-de4d-453b-8e73-3a447e197309-kube-api-access-cqz87\") pod \"maas-api-revoke-keys-e2e-api-a-7d1e5c-sl2nh\" (UID: \"4089af4a-de4d-453b-8e73-3a447e197309\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7d1e5c-sl2nh" Oct 30 17:11:20.989661 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:20.939646 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4089af4a-de4d-453b-8e73-3a447e197309-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7d1e5c-sl2nh\" (UID: \"4089af4a-de4d-453b-8e73-3a447e197309\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7d1e5c-sl2nh" Oct 30 17:11:21.070342 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:21.067331 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4089af4a-de4d-453b-8e73-3a447e197309-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7d1e5c-sl2nh\" (UID: \"4089af4a-de4d-453b-8e73-3a447e197309\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7d1e5c-sl2nh" Oct 30 17:11:21.070342 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:21.069339 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqz87\" (UniqueName: \"kubernetes.io/projected/4089af4a-de4d-453b-8e73-3a447e197309-kube-api-access-cqz87\") pod \"maas-api-revoke-keys-e2e-api-a-7d1e5c-sl2nh\" (UID: \"4089af4a-de4d-453b-8e73-3a447e197309\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7d1e5c-sl2nh" Oct 30 17:11:21.367806 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:21.367765 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7d1e5c-sl2nh" Oct 30 17:11:21.486350 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:21.478692 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" podUID="2f1e3bbb-ab1c-455d-a05d-d62f72c5d009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 17:11:21.662665 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:21.577855 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/distinct-w2-b4f406-kserve-5db4cc6969-7n846" podUID="b8810dbc-b769-46a4-90b9-72a68592bc1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:21.662665 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:21.610777 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" podUID="9bfbc6d3-91fd-41a7-9585-40a8f252f743" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:21.662665 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:21.611076 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq" podUID="f18303fb-6d27-4ff1-9cc9-ed032e43d098" containerName="main" probeResult="failure" output="Get \"https://10.134.0.120:8000/ready\": context deadline exceeded" Oct 30 17:11:21.817327 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:21.816685 4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" podUID="9bfbc6d3-91fd-41a7-9585-40a8f252f743" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:21.853338 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:21.845158 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" podUID="a3e5e732-2c67-4377-9755-e2178bf33772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:22.133328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:22.130386 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" podUID="75321c5d-8bf8-45c6-8811-81981fe3cad4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:22.133328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:22.130602 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" podUID="0b832781-3d41-4496-aa86-e4d92792e590" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 17:11:22.133714 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:22.133682 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" podUID="4b4ad752-e053-4cea-a3bc-fe2be6f92021" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:22.235360 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:22.220400 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/distinct-w1-04248e-kserve-5859d49965-kgq46" podUID="8dda5ac4-30dd-432e-81eb-cb6d39800628" containerName="main" probeResult="failure" output="Get \"https://10.134.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:22.684541 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:22.634443 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq" podUID="f18303fb-6d27-4ff1-9cc9-ed032e43d098" containerName="main" probeResult="failure" output="Get \"https://10.134.0.120:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:22.684541 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:22.670029 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/distinct-w2-b4f406-kserve-5db4cc6969-7n846" podUID="b8810dbc-b769-46a4-90b9-72a68592bc1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:22.684541 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:22.670405 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" podUID="9bfbc6d3-91fd-41a7-9585-40a8f252f743" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 30 17:11:22.684541 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:22.672845 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5" podUID="4360e9dc-c001-4102-8af6-b55b692fcf2c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:22.684541 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:22.672954 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" podUID="8a6a85ad-c944-450f-b0bb-e3b64b0f56f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 17:11:22.684541 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:22.673256 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" podUID="2f1e3bbb-ab1c-455d-a05d-d62f72c5d009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 17:11:22.727390 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:22.718508 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" podUID="c9ad71b7-ad90-4c8c-bf27-0f10b23f1593" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 17:11:22.863578 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:22.801089 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04248e/distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq" Oct 30 17:11:22.863578 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:22.801191 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b4f406/distinct-w2-b4f406-kserve-5db4cc6969-7n846" Oct 30 17:11:22.863578 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:22.801254 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" Oct 30 17:11:22.863578 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:22.801333 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" Oct 30 17:11:22.863578 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:22.819746 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb" podUID="d86fded1-2e04-4d73-bb22-ea0d9a61b961" containerName="main" probeResult="failure" output="Get \"https://10.134.0.127:8000/ready\": context deadline exceeded" Oct 30 17:11:22.885700 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:22.885659 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" podUID="43ca764d-ae0e-4858-9841-5bbd0ffdbb85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:22.928008 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:22.927963 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" podUID="a3e5e732-2c67-4377-9755-e2178bf33772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:22.980437 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:22.977377 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" Oct 30 17:11:23.059817 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:23.059783 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" Oct 30 17:11:23.163328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:23.131359 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" podUID="0b832781-3d41-4496-aa86-e4d92792e590" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 17:11:23.169330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:23.133219 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" podUID="75321c5d-8bf8-45c6-8811-81981fe3cad4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:23.169330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:23.143582 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" podUID="4b4ad752-e053-4cea-a3bc-fe2be6f92021" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Oct 30 17:11:23.169330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:23.168390 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" Oct 30 17:11:23.169330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:23.168518 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" Oct 30 17:11:23.169330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:23.168577 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" Oct 30 17:11:23.192461 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:23.192417 4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" podUID="2f1e3bbb-ab1c-455d-a05d-d62f72c5d009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:23.273136 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:23.257803 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04248e/distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq" Oct 30 17:11:23.273136 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:23.266639 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" Oct 30 17:11:23.273136 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:23.271394 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b4f406/distinct-w2-b4f406-kserve-5db4cc6969-7n846" Oct 30 17:11:23.273136 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:23.271449 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" Oct 30 17:11:23.274740 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:23.273216 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" Oct 30 17:11:23.274947 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:23.274925 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" Oct 30 17:11:23.276194 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:23.276127 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" Oct 30 17:11:23.377375 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:23.377347 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7d1e5c-sl2nh"] Oct 30 17:11:23.861950 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:23.861922 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9"] Oct 30 17:11:23.862944 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:23.862914 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9" podUID="79ea6704-625a-4af9-83c6-7f5214a6b8b4" containerName="istio-proxy" containerID="cri-o://b30e42bcea4c35136a893f35f69ba397516426654b5cf251fedc902b3388ac0c" gracePeriod=30 Oct 30 17:11:23.869451 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:23.869429 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh"] Oct 30 17:11:23.869808 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:23.869778 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh" podUID="aa811185-1eb8-451b-afb7-9580783a30f2" containerName="istio-proxy" containerID="cri-o://ee7a7bb2cd70fb9312ee4948bd61f60505bd923fdced7d9ce6d77d059886eb2e" gracePeriod=30 Oct 30 17:11:23.874608 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:23.874588 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc"] Oct 30 17:11:23.875010 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:23.874983 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc" podUID="c99fb3fd-1362-414e-82bd-88f20c055166" containerName="istio-proxy" containerID="cri-o://a72e5b4dcef303d00a23928b773b6cce77fedcebe220e88403df980275c177bd" gracePeriod=30 Oct 30 17:11:24.311332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:24.307665 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7d1e5c-sl2nh" event={"ID":"4089af4a-de4d-453b-8e73-3a447e197309","Type":"ContainerStarted","Data":"961dc7a7a9dfa0767419f46e4a780f01a8e2547f066471f56db69e4b032dfe71"} Oct 30 17:11:24.323435 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:24.323398 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7d1e5c-sl2nh" event={"ID":"4089af4a-de4d-453b-8e73-3a447e197309","Type":"ContainerStarted","Data":"c6e1fc3ad3248d45128293ce3d507986d634586c3a08f72e45780bf80cd2482c"} Oct 30 17:11:24.378355 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:24.378278 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7d1e5c-sl2nh" podStartSLOduration=4.378261545 podStartE2EDuration="4.378261545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:11:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:11:24.370676196 +0000 UTC m=+1458.371431118" watchObservedRunningTime="2026-10-30 17:11:24.378261545 +0000 UTC m=+1458.379016505" Oct 30 17:11:25.371326 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:25.371014 4522 generic.go:356] "Generic (PLEG): container finished" podID="4089af4a-de4d-453b-8e73-3a447e197309" containerID="961dc7a7a9dfa0767419f46e4a780f01a8e2547f066471f56db69e4b032dfe71" exitCode=0 Oct 30 17:11:25.371746 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:25.371339 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7d1e5c-sl2nh" event={"ID":"4089af4a-de4d-453b-8e73-3a447e197309","Type":"ContainerDied","Data":"961dc7a7a9dfa0767419f46e4a780f01a8e2547f066471f56db69e4b032dfe71"} Oct 30 17:11:26.656899 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:26.656826 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7d1e5c-sl2nh" Oct 30 17:11:26.671615 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:26.671586 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4089af4a-de4d-453b-8e73-3a447e197309-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4089af4a-de4d-453b-8e73-3a447e197309-maas-api-openshift-ca\") pod \"4089af4a-de4d-453b-8e73-3a447e197309\" (UID: \"4089af4a-de4d-453b-8e73-3a447e197309\") " Oct 30 17:11:26.671803 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:26.671785 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4089af4a-de4d-453b-8e73-3a447e197309-kube-api-access-cqz87\" (UniqueName: \"kubernetes.io/projected/4089af4a-de4d-453b-8e73-3a447e197309-kube-api-access-cqz87\") pod \"4089af4a-de4d-453b-8e73-3a447e197309\" (UID: \"4089af4a-de4d-453b-8e73-3a447e197309\") " Oct 30 17:11:26.671971 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:26.671950 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4089af4a-de4d-453b-8e73-3a447e197309-maas-api-openshift-ca" pod "4089af4a-de4d-453b-8e73-3a447e197309" (UID: "4089af4a-de4d-453b-8e73-3a447e197309"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:11:26.672068 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:26.672051 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4089af4a-de4d-453b-8e73-3a447e197309-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4089af4a-de4d-453b-8e73-3a447e197309-maas-api-xks-ca\") pod \"4089af4a-de4d-453b-8e73-3a447e197309\" (UID: \"4089af4a-de4d-453b-8e73-3a447e197309\") " Oct 30 17:11:26.672761 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:26.672739 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4089af4a-de4d-453b-8e73-3a447e197309-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:26.674037 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:26.674002 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4089af4a-de4d-453b-8e73-3a447e197309-kube-api-access-cqz87" pod "4089af4a-de4d-453b-8e73-3a447e197309" (UID: "4089af4a-de4d-453b-8e73-3a447e197309"). InnerVolumeSpecName "kube-api-access-cqz87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:11:26.674444 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:26.674423 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4089af4a-de4d-453b-8e73-3a447e197309-maas-api-xks-ca" pod "4089af4a-de4d-453b-8e73-3a447e197309" (UID: "4089af4a-de4d-453b-8e73-3a447e197309"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:11:26.774560 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:26.774508 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4089af4a-de4d-453b-8e73-3a447e197309-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:26.774560 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:26.774540 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqz87\" (UniqueName: \"kubernetes.io/projected/4089af4a-de4d-453b-8e73-3a447e197309-kube-api-access-cqz87\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:27.393983 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:27.393948 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7d1e5c-sl2nh" Oct 30 17:11:27.393983 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:27.393977 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7d1e5c-sl2nh" event={"ID":"4089af4a-de4d-453b-8e73-3a447e197309","Type":"ContainerDied","Data":"c6e1fc3ad3248d45128293ce3d507986d634586c3a08f72e45780bf80cd2482c"} Oct 30 17:11:27.394241 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:27.394028 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6e1fc3ad3248d45128293ce3d507986d634586c3a08f72e45780bf80cd2482c" Oct 30 17:11:29.076192 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.076160 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h"] Oct 30 17:11:29.077343 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.077301 4522 cpu_manager.go:414] "containerMap: removing container" podUID="4089af4a-de4d-453b-8e73-3a447e197309" containerName="revoke-keys" Oct 30 17:11:29.077399 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.077347 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4089af4a-de4d-453b-8e73-3a447e197309" containerName="revoke-keys" Oct 30 17:11:29.077624 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.077607 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="b590c8f6-9d91-49bb-8e0f-5e65eb24e631" containerName="istio-proxy" Oct 30 17:11:29.077670 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.077646 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="4089af4a-de4d-453b-8e73-3a447e197309" containerName="revoke-keys" Oct 30 17:11:29.120225 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.119541 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h" Oct 30 17:11:29.120446 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.119434 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h"] Oct 30 17:11:29.140568 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.139265 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-c4c2e4-openshift-default-dockercfg-7cm86\"" Oct 30 17:11:29.220824 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.220090 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ef91484-ab4c-4b90-9802-0ead2c544bf3-istio-podinfo\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h\" (UID: \"6ef91484-ab4c-4b90-9802-0ead2c544bf3\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h" Oct 30 17:11:29.220824 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.220170 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ef91484-ab4c-4b90-9802-0ead2c544bf3-credential-socket\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h\" (UID: \"6ef91484-ab4c-4b90-9802-0ead2c544bf3\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h" Oct 30 17:11:29.220824 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.220421 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ef91484-ab4c-4b90-9802-0ead2c544bf3-istio-envoy\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h\" (UID: \"6ef91484-ab4c-4b90-9802-0ead2c544bf3\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h" Oct 30 17:11:29.220824 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.220487 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ef91484-ab4c-4b90-9802-0ead2c544bf3-workload-socket\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h\" (UID: \"6ef91484-ab4c-4b90-9802-0ead2c544bf3\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h" Oct 30 17:11:29.220824 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.220684 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ef91484-ab4c-4b90-9802-0ead2c544bf3-istio-token\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h\" (UID: \"6ef91484-ab4c-4b90-9802-0ead2c544bf3\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h" Oct 30 17:11:29.220824 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.220810 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ef91484-ab4c-4b90-9802-0ead2c544bf3-workload-certs\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h\" (UID: \"6ef91484-ab4c-4b90-9802-0ead2c544bf3\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h" Oct 30 17:11:29.221256 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.220884 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmw8n\" (UniqueName: \"kubernetes.io/projected/6ef91484-ab4c-4b90-9802-0ead2c544bf3-kube-api-access-qmw8n\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h\" (UID: \"6ef91484-ab4c-4b90-9802-0ead2c544bf3\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h" Oct 30 17:11:29.221256 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.220936 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ef91484-ab4c-4b90-9802-0ead2c544bf3-istiod-ca-cert\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h\" (UID: \"6ef91484-ab4c-4b90-9802-0ead2c544bf3\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h" Oct 30 17:11:29.221256 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.220973 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ef91484-ab4c-4b90-9802-0ead2c544bf3-istio-data\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h\" (UID: \"6ef91484-ab4c-4b90-9802-0ead2c544bf3\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h" Oct 30 17:11:29.344393 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.312438 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9" Oct 30 17:11:29.344393 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.323873 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79ea6704-625a-4af9-83c6-7f5214a6b8b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ea6704-625a-4af9-83c6-7f5214a6b8b4-credential-socket\") pod \"79ea6704-625a-4af9-83c6-7f5214a6b8b4\" (UID: \"79ea6704-625a-4af9-83c6-7f5214a6b8b4\") " Oct 30 17:11:29.344393 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.323918 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79ea6704-625a-4af9-83c6-7f5214a6b8b4-kube-api-access-dw55g\" (UniqueName: \"kubernetes.io/projected/79ea6704-625a-4af9-83c6-7f5214a6b8b4-kube-api-access-dw55g\") pod \"79ea6704-625a-4af9-83c6-7f5214a6b8b4\" (UID: \"79ea6704-625a-4af9-83c6-7f5214a6b8b4\") " Oct 30 17:11:29.344393 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.323953 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79ea6704-625a-4af9-83c6-7f5214a6b8b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79ea6704-625a-4af9-83c6-7f5214a6b8b4-istiod-ca-cert\") pod \"79ea6704-625a-4af9-83c6-7f5214a6b8b4\" (UID: \"79ea6704-625a-4af9-83c6-7f5214a6b8b4\") " Oct 30 17:11:29.344393 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.324102 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79ea6704-625a-4af9-83c6-7f5214a6b8b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79ea6704-625a-4af9-83c6-7f5214a6b8b4-istio-data\") pod \"79ea6704-625a-4af9-83c6-7f5214a6b8b4\" (UID: \"79ea6704-625a-4af9-83c6-7f5214a6b8b4\") " Oct 30 17:11:29.344393 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.324215 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79ea6704-625a-4af9-83c6-7f5214a6b8b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ea6704-625a-4af9-83c6-7f5214a6b8b4-workload-socket\") pod \"79ea6704-625a-4af9-83c6-7f5214a6b8b4\" (UID: \"79ea6704-625a-4af9-83c6-7f5214a6b8b4\") " Oct 30 17:11:29.344393 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.324319 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79ea6704-625a-4af9-83c6-7f5214a6b8b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79ea6704-625a-4af9-83c6-7f5214a6b8b4-istio-envoy\") pod \"79ea6704-625a-4af9-83c6-7f5214a6b8b4\" (UID: \"79ea6704-625a-4af9-83c6-7f5214a6b8b4\") " Oct 30 17:11:29.344393 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.324360 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79ea6704-625a-4af9-83c6-7f5214a6b8b4-istio-token\" (UniqueName: \"kubernetes.io/projected/79ea6704-625a-4af9-83c6-7f5214a6b8b4-istio-token\") pod \"79ea6704-625a-4af9-83c6-7f5214a6b8b4\" (UID: \"79ea6704-625a-4af9-83c6-7f5214a6b8b4\") " Oct 30 17:11:29.344393 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.324456 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79ea6704-625a-4af9-83c6-7f5214a6b8b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79ea6704-625a-4af9-83c6-7f5214a6b8b4-workload-certs\") pod \"79ea6704-625a-4af9-83c6-7f5214a6b8b4\" (UID: \"79ea6704-625a-4af9-83c6-7f5214a6b8b4\") " Oct 30 17:11:29.344393 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.324526 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79ea6704-625a-4af9-83c6-7f5214a6b8b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79ea6704-625a-4af9-83c6-7f5214a6b8b4-istio-podinfo\") pod \"79ea6704-625a-4af9-83c6-7f5214a6b8b4\" (UID: \"79ea6704-625a-4af9-83c6-7f5214a6b8b4\") " Oct 30 17:11:29.344393 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.324843 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ef91484-ab4c-4b90-9802-0ead2c544bf3-istio-envoy\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h\" (UID: \"6ef91484-ab4c-4b90-9802-0ead2c544bf3\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h" Oct 30 17:11:29.344393 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.324899 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ef91484-ab4c-4b90-9802-0ead2c544bf3-workload-socket\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h\" (UID: \"6ef91484-ab4c-4b90-9802-0ead2c544bf3\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h" Oct 30 17:11:29.344393 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.325053 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ef91484-ab4c-4b90-9802-0ead2c544bf3-istio-token\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h\" (UID: \"6ef91484-ab4c-4b90-9802-0ead2c544bf3\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h" Oct 30 17:11:29.344393 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.325143 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ef91484-ab4c-4b90-9802-0ead2c544bf3-workload-certs\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h\" (UID: \"6ef91484-ab4c-4b90-9802-0ead2c544bf3\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h" Oct 30 17:11:29.344393 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.325226 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmw8n\" (UniqueName: \"kubernetes.io/projected/6ef91484-ab4c-4b90-9802-0ead2c544bf3-kube-api-access-qmw8n\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h\" (UID: \"6ef91484-ab4c-4b90-9802-0ead2c544bf3\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h" Oct 30 17:11:29.344393 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.325281 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ef91484-ab4c-4b90-9802-0ead2c544bf3-istiod-ca-cert\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h\" (UID: \"6ef91484-ab4c-4b90-9802-0ead2c544bf3\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h" Oct 30 17:11:29.344393 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.325339 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ef91484-ab4c-4b90-9802-0ead2c544bf3-istio-data\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h\" (UID: \"6ef91484-ab4c-4b90-9802-0ead2c544bf3\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h" Oct 30 17:11:29.344393 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.325447 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ef91484-ab4c-4b90-9802-0ead2c544bf3-istio-podinfo\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h\" (UID: \"6ef91484-ab4c-4b90-9802-0ead2c544bf3\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h" Oct 30 17:11:29.344393 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.325485 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ef91484-ab4c-4b90-9802-0ead2c544bf3-credential-socket\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h\" (UID: \"6ef91484-ab4c-4b90-9802-0ead2c544bf3\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h" Oct 30 17:11:29.344393 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.326100 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ef91484-ab4c-4b90-9802-0ead2c544bf3-credential-socket\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h\" (UID: \"6ef91484-ab4c-4b90-9802-0ead2c544bf3\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h" Oct 30 17:11:29.344393 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.326537 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ef91484-ab4c-4b90-9802-0ead2c544bf3-workload-certs\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h\" (UID: \"6ef91484-ab4c-4b90-9802-0ead2c544bf3\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h" Oct 30 17:11:29.344393 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.325222 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79ea6704-625a-4af9-83c6-7f5214a6b8b4-istio-data" pod "79ea6704-625a-4af9-83c6-7f5214a6b8b4" (UID: "79ea6704-625a-4af9-83c6-7f5214a6b8b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:29.344393 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.325414 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79ea6704-625a-4af9-83c6-7f5214a6b8b4-workload-socket" pod "79ea6704-625a-4af9-83c6-7f5214a6b8b4" (UID: "79ea6704-625a-4af9-83c6-7f5214a6b8b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:29.344393 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.327793 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ef91484-ab4c-4b90-9802-0ead2c544bf3-istiod-ca-cert\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h\" (UID: \"6ef91484-ab4c-4b90-9802-0ead2c544bf3\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h" Oct 30 17:11:29.344393 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.328072 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ef91484-ab4c-4b90-9802-0ead2c544bf3-istio-data\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h\" (UID: \"6ef91484-ab4c-4b90-9802-0ead2c544bf3\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h" Oct 30 17:11:29.344393 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.328453 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79ea6704-625a-4af9-83c6-7f5214a6b8b4-workload-certs" pod "79ea6704-625a-4af9-83c6-7f5214a6b8b4" (UID: "79ea6704-625a-4af9-83c6-7f5214a6b8b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:29.344393 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.328743 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ef91484-ab4c-4b90-9802-0ead2c544bf3-workload-socket\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h\" (UID: \"6ef91484-ab4c-4b90-9802-0ead2c544bf3\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h" Oct 30 17:11:29.344393 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.329502 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79ea6704-625a-4af9-83c6-7f5214a6b8b4-istiod-ca-cert" pod "79ea6704-625a-4af9-83c6-7f5214a6b8b4" (UID: "79ea6704-625a-4af9-83c6-7f5214a6b8b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:11:29.344393 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.334954 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79ea6704-625a-4af9-83c6-7f5214a6b8b4-credential-socket" pod "79ea6704-625a-4af9-83c6-7f5214a6b8b4" (UID: "79ea6704-625a-4af9-83c6-7f5214a6b8b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:29.351097 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.350793 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ef91484-ab4c-4b90-9802-0ead2c544bf3-istio-envoy\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h\" (UID: \"6ef91484-ab4c-4b90-9802-0ead2c544bf3\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h" Oct 30 17:11:29.354501 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.354470 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79ea6704-625a-4af9-83c6-7f5214a6b8b4-kube-api-access-dw55g" pod "79ea6704-625a-4af9-83c6-7f5214a6b8b4" (UID: "79ea6704-625a-4af9-83c6-7f5214a6b8b4"). InnerVolumeSpecName "kube-api-access-dw55g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:11:29.357256 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.357226 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmw8n\" (UniqueName: \"kubernetes.io/projected/6ef91484-ab4c-4b90-9802-0ead2c544bf3-kube-api-access-qmw8n\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h\" (UID: \"6ef91484-ab4c-4b90-9802-0ead2c544bf3\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h" Oct 30 17:11:29.357792 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.357768 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ef91484-ab4c-4b90-9802-0ead2c544bf3-istio-podinfo\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h\" (UID: \"6ef91484-ab4c-4b90-9802-0ead2c544bf3\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h" Oct 30 17:11:29.359972 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.359620 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79ea6704-625a-4af9-83c6-7f5214a6b8b4-istio-envoy" pod "79ea6704-625a-4af9-83c6-7f5214a6b8b4" (UID: "79ea6704-625a-4af9-83c6-7f5214a6b8b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:29.359972 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.359935 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79ea6704-625a-4af9-83c6-7f5214a6b8b4-istio-podinfo" pod "79ea6704-625a-4af9-83c6-7f5214a6b8b4" (UID: "79ea6704-625a-4af9-83c6-7f5214a6b8b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:11:29.359972 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.359966 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79ea6704-625a-4af9-83c6-7f5214a6b8b4-istio-token" pod "79ea6704-625a-4af9-83c6-7f5214a6b8b4" (UID: "79ea6704-625a-4af9-83c6-7f5214a6b8b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:11:29.363744 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.363718 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ef91484-ab4c-4b90-9802-0ead2c544bf3-istio-token\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h\" (UID: \"6ef91484-ab4c-4b90-9802-0ead2c544bf3\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h" Oct 30 17:11:29.385656 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.385133 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc" Oct 30 17:11:29.429133 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.428642 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c99fb3fd-1362-414e-82bd-88f20c055166-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c99fb3fd-1362-414e-82bd-88f20c055166-istio-envoy\") pod \"c99fb3fd-1362-414e-82bd-88f20c055166\" (UID: \"c99fb3fd-1362-414e-82bd-88f20c055166\") " Oct 30 17:11:29.429133 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.428793 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c99fb3fd-1362-414e-82bd-88f20c055166-kube-api-access-qx2gn\" (UniqueName: \"kubernetes.io/projected/c99fb3fd-1362-414e-82bd-88f20c055166-kube-api-access-qx2gn\") pod \"c99fb3fd-1362-414e-82bd-88f20c055166\" (UID: \"c99fb3fd-1362-414e-82bd-88f20c055166\") " Oct 30 17:11:29.429133 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.428884 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c99fb3fd-1362-414e-82bd-88f20c055166-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c99fb3fd-1362-414e-82bd-88f20c055166-istiod-ca-cert\") pod \"c99fb3fd-1362-414e-82bd-88f20c055166\" (UID: \"c99fb3fd-1362-414e-82bd-88f20c055166\") " Oct 30 17:11:29.429133 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.428939 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c99fb3fd-1362-414e-82bd-88f20c055166-istio-token\" (UniqueName: \"kubernetes.io/projected/c99fb3fd-1362-414e-82bd-88f20c055166-istio-token\") pod \"c99fb3fd-1362-414e-82bd-88f20c055166\" (UID: \"c99fb3fd-1362-414e-82bd-88f20c055166\") " Oct 30 17:11:29.429133 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.428987 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c99fb3fd-1362-414e-82bd-88f20c055166-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c99fb3fd-1362-414e-82bd-88f20c055166-credential-socket\") pod \"c99fb3fd-1362-414e-82bd-88f20c055166\" (UID: \"c99fb3fd-1362-414e-82bd-88f20c055166\") " Oct 30 17:11:29.429133 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.429020 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c99fb3fd-1362-414e-82bd-88f20c055166-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c99fb3fd-1362-414e-82bd-88f20c055166-workload-socket\") pod \"c99fb3fd-1362-414e-82bd-88f20c055166\" (UID: \"c99fb3fd-1362-414e-82bd-88f20c055166\") " Oct 30 17:11:29.429133 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.429058 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c99fb3fd-1362-414e-82bd-88f20c055166-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c99fb3fd-1362-414e-82bd-88f20c055166-istio-podinfo\") pod \"c99fb3fd-1362-414e-82bd-88f20c055166\" (UID: \"c99fb3fd-1362-414e-82bd-88f20c055166\") " Oct 30 17:11:29.429133 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.429083 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c99fb3fd-1362-414e-82bd-88f20c055166-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c99fb3fd-1362-414e-82bd-88f20c055166-workload-certs\") pod \"c99fb3fd-1362-414e-82bd-88f20c055166\" (UID: \"c99fb3fd-1362-414e-82bd-88f20c055166\") " Oct 30 17:11:29.432219 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.431491 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c99fb3fd-1362-414e-82bd-88f20c055166-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c99fb3fd-1362-414e-82bd-88f20c055166-istio-data\") pod \"c99fb3fd-1362-414e-82bd-88f20c055166\" (UID: \"c99fb3fd-1362-414e-82bd-88f20c055166\") " Oct 30 17:11:29.433973 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.433640 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c99fb3fd-1362-414e-82bd-88f20c055166-istio-token" pod "c99fb3fd-1362-414e-82bd-88f20c055166" (UID: "c99fb3fd-1362-414e-82bd-88f20c055166"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:11:29.434785 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.434418 4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79ea6704-625a-4af9-83c6-7f5214a6b8b4-istio-podinfo\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:29.434785 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.434446 4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ea6704-625a-4af9-83c6-7f5214a6b8b4-credential-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:29.434785 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.434460 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw55g\" (UniqueName: \"kubernetes.io/projected/79ea6704-625a-4af9-83c6-7f5214a6b8b4-kube-api-access-dw55g\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:29.434785 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.434473 4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79ea6704-625a-4af9-83c6-7f5214a6b8b4-istiod-ca-cert\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:29.434785 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.434487 4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79ea6704-625a-4af9-83c6-7f5214a6b8b4-istio-data\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:29.434785 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.434500 4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ea6704-625a-4af9-83c6-7f5214a6b8b4-workload-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:29.434785 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.434514 4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c99fb3fd-1362-414e-82bd-88f20c055166-istio-token\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:29.434785 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.434529 4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79ea6704-625a-4af9-83c6-7f5214a6b8b4-istio-envoy\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:29.434785 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.434541 4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79ea6704-625a-4af9-83c6-7f5214a6b8b4-istio-token\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:29.434785 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.434554 4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79ea6704-625a-4af9-83c6-7f5214a6b8b4-workload-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:29.435874 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.435826 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c99fb3fd-1362-414e-82bd-88f20c055166-credential-socket" pod "c99fb3fd-1362-414e-82bd-88f20c055166" (UID: "c99fb3fd-1362-414e-82bd-88f20c055166"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:29.437295 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.436145 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c99fb3fd-1362-414e-82bd-88f20c055166-workload-socket" pod "c99fb3fd-1362-414e-82bd-88f20c055166" (UID: "c99fb3fd-1362-414e-82bd-88f20c055166"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:29.438879 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.438562 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c99fb3fd-1362-414e-82bd-88f20c055166-istio-envoy" pod "c99fb3fd-1362-414e-82bd-88f20c055166" (UID: "c99fb3fd-1362-414e-82bd-88f20c055166"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:29.438879 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.438658 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c99fb3fd-1362-414e-82bd-88f20c055166-istio-data" pod "c99fb3fd-1362-414e-82bd-88f20c055166" (UID: "c99fb3fd-1362-414e-82bd-88f20c055166"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:29.438879 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.438707 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c99fb3fd-1362-414e-82bd-88f20c055166-workload-certs" pod "c99fb3fd-1362-414e-82bd-88f20c055166" (UID: "c99fb3fd-1362-414e-82bd-88f20c055166"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:29.438879 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.438853 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c99fb3fd-1362-414e-82bd-88f20c055166-istiod-ca-cert" pod "c99fb3fd-1362-414e-82bd-88f20c055166" (UID: "c99fb3fd-1362-414e-82bd-88f20c055166"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:11:29.455806 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.455776 4522 generic.go:356] "Generic (PLEG): container finished" podID="79ea6704-625a-4af9-83c6-7f5214a6b8b4" containerID="b30e42bcea4c35136a893f35f69ba397516426654b5cf251fedc902b3388ac0c" exitCode=0 Oct 30 17:11:29.455979 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.455922 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9" event={"ID":"79ea6704-625a-4af9-83c6-7f5214a6b8b4","Type":"ContainerDied","Data":"b30e42bcea4c35136a893f35f69ba397516426654b5cf251fedc902b3388ac0c"} Oct 30 17:11:29.455979 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.455976 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9" event={"ID":"79ea6704-625a-4af9-83c6-7f5214a6b8b4","Type":"ContainerDied","Data":"027ae8786d72b07ed8a2aa8d93e3ee842cd51c9c5fbc5e0b0ea994c56fe57090"} Oct 30 17:11:29.456097 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.456005 4522 scope.go:122] "RemoveContainer" containerID="b30e42bcea4c35136a893f35f69ba397516426654b5cf251fedc902b3388ac0c" Oct 30 17:11:29.456187 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.456172 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9" Oct 30 17:11:29.465905 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.465722 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c99fb3fd-1362-414e-82bd-88f20c055166-kube-api-access-qx2gn" pod "c99fb3fd-1362-414e-82bd-88f20c055166" (UID: "c99fb3fd-1362-414e-82bd-88f20c055166"). InnerVolumeSpecName "kube-api-access-qx2gn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:11:29.466242 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.466018 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h" Oct 30 17:11:29.467229 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.467207 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c99fb3fd-1362-414e-82bd-88f20c055166-istio-podinfo" pod "c99fb3fd-1362-414e-82bd-88f20c055166" (UID: "c99fb3fd-1362-414e-82bd-88f20c055166"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:11:29.467994 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.467850 4522 generic.go:356] "Generic (PLEG): container finished" podID="aa811185-1eb8-451b-afb7-9580783a30f2" containerID="ee7a7bb2cd70fb9312ee4948bd61f60505bd923fdced7d9ce6d77d059886eb2e" exitCode=0 Oct 30 17:11:29.467994 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.467955 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh" event={"ID":"aa811185-1eb8-451b-afb7-9580783a30f2","Type":"ContainerDied","Data":"ee7a7bb2cd70fb9312ee4948bd61f60505bd923fdced7d9ce6d77d059886eb2e"} Oct 30 17:11:29.470190 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.470164 4522 generic.go:356] "Generic (PLEG): container finished" podID="c99fb3fd-1362-414e-82bd-88f20c055166" containerID="a72e5b4dcef303d00a23928b773b6cce77fedcebe220e88403df980275c177bd" exitCode=0 Oct 30 17:11:29.470292 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.470272 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc" Oct 30 17:11:29.470387 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.470339 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc" event={"ID":"c99fb3fd-1362-414e-82bd-88f20c055166","Type":"ContainerDied","Data":"a72e5b4dcef303d00a23928b773b6cce77fedcebe220e88403df980275c177bd"} Oct 30 17:11:29.470434 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.470396 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc" event={"ID":"c99fb3fd-1362-414e-82bd-88f20c055166","Type":"ContainerDied","Data":"3fdf04d56684b580224bfe104161a2d90a2ba160d9c46bcb788ac4ecfdb74961"} Oct 30 17:11:29.481269 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.481247 4522 scope.go:122] "RemoveContainer" containerID="b30e42bcea4c35136a893f35f69ba397516426654b5cf251fedc902b3388ac0c" Oct 30 17:11:29.481726 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:11:29.481567 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b30e42bcea4c35136a893f35f69ba397516426654b5cf251fedc902b3388ac0c\": container with ID starting with b30e42bcea4c35136a893f35f69ba397516426654b5cf251fedc902b3388ac0c not found: ID does not exist" containerID="b30e42bcea4c35136a893f35f69ba397516426654b5cf251fedc902b3388ac0c" Oct 30 17:11:29.481726 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.481611 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b30e42bcea4c35136a893f35f69ba397516426654b5cf251fedc902b3388ac0c"} err="failed to get container status \"b30e42bcea4c35136a893f35f69ba397516426654b5cf251fedc902b3388ac0c\": rpc error: code = NotFound desc = could not find container \"b30e42bcea4c35136a893f35f69ba397516426654b5cf251fedc902b3388ac0c\": container with ID starting with b30e42bcea4c35136a893f35f69ba397516426654b5cf251fedc902b3388ac0c not found: ID does not exist" Oct 30 17:11:29.481726 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.481636 4522 scope.go:122] "RemoveContainer" containerID="a72e5b4dcef303d00a23928b773b6cce77fedcebe220e88403df980275c177bd" Oct 30 17:11:29.535701 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.535668 4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c99fb3fd-1362-414e-82bd-88f20c055166-istio-envoy\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:29.535701 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.535700 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx2gn\" (UniqueName: \"kubernetes.io/projected/c99fb3fd-1362-414e-82bd-88f20c055166-kube-api-access-qx2gn\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:29.535913 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.535712 4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c99fb3fd-1362-414e-82bd-88f20c055166-istiod-ca-cert\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:29.535913 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.535728 4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c99fb3fd-1362-414e-82bd-88f20c055166-credential-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:29.535913 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.535741 4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c99fb3fd-1362-414e-82bd-88f20c055166-workload-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:29.535913 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.535753 4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c99fb3fd-1362-414e-82bd-88f20c055166-istio-podinfo\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:29.535913 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.535765 4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c99fb3fd-1362-414e-82bd-88f20c055166-workload-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:29.535913 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.535777 4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c99fb3fd-1362-414e-82bd-88f20c055166-istio-data\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:29.613783 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.613756 4522 scope.go:122] "RemoveContainer" containerID="a72e5b4dcef303d00a23928b773b6cce77fedcebe220e88403df980275c177bd" Oct 30 17:11:29.614296 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:11:29.614265 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a72e5b4dcef303d00a23928b773b6cce77fedcebe220e88403df980275c177bd\": container with ID starting with a72e5b4dcef303d00a23928b773b6cce77fedcebe220e88403df980275c177bd not found: ID does not exist" containerID="a72e5b4dcef303d00a23928b773b6cce77fedcebe220e88403df980275c177bd" Oct 30 17:11:29.614422 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.614337 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a72e5b4dcef303d00a23928b773b6cce77fedcebe220e88403df980275c177bd"} err="failed to get container status \"a72e5b4dcef303d00a23928b773b6cce77fedcebe220e88403df980275c177bd\": rpc error: code = NotFound desc = could not find container \"a72e5b4dcef303d00a23928b773b6cce77fedcebe220e88403df980275c177bd\": container with ID starting with a72e5b4dcef303d00a23928b773b6cce77fedcebe220e88403df980275c177bd not found: ID does not exist" Oct 30 17:11:29.633967 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.633938 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh" Oct 30 17:11:29.667591 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.667498 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc"] Oct 30 17:11:29.674981 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.674948 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-khtzc"] Oct 30 17:11:29.722013 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.721973 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9"] Oct 30 17:11:29.730154 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.730118 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2r2t9"] Oct 30 17:11:29.739095 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.739057 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa811185-1eb8-451b-afb7-9580783a30f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa811185-1eb8-451b-afb7-9580783a30f2-istio-envoy\") pod \"aa811185-1eb8-451b-afb7-9580783a30f2\" (UID: \"aa811185-1eb8-451b-afb7-9580783a30f2\") " Oct 30 17:11:29.739256 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.739118 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa811185-1eb8-451b-afb7-9580783a30f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa811185-1eb8-451b-afb7-9580783a30f2-workload-socket\") pod \"aa811185-1eb8-451b-afb7-9580783a30f2\" (UID: \"aa811185-1eb8-451b-afb7-9580783a30f2\") " Oct 30 17:11:29.739256 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.739181 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa811185-1eb8-451b-afb7-9580783a30f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa811185-1eb8-451b-afb7-9580783a30f2-workload-certs\") pod \"aa811185-1eb8-451b-afb7-9580783a30f2\" (UID: \"aa811185-1eb8-451b-afb7-9580783a30f2\") " Oct 30 17:11:29.739399 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.739259 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa811185-1eb8-451b-afb7-9580783a30f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa811185-1eb8-451b-afb7-9580783a30f2-istiod-ca-cert\") pod \"aa811185-1eb8-451b-afb7-9580783a30f2\" (UID: \"aa811185-1eb8-451b-afb7-9580783a30f2\") " Oct 30 17:11:29.739399 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.739376 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa811185-1eb8-451b-afb7-9580783a30f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa811185-1eb8-451b-afb7-9580783a30f2-credential-socket\") pod \"aa811185-1eb8-451b-afb7-9580783a30f2\" (UID: \"aa811185-1eb8-451b-afb7-9580783a30f2\") " Oct 30 17:11:29.739509 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.739454 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa811185-1eb8-451b-afb7-9580783a30f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa811185-1eb8-451b-afb7-9580783a30f2-istio-data\") pod \"aa811185-1eb8-451b-afb7-9580783a30f2\" (UID: \"aa811185-1eb8-451b-afb7-9580783a30f2\") " Oct 30 17:11:29.739566 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.739518 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa811185-1eb8-451b-afb7-9580783a30f2-kube-api-access-pf247\" (UniqueName: \"kubernetes.io/projected/aa811185-1eb8-451b-afb7-9580783a30f2-kube-api-access-pf247\") pod \"aa811185-1eb8-451b-afb7-9580783a30f2\" (UID: \"aa811185-1eb8-451b-afb7-9580783a30f2\") " Oct 30 17:11:29.739566 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.739555 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa811185-1eb8-451b-afb7-9580783a30f2-istio-token\" (UniqueName: \"kubernetes.io/projected/aa811185-1eb8-451b-afb7-9580783a30f2-istio-token\") pod \"aa811185-1eb8-451b-afb7-9580783a30f2\" (UID: \"aa811185-1eb8-451b-afb7-9580783a30f2\") " Oct 30 17:11:29.739662 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.739582 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa811185-1eb8-451b-afb7-9580783a30f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa811185-1eb8-451b-afb7-9580783a30f2-istio-podinfo\") pod \"aa811185-1eb8-451b-afb7-9580783a30f2\" (UID: \"aa811185-1eb8-451b-afb7-9580783a30f2\") " Oct 30 17:11:29.739712 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.739675 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa811185-1eb8-451b-afb7-9580783a30f2-workload-certs" pod "aa811185-1eb8-451b-afb7-9580783a30f2" (UID: "aa811185-1eb8-451b-afb7-9580783a30f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:29.740114 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.739926 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa811185-1eb8-451b-afb7-9580783a30f2-workload-socket" pod "aa811185-1eb8-451b-afb7-9580783a30f2" (UID: "aa811185-1eb8-451b-afb7-9580783a30f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:29.740235 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.740141 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa811185-1eb8-451b-afb7-9580783a30f2-credential-socket" pod "aa811185-1eb8-451b-afb7-9580783a30f2" (UID: "aa811185-1eb8-451b-afb7-9580783a30f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:29.740298 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.740232 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa811185-1eb8-451b-afb7-9580783a30f2-istio-data" pod "aa811185-1eb8-451b-afb7-9580783a30f2" (UID: "aa811185-1eb8-451b-afb7-9580783a30f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:29.740686 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.740658 4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa811185-1eb8-451b-afb7-9580783a30f2-istio-data\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:29.740796 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.740690 4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa811185-1eb8-451b-afb7-9580783a30f2-workload-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:29.740796 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.740706 4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa811185-1eb8-451b-afb7-9580783a30f2-workload-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:29.740796 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.740720 4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa811185-1eb8-451b-afb7-9580783a30f2-credential-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:29.740967 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.740882 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa811185-1eb8-451b-afb7-9580783a30f2-istiod-ca-cert" pod "aa811185-1eb8-451b-afb7-9580783a30f2" (UID: "aa811185-1eb8-451b-afb7-9580783a30f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:11:29.742006 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.741971 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa811185-1eb8-451b-afb7-9580783a30f2-istio-envoy" pod "aa811185-1eb8-451b-afb7-9580783a30f2" (UID: "aa811185-1eb8-451b-afb7-9580783a30f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:29.743967 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.743940 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa811185-1eb8-451b-afb7-9580783a30f2-istio-token" pod "aa811185-1eb8-451b-afb7-9580783a30f2" (UID: "aa811185-1eb8-451b-afb7-9580783a30f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:11:29.744742 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.744709 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa811185-1eb8-451b-afb7-9580783a30f2-kube-api-access-pf247" pod "aa811185-1eb8-451b-afb7-9580783a30f2" (UID: "aa811185-1eb8-451b-afb7-9580783a30f2"). InnerVolumeSpecName "kube-api-access-pf247". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:11:29.744838 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.744711 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa811185-1eb8-451b-afb7-9580783a30f2-istio-podinfo" pod "aa811185-1eb8-451b-afb7-9580783a30f2" (UID: "aa811185-1eb8-451b-afb7-9580783a30f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:11:29.842640 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.842600 4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa811185-1eb8-451b-afb7-9580783a30f2-istio-envoy\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:29.842640 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.842639 4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa811185-1eb8-451b-afb7-9580783a30f2-istiod-ca-cert\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:29.842891 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.842655 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf247\" (UniqueName: \"kubernetes.io/projected/aa811185-1eb8-451b-afb7-9580783a30f2-kube-api-access-pf247\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:29.842891 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.842672 4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa811185-1eb8-451b-afb7-9580783a30f2-istio-token\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:29.842891 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.842687 4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa811185-1eb8-451b-afb7-9580783a30f2-istio-podinfo\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:29.956634 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:11:29.956600 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ef91484_ab4c_4b90_9802_0ead2c544bf3.slice/crio-0d9123e14519b5b9479d2b7b27098f572848e7b273dc64f8f60bfcd443bae472 WatchSource:0}: Error finding container 0d9123e14519b5b9479d2b7b27098f572848e7b273dc64f8f60bfcd443bae472: Status 404 returned error can't find the container with id 0d9123e14519b5b9479d2b7b27098f572848e7b273dc64f8f60bfcd443bae472 Oct 30 17:11:29.959375 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.959350 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h"] Oct 30 17:11:29.959907 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.959867 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:11:29.959974 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.959958 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:11:29.960037 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:29.960022 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:11:30.152693 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:30.152645 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79ea6704-625a-4af9-83c6-7f5214a6b8b4" path="/var/lib/kubelet/pods/79ea6704-625a-4af9-83c6-7f5214a6b8b4/volumes" Oct 30 17:11:30.153633 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:30.153607 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c99fb3fd-1362-414e-82bd-88f20c055166" path="/var/lib/kubelet/pods/c99fb3fd-1362-414e-82bd-88f20c055166/volumes" Oct 30 17:11:30.482065 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:30.482028 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh" Oct 30 17:11:30.482932 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:30.482025 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh" event={"ID":"aa811185-1eb8-451b-afb7-9580783a30f2","Type":"ContainerDied","Data":"ce618febfc7802b12ff2c3798c002ad6c97214188e8cd4ca4a37ba622b2b20c8"} Oct 30 17:11:30.483060 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:30.482964 4522 scope.go:122] "RemoveContainer" containerID="ee7a7bb2cd70fb9312ee4948bd61f60505bd923fdced7d9ce6d77d059886eb2e" Oct 30 17:11:30.487927 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:30.487159 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h" event={"ID":"6ef91484-ab4c-4b90-9802-0ead2c544bf3","Type":"ContainerStarted","Data":"f0790e925b052e5a80917a8801576521214dc41e97bdf1a8c08f62309ce4de04"} Oct 30 17:11:30.487927 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:30.487224 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h" event={"ID":"6ef91484-ab4c-4b90-9802-0ead2c544bf3","Type":"ContainerStarted","Data":"0d9123e14519b5b9479d2b7b27098f572848e7b273dc64f8f60bfcd443bae472"} Oct 30 17:11:30.547357 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:30.547298 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7d1e5c-547b5cccb8-r8865"] Oct 30 17:11:30.557088 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:30.557017 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h" podStartSLOduration=1.556998997 podStartE2EDuration="1.556998997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:11:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:11:30.553173815 +0000 UTC m=+1464.553928760" watchObservedRunningTime="2026-10-30 17:11:30.556998997 +0000 UTC m=+1464.557753899" Oct 30 17:11:30.583265 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:30.583228 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7d1e5c-79c87484bc-6dx5v"] Oct 30 17:11:30.603223 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:30.603188 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh"] Oct 30 17:11:30.610825 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:30.610799 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-2sjkh"] Oct 30 17:11:31.066910 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:31.066872 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eadf55b1-xznvl"] Oct 30 17:11:31.195602 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:31.073965 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eadf55b1-xznvl"] Oct 30 17:11:31.477369 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:31.474005 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h" Oct 30 17:11:31.477369 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:31.474137 4522 patch_prober.go:29] interesting pod/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 30 17:11:31.477369 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:31.474184 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h" podUID="6ef91484-ab4c-4b90-9802-0ead2c544bf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 30 17:11:31.599544 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:31.599446 4522 patch_prober.go:29] interesting pod/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:11:31.599816 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:31.599777 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m" podUID="4d683a0f-9de0-46dc-9772-d0915b0cb985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:31.651995 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:31.651927 4522 generic.go:356] "Generic (PLEG): container finished" podID="827251e7-f88d-4ed1-81bb-ce4960304aed" containerID="b686e611a6d354837289bab219e846e56d5bfc25c70979e7d7e2df9e7faf90a1" exitCode=0 Oct 30 17:11:31.655246 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:31.654006 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql" event={"ID":"827251e7-f88d-4ed1-81bb-ce4960304aed","Type":"ContainerDied","Data":"b686e611a6d354837289bab219e846e56d5bfc25c70979e7d7e2df9e7faf90a1"} Oct 30 17:11:31.657209 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:31.656374 4522 scope.go:122] "RemoveContainer" containerID="b686e611a6d354837289bab219e846e56d5bfc25c70979e7d7e2df9e7faf90a1" Oct 30 17:11:31.662270 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:31.660151 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:11:31.662270 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:31.660247 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:11:31.662270 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:31.660292 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:11:32.160192 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:32.160118 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa811185-1eb8-451b-afb7-9580783a30f2" path="/var/lib/kubelet/pods/aa811185-1eb8-451b-afb7-9580783a30f2/volumes" Oct 30 17:11:32.223915 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:32.160666 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd03a682-5ace-47b9-b539-9d0ddf3cbfdb" path="/var/lib/kubelet/pods/fd03a682-5ace-47b9-b539-9d0ddf3cbfdb/volumes" Oct 30 17:11:32.467093 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:32.467061 4522 patch_prober.go:29] interesting pod/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 30 17:11:32.467289 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:32.467133 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h" podUID="6ef91484-ab4c-4b90-9802-0ead2c544bf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 30 17:11:32.673897 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:32.673868 4522 generic.go:356] "Generic (PLEG): container finished" podID="4d683a0f-9de0-46dc-9772-d0915b0cb985" containerID="35d32d97fd6f7931d29ae91feb87af869106f6d3cf72391bf7f35b37dbd5e81a" exitCode=0 Oct 30 17:11:32.674113 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:32.673948 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m" event={"ID":"4d683a0f-9de0-46dc-9772-d0915b0cb985","Type":"ContainerDied","Data":"35d32d97fd6f7931d29ae91feb87af869106f6d3cf72391bf7f35b37dbd5e81a"} Oct 30 17:11:32.674424 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:32.674402 4522 scope.go:122] "RemoveContainer" containerID="35d32d97fd6f7931d29ae91feb87af869106f6d3cf72391bf7f35b37dbd5e81a" Oct 30 17:11:32.676119 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:32.676096 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql" event={"ID":"827251e7-f88d-4ed1-81bb-ce4960304aed","Type":"ContainerStarted","Data":"b5d6f8e901f3808466f8a618688cb7b2a63835c6ba874496af0c889333c51f2f"} Oct 30 17:11:32.676197 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:32.676162 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:11:32.676237 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:32.676224 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:11:32.676279 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:32.676258 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:11:32.686278 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:32.686252 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql" Oct 30 17:11:32.687103 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:32.687073 4522 patch_prober.go:29] interesting pod/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 30 17:11:32.687199 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:32.687125 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql" podUID="827251e7-f88d-4ed1-81bb-ce4960304aed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 30 17:11:33.467040 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:33.467007 4522 patch_prober.go:29] interesting pod/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 30 17:11:33.467488 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:33.467059 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h" podUID="6ef91484-ab4c-4b90-9802-0ead2c544bf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 30 17:11:33.687176 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:33.686976 4522 patch_prober.go:29] interesting pod/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 30 17:11:33.687176 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:33.687032 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql" podUID="827251e7-f88d-4ed1-81bb-ce4960304aed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 30 17:11:33.688527 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:33.688501 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m" event={"ID":"4d683a0f-9de0-46dc-9772-d0915b0cb985","Type":"ContainerStarted","Data":"d8434a2a67396bb270b71f833213bbe6b7fb4eacbbbd26781a03bce6f5eacbf7"} Oct 30 17:11:34.073472 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:34.073419 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m" Oct 30 17:11:34.073766 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:34.073742 4522 patch_prober.go:29] interesting pod/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 30 17:11:34.073849 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:34.073791 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m" podUID="4d683a0f-9de0-46dc-9772-d0915b0cb985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 30 17:11:34.472084 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:34.472041 4522 patch_prober.go:29] interesting pod/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 30 17:11:34.472640 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:34.472116 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h" podUID="6ef91484-ab4c-4b90-9802-0ead2c544bf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 30 17:11:34.709009 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:34.708928 4522 patch_prober.go:29] interesting pod/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 30 17:11:34.709320 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:34.709269 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql" podUID="827251e7-f88d-4ed1-81bb-ce4960304aed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 30 17:11:35.073951 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:35.073908 4522 patch_prober.go:29] interesting pod/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 30 17:11:35.074144 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:35.073972 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m" podUID="4d683a0f-9de0-46dc-9772-d0915b0cb985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 30 17:11:35.473331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:35.468517 4522 patch_prober.go:29] interesting pod/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 30 17:11:35.473331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:35.468580 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h" podUID="6ef91484-ab4c-4b90-9802-0ead2c544bf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 30 17:11:35.548330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:35.548173 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-7d1e5c-547b5cccb8-r8865" podUID="ee88e1ad-1e39-49b4-8a3e-70455749bbe6" containerName="payload-processing" containerID="cri-o://ef0bf74cb880993cfa2ba7f0854c7ffbf3807baede26a4208b4980c050f4e001" gracePeriod=25 Oct 30 17:11:35.584222 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:35.583694 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d1e5c-79c87484bc-6dx5v" podUID="d40e76c4-53d4-4ad0-8952-46a6bac79333" containerName="payload-pre-processing" containerID="cri-o://cd6d80ca8b79a32b4feaa6590c4957e302ef0709b0a25a16539f54025cc668e4" gracePeriod=25 Oct 30 17:11:35.687648 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:35.687414 4522 patch_prober.go:29] interesting pod/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 30 17:11:35.687648 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:35.687464 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql" podUID="827251e7-f88d-4ed1-81bb-ce4960304aed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 30 17:11:35.790712 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:35.790223 4522 generic.go:356] "Generic (PLEG): container finished" podID="ee88e1ad-1e39-49b4-8a3e-70455749bbe6" containerID="ef0bf74cb880993cfa2ba7f0854c7ffbf3807baede26a4208b4980c050f4e001" exitCode=0 Oct 30 17:11:35.790712 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:35.790335 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7d1e5c-547b5cccb8-r8865" event={"ID":"ee88e1ad-1e39-49b4-8a3e-70455749bbe6","Type":"ContainerDied","Data":"ef0bf74cb880993cfa2ba7f0854c7ffbf3807baede26a4208b4980c050f4e001"} Oct 30 17:11:35.803430 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:35.802989 4522 generic.go:356] "Generic (PLEG): container finished" podID="d40e76c4-53d4-4ad0-8952-46a6bac79333" containerID="cd6d80ca8b79a32b4feaa6590c4957e302ef0709b0a25a16539f54025cc668e4" exitCode=0 Oct 30 17:11:35.803430 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:35.803344 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d1e5c-79c87484bc-6dx5v" event={"ID":"d40e76c4-53d4-4ad0-8952-46a6bac79333","Type":"ContainerDied","Data":"cd6d80ca8b79a32b4feaa6590c4957e302ef0709b0a25a16539f54025cc668e4"} Oct 30 17:11:36.085637 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:36.085369 4522 patch_prober.go:29] interesting pod/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 30 17:11:36.085637 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:36.085433 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m" podUID="4d683a0f-9de0-46dc-9772-d0915b0cb985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 30 17:11:36.476917 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:36.471717 4522 patch_prober.go:29] interesting pod/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 30 17:11:36.476917 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:36.471776 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h" podUID="6ef91484-ab4c-4b90-9802-0ead2c544bf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 30 17:11:36.481912 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:36.481885 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7d1e5c-547b5cccb8-r8865" Oct 30 17:11:36.547812 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:36.547776 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee88e1ad-1e39-49b4-8a3e-70455749bbe6-kube-api-access-2x72w\" (UniqueName: \"kubernetes.io/projected/ee88e1ad-1e39-49b4-8a3e-70455749bbe6-kube-api-access-2x72w\") pod \"ee88e1ad-1e39-49b4-8a3e-70455749bbe6\" (UID: \"ee88e1ad-1e39-49b4-8a3e-70455749bbe6\") " Oct 30 17:11:36.548154 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:36.548134 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee88e1ad-1e39-49b4-8a3e-70455749bbe6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee88e1ad-1e39-49b4-8a3e-70455749bbe6-plugins-config-volume\") pod \"ee88e1ad-1e39-49b4-8a3e-70455749bbe6\" (UID: \"ee88e1ad-1e39-49b4-8a3e-70455749bbe6\") " Oct 30 17:11:36.565427 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:36.565386 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee88e1ad-1e39-49b4-8a3e-70455749bbe6-plugins-config-volume" pod "ee88e1ad-1e39-49b4-8a3e-70455749bbe6" (UID: "ee88e1ad-1e39-49b4-8a3e-70455749bbe6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:11:36.590520 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:36.590465 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee88e1ad-1e39-49b4-8a3e-70455749bbe6-kube-api-access-2x72w" pod "ee88e1ad-1e39-49b4-8a3e-70455749bbe6" (UID: "ee88e1ad-1e39-49b4-8a3e-70455749bbe6"). InnerVolumeSpecName "kube-api-access-2x72w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:11:36.661639 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:36.661549 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2x72w\" (UniqueName: \"kubernetes.io/projected/ee88e1ad-1e39-49b4-8a3e-70455749bbe6-kube-api-access-2x72w\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:36.661853 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:36.661836 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee88e1ad-1e39-49b4-8a3e-70455749bbe6-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:36.686213 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:36.686182 4522 patch_prober.go:29] interesting pod/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 30 17:11:36.686396 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:36.686237 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql" podUID="827251e7-f88d-4ed1-81bb-ce4960304aed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 30 17:11:36.760342 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:36.758536 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d1e5c-79c87484bc-6dx5v" Oct 30 17:11:36.866252 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:36.866222 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d40e76c4-53d4-4ad0-8952-46a6bac79333-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d40e76c4-53d4-4ad0-8952-46a6bac79333-plugins-config-volume\") pod \"d40e76c4-53d4-4ad0-8952-46a6bac79333\" (UID: \"d40e76c4-53d4-4ad0-8952-46a6bac79333\") " Oct 30 17:11:36.878725 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:36.871367 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d40e76c4-53d4-4ad0-8952-46a6bac79333-plugins-config-volume" pod "d40e76c4-53d4-4ad0-8952-46a6bac79333" (UID: "d40e76c4-53d4-4ad0-8952-46a6bac79333"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:11:36.878725 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:36.876210 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7d1e5c-547b5cccb8-r8865" event={"ID":"ee88e1ad-1e39-49b4-8a3e-70455749bbe6","Type":"ContainerDied","Data":"951d20ca5c1bed7e6f3e0e666bdf10df466157b358f13ea9dd573b392b3fbcd6"} Oct 30 17:11:36.878725 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:36.876330 4522 scope.go:122] "RemoveContainer" containerID="ef0bf74cb880993cfa2ba7f0854c7ffbf3807baede26a4208b4980c050f4e001" Oct 30 17:11:36.878725 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:36.876584 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7d1e5c-547b5cccb8-r8865" Oct 30 17:11:36.879543 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:36.879515 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d40e76c4-53d4-4ad0-8952-46a6bac79333-kube-api-access-d7x8f\" (UniqueName: \"kubernetes.io/projected/d40e76c4-53d4-4ad0-8952-46a6bac79333-kube-api-access-d7x8f\") pod \"d40e76c4-53d4-4ad0-8952-46a6bac79333\" (UID: \"d40e76c4-53d4-4ad0-8952-46a6bac79333\") " Oct 30 17:11:36.880873 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:36.880734 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d40e76c4-53d4-4ad0-8952-46a6bac79333-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:36.961865 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:36.961398 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d1e5c-79c87484bc-6dx5v" event={"ID":"d40e76c4-53d4-4ad0-8952-46a6bac79333","Type":"ContainerDied","Data":"6f662a2b61f768afae5b9811f248a978bb29e2d5f0d046dccc8cd27c1387a5c1"} Oct 30 17:11:36.961865 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:36.961452 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d1e5c-79c87484bc-6dx5v" Oct 30 17:11:36.961865 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:36.961470 4522 scope.go:122] "RemoveContainer" containerID="cd6d80ca8b79a32b4feaa6590c4957e302ef0709b0a25a16539f54025cc668e4" Oct 30 17:11:36.962522 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:36.962495 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d40e76c4-53d4-4ad0-8952-46a6bac79333-kube-api-access-d7x8f" pod "d40e76c4-53d4-4ad0-8952-46a6bac79333" (UID: "d40e76c4-53d4-4ad0-8952-46a6bac79333"). InnerVolumeSpecName "kube-api-access-d7x8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:11:36.983264 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:36.983232 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7x8f\" (UniqueName: \"kubernetes.io/projected/d40e76c4-53d4-4ad0-8952-46a6bac79333-kube-api-access-d7x8f\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:37.058384 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:37.058090 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7d1e5c-547b5cccb8-r8865"] Oct 30 17:11:37.063495 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:37.063469 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7d1e5c-547b5cccb8-r8865"] Oct 30 17:11:37.074847 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:37.074817 4522 patch_prober.go:29] interesting pod/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 30 17:11:37.074941 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:37.074876 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m" podUID="4d683a0f-9de0-46dc-9772-d0915b0cb985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 30 17:11:37.352413 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:37.352367 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7d1e5c-79c87484bc-6dx5v"] Oct 30 17:11:37.357852 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:37.357799 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7d1e5c-79c87484bc-6dx5v"] Oct 30 17:11:37.512710 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:37.512673 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h" Oct 30 17:11:37.514961 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:37.514938 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h" Oct 30 17:11:37.533389 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:37.526569 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h" Oct 30 17:11:37.686983 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:37.686778 4522 patch_prober.go:29] interesting pod/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 30 17:11:37.686983 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:37.686831 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql" podUID="827251e7-f88d-4ed1-81bb-ce4960304aed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 30 17:11:37.911388 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:37.903409 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql" Oct 30 17:11:38.074441 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:38.074407 4522 patch_prober.go:29] interesting pod/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 30 17:11:38.074667 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:38.074646 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m" podUID="4d683a0f-9de0-46dc-9772-d0915b0cb985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 30 17:11:38.154892 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:38.154862 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d40e76c4-53d4-4ad0-8952-46a6bac79333" path="/var/lib/kubelet/pods/d40e76c4-53d4-4ad0-8952-46a6bac79333/volumes" Oct 30 17:11:38.155619 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:38.155601 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee88e1ad-1e39-49b4-8a3e-70455749bbe6" path="/var/lib/kubelet/pods/ee88e1ad-1e39-49b4-8a3e-70455749bbe6/volumes" Oct 30 17:11:38.694419 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:38.691600 4522 patch_prober.go:29] interesting pod/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 30 17:11:38.694419 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:38.691657 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql" podUID="827251e7-f88d-4ed1-81bb-ce4960304aed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 30 17:11:38.859337 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:38.857935 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw"] Oct 30 17:11:38.859337 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:38.858230 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw" podUID="0e123c50-767e-478c-a7e5-0d2476ccbc3b" containerName="istio-proxy" containerID="cri-o://5a55745e4b054f0015ed3df749d3c576744a49e28d2dc31ba121e8a6bfa4674c" gracePeriod=30 Oct 30 17:11:39.074242 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:39.074207 4522 patch_prober.go:29] interesting pod/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 30 17:11:39.074468 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:39.074261 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m" podUID="4d683a0f-9de0-46dc-9772-d0915b0cb985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 30 17:11:39.691146 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:39.691111 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql" Oct 30 17:11:39.692334 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:39.692238 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql" Oct 30 17:11:40.078079 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:40.078042 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m" Oct 30 17:11:40.078564 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:40.078549 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m" Oct 30 17:11:40.079157 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:40.079140 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m" Oct 30 17:11:42.043448 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:42.043417 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11c7f0ab-m929r"] Oct 30 17:11:42.056398 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:42.056371 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11c7f0ab-m929r"] Oct 30 17:11:42.149574 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:42.149535 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db670098-c2e3-49bd-a107-8a9e72610845" path="/var/lib/kubelet/pods/db670098-c2e3-49bd-a107-8a9e72610845/volumes" Oct 30 17:11:42.932597 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:42.932564 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-409387-vhhd7"] Oct 30 17:11:42.933296 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:42.933281 4522 cpu_manager.go:414] "containerMap: removing container" podUID="c99fb3fd-1362-414e-82bd-88f20c055166" containerName="istio-proxy" Oct 30 17:11:42.933382 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:42.933297 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c99fb3fd-1362-414e-82bd-88f20c055166" containerName="istio-proxy" Oct 30 17:11:42.933382 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:42.933327 4522 cpu_manager.go:414] "containerMap: removing container" podUID="aa811185-1eb8-451b-afb7-9580783a30f2" containerName="istio-proxy" Oct 30 17:11:42.933382 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:42.933332 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa811185-1eb8-451b-afb7-9580783a30f2" containerName="istio-proxy" Oct 30 17:11:42.933382 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:42.933339 4522 cpu_manager.go:414] "containerMap: removing container" podUID="ee88e1ad-1e39-49b4-8a3e-70455749bbe6" containerName="payload-processing" Oct 30 17:11:42.933382 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:42.933344 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee88e1ad-1e39-49b4-8a3e-70455749bbe6" containerName="payload-processing" Oct 30 17:11:42.933382 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:42.933359 4522 cpu_manager.go:414] "containerMap: removing container" podUID="79ea6704-625a-4af9-83c6-7f5214a6b8b4" containerName="istio-proxy" Oct 30 17:11:42.933382 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:42.933364 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79ea6704-625a-4af9-83c6-7f5214a6b8b4" containerName="istio-proxy" Oct 30 17:11:42.933595 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:42.933400 4522 cpu_manager.go:414] "containerMap: removing container" podUID="d40e76c4-53d4-4ad0-8952-46a6bac79333" containerName="payload-pre-processing" Oct 30 17:11:42.933595 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:42.933405 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d40e76c4-53d4-4ad0-8952-46a6bac79333" containerName="payload-pre-processing" Oct 30 17:11:42.933595 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:42.933510 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa811185-1eb8-451b-afb7-9580783a30f2" containerName="istio-proxy" Oct 30 17:11:42.933595 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:42.933520 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee88e1ad-1e39-49b4-8a3e-70455749bbe6" containerName="payload-processing" Oct 30 17:11:42.933595 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:42.933527 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="79ea6704-625a-4af9-83c6-7f5214a6b8b4" containerName="istio-proxy" Oct 30 17:11:42.933595 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:42.933540 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="d40e76c4-53d4-4ad0-8952-46a6bac79333" containerName="payload-pre-processing" Oct 30 17:11:42.933595 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:42.933547 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="c99fb3fd-1362-414e-82bd-88f20c055166" containerName="istio-proxy" Oct 30 17:11:42.937281 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:42.937261 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-409387-vhhd7" Oct 30 17:11:42.952714 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:11:42.952686 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:11:42.953155 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:42.953129 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:11:42.957240 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:42.957214 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-409387-vhhd7"] Oct 30 17:11:42.965080 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:42.965056 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:11:42.996689 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:42.996657 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0e64808-7b16-4675-a2d6-c6109477d027-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-409387-vhhd7\" (UID: \"a0e64808-7b16-4675-a2d6-c6109477d027\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-409387-vhhd7" Oct 30 17:11:42.997024 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:42.997002 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0e64808-7b16-4675-a2d6-c6109477d027-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-409387-vhhd7\" (UID: \"a0e64808-7b16-4675-a2d6-c6109477d027\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-409387-vhhd7" Oct 30 17:11:42.997137 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:42.997119 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd8mh\" (UniqueName: \"kubernetes.io/projected/a0e64808-7b16-4675-a2d6-c6109477d027-kube-api-access-dd8mh\") pod \"maas-api-revoke-keys-e2e-api-a-409387-vhhd7\" (UID: \"a0e64808-7b16-4675-a2d6-c6109477d027\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-409387-vhhd7" Oct 30 17:11:43.098915 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:43.098877 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0e64808-7b16-4675-a2d6-c6109477d027-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-409387-vhhd7\" (UID: \"a0e64808-7b16-4675-a2d6-c6109477d027\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-409387-vhhd7" Oct 30 17:11:43.099455 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:43.098990 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd8mh\" (UniqueName: \"kubernetes.io/projected/a0e64808-7b16-4675-a2d6-c6109477d027-kube-api-access-dd8mh\") pod \"maas-api-revoke-keys-e2e-api-a-409387-vhhd7\" (UID: \"a0e64808-7b16-4675-a2d6-c6109477d027\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-409387-vhhd7" Oct 30 17:11:43.099455 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:43.099096 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0e64808-7b16-4675-a2d6-c6109477d027-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-409387-vhhd7\" (UID: \"a0e64808-7b16-4675-a2d6-c6109477d027\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-409387-vhhd7" Oct 30 17:11:43.099838 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:43.099813 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0e64808-7b16-4675-a2d6-c6109477d027-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-409387-vhhd7\" (UID: \"a0e64808-7b16-4675-a2d6-c6109477d027\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-409387-vhhd7" Oct 30 17:11:43.101990 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:43.101965 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0e64808-7b16-4675-a2d6-c6109477d027-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-409387-vhhd7\" (UID: \"a0e64808-7b16-4675-a2d6-c6109477d027\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-409387-vhhd7" Oct 30 17:11:43.125107 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:43.125068 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd8mh\" (UniqueName: \"kubernetes.io/projected/a0e64808-7b16-4675-a2d6-c6109477d027-kube-api-access-dd8mh\") pod \"maas-api-revoke-keys-e2e-api-a-409387-vhhd7\" (UID: \"a0e64808-7b16-4675-a2d6-c6109477d027\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-409387-vhhd7" Oct 30 17:11:43.253026 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:43.252997 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-409387-vhhd7" Oct 30 17:11:43.575489 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:11:43.575453 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0e64808_7b16_4675_a2d6_c6109477d027.slice/crio-5b3edaa139db2bbb1c8c6d7b38c69db29d2cdd1d2a9430f8551c3361eeba37ee WatchSource:0}: Error finding container 5b3edaa139db2bbb1c8c6d7b38c69db29d2cdd1d2a9430f8551c3361eeba37ee: Status 404 returned error can't find the container with id 5b3edaa139db2bbb1c8c6d7b38c69db29d2cdd1d2a9430f8551c3361eeba37ee Oct 30 17:11:43.576855 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:43.576834 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-409387-vhhd7"] Oct 30 17:11:44.063936 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:44.063908 4522 generic.go:356] "Generic (PLEG): container finished" podID="0e123c50-767e-478c-a7e5-0d2476ccbc3b" containerID="5a55745e4b054f0015ed3df749d3c576744a49e28d2dc31ba121e8a6bfa4674c" exitCode=0 Oct 30 17:11:44.064085 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:44.063977 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw" event={"ID":"0e123c50-767e-478c-a7e5-0d2476ccbc3b","Type":"ContainerDied","Data":"5a55745e4b054f0015ed3df749d3c576744a49e28d2dc31ba121e8a6bfa4674c"} Oct 30 17:11:44.065589 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:44.065560 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-409387-vhhd7" event={"ID":"a0e64808-7b16-4675-a2d6-c6109477d027","Type":"ContainerStarted","Data":"4f6d281faa42f5af7a398425b5ef284424679fd73c868a7aedc368eac32a3a13"} Oct 30 17:11:44.065719 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:44.065615 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-409387-vhhd7" event={"ID":"a0e64808-7b16-4675-a2d6-c6109477d027","Type":"ContainerStarted","Data":"5b3edaa139db2bbb1c8c6d7b38c69db29d2cdd1d2a9430f8551c3361eeba37ee"} Oct 30 17:11:44.113910 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:44.113846 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-409387-vhhd7" podStartSLOduration=2.113828305 podStartE2EDuration="2.113828305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:11:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:11:44.108734894 +0000 UTC m=+1478.109489802" watchObservedRunningTime="2026-10-30 17:11:44.113828305 +0000 UTC m=+1478.114583205" Oct 30 17:11:44.672020 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:44.671991 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw" Oct 30 17:11:44.732562 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:44.732524 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e123c50-767e-478c-a7e5-0d2476ccbc3b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e123c50-767e-478c-a7e5-0d2476ccbc3b-istio-data\") pod \"0e123c50-767e-478c-a7e5-0d2476ccbc3b\" (UID: \"0e123c50-767e-478c-a7e5-0d2476ccbc3b\") " Oct 30 17:11:44.732744 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:44.732686 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e123c50-767e-478c-a7e5-0d2476ccbc3b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e123c50-767e-478c-a7e5-0d2476ccbc3b-istio-podinfo\") pod \"0e123c50-767e-478c-a7e5-0d2476ccbc3b\" (UID: \"0e123c50-767e-478c-a7e5-0d2476ccbc3b\") " Oct 30 17:11:44.732866 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:44.732803 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e123c50-767e-478c-a7e5-0d2476ccbc3b-istio-data" pod "0e123c50-767e-478c-a7e5-0d2476ccbc3b" (UID: "0e123c50-767e-478c-a7e5-0d2476ccbc3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:44.732866 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:44.732844 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e123c50-767e-478c-a7e5-0d2476ccbc3b-istio-token\" (UniqueName: \"kubernetes.io/projected/0e123c50-767e-478c-a7e5-0d2476ccbc3b-istio-token\") pod \"0e123c50-767e-478c-a7e5-0d2476ccbc3b\" (UID: \"0e123c50-767e-478c-a7e5-0d2476ccbc3b\") " Oct 30 17:11:44.732954 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:44.732894 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e123c50-767e-478c-a7e5-0d2476ccbc3b-kube-api-access-bk7b4\" (UniqueName: \"kubernetes.io/projected/0e123c50-767e-478c-a7e5-0d2476ccbc3b-kube-api-access-bk7b4\") pod \"0e123c50-767e-478c-a7e5-0d2476ccbc3b\" (UID: \"0e123c50-767e-478c-a7e5-0d2476ccbc3b\") " Oct 30 17:11:44.732954 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:44.732922 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e123c50-767e-478c-a7e5-0d2476ccbc3b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e123c50-767e-478c-a7e5-0d2476ccbc3b-istiod-ca-cert\") pod \"0e123c50-767e-478c-a7e5-0d2476ccbc3b\" (UID: \"0e123c50-767e-478c-a7e5-0d2476ccbc3b\") " Oct 30 17:11:44.733033 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:44.732965 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e123c50-767e-478c-a7e5-0d2476ccbc3b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e123c50-767e-478c-a7e5-0d2476ccbc3b-istio-envoy\") pod \"0e123c50-767e-478c-a7e5-0d2476ccbc3b\" (UID: \"0e123c50-767e-478c-a7e5-0d2476ccbc3b\") " Oct 30 17:11:44.733033 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:44.732996 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e123c50-767e-478c-a7e5-0d2476ccbc3b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e123c50-767e-478c-a7e5-0d2476ccbc3b-credential-socket\") pod \"0e123c50-767e-478c-a7e5-0d2476ccbc3b\" (UID: \"0e123c50-767e-478c-a7e5-0d2476ccbc3b\") " Oct 30 17:11:44.733110 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:44.733078 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e123c50-767e-478c-a7e5-0d2476ccbc3b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e123c50-767e-478c-a7e5-0d2476ccbc3b-workload-socket\") pod \"0e123c50-767e-478c-a7e5-0d2476ccbc3b\" (UID: \"0e123c50-767e-478c-a7e5-0d2476ccbc3b\") " Oct 30 17:11:44.733148 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:44.733127 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e123c50-767e-478c-a7e5-0d2476ccbc3b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e123c50-767e-478c-a7e5-0d2476ccbc3b-workload-certs\") pod \"0e123c50-767e-478c-a7e5-0d2476ccbc3b\" (UID: \"0e123c50-767e-478c-a7e5-0d2476ccbc3b\") " Oct 30 17:11:44.733576 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:44.733484 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e123c50-767e-478c-a7e5-0d2476ccbc3b-workload-socket" pod "0e123c50-767e-478c-a7e5-0d2476ccbc3b" (UID: "0e123c50-767e-478c-a7e5-0d2476ccbc3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:44.733576 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:44.733522 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e123c50-767e-478c-a7e5-0d2476ccbc3b-credential-socket" pod "0e123c50-767e-478c-a7e5-0d2476ccbc3b" (UID: "0e123c50-767e-478c-a7e5-0d2476ccbc3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:44.733759 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:44.733628 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e123c50-767e-478c-a7e5-0d2476ccbc3b-istiod-ca-cert" pod "0e123c50-767e-478c-a7e5-0d2476ccbc3b" (UID: "0e123c50-767e-478c-a7e5-0d2476ccbc3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:11:44.733759 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:44.733721 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e123c50-767e-478c-a7e5-0d2476ccbc3b-workload-certs" pod "0e123c50-767e-478c-a7e5-0d2476ccbc3b" (UID: "0e123c50-767e-478c-a7e5-0d2476ccbc3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:44.733932 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:44.733910 4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e123c50-767e-478c-a7e5-0d2476ccbc3b-istiod-ca-cert\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:44.733997 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:44.733939 4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e123c50-767e-478c-a7e5-0d2476ccbc3b-credential-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:44.733997 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:44.733954 4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e123c50-767e-478c-a7e5-0d2476ccbc3b-workload-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:44.733997 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:44.733970 4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e123c50-767e-478c-a7e5-0d2476ccbc3b-workload-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:44.733997 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:44.733983 4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e123c50-767e-478c-a7e5-0d2476ccbc3b-istio-data\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:44.735640 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:44.735614 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e123c50-767e-478c-a7e5-0d2476ccbc3b-kube-api-access-bk7b4" pod "0e123c50-767e-478c-a7e5-0d2476ccbc3b" (UID: "0e123c50-767e-478c-a7e5-0d2476ccbc3b"). InnerVolumeSpecName "kube-api-access-bk7b4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:11:44.735899 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:44.735881 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e123c50-767e-478c-a7e5-0d2476ccbc3b-istio-token" pod "0e123c50-767e-478c-a7e5-0d2476ccbc3b" (UID: "0e123c50-767e-478c-a7e5-0d2476ccbc3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:11:44.735983 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:44.735925 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e123c50-767e-478c-a7e5-0d2476ccbc3b-istio-envoy" pod "0e123c50-767e-478c-a7e5-0d2476ccbc3b" (UID: "0e123c50-767e-478c-a7e5-0d2476ccbc3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:44.735983 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:44.735956 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e123c50-767e-478c-a7e5-0d2476ccbc3b-istio-podinfo" pod "0e123c50-767e-478c-a7e5-0d2476ccbc3b" (UID: "0e123c50-767e-478c-a7e5-0d2476ccbc3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:11:44.835595 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:44.835561 4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e123c50-767e-478c-a7e5-0d2476ccbc3b-istio-token\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:44.835595 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:44.835590 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bk7b4\" (UniqueName: \"kubernetes.io/projected/0e123c50-767e-478c-a7e5-0d2476ccbc3b-kube-api-access-bk7b4\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:44.835595 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:44.835600 4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e123c50-767e-478c-a7e5-0d2476ccbc3b-istio-envoy\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:44.835834 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:44.835610 4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e123c50-767e-478c-a7e5-0d2476ccbc3b-istio-podinfo\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:45.074776 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:45.074748 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw" Oct 30 17:11:45.074982 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:45.074755 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw" event={"ID":"0e123c50-767e-478c-a7e5-0d2476ccbc3b","Type":"ContainerDied","Data":"86b8c2afe64a24c7e9a3b0c6bf524a8241f24195f92936d8e8f436fa2a503d19"} Oct 30 17:11:45.074982 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:45.074880 4522 scope.go:122] "RemoveContainer" containerID="5a55745e4b054f0015ed3df749d3c576744a49e28d2dc31ba121e8a6bfa4674c" Oct 30 17:11:45.076679 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:45.076542 4522 generic.go:356] "Generic (PLEG): container finished" podID="a0e64808-7b16-4675-a2d6-c6109477d027" containerID="4f6d281faa42f5af7a398425b5ef284424679fd73c868a7aedc368eac32a3a13" exitCode=0 Oct 30 17:11:45.076679 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:45.076616 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-409387-vhhd7" event={"ID":"a0e64808-7b16-4675-a2d6-c6109477d027","Type":"ContainerDied","Data":"4f6d281faa42f5af7a398425b5ef284424679fd73c868a7aedc368eac32a3a13"} Oct 30 17:11:45.187803 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:45.187765 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw"] Oct 30 17:11:45.191943 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:45.191915 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-wgdqw"] Oct 30 17:11:46.147399 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:46.147364 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e123c50-767e-478c-a7e5-0d2476ccbc3b" path="/var/lib/kubelet/pods/0e123c50-767e-478c-a7e5-0d2476ccbc3b/volumes" Oct 30 17:11:46.252953 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:46.252925 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-409387-vhhd7" Oct 30 17:11:46.359756 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:46.359669 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0e64808-7b16-4675-a2d6-c6109477d027-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0e64808-7b16-4675-a2d6-c6109477d027-maas-api-xks-ca\") pod \"a0e64808-7b16-4675-a2d6-c6109477d027\" (UID: \"a0e64808-7b16-4675-a2d6-c6109477d027\") " Oct 30 17:11:46.359756 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:46.359742 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0e64808-7b16-4675-a2d6-c6109477d027-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0e64808-7b16-4675-a2d6-c6109477d027-maas-api-openshift-ca\") pod \"a0e64808-7b16-4675-a2d6-c6109477d027\" (UID: \"a0e64808-7b16-4675-a2d6-c6109477d027\") " Oct 30 17:11:46.359973 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:46.359770 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0e64808-7b16-4675-a2d6-c6109477d027-kube-api-access-dd8mh\" (UniqueName: \"kubernetes.io/projected/a0e64808-7b16-4675-a2d6-c6109477d027-kube-api-access-dd8mh\") pod \"a0e64808-7b16-4675-a2d6-c6109477d027\" (UID: \"a0e64808-7b16-4675-a2d6-c6109477d027\") " Oct 30 17:11:46.360176 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:46.360149 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0e64808-7b16-4675-a2d6-c6109477d027-maas-api-openshift-ca" pod "a0e64808-7b16-4675-a2d6-c6109477d027" (UID: "a0e64808-7b16-4675-a2d6-c6109477d027"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:11:46.360559 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:46.360543 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0e64808-7b16-4675-a2d6-c6109477d027-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:46.362434 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:46.362408 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0e64808-7b16-4675-a2d6-c6109477d027-maas-api-xks-ca" pod "a0e64808-7b16-4675-a2d6-c6109477d027" (UID: "a0e64808-7b16-4675-a2d6-c6109477d027"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:11:46.362521 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:46.362460 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0e64808-7b16-4675-a2d6-c6109477d027-kube-api-access-dd8mh" pod "a0e64808-7b16-4675-a2d6-c6109477d027" (UID: "a0e64808-7b16-4675-a2d6-c6109477d027"). InnerVolumeSpecName "kube-api-access-dd8mh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:11:46.462536 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:46.462500 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0e64808-7b16-4675-a2d6-c6109477d027-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:46.462536 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:46.462531 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dd8mh\" (UniqueName: \"kubernetes.io/projected/a0e64808-7b16-4675-a2d6-c6109477d027-kube-api-access-dd8mh\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:47.096864 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:47.096825 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-409387-vhhd7" event={"ID":"a0e64808-7b16-4675-a2d6-c6109477d027","Type":"ContainerDied","Data":"5b3edaa139db2bbb1c8c6d7b38c69db29d2cdd1d2a9430f8551c3361eeba37ee"} Oct 30 17:11:47.097025 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:47.096871 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b3edaa139db2bbb1c8c6d7b38c69db29d2cdd1d2a9430f8551c3361eeba37ee" Oct 30 17:11:47.097025 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:47.096840 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-409387-vhhd7" Oct 30 17:11:48.698292 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:48.698259 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-db8417-74d9d84dc9-ddws5"] Oct 30 17:11:48.699067 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:48.699050 4522 cpu_manager.go:414] "containerMap: removing container" podUID="a0e64808-7b16-4675-a2d6-c6109477d027" containerName="revoke-keys" Oct 30 17:11:48.699113 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:48.699068 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0e64808-7b16-4675-a2d6-c6109477d027" containerName="revoke-keys" Oct 30 17:11:48.699113 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:48.699102 4522 cpu_manager.go:414] "containerMap: removing container" podUID="0e123c50-767e-478c-a7e5-0d2476ccbc3b" containerName="istio-proxy" Oct 30 17:11:48.699113 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:48.699108 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e123c50-767e-478c-a7e5-0d2476ccbc3b" containerName="istio-proxy" Oct 30 17:11:48.699225 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:48.699216 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0e64808-7b16-4675-a2d6-c6109477d027" containerName="revoke-keys" Oct 30 17:11:48.699257 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:48.699233 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e123c50-767e-478c-a7e5-0d2476ccbc3b" containerName="istio-proxy" Oct 30 17:11:48.703747 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:48.703723 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-db8417-74d9d84dc9-ddws5" Oct 30 17:11:48.714743 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:48.714711 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-db8417-74d9d84dc9-ddws5"] Oct 30 17:11:48.723041 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:48.723013 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-db8417\"" Oct 30 17:11:48.723212 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:48.723133 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-db8417-dockercfg-5jlrr\"" Oct 30 17:11:48.748169 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:48.748122 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-db8417-bfdb94d4d-frgtn"] Oct 30 17:11:48.836606 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:48.834992 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbtjr\" (UniqueName: \"kubernetes.io/projected/e6e99dad-79f2-4b63-87f9-1c761080cada-kube-api-access-mbtjr\") pod \"payload-pre-processing-e2e-shared-b-w3-db8417-74d9d84dc9-ddws5\" (UID: \"e6e99dad-79f2-4b63-87f9-1c761080cada\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-db8417-74d9d84dc9-ddws5" Oct 30 17:11:48.836606 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:48.835259 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6e99dad-79f2-4b63-87f9-1c761080cada-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-db8417-74d9d84dc9-ddws5\" (UID: \"e6e99dad-79f2-4b63-87f9-1c761080cada\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-db8417-74d9d84dc9-ddws5" Oct 30 17:11:48.862688 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:48.862656 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-db8417-bfdb94d4d-frgtn" Oct 30 17:11:48.995625 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:48.943536 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6e99dad-79f2-4b63-87f9-1c761080cada-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-db8417-74d9d84dc9-ddws5\" (UID: \"e6e99dad-79f2-4b63-87f9-1c761080cada\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-db8417-74d9d84dc9-ddws5" Oct 30 17:11:48.995625 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:48.943792 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbtjr\" (UniqueName: \"kubernetes.io/projected/e6e99dad-79f2-4b63-87f9-1c761080cada-kube-api-access-mbtjr\") pod \"payload-pre-processing-e2e-shared-b-w3-db8417-74d9d84dc9-ddws5\" (UID: \"e6e99dad-79f2-4b63-87f9-1c761080cada\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-db8417-74d9d84dc9-ddws5" Oct 30 17:11:48.995625 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:48.953811 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6e99dad-79f2-4b63-87f9-1c761080cada-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-db8417-74d9d84dc9-ddws5\" (UID: \"e6e99dad-79f2-4b63-87f9-1c761080cada\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-db8417-74d9d84dc9-ddws5" Oct 30 17:11:49.003956 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:49.003022 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbtjr\" (UniqueName: \"kubernetes.io/projected/e6e99dad-79f2-4b63-87f9-1c761080cada-kube-api-access-mbtjr\") pod \"payload-pre-processing-e2e-shared-b-w3-db8417-74d9d84dc9-ddws5\" (UID: \"e6e99dad-79f2-4b63-87f9-1c761080cada\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-db8417-74d9d84dc9-ddws5" Oct 30 17:11:49.009868 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:49.009789 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-db8417-bfdb94d4d-frgtn"] Oct 30 17:11:49.028955 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:49.028689 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-db8417-74d9d84dc9-ddws5" Oct 30 17:11:49.045417 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:49.045388 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47fa8756-130e-48fc-bcf9-008ab31f95cf-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-db8417-bfdb94d4d-frgtn\" (UID: \"47fa8756-130e-48fc-bcf9-008ab31f95cf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-db8417-bfdb94d4d-frgtn" Oct 30 17:11:49.045576 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:49.045437 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt5qx\" (UniqueName: \"kubernetes.io/projected/47fa8756-130e-48fc-bcf9-008ab31f95cf-kube-api-access-mt5qx\") pod \"payload-processing-e2e-shared-b-w3-db8417-bfdb94d4d-frgtn\" (UID: \"47fa8756-130e-48fc-bcf9-008ab31f95cf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-db8417-bfdb94d4d-frgtn" Oct 30 17:11:49.147798 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:49.147759 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47fa8756-130e-48fc-bcf9-008ab31f95cf-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-db8417-bfdb94d4d-frgtn\" (UID: \"47fa8756-130e-48fc-bcf9-008ab31f95cf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-db8417-bfdb94d4d-frgtn" Oct 30 17:11:49.148006 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:49.147829 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt5qx\" (UniqueName: \"kubernetes.io/projected/47fa8756-130e-48fc-bcf9-008ab31f95cf-kube-api-access-mt5qx\") pod \"payload-processing-e2e-shared-b-w3-db8417-bfdb94d4d-frgtn\" (UID: \"47fa8756-130e-48fc-bcf9-008ab31f95cf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-db8417-bfdb94d4d-frgtn" Oct 30 17:11:49.149497 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:49.149469 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47fa8756-130e-48fc-bcf9-008ab31f95cf-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-db8417-bfdb94d4d-frgtn\" (UID: \"47fa8756-130e-48fc-bcf9-008ab31f95cf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-db8417-bfdb94d4d-frgtn" Oct 30 17:11:49.181266 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:49.181136 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt5qx\" (UniqueName: \"kubernetes.io/projected/47fa8756-130e-48fc-bcf9-008ab31f95cf-kube-api-access-mt5qx\") pod \"payload-processing-e2e-shared-b-w3-db8417-bfdb94d4d-frgtn\" (UID: \"47fa8756-130e-48fc-bcf9-008ab31f95cf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-db8417-bfdb94d4d-frgtn" Oct 30 17:11:49.332439 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:49.332362 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-db8417-bfdb94d4d-frgtn" Oct 30 17:11:49.356341 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:11:49.356296 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6e99dad_79f2_4b63_87f9_1c761080cada.slice/crio-f108427ed3d5d904c10fe00da1547b90b263beb2c54e3dc45b8a969e86bc7ff7 WatchSource:0}: Error finding container f108427ed3d5d904c10fe00da1547b90b263beb2c54e3dc45b8a969e86bc7ff7: Status 404 returned error can't find the container with id f108427ed3d5d904c10fe00da1547b90b263beb2c54e3dc45b8a969e86bc7ff7 Oct 30 17:11:49.360566 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:49.360539 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-db8417-74d9d84dc9-ddws5"] Oct 30 17:11:49.645890 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:11:49.645859 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47fa8756_130e_48fc_bcf9_008ab31f95cf.slice/crio-ef46a8683eb286b22516ae5662928b65954917ca4eca9c1f6c97795b2593ddef WatchSource:0}: Error finding container ef46a8683eb286b22516ae5662928b65954917ca4eca9c1f6c97795b2593ddef: Status 404 returned error can't find the container with id ef46a8683eb286b22516ae5662928b65954917ca4eca9c1f6c97795b2593ddef Oct 30 17:11:49.648627 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:49.648603 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-db8417-bfdb94d4d-frgtn"] Oct 30 17:11:50.146231 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:50.146199 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-db8417-74d9d84dc9-ddws5" Oct 30 17:11:50.146650 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:50.146251 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-db8417-bfdb94d4d-frgtn" Oct 30 17:11:50.146650 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:50.146272 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-db8417-74d9d84dc9-ddws5" event={"ID":"e6e99dad-79f2-4b63-87f9-1c761080cada","Type":"ContainerStarted","Data":"a0df871312b2d488be253a98a94a9998a29369b3f162066331873f625d033746"} Oct 30 17:11:50.146650 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:50.146290 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-db8417-74d9d84dc9-ddws5" event={"ID":"e6e99dad-79f2-4b63-87f9-1c761080cada","Type":"ContainerStarted","Data":"f108427ed3d5d904c10fe00da1547b90b263beb2c54e3dc45b8a969e86bc7ff7"} Oct 30 17:11:50.146650 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:50.146321 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-db8417-bfdb94d4d-frgtn" event={"ID":"47fa8756-130e-48fc-bcf9-008ab31f95cf","Type":"ContainerStarted","Data":"eaad7e2d0590e24706f9d1084db25ae7b57647af207968ccc4f4d99782071560"} Oct 30 17:11:50.146650 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:50.146342 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-db8417-bfdb94d4d-frgtn" event={"ID":"47fa8756-130e-48fc-bcf9-008ab31f95cf","Type":"ContainerStarted","Data":"ef46a8683eb286b22516ae5662928b65954917ca4eca9c1f6c97795b2593ddef"} Oct 30 17:11:50.180907 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:50.180855 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-db8417-74d9d84dc9-ddws5" podStartSLOduration=2.180842256 podStartE2EDuration="2.180842256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:11:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:11:50.179385862 +0000 UTC m=+1484.180140780" watchObservedRunningTime="2026-10-30 17:11:50.180842256 +0000 UTC m=+1484.181597225" Oct 30 17:11:50.225146 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:50.225101 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-db8417-bfdb94d4d-frgtn" podStartSLOduration=2.225088139 podStartE2EDuration="2.225088139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:11:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:11:50.221885284 +0000 UTC m=+1484.222640185" watchObservedRunningTime="2026-10-30 17:11:50.225088139 +0000 UTC m=+1484.225843040" Oct 30 17:11:53.082654 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:53.082621 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-409387-6487549584-9fc97"] Oct 30 17:11:53.122431 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:53.122403 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-409387-6d8f8cdb55-qm6nw"] Oct 30 17:11:56.035541 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:56.035506 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efac793c-fcjzh"] Oct 30 17:11:56.047844 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:56.047812 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efac793c-fcjzh"] Oct 30 17:11:56.146713 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:56.146682 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2a2c63d-154f-4950-95d6-55694fedd0c0" path="/var/lib/kubelet/pods/a2a2c63d-154f-4950-95d6-55694fedd0c0/volumes" Oct 30 17:11:58.083341 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:58.083266 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-409387-6487549584-9fc97" podUID="d3cd810a-a584-4693-abbc-9146035416f3" containerName="payload-processing" containerID="cri-o://030f89508093cb320ad3e8bc90f04d649015523d93506e83151ed5f0ea1adafe" gracePeriod=25 Oct 30 17:11:58.125870 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:58.125450 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-409387-6d8f8cdb55-qm6nw" podUID="e5551ac9-3a13-4b7f-9110-b93923d75d26" containerName="payload-pre-processing" containerID="cri-o://73465da99ae00b79839e4ac5c64cd1628309e80dfb012a7ca969b5b001b67461" gracePeriod=25 Oct 30 17:11:58.569468 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:58.569442 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-409387-6487549584-9fc97" Oct 30 17:11:58.605172 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:58.605149 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-409387-6d8f8cdb55-qm6nw" Oct 30 17:11:58.664520 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:58.663639 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3cd810a-a584-4693-abbc-9146035416f3-kube-api-access-dmqdj\" (UniqueName: \"kubernetes.io/projected/d3cd810a-a584-4693-abbc-9146035416f3-kube-api-access-dmqdj\") pod \"d3cd810a-a584-4693-abbc-9146035416f3\" (UID: \"d3cd810a-a584-4693-abbc-9146035416f3\") " Oct 30 17:11:58.664520 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:58.663781 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3cd810a-a584-4693-abbc-9146035416f3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3cd810a-a584-4693-abbc-9146035416f3-plugins-config-volume\") pod \"d3cd810a-a584-4693-abbc-9146035416f3\" (UID: \"d3cd810a-a584-4693-abbc-9146035416f3\") " Oct 30 17:11:58.664520 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:58.664463 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3cd810a-a584-4693-abbc-9146035416f3-plugins-config-volume" pod "d3cd810a-a584-4693-abbc-9146035416f3" (UID: "d3cd810a-a584-4693-abbc-9146035416f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:11:58.667301 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:58.667264 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3cd810a-a584-4693-abbc-9146035416f3-kube-api-access-dmqdj" pod "d3cd810a-a584-4693-abbc-9146035416f3" (UID: "d3cd810a-a584-4693-abbc-9146035416f3"). InnerVolumeSpecName "kube-api-access-dmqdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:11:58.765686 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:58.765651 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5551ac9-3a13-4b7f-9110-b93923d75d26-kube-api-access-9l89z\" (UniqueName: \"kubernetes.io/projected/e5551ac9-3a13-4b7f-9110-b93923d75d26-kube-api-access-9l89z\") pod \"e5551ac9-3a13-4b7f-9110-b93923d75d26\" (UID: \"e5551ac9-3a13-4b7f-9110-b93923d75d26\") " Oct 30 17:11:58.765865 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:58.765811 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5551ac9-3a13-4b7f-9110-b93923d75d26-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5551ac9-3a13-4b7f-9110-b93923d75d26-plugins-config-volume\") pod \"e5551ac9-3a13-4b7f-9110-b93923d75d26\" (UID: \"e5551ac9-3a13-4b7f-9110-b93923d75d26\") " Oct 30 17:11:58.766233 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:58.766208 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5551ac9-3a13-4b7f-9110-b93923d75d26-plugins-config-volume" pod "e5551ac9-3a13-4b7f-9110-b93923d75d26" (UID: "e5551ac9-3a13-4b7f-9110-b93923d75d26"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:11:58.766458 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:58.766437 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5551ac9-3a13-4b7f-9110-b93923d75d26-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:58.766532 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:58.766459 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmqdj\" (UniqueName: \"kubernetes.io/projected/d3cd810a-a584-4693-abbc-9146035416f3-kube-api-access-dmqdj\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:58.766532 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:58.766471 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3cd810a-a584-4693-abbc-9146035416f3-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:58.767920 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:58.767889 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5551ac9-3a13-4b7f-9110-b93923d75d26-kube-api-access-9l89z" pod "e5551ac9-3a13-4b7f-9110-b93923d75d26" (UID: "e5551ac9-3a13-4b7f-9110-b93923d75d26"). InnerVolumeSpecName "kube-api-access-9l89z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:11:58.868336 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:58.868282 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l89z\" (UniqueName: \"kubernetes.io/projected/e5551ac9-3a13-4b7f-9110-b93923d75d26-kube-api-access-9l89z\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:11:59.292145 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:59.292114 4522 generic.go:356] "Generic (PLEG): container finished" podID="e5551ac9-3a13-4b7f-9110-b93923d75d26" containerID="73465da99ae00b79839e4ac5c64cd1628309e80dfb012a7ca969b5b001b67461" exitCode=0 Oct 30 17:11:59.292575 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:59.292177 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-409387-6d8f8cdb55-qm6nw" Oct 30 17:11:59.292575 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:59.292196 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-409387-6d8f8cdb55-qm6nw" event={"ID":"e5551ac9-3a13-4b7f-9110-b93923d75d26","Type":"ContainerDied","Data":"73465da99ae00b79839e4ac5c64cd1628309e80dfb012a7ca969b5b001b67461"} Oct 30 17:11:59.292575 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:59.292280 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-409387-6d8f8cdb55-qm6nw" event={"ID":"e5551ac9-3a13-4b7f-9110-b93923d75d26","Type":"ContainerDied","Data":"de46503e09d85f80c5311b1d4f2304936b268eae30a01a3e4a0e0058675f532e"} Oct 30 17:11:59.292575 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:59.292327 4522 scope.go:122] "RemoveContainer" containerID="73465da99ae00b79839e4ac5c64cd1628309e80dfb012a7ca969b5b001b67461" Oct 30 17:11:59.293969 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:59.293948 4522 generic.go:356] "Generic (PLEG): container finished" podID="d3cd810a-a584-4693-abbc-9146035416f3" containerID="030f89508093cb320ad3e8bc90f04d649015523d93506e83151ed5f0ea1adafe" exitCode=0 Oct 30 17:11:59.294100 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:59.293993 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-409387-6487549584-9fc97" event={"ID":"d3cd810a-a584-4693-abbc-9146035416f3","Type":"ContainerDied","Data":"030f89508093cb320ad3e8bc90f04d649015523d93506e83151ed5f0ea1adafe"} Oct 30 17:11:59.294100 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:59.294009 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-409387-6487549584-9fc97" Oct 30 17:11:59.294100 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:59.294022 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-409387-6487549584-9fc97" event={"ID":"d3cd810a-a584-4693-abbc-9146035416f3","Type":"ContainerDied","Data":"fc6d6e85dae3c1ad031f6f84244fc8d2ff8237529083f3e51cde0a181808a3c6"} Oct 30 17:11:59.304171 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:59.304153 4522 scope.go:122] "RemoveContainer" containerID="73465da99ae00b79839e4ac5c64cd1628309e80dfb012a7ca969b5b001b67461" Oct 30 17:11:59.304518 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:11:59.304496 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73465da99ae00b79839e4ac5c64cd1628309e80dfb012a7ca969b5b001b67461\": container with ID starting with 73465da99ae00b79839e4ac5c64cd1628309e80dfb012a7ca969b5b001b67461 not found: ID does not exist" containerID="73465da99ae00b79839e4ac5c64cd1628309e80dfb012a7ca969b5b001b67461" Oct 30 17:11:59.304594 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:59.304527 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73465da99ae00b79839e4ac5c64cd1628309e80dfb012a7ca969b5b001b67461"} err="failed to get container status \"73465da99ae00b79839e4ac5c64cd1628309e80dfb012a7ca969b5b001b67461\": rpc error: code = NotFound desc = could not find container \"73465da99ae00b79839e4ac5c64cd1628309e80dfb012a7ca969b5b001b67461\": container with ID starting with 73465da99ae00b79839e4ac5c64cd1628309e80dfb012a7ca969b5b001b67461 not found: ID does not exist" Oct 30 17:11:59.304594 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:59.304544 4522 scope.go:122] "RemoveContainer" containerID="030f89508093cb320ad3e8bc90f04d649015523d93506e83151ed5f0ea1adafe" Oct 30 17:11:59.314930 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:59.314908 4522 scope.go:122] "RemoveContainer" containerID="030f89508093cb320ad3e8bc90f04d649015523d93506e83151ed5f0ea1adafe" Oct 30 17:11:59.315238 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:11:59.315216 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"030f89508093cb320ad3e8bc90f04d649015523d93506e83151ed5f0ea1adafe\": container with ID starting with 030f89508093cb320ad3e8bc90f04d649015523d93506e83151ed5f0ea1adafe not found: ID does not exist" containerID="030f89508093cb320ad3e8bc90f04d649015523d93506e83151ed5f0ea1adafe" Oct 30 17:11:59.315285 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:59.315249 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"030f89508093cb320ad3e8bc90f04d649015523d93506e83151ed5f0ea1adafe"} err="failed to get container status \"030f89508093cb320ad3e8bc90f04d649015523d93506e83151ed5f0ea1adafe\": rpc error: code = NotFound desc = could not find container \"030f89508093cb320ad3e8bc90f04d649015523d93506e83151ed5f0ea1adafe\": container with ID starting with 030f89508093cb320ad3e8bc90f04d649015523d93506e83151ed5f0ea1adafe not found: ID does not exist" Oct 30 17:11:59.348642 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:59.348614 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-409387-6d8f8cdb55-qm6nw"] Oct 30 17:11:59.361624 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:59.361557 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-409387-6d8f8cdb55-qm6nw"] Oct 30 17:11:59.390714 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:59.390684 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-409387-6487549584-9fc97"] Oct 30 17:11:59.394793 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:11:59.394771 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-409387-6487549584-9fc97"] Oct 30 17:12:00.171658 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:00.171560 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3cd810a-a584-4693-abbc-9146035416f3" path="/var/lib/kubelet/pods/d3cd810a-a584-4693-abbc-9146035416f3/volumes" Oct 30 17:12:00.172238 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:00.172215 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5551ac9-3a13-4b7f-9110-b93923d75d26" path="/var/lib/kubelet/pods/e5551ac9-3a13-4b7f-9110-b93923d75d26/volumes" Oct 30 17:12:02.038545 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:02.038510 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3451d3b5-cm2tc"] Oct 30 17:12:02.050756 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:02.050731 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3451d3b5-cm2tc"] Oct 30 17:12:02.146735 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:02.146688 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8950018-69d7-41f4-85ee-e56a6665f22a" path="/var/lib/kubelet/pods/e8950018-69d7-41f4-85ee-e56a6665f22a/volumes" Oct 30 17:12:02.158991 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:02.158965 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-db8417-bfdb94d4d-frgtn" Oct 30 17:12:02.159166 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:02.159030 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-db8417-74d9d84dc9-ddws5" Oct 30 17:12:04.390514 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:04.390480 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b2c3fb-7698878959-chvds"] Oct 30 17:12:04.391247 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:04.391233 4522 cpu_manager.go:414] "containerMap: removing container" podUID="d3cd810a-a584-4693-abbc-9146035416f3" containerName="payload-processing" Oct 30 17:12:04.391297 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:04.391248 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3cd810a-a584-4693-abbc-9146035416f3" containerName="payload-processing" Oct 30 17:12:04.391297 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:04.391258 4522 cpu_manager.go:414] "containerMap: removing container" podUID="e5551ac9-3a13-4b7f-9110-b93923d75d26" containerName="payload-pre-processing" Oct 30 17:12:04.391297 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:04.391263 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5551ac9-3a13-4b7f-9110-b93923d75d26" containerName="payload-pre-processing" Oct 30 17:12:04.391442 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:04.391395 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3cd810a-a584-4693-abbc-9146035416f3" containerName="payload-processing" Oct 30 17:12:04.391442 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:04.391402 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5551ac9-3a13-4b7f-9110-b93923d75d26" containerName="payload-pre-processing" Oct 30 17:12:04.394749 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:04.394731 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b2c3fb-7698878959-chvds" Oct 30 17:12:04.417731 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:04.417697 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b2c3fb-7698878959-chvds"] Oct 30 17:12:04.425720 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:04.425690 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-b2c3fb-dockercfg-zbv9f\"" Oct 30 17:12:04.426540 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:04.425754 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-b2c3fb\"" Oct 30 17:12:04.521331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:04.496543 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzd6g\" (UniqueName: \"kubernetes.io/projected/90af553e-d49b-40d0-b182-8177a514e1eb-kube-api-access-gzd6g\") pod \"payload-pre-processing-e2e-api-b-b2c3fb-7698878959-chvds\" (UID: \"90af553e-d49b-40d0-b182-8177a514e1eb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b2c3fb-7698878959-chvds" Oct 30 17:12:04.542968 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:04.542927 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90af553e-d49b-40d0-b182-8177a514e1eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b2c3fb-7698878959-chvds\" (UID: \"90af553e-d49b-40d0-b182-8177a514e1eb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b2c3fb-7698878959-chvds" Oct 30 17:12:04.796256 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:04.654886 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzd6g\" (UniqueName: \"kubernetes.io/projected/90af553e-d49b-40d0-b182-8177a514e1eb-kube-api-access-gzd6g\") pod \"payload-pre-processing-e2e-api-b-b2c3fb-7698878959-chvds\" (UID: \"90af553e-d49b-40d0-b182-8177a514e1eb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b2c3fb-7698878959-chvds" Oct 30 17:12:04.796256 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:04.667548 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90af553e-d49b-40d0-b182-8177a514e1eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b2c3fb-7698878959-chvds\" (UID: \"90af553e-d49b-40d0-b182-8177a514e1eb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b2c3fb-7698878959-chvds" Oct 30 17:12:04.796256 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:04.669810 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90af553e-d49b-40d0-b182-8177a514e1eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b2c3fb-7698878959-chvds\" (UID: \"90af553e-d49b-40d0-b182-8177a514e1eb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b2c3fb-7698878959-chvds" Oct 30 17:12:04.875154 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:04.871391 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzd6g\" (UniqueName: \"kubernetes.io/projected/90af553e-d49b-40d0-b182-8177a514e1eb-kube-api-access-gzd6g\") pod \"payload-pre-processing-e2e-api-b-b2c3fb-7698878959-chvds\" (UID: \"90af553e-d49b-40d0-b182-8177a514e1eb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b2c3fb-7698878959-chvds" Oct 30 17:12:04.909719 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:04.909688 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b2c3fb-5f877958dd-99vkh"] Oct 30 17:12:04.926360 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:04.922339 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b2c3fb-5f877958dd-99vkh"] Oct 30 17:12:04.926360 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:04.922440 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b2c3fb-5f877958dd-99vkh" Oct 30 17:12:04.944246 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:04.944200 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqcvj\" (UniqueName: \"kubernetes.io/projected/42e57fa2-479b-4655-9ce4-c49ee4757cc4-kube-api-access-bqcvj\") pod \"payload-processing-e2e-api-b-b2c3fb-5f877958dd-99vkh\" (UID: \"42e57fa2-479b-4655-9ce4-c49ee4757cc4\") " pod="openshift-ingress/payload-processing-e2e-api-b-b2c3fb-5f877958dd-99vkh" Oct 30 17:12:04.944413 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:04.944397 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42e57fa2-479b-4655-9ce4-c49ee4757cc4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b2c3fb-5f877958dd-99vkh\" (UID: \"42e57fa2-479b-4655-9ce4-c49ee4757cc4\") " pod="openshift-ingress/payload-processing-e2e-api-b-b2c3fb-5f877958dd-99vkh" Oct 30 17:12:05.008955 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:05.008922 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b2c3fb-7698878959-chvds" Oct 30 17:12:05.046220 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:05.046180 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqcvj\" (UniqueName: \"kubernetes.io/projected/42e57fa2-479b-4655-9ce4-c49ee4757cc4-kube-api-access-bqcvj\") pod \"payload-processing-e2e-api-b-b2c3fb-5f877958dd-99vkh\" (UID: \"42e57fa2-479b-4655-9ce4-c49ee4757cc4\") " pod="openshift-ingress/payload-processing-e2e-api-b-b2c3fb-5f877958dd-99vkh" Oct 30 17:12:05.046823 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:05.046434 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42e57fa2-479b-4655-9ce4-c49ee4757cc4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b2c3fb-5f877958dd-99vkh\" (UID: \"42e57fa2-479b-4655-9ce4-c49ee4757cc4\") " pod="openshift-ingress/payload-processing-e2e-api-b-b2c3fb-5f877958dd-99vkh" Oct 30 17:12:05.047814 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:05.047687 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42e57fa2-479b-4655-9ce4-c49ee4757cc4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b2c3fb-5f877958dd-99vkh\" (UID: \"42e57fa2-479b-4655-9ce4-c49ee4757cc4\") " pod="openshift-ingress/payload-processing-e2e-api-b-b2c3fb-5f877958dd-99vkh" Oct 30 17:12:05.072642 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:05.072611 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqcvj\" (UniqueName: \"kubernetes.io/projected/42e57fa2-479b-4655-9ce4-c49ee4757cc4-kube-api-access-bqcvj\") pod \"payload-processing-e2e-api-b-b2c3fb-5f877958dd-99vkh\" (UID: \"42e57fa2-479b-4655-9ce4-c49ee4757cc4\") " pod="openshift-ingress/payload-processing-e2e-api-b-b2c3fb-5f877958dd-99vkh" Oct 30 17:12:05.243442 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:05.243408 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b2c3fb-5f877958dd-99vkh" Oct 30 17:12:05.342658 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:12:05.342608 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90af553e_d49b_40d0_b182_8177a514e1eb.slice/crio-88f375b05345e088b063c7e9d7ef1cd96d9dff07933468bba6c56f7a48c70d3d WatchSource:0}: Error finding container 88f375b05345e088b063c7e9d7ef1cd96d9dff07933468bba6c56f7a48c70d3d: Status 404 returned error can't find the container with id 88f375b05345e088b063c7e9d7ef1cd96d9dff07933468bba6c56f7a48c70d3d Oct 30 17:12:05.344340 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:05.344296 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b2c3fb-7698878959-chvds"] Oct 30 17:12:05.359108 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:05.359076 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b2c3fb-7698878959-chvds" event={"ID":"90af553e-d49b-40d0-b182-8177a514e1eb","Type":"ContainerStarted","Data":"88f375b05345e088b063c7e9d7ef1cd96d9dff07933468bba6c56f7a48c70d3d"} Oct 30 17:12:05.566255 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:12:05.566200 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42e57fa2_479b_4655_9ce4_c49ee4757cc4.slice/crio-a4728d490b3bca62de83699f05b3694b28570ac0630925da55f662cc1d4c77db WatchSource:0}: Error finding container a4728d490b3bca62de83699f05b3694b28570ac0630925da55f662cc1d4c77db: Status 404 returned error can't find the container with id a4728d490b3bca62de83699f05b3694b28570ac0630925da55f662cc1d4c77db Oct 30 17:12:05.568724 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:05.568700 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b2c3fb-5f877958dd-99vkh"] Oct 30 17:12:06.382194 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:06.381962 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b2c3fb-7698878959-chvds" event={"ID":"90af553e-d49b-40d0-b182-8177a514e1eb","Type":"ContainerStarted","Data":"19aa7d1ddbbea0692fdd28ce84ae0399cc3513ede37d257eb2fe0b427915c624"} Oct 30 17:12:06.382500 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:06.382476 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b2c3fb-7698878959-chvds" Oct 30 17:12:06.385580 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:06.385464 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b2c3fb-5f877958dd-99vkh" event={"ID":"42e57fa2-479b-4655-9ce4-c49ee4757cc4","Type":"ContainerStarted","Data":"bed897987805e59b26129f538252a1ab876ddf29ab8d61933a84d09d7eaad5b0"} Oct 30 17:12:06.385580 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:06.385521 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-b2c3fb-5f877958dd-99vkh" Oct 30 17:12:06.385580 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:06.385546 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b2c3fb-5f877958dd-99vkh" event={"ID":"42e57fa2-479b-4655-9ce4-c49ee4757cc4","Type":"ContainerStarted","Data":"a4728d490b3bca62de83699f05b3694b28570ac0630925da55f662cc1d4c77db"} Oct 30 17:12:06.433575 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:06.433517 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b2c3fb-7698878959-chvds" podStartSLOduration=2.433497778 podStartE2EDuration="2.433497778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:12:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:06.432975918 +0000 UTC m=+1500.433730856" watchObservedRunningTime="2026-10-30 17:12:06.433497778 +0000 UTC m=+1500.434252681" Oct 30 17:12:06.479644 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:06.479586 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-b2c3fb-5f877958dd-99vkh" podStartSLOduration=2.479567361 podStartE2EDuration="2.479567361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:12:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:06.476101952 +0000 UTC m=+1500.476856875" watchObservedRunningTime="2026-10-30 17:12:06.479567361 +0000 UTC m=+1500.480322263" Oct 30 17:12:07.472329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:07.471705 4522 scope.go:122] "RemoveContainer" containerID="73ebb99cddb09eb2591357918620a190199d0bc8fd4ca16cbb572ffbed0fc7b1" Oct 30 17:12:07.485268 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:07.485239 4522 scope.go:122] "RemoveContainer" containerID="734f2ae61c97009cf9b9862a641563555211e36619d7be1efc805a17c7a20c16" Oct 30 17:12:07.576454 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:07.576424 4522 scope.go:122] "RemoveContainer" containerID="2c65870447f84761427395653edde21d0fc029d1143bb4ff7ab26bd4e92b0e73" Oct 30 17:12:07.588800 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:07.588781 4522 scope.go:122] "RemoveContainer" containerID="2ff694d90998f96a8da26f7eb9973b6b4f27f48179e082bc271ef5206b197a81" Oct 30 17:12:12.951676 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:12:12.951637 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:12:15.576955 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:15.576915 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7d1e5c-sl2nh"] Oct 30 17:12:15.591868 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:15.591825 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7d1e5c-sl2nh"] Oct 30 17:12:16.148065 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:16.148034 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4089af4a-de4d-453b-8e73-3a447e197309" path="/var/lib/kubelet/pods/4089af4a-de4d-453b-8e73-3a447e197309/volumes" Oct 30 17:12:17.459572 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:17.459543 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-b2c3fb-5f877958dd-99vkh" Oct 30 17:12:17.459951 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:17.459611 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b2c3fb-7698878959-chvds" Oct 30 17:12:22.748519 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:22.748479 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr"] Oct 30 17:12:22.749010 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:22.748739 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr" podUID="073c8bab-6c64-4fa5-8075-80f71c994845" containerName="istio-proxy" containerID="cri-o://6edf92bc06a4e6acf907fb52c588cd0d0f7b6cd017647bb93529b310fc8f7952" gracePeriod=30 Oct 30 17:12:23.334974 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:23.334936 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-362bbf-pbh42"] Oct 30 17:12:23.340665 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:23.340642 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-362bbf-pbh42" Oct 30 17:12:23.360654 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:23.360622 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-362bbf-pbh42"] Oct 30 17:12:23.397357 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:23.393636 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:12:23.397357 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:23.393692 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:12:23.473029 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:23.473000 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgcjm\" (UniqueName: \"kubernetes.io/projected/19794b16-e65f-452d-b69b-15435649d07e-kube-api-access-bgcjm\") pod \"maas-api-revoke-keys-e2e-api-b-362bbf-pbh42\" (UID: \"19794b16-e65f-452d-b69b-15435649d07e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-362bbf-pbh42" Oct 30 17:12:23.473199 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:23.473151 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19794b16-e65f-452d-b69b-15435649d07e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-362bbf-pbh42\" (UID: \"19794b16-e65f-452d-b69b-15435649d07e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-362bbf-pbh42" Oct 30 17:12:23.473262 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:23.473211 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19794b16-e65f-452d-b69b-15435649d07e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-362bbf-pbh42\" (UID: \"19794b16-e65f-452d-b69b-15435649d07e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-362bbf-pbh42" Oct 30 17:12:23.574235 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:23.574200 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19794b16-e65f-452d-b69b-15435649d07e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-362bbf-pbh42\" (UID: \"19794b16-e65f-452d-b69b-15435649d07e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-362bbf-pbh42" Oct 30 17:12:23.574557 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:23.574536 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgcjm\" (UniqueName: \"kubernetes.io/projected/19794b16-e65f-452d-b69b-15435649d07e-kube-api-access-bgcjm\") pod \"maas-api-revoke-keys-e2e-api-b-362bbf-pbh42\" (UID: \"19794b16-e65f-452d-b69b-15435649d07e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-362bbf-pbh42" Oct 30 17:12:23.574688 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:23.574670 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19794b16-e65f-452d-b69b-15435649d07e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-362bbf-pbh42\" (UID: \"19794b16-e65f-452d-b69b-15435649d07e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-362bbf-pbh42" Oct 30 17:12:23.575268 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:23.575248 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19794b16-e65f-452d-b69b-15435649d07e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-362bbf-pbh42\" (UID: \"19794b16-e65f-452d-b69b-15435649d07e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-362bbf-pbh42" Oct 30 17:12:23.576912 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:23.576884 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19794b16-e65f-452d-b69b-15435649d07e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-362bbf-pbh42\" (UID: \"19794b16-e65f-452d-b69b-15435649d07e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-362bbf-pbh42" Oct 30 17:12:23.598647 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:23.598587 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgcjm\" (UniqueName: \"kubernetes.io/projected/19794b16-e65f-452d-b69b-15435649d07e-kube-api-access-bgcjm\") pod \"maas-api-revoke-keys-e2e-api-b-362bbf-pbh42\" (UID: \"19794b16-e65f-452d-b69b-15435649d07e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-362bbf-pbh42" Oct 30 17:12:23.656723 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:23.656695 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-362bbf-pbh42" Oct 30 17:12:23.963138 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:12:23.963106 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19794b16_e65f_452d_b69b_15435649d07e.slice/crio-a8f5265e43033aa7f60ccc0ecfb7438b5875c733c20a5ad77c07139173c60134 WatchSource:0}: Error finding container a8f5265e43033aa7f60ccc0ecfb7438b5875c733c20a5ad77c07139173c60134: Status 404 returned error can't find the container with id a8f5265e43033aa7f60ccc0ecfb7438b5875c733c20a5ad77c07139173c60134 Oct 30 17:12:23.965965 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:23.965938 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-362bbf-pbh42"] Oct 30 17:12:24.616889 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:24.616793 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-362bbf-pbh42" event={"ID":"19794b16-e65f-452d-b69b-15435649d07e","Type":"ContainerStarted","Data":"8f8b1f55862d413d4b68cd634112b25f4850bd62a62c2e0c6b06d94ee6743796"} Oct 30 17:12:24.616889 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:24.616855 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-362bbf-pbh42" event={"ID":"19794b16-e65f-452d-b69b-15435649d07e","Type":"ContainerStarted","Data":"a8f5265e43033aa7f60ccc0ecfb7438b5875c733c20a5ad77c07139173c60134"} Oct 30 17:12:24.660087 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:24.660023 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-362bbf-pbh42" podStartSLOduration=1.66000897 podStartE2EDuration="1.66000897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:12:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:24.65806319 +0000 UTC m=+1518.658818091" watchObservedRunningTime="2026-10-30 17:12:24.66000897 +0000 UTC m=+1518.660763870" Oct 30 17:12:25.628138 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:25.628108 4522 generic.go:356] "Generic (PLEG): container finished" podID="19794b16-e65f-452d-b69b-15435649d07e" containerID="8f8b1f55862d413d4b68cd634112b25f4850bd62a62c2e0c6b06d94ee6743796" exitCode=0 Oct 30 17:12:25.628566 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:25.628201 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-362bbf-pbh42" event={"ID":"19794b16-e65f-452d-b69b-15435649d07e","Type":"ContainerDied","Data":"8f8b1f55862d413d4b68cd634112b25f4850bd62a62c2e0c6b06d94ee6743796"} Oct 30 17:12:26.670361 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:26.670186 4522 patch_prober.go:29] interesting pod/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:12:26.670361 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:26.670243 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr" podUID="073c8bab-6c64-4fa5-8075-80f71c994845" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:12:26.816177 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:26.816153 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-362bbf-pbh42" Oct 30 17:12:26.932361 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:26.932259 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19794b16-e65f-452d-b69b-15435649d07e-kube-api-access-bgcjm\" (UniqueName: \"kubernetes.io/projected/19794b16-e65f-452d-b69b-15435649d07e-kube-api-access-bgcjm\") pod \"19794b16-e65f-452d-b69b-15435649d07e\" (UID: \"19794b16-e65f-452d-b69b-15435649d07e\") " Oct 30 17:12:26.932531 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:26.932427 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19794b16-e65f-452d-b69b-15435649d07e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19794b16-e65f-452d-b69b-15435649d07e-maas-api-openshift-ca\") pod \"19794b16-e65f-452d-b69b-15435649d07e\" (UID: \"19794b16-e65f-452d-b69b-15435649d07e\") " Oct 30 17:12:26.932704 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:26.932685 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19794b16-e65f-452d-b69b-15435649d07e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19794b16-e65f-452d-b69b-15435649d07e-maas-api-xks-ca\") pod \"19794b16-e65f-452d-b69b-15435649d07e\" (UID: \"19794b16-e65f-452d-b69b-15435649d07e\") " Oct 30 17:12:26.932799 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:26.932772 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19794b16-e65f-452d-b69b-15435649d07e-maas-api-openshift-ca" pod "19794b16-e65f-452d-b69b-15435649d07e" (UID: "19794b16-e65f-452d-b69b-15435649d07e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:12:26.933382 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:26.933361 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19794b16-e65f-452d-b69b-15435649d07e-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:12:26.934489 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:26.934464 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19794b16-e65f-452d-b69b-15435649d07e-kube-api-access-bgcjm" pod "19794b16-e65f-452d-b69b-15435649d07e" (UID: "19794b16-e65f-452d-b69b-15435649d07e"). InnerVolumeSpecName "kube-api-access-bgcjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:12:26.934613 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:26.934595 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19794b16-e65f-452d-b69b-15435649d07e-maas-api-xks-ca" pod "19794b16-e65f-452d-b69b-15435649d07e" (UID: "19794b16-e65f-452d-b69b-15435649d07e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:12:27.034768 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:27.034727 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19794b16-e65f-452d-b69b-15435649d07e-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:12:27.034768 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:27.034760 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgcjm\" (UniqueName: \"kubernetes.io/projected/19794b16-e65f-452d-b69b-15435649d07e-kube-api-access-bgcjm\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:12:27.646422 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:27.646386 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-362bbf-pbh42" Oct 30 17:12:27.646422 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:27.646417 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-362bbf-pbh42" event={"ID":"19794b16-e65f-452d-b69b-15435649d07e","Type":"ContainerDied","Data":"a8f5265e43033aa7f60ccc0ecfb7438b5875c733c20a5ad77c07139173c60134"} Oct 30 17:12:27.646641 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:27.646467 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8f5265e43033aa7f60ccc0ecfb7438b5875c733c20a5ad77c07139173c60134" Oct 30 17:12:28.007821 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:28.006408 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ebeb6e1-qfdc7"] Oct 30 17:12:28.007821 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:28.007671 4522 cpu_manager.go:414] "containerMap: removing container" podUID="19794b16-e65f-452d-b69b-15435649d07e" containerName="revoke-keys" Oct 30 17:12:28.007821 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:28.007694 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19794b16-e65f-452d-b69b-15435649d07e" containerName="revoke-keys" Oct 30 17:12:28.008501 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:28.008034 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="19794b16-e65f-452d-b69b-15435649d07e" containerName="revoke-keys" Oct 30 17:12:28.013876 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:28.013850 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ebeb6e1-qfdc7"] Oct 30 17:12:28.013982 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:28.013964 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ebeb6e1-qfdc7" Oct 30 17:12:28.038430 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:28.038402 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:12:28.038614 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:28.038421 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:12:28.152941 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:28.152910 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr" Oct 30 17:12:28.155856 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:28.155819 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18d1b373-ad7a-466e-9a6c-ed7f557c386a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-7ebeb6e1-qfdc7\" (UID: \"18d1b373-ad7a-466e-9a6c-ed7f557c386a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ebeb6e1-qfdc7" Oct 30 17:12:28.156001 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:28.155977 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtbpl\" (UniqueName: \"kubernetes.io/projected/18d1b373-ad7a-466e-9a6c-ed7f557c386a-kube-api-access-dtbpl\") pod \"maas-api-revoke-keys-e2e-mt-7ebeb6e1-qfdc7\" (UID: \"18d1b373-ad7a-466e-9a6c-ed7f557c386a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ebeb6e1-qfdc7" Oct 30 17:12:28.156148 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:28.156128 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18d1b373-ad7a-466e-9a6c-ed7f557c386a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-7ebeb6e1-qfdc7\" (UID: \"18d1b373-ad7a-466e-9a6c-ed7f557c386a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ebeb6e1-qfdc7" Oct 30 17:12:28.257129 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:28.257018 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/073c8bab-6c64-4fa5-8075-80f71c994845-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/073c8bab-6c64-4fa5-8075-80f71c994845-workload-socket\") pod \"073c8bab-6c64-4fa5-8075-80f71c994845\" (UID: \"073c8bab-6c64-4fa5-8075-80f71c994845\") " Oct 30 17:12:28.257332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:28.257201 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/073c8bab-6c64-4fa5-8075-80f71c994845-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/073c8bab-6c64-4fa5-8075-80f71c994845-istio-data\") pod \"073c8bab-6c64-4fa5-8075-80f71c994845\" (UID: \"073c8bab-6c64-4fa5-8075-80f71c994845\") " Oct 30 17:12:28.257332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:28.257321 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/073c8bab-6c64-4fa5-8075-80f71c994845-kube-api-access-6h8zd\" (UniqueName: \"kubernetes.io/projected/073c8bab-6c64-4fa5-8075-80f71c994845-kube-api-access-6h8zd\") pod \"073c8bab-6c64-4fa5-8075-80f71c994845\" (UID: \"073c8bab-6c64-4fa5-8075-80f71c994845\") " Oct 30 17:12:28.257472 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:28.257375 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/073c8bab-6c64-4fa5-8075-80f71c994845-istio-token\" (UniqueName: \"kubernetes.io/projected/073c8bab-6c64-4fa5-8075-80f71c994845-istio-token\") pod \"073c8bab-6c64-4fa5-8075-80f71c994845\" (UID: \"073c8bab-6c64-4fa5-8075-80f71c994845\") " Oct 30 17:12:28.257472 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:28.257369 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/073c8bab-6c64-4fa5-8075-80f71c994845-workload-socket" pod "073c8bab-6c64-4fa5-8075-80f71c994845" (UID: "073c8bab-6c64-4fa5-8075-80f71c994845"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:28.257472 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:28.257419 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/073c8bab-6c64-4fa5-8075-80f71c994845-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/073c8bab-6c64-4fa5-8075-80f71c994845-istiod-ca-cert\") pod \"073c8bab-6c64-4fa5-8075-80f71c994845\" (UID: \"073c8bab-6c64-4fa5-8075-80f71c994845\") " Oct 30 17:12:28.257472 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:28.257449 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/073c8bab-6c64-4fa5-8075-80f71c994845-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/073c8bab-6c64-4fa5-8075-80f71c994845-istio-podinfo\") pod \"073c8bab-6c64-4fa5-8075-80f71c994845\" (UID: \"073c8bab-6c64-4fa5-8075-80f71c994845\") " Oct 30 17:12:28.257683 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:28.257513 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/073c8bab-6c64-4fa5-8075-80f71c994845-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/073c8bab-6c64-4fa5-8075-80f71c994845-credential-socket\") pod \"073c8bab-6c64-4fa5-8075-80f71c994845\" (UID: \"073c8bab-6c64-4fa5-8075-80f71c994845\") " Oct 30 17:12:28.257683 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:28.257625 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/073c8bab-6c64-4fa5-8075-80f71c994845-istio-data" pod "073c8bab-6c64-4fa5-8075-80f71c994845" (UID: "073c8bab-6c64-4fa5-8075-80f71c994845"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:28.257786 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:28.257667 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/073c8bab-6c64-4fa5-8075-80f71c994845-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/073c8bab-6c64-4fa5-8075-80f71c994845-workload-certs\") pod \"073c8bab-6c64-4fa5-8075-80f71c994845\" (UID: \"073c8bab-6c64-4fa5-8075-80f71c994845\") " Oct 30 17:12:28.257786 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:28.257763 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/073c8bab-6c64-4fa5-8075-80f71c994845-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/073c8bab-6c64-4fa5-8075-80f71c994845-istio-envoy\") pod \"073c8bab-6c64-4fa5-8075-80f71c994845\" (UID: \"073c8bab-6c64-4fa5-8075-80f71c994845\") " Oct 30 17:12:28.258165 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:28.258142 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/073c8bab-6c64-4fa5-8075-80f71c994845-credential-socket" pod "073c8bab-6c64-4fa5-8075-80f71c994845" (UID: "073c8bab-6c64-4fa5-8075-80f71c994845"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:28.258248 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:28.258225 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18d1b373-ad7a-466e-9a6c-ed7f557c386a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-7ebeb6e1-qfdc7\" (UID: \"18d1b373-ad7a-466e-9a6c-ed7f557c386a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ebeb6e1-qfdc7" Oct 30 17:12:28.258666 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:28.258642 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18d1b373-ad7a-466e-9a6c-ed7f557c386a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-7ebeb6e1-qfdc7\" (UID: \"18d1b373-ad7a-466e-9a6c-ed7f557c386a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ebeb6e1-qfdc7" Oct 30 17:12:28.259021 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:28.258991 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtbpl\" (UniqueName: \"kubernetes.io/projected/18d1b373-ad7a-466e-9a6c-ed7f557c386a-kube-api-access-dtbpl\") pod \"maas-api-revoke-keys-e2e-mt-7ebeb6e1-qfdc7\" (UID: \"18d1b373-ad7a-466e-9a6c-ed7f557c386a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ebeb6e1-qfdc7" Oct 30 17:12:28.259241 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:28.259219 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18d1b373-ad7a-466e-9a6c-ed7f557c386a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-7ebeb6e1-qfdc7\" (UID: \"18d1b373-ad7a-466e-9a6c-ed7f557c386a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ebeb6e1-qfdc7" Oct 30 17:12:28.259366 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:28.259228 4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/073c8bab-6c64-4fa5-8075-80f71c994845-workload-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:12:28.259366 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:28.259278 4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/073c8bab-6c64-4fa5-8075-80f71c994845-istio-data\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:12:28.259366 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:28.259297 4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/073c8bab-6c64-4fa5-8075-80f71c994845-credential-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:12:28.259366 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:28.258995 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/073c8bab-6c64-4fa5-8075-80f71c994845-istiod-ca-cert" pod "073c8bab-6c64-4fa5-8075-80f71c994845" (UID: "073c8bab-6c64-4fa5-8075-80f71c994845"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:12:28.259606 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:28.259412 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/073c8bab-6c64-4fa5-8075-80f71c994845-workload-certs" pod "073c8bab-6c64-4fa5-8075-80f71c994845" (UID: "073c8bab-6c64-4fa5-8075-80f71c994845"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:28.260745 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:28.260719 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/073c8bab-6c64-4fa5-8075-80f71c994845-istio-token" pod "073c8bab-6c64-4fa5-8075-80f71c994845" (UID: "073c8bab-6c64-4fa5-8075-80f71c994845"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:12:28.261127 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:28.261098 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/073c8bab-6c64-4fa5-8075-80f71c994845-kube-api-access-6h8zd" pod "073c8bab-6c64-4fa5-8075-80f71c994845" (UID: "073c8bab-6c64-4fa5-8075-80f71c994845"). InnerVolumeSpecName "kube-api-access-6h8zd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:12:28.261812 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:28.261784 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/073c8bab-6c64-4fa5-8075-80f71c994845-istio-envoy" pod "073c8bab-6c64-4fa5-8075-80f71c994845" (UID: "073c8bab-6c64-4fa5-8075-80f71c994845"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:28.262461 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:28.262434 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/073c8bab-6c64-4fa5-8075-80f71c994845-istio-podinfo" pod "073c8bab-6c64-4fa5-8075-80f71c994845" (UID: "073c8bab-6c64-4fa5-8075-80f71c994845"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:12:28.263318 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:28.263279 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18d1b373-ad7a-466e-9a6c-ed7f557c386a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-7ebeb6e1-qfdc7\" (UID: \"18d1b373-ad7a-466e-9a6c-ed7f557c386a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ebeb6e1-qfdc7" Oct 30 17:12:28.293383 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:28.293281 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtbpl\" (UniqueName: \"kubernetes.io/projected/18d1b373-ad7a-466e-9a6c-ed7f557c386a-kube-api-access-dtbpl\") pod \"maas-api-revoke-keys-e2e-mt-7ebeb6e1-qfdc7\" (UID: \"18d1b373-ad7a-466e-9a6c-ed7f557c386a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ebeb6e1-qfdc7" Oct 30 17:12:28.355140 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:28.355099 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ebeb6e1-qfdc7" Oct 30 17:12:28.360542 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:28.360516 4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/073c8bab-6c64-4fa5-8075-80f71c994845-workload-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:12:28.360542 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:28.360543 4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/073c8bab-6c64-4fa5-8075-80f71c994845-istio-envoy\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:12:28.360670 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:28.360556 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h8zd\" (UniqueName: \"kubernetes.io/projected/073c8bab-6c64-4fa5-8075-80f71c994845-kube-api-access-6h8zd\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:12:28.360670 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:28.360570 4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/073c8bab-6c64-4fa5-8075-80f71c994845-istio-token\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:12:28.360670 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:28.360583 4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/073c8bab-6c64-4fa5-8075-80f71c994845-istiod-ca-cert\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:12:28.360670 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:28.360595 4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/073c8bab-6c64-4fa5-8075-80f71c994845-istio-podinfo\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:12:28.690496 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:12:28.685635 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18d1b373_ad7a_466e_9a6c_ed7f557c386a.slice/crio-48da88a2b04448f8fd5a0ce21d6ab46f602e0443b1820abe564c2e06b27a713c WatchSource:0}: Error finding container 48da88a2b04448f8fd5a0ce21d6ab46f602e0443b1820abe564c2e06b27a713c: Status 404 returned error can't find the container with id 48da88a2b04448f8fd5a0ce21d6ab46f602e0443b1820abe564c2e06b27a713c Oct 30 17:12:28.690496 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:28.686792 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ebeb6e1-qfdc7"] Oct 30 17:12:28.690496 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:28.688266 4522 generic.go:356] "Generic (PLEG): container finished" podID="073c8bab-6c64-4fa5-8075-80f71c994845" containerID="6edf92bc06a4e6acf907fb52c588cd0d0f7b6cd017647bb93529b310fc8f7952" exitCode=0 Oct 30 17:12:28.690496 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:28.688510 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr" Oct 30 17:12:28.690496 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:28.688600 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr" event={"ID":"073c8bab-6c64-4fa5-8075-80f71c994845","Type":"ContainerDied","Data":"6edf92bc06a4e6acf907fb52c588cd0d0f7b6cd017647bb93529b310fc8f7952"} Oct 30 17:12:28.690496 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:28.688665 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr" event={"ID":"073c8bab-6c64-4fa5-8075-80f71c994845","Type":"ContainerDied","Data":"8693cda88f5e5e9b795721ff2b56f8091097adf4e1d139f67aec82c4e7cebf6e"} Oct 30 17:12:28.690496 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:28.688698 4522 scope.go:122] "RemoveContainer" containerID="6edf92bc06a4e6acf907fb52c588cd0d0f7b6cd017647bb93529b310fc8f7952" Oct 30 17:12:28.707871 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:28.707853 4522 scope.go:122] "RemoveContainer" containerID="fa24750506a1ec43eac352c1a2660077859f7a98e44268cf9240cc97136bb8d7" Oct 30 17:12:28.725500 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:28.725474 4522 scope.go:122] "RemoveContainer" containerID="6edf92bc06a4e6acf907fb52c588cd0d0f7b6cd017647bb93529b310fc8f7952" Oct 30 17:12:28.725854 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:12:28.725829 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6edf92bc06a4e6acf907fb52c588cd0d0f7b6cd017647bb93529b310fc8f7952\": container with ID starting with 6edf92bc06a4e6acf907fb52c588cd0d0f7b6cd017647bb93529b310fc8f7952 not found: ID does not exist" containerID="6edf92bc06a4e6acf907fb52c588cd0d0f7b6cd017647bb93529b310fc8f7952" Oct 30 17:12:28.725931 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:28.725863 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6edf92bc06a4e6acf907fb52c588cd0d0f7b6cd017647bb93529b310fc8f7952"} err="failed to get container status \"6edf92bc06a4e6acf907fb52c588cd0d0f7b6cd017647bb93529b310fc8f7952\": rpc error: code = NotFound desc = could not find container \"6edf92bc06a4e6acf907fb52c588cd0d0f7b6cd017647bb93529b310fc8f7952\": container with ID starting with 6edf92bc06a4e6acf907fb52c588cd0d0f7b6cd017647bb93529b310fc8f7952 not found: ID does not exist" Oct 30 17:12:28.725931 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:28.725883 4522 scope.go:122] "RemoveContainer" containerID="fa24750506a1ec43eac352c1a2660077859f7a98e44268cf9240cc97136bb8d7" Oct 30 17:12:28.726278 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:12:28.726261 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa24750506a1ec43eac352c1a2660077859f7a98e44268cf9240cc97136bb8d7\": container with ID starting with fa24750506a1ec43eac352c1a2660077859f7a98e44268cf9240cc97136bb8d7 not found: ID does not exist" containerID="fa24750506a1ec43eac352c1a2660077859f7a98e44268cf9240cc97136bb8d7" Oct 30 17:12:28.726382 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:28.726287 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa24750506a1ec43eac352c1a2660077859f7a98e44268cf9240cc97136bb8d7"} err="failed to get container status \"fa24750506a1ec43eac352c1a2660077859f7a98e44268cf9240cc97136bb8d7\": rpc error: code = NotFound desc = could not find container \"fa24750506a1ec43eac352c1a2660077859f7a98e44268cf9240cc97136bb8d7\": container with ID starting with fa24750506a1ec43eac352c1a2660077859f7a98e44268cf9240cc97136bb8d7 not found: ID does not exist" Oct 30 17:12:28.758480 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:28.758436 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr"] Oct 30 17:12:28.764932 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:28.764901 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-7d1e5c-openshift-default-b9f445d7-vmqcr"] Oct 30 17:12:29.699007 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:29.698979 4522 generic.go:356] "Generic (PLEG): container finished" podID="18d1b373-ad7a-466e-9a6c-ed7f557c386a" containerID="1efd4e01714cb87d3bed11a295abf1528a19c909ece6233634556584e6118364" exitCode=0 Oct 30 17:12:29.699454 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:29.699064 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ebeb6e1-qfdc7" event={"ID":"18d1b373-ad7a-466e-9a6c-ed7f557c386a","Type":"ContainerDied","Data":"1efd4e01714cb87d3bed11a295abf1528a19c909ece6233634556584e6118364"} Oct 30 17:12:29.699454 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:29.699126 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ebeb6e1-qfdc7" event={"ID":"18d1b373-ad7a-466e-9a6c-ed7f557c386a","Type":"ContainerStarted","Data":"48da88a2b04448f8fd5a0ce21d6ab46f602e0443b1820abe564c2e06b27a713c"} Oct 30 17:12:30.148242 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:30.148200 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="073c8bab-6c64-4fa5-8075-80f71c994845" path="/var/lib/kubelet/pods/073c8bab-6c64-4fa5-8075-80f71c994845/volumes" Oct 30 17:12:30.872553 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:30.872529 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ebeb6e1-qfdc7" Oct 30 17:12:30.915069 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:30.915033 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18d1b373-ad7a-466e-9a6c-ed7f557c386a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18d1b373-ad7a-466e-9a6c-ed7f557c386a-maas-api-xks-ca\") pod \"18d1b373-ad7a-466e-9a6c-ed7f557c386a\" (UID: \"18d1b373-ad7a-466e-9a6c-ed7f557c386a\") " Oct 30 17:12:30.921555 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:30.921091 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18d1b373-ad7a-466e-9a6c-ed7f557c386a-maas-api-xks-ca" pod "18d1b373-ad7a-466e-9a6c-ed7f557c386a" (UID: "18d1b373-ad7a-466e-9a6c-ed7f557c386a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:12:31.016594 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:31.016501 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18d1b373-ad7a-466e-9a6c-ed7f557c386a-kube-api-access-dtbpl\" (UniqueName: \"kubernetes.io/projected/18d1b373-ad7a-466e-9a6c-ed7f557c386a-kube-api-access-dtbpl\") pod \"18d1b373-ad7a-466e-9a6c-ed7f557c386a\" (UID: \"18d1b373-ad7a-466e-9a6c-ed7f557c386a\") " Oct 30 17:12:31.016594 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:31.016541 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18d1b373-ad7a-466e-9a6c-ed7f557c386a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18d1b373-ad7a-466e-9a6c-ed7f557c386a-maas-api-openshift-ca\") pod \"18d1b373-ad7a-466e-9a6c-ed7f557c386a\" (UID: \"18d1b373-ad7a-466e-9a6c-ed7f557c386a\") " Oct 30 17:12:31.017003 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:31.016978 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18d1b373-ad7a-466e-9a6c-ed7f557c386a-maas-api-openshift-ca" pod "18d1b373-ad7a-466e-9a6c-ed7f557c386a" (UID: "18d1b373-ad7a-466e-9a6c-ed7f557c386a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:12:31.017128 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:31.017108 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18d1b373-ad7a-466e-9a6c-ed7f557c386a-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:12:31.017238 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:31.017130 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18d1b373-ad7a-466e-9a6c-ed7f557c386a-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:12:31.018797 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:31.018773 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18d1b373-ad7a-466e-9a6c-ed7f557c386a-kube-api-access-dtbpl" pod "18d1b373-ad7a-466e-9a6c-ed7f557c386a" (UID: "18d1b373-ad7a-466e-9a6c-ed7f557c386a"). InnerVolumeSpecName "kube-api-access-dtbpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:12:31.118135 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:31.118099 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtbpl\" (UniqueName: \"kubernetes.io/projected/18d1b373-ad7a-466e-9a6c-ed7f557c386a-kube-api-access-dtbpl\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:12:31.718249 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:31.718221 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ebeb6e1-qfdc7" Oct 30 17:12:31.718249 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:31.718232 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ebeb6e1-qfdc7" event={"ID":"18d1b373-ad7a-466e-9a6c-ed7f557c386a","Type":"ContainerDied","Data":"48da88a2b04448f8fd5a0ce21d6ab46f602e0443b1820abe564c2e06b27a713c"} Oct 30 17:12:31.718476 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:31.718284 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48da88a2b04448f8fd5a0ce21d6ab46f602e0443b1820abe564c2e06b27a713c" Oct 30 17:12:32.296641 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:32.296602 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16704da5-9m4g2"] Oct 30 17:12:32.297413 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:32.297396 4522 cpu_manager.go:414] "containerMap: removing container" podUID="073c8bab-6c64-4fa5-8075-80f71c994845" containerName="istio-proxy" Oct 30 17:12:32.297413 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:32.297413 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="073c8bab-6c64-4fa5-8075-80f71c994845" containerName="istio-proxy" Oct 30 17:12:32.297486 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:32.297427 4522 cpu_manager.go:414] "containerMap: removing container" podUID="18d1b373-ad7a-466e-9a6c-ed7f557c386a" containerName="revoke-keys" Oct 30 17:12:32.297486 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:32.297435 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18d1b373-ad7a-466e-9a6c-ed7f557c386a" containerName="revoke-keys" Oct 30 17:12:32.297486 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:32.297443 4522 cpu_manager.go:414] "containerMap: removing container" podUID="073c8bab-6c64-4fa5-8075-80f71c994845" containerName="istio-proxy" Oct 30 17:12:32.297486 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:32.297447 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="073c8bab-6c64-4fa5-8075-80f71c994845" containerName="istio-proxy" Oct 30 17:12:32.297610 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:32.297544 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="18d1b373-ad7a-466e-9a6c-ed7f557c386a" containerName="revoke-keys" Oct 30 17:12:32.297610 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:32.297554 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="073c8bab-6c64-4fa5-8075-80f71c994845" containerName="istio-proxy" Oct 30 17:12:32.300197 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:32.300179 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16704da5-9m4g2" Oct 30 17:12:32.317930 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:32.317911 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:12:32.339071 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:32.339035 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16704da5-9m4g2"] Oct 30 17:12:32.339996 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:32.339973 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46502a59-2f4b-4dbe-aef2-196aa1be6893-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16704da5-9m4g2\" (UID: \"46502a59-2f4b-4dbe-aef2-196aa1be6893\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16704da5-9m4g2" Oct 30 17:12:32.340137 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:32.340114 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wxdv\" (UniqueName: \"kubernetes.io/projected/46502a59-2f4b-4dbe-aef2-196aa1be6893-kube-api-access-6wxdv\") pod \"maas-api-revoke-sub-16704da5-9m4g2\" (UID: \"46502a59-2f4b-4dbe-aef2-196aa1be6893\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16704da5-9m4g2" Oct 30 17:12:32.340407 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:32.340389 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46502a59-2f4b-4dbe-aef2-196aa1be6893-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16704da5-9m4g2\" (UID: \"46502a59-2f4b-4dbe-aef2-196aa1be6893\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16704da5-9m4g2" Oct 30 17:12:32.418034 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:32.418003 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:12:32.441909 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:32.441876 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46502a59-2f4b-4dbe-aef2-196aa1be6893-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16704da5-9m4g2\" (UID: \"46502a59-2f4b-4dbe-aef2-196aa1be6893\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16704da5-9m4g2" Oct 30 17:12:32.442054 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:32.441978 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46502a59-2f4b-4dbe-aef2-196aa1be6893-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16704da5-9m4g2\" (UID: \"46502a59-2f4b-4dbe-aef2-196aa1be6893\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16704da5-9m4g2" Oct 30 17:12:32.442099 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:32.442073 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wxdv\" (UniqueName: \"kubernetes.io/projected/46502a59-2f4b-4dbe-aef2-196aa1be6893-kube-api-access-6wxdv\") pod \"maas-api-revoke-sub-16704da5-9m4g2\" (UID: \"46502a59-2f4b-4dbe-aef2-196aa1be6893\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16704da5-9m4g2" Oct 30 17:12:32.442649 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:32.442625 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46502a59-2f4b-4dbe-aef2-196aa1be6893-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16704da5-9m4g2\" (UID: \"46502a59-2f4b-4dbe-aef2-196aa1be6893\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16704da5-9m4g2" Oct 30 17:12:32.444397 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:32.444375 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46502a59-2f4b-4dbe-aef2-196aa1be6893-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16704da5-9m4g2\" (UID: \"46502a59-2f4b-4dbe-aef2-196aa1be6893\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16704da5-9m4g2" Oct 30 17:12:32.469011 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:32.468983 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wxdv\" (UniqueName: \"kubernetes.io/projected/46502a59-2f4b-4dbe-aef2-196aa1be6893-kube-api-access-6wxdv\") pod \"maas-api-revoke-sub-16704da5-9m4g2\" (UID: \"46502a59-2f4b-4dbe-aef2-196aa1be6893\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16704da5-9m4g2" Oct 30 17:12:32.613984 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:32.613906 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16704da5-9m4g2" Oct 30 17:12:32.920043 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:12:32.920006 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46502a59_2f4b_4dbe_aef2_196aa1be6893.slice/crio-a322f1890330dbbee82b5120bc67980d0b5187c1e140f96495dbffb81a4f5740 WatchSource:0}: Error finding container a322f1890330dbbee82b5120bc67980d0b5187c1e140f96495dbffb81a4f5740: Status 404 returned error can't find the container with id a322f1890330dbbee82b5120bc67980d0b5187c1e140f96495dbffb81a4f5740 Oct 30 17:12:32.922685 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:32.922661 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16704da5-9m4g2"] Oct 30 17:12:33.594513 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:33.594419 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-362bbf-85765567b7-kqfrd"] Oct 30 17:12:33.739918 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:33.739885 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16704da5-9m4g2" event={"ID":"46502a59-2f4b-4dbe-aef2-196aa1be6893","Type":"ContainerStarted","Data":"3a213048b12519db261e4a792b0ba9c8ea3111371369a560000c4068c193e8d4"} Oct 30 17:12:33.740096 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:33.739936 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16704da5-9m4g2" event={"ID":"46502a59-2f4b-4dbe-aef2-196aa1be6893","Type":"ContainerStarted","Data":"a322f1890330dbbee82b5120bc67980d0b5187c1e140f96495dbffb81a4f5740"} Oct 30 17:12:33.789773 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:33.789722 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16704da5-9m4g2" podStartSLOduration=1.7897087969999999 podStartE2EDuration="1.789708797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:12:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:33.785898766 +0000 UTC m=+1527.786653692" watchObservedRunningTime="2026-10-30 17:12:33.789708797 +0000 UTC m=+1527.790463699" Oct 30 17:12:34.748769 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:34.748734 4522 generic.go:356] "Generic (PLEG): container finished" podID="46502a59-2f4b-4dbe-aef2-196aa1be6893" containerID="3a213048b12519db261e4a792b0ba9c8ea3111371369a560000c4068c193e8d4" exitCode=0 Oct 30 17:12:34.749221 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:34.748782 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16704da5-9m4g2" event={"ID":"46502a59-2f4b-4dbe-aef2-196aa1be6893","Type":"ContainerDied","Data":"3a213048b12519db261e4a792b0ba9c8ea3111371369a560000c4068c193e8d4"} Oct 30 17:12:35.927069 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:35.927039 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16704da5-9m4g2" Oct 30 17:12:36.001433 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:36.001402 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46502a59-2f4b-4dbe-aef2-196aa1be6893-kube-api-access-6wxdv\" (UniqueName: \"kubernetes.io/projected/46502a59-2f4b-4dbe-aef2-196aa1be6893-kube-api-access-6wxdv\") pod \"46502a59-2f4b-4dbe-aef2-196aa1be6893\" (UID: \"46502a59-2f4b-4dbe-aef2-196aa1be6893\") " Oct 30 17:12:36.001629 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:36.001450 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46502a59-2f4b-4dbe-aef2-196aa1be6893-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46502a59-2f4b-4dbe-aef2-196aa1be6893-maas-api-openshift-ca\") pod \"46502a59-2f4b-4dbe-aef2-196aa1be6893\" (UID: \"46502a59-2f4b-4dbe-aef2-196aa1be6893\") " Oct 30 17:12:36.001629 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:36.001479 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46502a59-2f4b-4dbe-aef2-196aa1be6893-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46502a59-2f4b-4dbe-aef2-196aa1be6893-maas-api-xks-ca\") pod \"46502a59-2f4b-4dbe-aef2-196aa1be6893\" (UID: \"46502a59-2f4b-4dbe-aef2-196aa1be6893\") " Oct 30 17:12:36.001915 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:36.001890 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46502a59-2f4b-4dbe-aef2-196aa1be6893-maas-api-openshift-ca" pod "46502a59-2f4b-4dbe-aef2-196aa1be6893" (UID: "46502a59-2f4b-4dbe-aef2-196aa1be6893"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:12:36.002407 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:36.002376 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46502a59-2f4b-4dbe-aef2-196aa1be6893-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:12:36.004259 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:36.004222 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46502a59-2f4b-4dbe-aef2-196aa1be6893-maas-api-xks-ca" pod "46502a59-2f4b-4dbe-aef2-196aa1be6893" (UID: "46502a59-2f4b-4dbe-aef2-196aa1be6893"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:12:36.004400 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:36.004374 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46502a59-2f4b-4dbe-aef2-196aa1be6893-kube-api-access-6wxdv" pod "46502a59-2f4b-4dbe-aef2-196aa1be6893" (UID: "46502a59-2f4b-4dbe-aef2-196aa1be6893"). InnerVolumeSpecName "kube-api-access-6wxdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:12:36.103431 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:36.103395 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wxdv\" (UniqueName: \"kubernetes.io/projected/46502a59-2f4b-4dbe-aef2-196aa1be6893-kube-api-access-6wxdv\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:12:36.103431 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:36.103425 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46502a59-2f4b-4dbe-aef2-196aa1be6893-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:12:36.765969 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:36.765927 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16704da5-9m4g2" Oct 30 17:12:36.765969 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:36.765957 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16704da5-9m4g2" event={"ID":"46502a59-2f4b-4dbe-aef2-196aa1be6893","Type":"ContainerDied","Data":"a322f1890330dbbee82b5120bc67980d0b5187c1e140f96495dbffb81a4f5740"} Oct 30 17:12:36.766202 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:36.766050 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a322f1890330dbbee82b5120bc67980d0b5187c1e140f96495dbffb81a4f5740" Oct 30 17:12:38.595745 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:38.595700 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-362bbf-85765567b7-kqfrd" podUID="8da017e0-da57-4687-96cc-505f32038b61" containerName="payload-pre-processing" containerID="cri-o://9d232138584436b3b41176d2e7988d2e2e60dc5b6477e722eaa8bc1792d215f0" gracePeriod=25 Oct 30 17:12:39.004574 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:39.000019 4522 generic.go:356] "Generic (PLEG): container finished" podID="8da017e0-da57-4687-96cc-505f32038b61" containerID="9d232138584436b3b41176d2e7988d2e2e60dc5b6477e722eaa8bc1792d215f0" exitCode=0 Oct 30 17:12:39.004574 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:39.000241 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-362bbf-85765567b7-kqfrd" event={"ID":"8da017e0-da57-4687-96cc-505f32038b61","Type":"ContainerDied","Data":"9d232138584436b3b41176d2e7988d2e2e60dc5b6477e722eaa8bc1792d215f0"} Oct 30 17:12:39.124492 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:39.124449 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-362bbf-85765567b7-kqfrd" Oct 30 17:12:39.194338 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:39.193641 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8da017e0-da57-4687-96cc-505f32038b61-kube-api-access-hbxcm\" (UniqueName: \"kubernetes.io/projected/8da017e0-da57-4687-96cc-505f32038b61-kube-api-access-hbxcm\") pod \"8da017e0-da57-4687-96cc-505f32038b61\" (UID: \"8da017e0-da57-4687-96cc-505f32038b61\") " Oct 30 17:12:39.194338 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:39.193940 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8da017e0-da57-4687-96cc-505f32038b61-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8da017e0-da57-4687-96cc-505f32038b61-plugins-config-volume\") pod \"8da017e0-da57-4687-96cc-505f32038b61\" (UID: \"8da017e0-da57-4687-96cc-505f32038b61\") " Oct 30 17:12:39.195817 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:39.195507 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8da017e0-da57-4687-96cc-505f32038b61-plugins-config-volume" pod "8da017e0-da57-4687-96cc-505f32038b61" (UID: "8da017e0-da57-4687-96cc-505f32038b61"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:12:39.205429 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:39.205237 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8da017e0-da57-4687-96cc-505f32038b61-kube-api-access-hbxcm" pod "8da017e0-da57-4687-96cc-505f32038b61" (UID: "8da017e0-da57-4687-96cc-505f32038b61"). InnerVolumeSpecName "kube-api-access-hbxcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:12:39.296889 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:39.296809 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8da017e0-da57-4687-96cc-505f32038b61-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:12:39.296889 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:39.296837 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbxcm\" (UniqueName: \"kubernetes.io/projected/8da017e0-da57-4687-96cc-505f32038b61-kube-api-access-hbxcm\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:12:39.870941 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:39.870898 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5"] Oct 30 17:12:39.871475 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:39.871248 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5" podUID="6e1c3c74-139f-4299-97c4-4609e0b9e030" containerName="istio-proxy" containerID="cri-o://a60d5b915fec64c6758b8ef572674b5753e55fc85993d77537df3276db0a3c87" gracePeriod=30 Oct 30 17:12:39.886134 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:39.886100 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq"] Oct 30 17:12:39.886441 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:39.886410 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq" podUID="46027d41-98ad-409d-a66f-4bc050bc2648" containerName="istio-proxy" containerID="cri-o://c09766a2cb5828462c8d758dc680c4b58133409e57b861a19da7f333f2604862" gracePeriod=30 Oct 30 17:12:39.895647 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:39.895610 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6"] Oct 30 17:12:39.895878 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:39.895852 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6" podUID="c9c903cd-d636-4311-9afe-e51e4cc3db8e" containerName="istio-proxy" containerID="cri-o://73b47ad2ff23e446c1c18cea703cd9198beb3a34408083ae602202c9a326d39f" gracePeriod=30 Oct 30 17:12:39.909491 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:39.909463 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m"] Oct 30 17:12:39.909697 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:39.909675 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m" podUID="da926b76-e144-4a03-bb81-67d2effff0a6" containerName="istio-proxy" containerID="cri-o://b10a9c1634fcda6bb2882eb293c4a023244cb460c922442454335a65a65d90c1" gracePeriod=30 Oct 30 17:12:40.009583 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:40.009552 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-362bbf-85765567b7-kqfrd" Oct 30 17:12:40.009782 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:40.009589 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-362bbf-85765567b7-kqfrd" event={"ID":"8da017e0-da57-4687-96cc-505f32038b61","Type":"ContainerDied","Data":"9fad12ccf903cc01702cf2f4d26084fc52ff608fec0f60ee9abdd7ec135c1924"} Oct 30 17:12:40.009782 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:40.009656 4522 scope.go:122] "RemoveContainer" containerID="9d232138584436b3b41176d2e7988d2e2e60dc5b6477e722eaa8bc1792d215f0" Oct 30 17:12:40.067230 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:40.067154 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-409387-vhhd7"] Oct 30 17:12:40.081676 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:40.080104 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-409387-vhhd7"] Oct 30 17:12:40.149789 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:40.149697 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0e64808-7b16-4675-a2d6-c6109477d027" path="/var/lib/kubelet/pods/a0e64808-7b16-4675-a2d6-c6109477d027/volumes" Oct 30 17:12:40.150196 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:40.150178 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-362bbf-85765567b7-kqfrd"] Oct 30 17:12:40.150246 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:40.150221 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-362bbf-85765567b7-kqfrd"] Oct 30 17:12:42.001166 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:42.001129 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b2c3fb-7c87c57c7c-pz9ls"] Oct 30 17:12:42.002247 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:42.002229 4522 cpu_manager.go:414] "containerMap: removing container" podUID="46502a59-2f4b-4dbe-aef2-196aa1be6893" containerName="revoke-keys" Oct 30 17:12:42.002372 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:42.002249 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46502a59-2f4b-4dbe-aef2-196aa1be6893" containerName="revoke-keys" Oct 30 17:12:42.002372 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:42.002273 4522 cpu_manager.go:414] "containerMap: removing container" podUID="8da017e0-da57-4687-96cc-505f32038b61" containerName="payload-pre-processing" Oct 30 17:12:42.002372 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:42.002281 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8da017e0-da57-4687-96cc-505f32038b61" containerName="payload-pre-processing" Oct 30 17:12:42.002503 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:42.002461 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="46502a59-2f4b-4dbe-aef2-196aa1be6893" containerName="revoke-keys" Oct 30 17:12:42.002503 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:42.002475 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="8da017e0-da57-4687-96cc-505f32038b61" containerName="payload-pre-processing" Oct 30 17:12:42.002503 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:42.002486 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="073c8bab-6c64-4fa5-8075-80f71c994845" containerName="istio-proxy" Oct 30 17:12:42.006579 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:42.006557 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b2c3fb-7c87c57c7c-pz9ls" Oct 30 17:12:42.034104 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:42.034071 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b2c3fb-7c87c57c7c-pz9ls"] Oct 30 17:12:42.035150 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:42.035125 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-b2c3fb-serving-cert\"" Oct 30 17:12:42.046986 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:42.046956 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96-metrics-certs\") pod \"maas-api-e2e-api-b-b2c3fb-7c87c57c7c-pz9ls\" (UID: \"60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b2c3fb-7c87c57c7c-pz9ls" Oct 30 17:12:42.047134 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:42.047103 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d6dw\" (UniqueName: \"kubernetes.io/projected/60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96-kube-api-access-9d6dw\") pod \"maas-api-e2e-api-b-b2c3fb-7c87c57c7c-pz9ls\" (UID: \"60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b2c3fb-7c87c57c7c-pz9ls" Oct 30 17:12:42.047290 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:42.047273 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96-maas-api-tls\") pod \"maas-api-e2e-api-b-b2c3fb-7c87c57c7c-pz9ls\" (UID: \"60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b2c3fb-7c87c57c7c-pz9ls" Oct 30 17:12:42.047444 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:42.047427 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96-openshift-service-ca\") pod \"maas-api-e2e-api-b-b2c3fb-7c87c57c7c-pz9ls\" (UID: \"60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b2c3fb-7c87c57c7c-pz9ls" Oct 30 17:12:42.147547 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:42.147508 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8da017e0-da57-4687-96cc-505f32038b61" path="/var/lib/kubelet/pods/8da017e0-da57-4687-96cc-505f32038b61/volumes" Oct 30 17:12:42.148805 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:42.148775 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96-maas-api-tls\") pod \"maas-api-e2e-api-b-b2c3fb-7c87c57c7c-pz9ls\" (UID: \"60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b2c3fb-7c87c57c7c-pz9ls" Oct 30 17:12:42.148943 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:42.148829 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96-openshift-service-ca\") pod \"maas-api-e2e-api-b-b2c3fb-7c87c57c7c-pz9ls\" (UID: \"60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b2c3fb-7c87c57c7c-pz9ls" Oct 30 17:12:42.148995 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:42.148983 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96-metrics-certs\") pod \"maas-api-e2e-api-b-b2c3fb-7c87c57c7c-pz9ls\" (UID: \"60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b2c3fb-7c87c57c7c-pz9ls" Oct 30 17:12:42.149066 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:42.149053 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d6dw\" (UniqueName: \"kubernetes.io/projected/60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96-kube-api-access-9d6dw\") pod \"maas-api-e2e-api-b-b2c3fb-7c87c57c7c-pz9ls\" (UID: \"60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b2c3fb-7c87c57c7c-pz9ls" Oct 30 17:12:42.149759 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:42.149737 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96-openshift-service-ca\") pod \"maas-api-e2e-api-b-b2c3fb-7c87c57c7c-pz9ls\" (UID: \"60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b2c3fb-7c87c57c7c-pz9ls" Oct 30 17:12:42.151446 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:42.151427 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96-maas-api-tls\") pod \"maas-api-e2e-api-b-b2c3fb-7c87c57c7c-pz9ls\" (UID: \"60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b2c3fb-7c87c57c7c-pz9ls" Oct 30 17:12:42.151604 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:42.151458 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96-metrics-certs\") pod \"maas-api-e2e-api-b-b2c3fb-7c87c57c7c-pz9ls\" (UID: \"60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b2c3fb-7c87c57c7c-pz9ls" Oct 30 17:12:42.175912 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:42.175879 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d6dw\" (UniqueName: \"kubernetes.io/projected/60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96-kube-api-access-9d6dw\") pod \"maas-api-e2e-api-b-b2c3fb-7c87c57c7c-pz9ls\" (UID: \"60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b2c3fb-7c87c57c7c-pz9ls" Oct 30 17:12:42.329102 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:42.329016 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b2c3fb-7c87c57c7c-pz9ls" Oct 30 17:12:42.649239 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:12:42.649208 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60a8f0d7_aea8_48ff_beb3_c8c0a7a8cf96.slice/crio-adca166344b239a1178badd7c393470a086ab5bba15250ec3580e2dd4564e11d WatchSource:0}: Error finding container adca166344b239a1178badd7c393470a086ab5bba15250ec3580e2dd4564e11d: Status 404 returned error can't find the container with id adca166344b239a1178badd7c393470a086ab5bba15250ec3580e2dd4564e11d Oct 30 17:12:42.650913 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:42.650886 4522 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:12:42.652361 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:42.652337 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b2c3fb-7c87c57c7c-pz9ls"] Oct 30 17:12:42.950533 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:12:42.950445 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:12:43.047689 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:43.047655 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b2c3fb-7c87c57c7c-pz9ls" event={"ID":"60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96","Type":"ContainerStarted","Data":"adca166344b239a1178badd7c393470a086ab5bba15250ec3580e2dd4564e11d"} Oct 30 17:12:44.057388 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:44.057345 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b2c3fb-7c87c57c7c-pz9ls" event={"ID":"60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96","Type":"ContainerStarted","Data":"8e00516e6bc728fe10a6c71fe46b6b1f3d35a3d376864451f9522c595f4ef903"} Oct 30 17:12:44.057804 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:44.057423 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b2c3fb-7c87c57c7c-pz9ls" Oct 30 17:12:44.124871 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:44.124803 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b2c3fb-7c87c57c7c-pz9ls" podStartSLOduration=2.673227037 podStartE2EDuration="3.124771981s" totalImagesPullingTime="451.544944ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:12:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:44.10880707 +0000 UTC m=+1538.109561972" watchObservedRunningTime="2026-10-30 17:12:44.124771981 +0000 UTC m=+1538.125526883" Oct 30 17:12:45.078408 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.076967 4522 generic.go:356] "Generic (PLEG): container finished" podID="6e1c3c74-139f-4299-97c4-4609e0b9e030" containerID="a60d5b915fec64c6758b8ef572674b5753e55fc85993d77537df3276db0a3c87" exitCode=0 Oct 30 17:12:45.078408 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.077134 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5" event={"ID":"6e1c3c74-139f-4299-97c4-4609e0b9e030","Type":"ContainerDied","Data":"a60d5b915fec64c6758b8ef572674b5753e55fc85993d77537df3276db0a3c87"} Oct 30 17:12:45.085147 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.079339 4522 generic.go:356] "Generic (PLEG): container finished" podID="c9c903cd-d636-4311-9afe-e51e4cc3db8e" containerID="73b47ad2ff23e446c1c18cea703cd9198beb3a34408083ae602202c9a326d39f" exitCode=0 Oct 30 17:12:45.085147 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.079438 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6" event={"ID":"c9c903cd-d636-4311-9afe-e51e4cc3db8e","Type":"ContainerDied","Data":"73b47ad2ff23e446c1c18cea703cd9198beb3a34408083ae602202c9a326d39f"} Oct 30 17:12:45.085147 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.081723 4522 generic.go:356] "Generic (PLEG): container finished" podID="da926b76-e144-4a03-bb81-67d2effff0a6" containerID="b10a9c1634fcda6bb2882eb293c4a023244cb460c922442454335a65a65d90c1" exitCode=0 Oct 30 17:12:45.085147 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.081889 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m" event={"ID":"da926b76-e144-4a03-bb81-67d2effff0a6","Type":"ContainerDied","Data":"b10a9c1634fcda6bb2882eb293c4a023244cb460c922442454335a65a65d90c1"} Oct 30 17:12:45.085147 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.083570 4522 generic.go:356] "Generic (PLEG): container finished" podID="46027d41-98ad-409d-a66f-4bc050bc2648" containerID="c09766a2cb5828462c8d758dc680c4b58133409e57b861a19da7f333f2604862" exitCode=0 Oct 30 17:12:45.085147 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.084409 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq" event={"ID":"46027d41-98ad-409d-a66f-4bc050bc2648","Type":"ContainerDied","Data":"c09766a2cb5828462c8d758dc680c4b58133409e57b861a19da7f333f2604862"} Oct 30 17:12:45.256282 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.255895 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq" Oct 30 17:12:45.261386 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.261275 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5" Oct 30 17:12:45.305472 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.305417 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m" Oct 30 17:12:45.309837 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.309813 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6" Oct 30 17:12:45.346821 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.346788 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46027d41-98ad-409d-a66f-4bc050bc2648-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46027d41-98ad-409d-a66f-4bc050bc2648-istio-envoy\") pod \"46027d41-98ad-409d-a66f-4bc050bc2648\" (UID: \"46027d41-98ad-409d-a66f-4bc050bc2648\") " Oct 30 17:12:45.347013 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.346873 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46027d41-98ad-409d-a66f-4bc050bc2648-istio-token\" (UniqueName: \"kubernetes.io/projected/46027d41-98ad-409d-a66f-4bc050bc2648-istio-token\") pod \"46027d41-98ad-409d-a66f-4bc050bc2648\" (UID: \"46027d41-98ad-409d-a66f-4bc050bc2648\") " Oct 30 17:12:45.347013 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.347003 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46027d41-98ad-409d-a66f-4bc050bc2648-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46027d41-98ad-409d-a66f-4bc050bc2648-workload-socket\") pod \"46027d41-98ad-409d-a66f-4bc050bc2648\" (UID: \"46027d41-98ad-409d-a66f-4bc050bc2648\") " Oct 30 17:12:45.347133 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.347032 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46027d41-98ad-409d-a66f-4bc050bc2648-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46027d41-98ad-409d-a66f-4bc050bc2648-istio-podinfo\") pod \"46027d41-98ad-409d-a66f-4bc050bc2648\" (UID: \"46027d41-98ad-409d-a66f-4bc050bc2648\") " Oct 30 17:12:45.347133 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.347111 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46027d41-98ad-409d-a66f-4bc050bc2648-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46027d41-98ad-409d-a66f-4bc050bc2648-istiod-ca-cert\") pod \"46027d41-98ad-409d-a66f-4bc050bc2648\" (UID: \"46027d41-98ad-409d-a66f-4bc050bc2648\") " Oct 30 17:12:45.347232 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.347201 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46027d41-98ad-409d-a66f-4bc050bc2648-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46027d41-98ad-409d-a66f-4bc050bc2648-credential-socket\") pod \"46027d41-98ad-409d-a66f-4bc050bc2648\" (UID: \"46027d41-98ad-409d-a66f-4bc050bc2648\") " Oct 30 17:12:45.347282 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.347271 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46027d41-98ad-409d-a66f-4bc050bc2648-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46027d41-98ad-409d-a66f-4bc050bc2648-istio-data\") pod \"46027d41-98ad-409d-a66f-4bc050bc2648\" (UID: \"46027d41-98ad-409d-a66f-4bc050bc2648\") " Oct 30 17:12:45.347418 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.347402 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46027d41-98ad-409d-a66f-4bc050bc2648-kube-api-access-6nnf2\" (UniqueName: \"kubernetes.io/projected/46027d41-98ad-409d-a66f-4bc050bc2648-kube-api-access-6nnf2\") pod \"46027d41-98ad-409d-a66f-4bc050bc2648\" (UID: \"46027d41-98ad-409d-a66f-4bc050bc2648\") " Oct 30 17:12:45.347515 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.347501 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46027d41-98ad-409d-a66f-4bc050bc2648-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46027d41-98ad-409d-a66f-4bc050bc2648-workload-certs\") pod \"46027d41-98ad-409d-a66f-4bc050bc2648\" (UID: \"46027d41-98ad-409d-a66f-4bc050bc2648\") " Oct 30 17:12:45.347849 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.347819 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46027d41-98ad-409d-a66f-4bc050bc2648-credential-socket" pod "46027d41-98ad-409d-a66f-4bc050bc2648" (UID: "46027d41-98ad-409d-a66f-4bc050bc2648"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:45.348041 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.348018 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46027d41-98ad-409d-a66f-4bc050bc2648-istiod-ca-cert" pod "46027d41-98ad-409d-a66f-4bc050bc2648" (UID: "46027d41-98ad-409d-a66f-4bc050bc2648"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:12:45.348132 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.348029 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46027d41-98ad-409d-a66f-4bc050bc2648-workload-certs" pod "46027d41-98ad-409d-a66f-4bc050bc2648" (UID: "46027d41-98ad-409d-a66f-4bc050bc2648"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:45.348391 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.348293 4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46027d41-98ad-409d-a66f-4bc050bc2648-workload-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.348391 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.348339 4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46027d41-98ad-409d-a66f-4bc050bc2648-istiod-ca-cert\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.348391 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.348353 4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46027d41-98ad-409d-a66f-4bc050bc2648-credential-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.348391 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.348364 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46027d41-98ad-409d-a66f-4bc050bc2648-workload-socket" pod "46027d41-98ad-409d-a66f-4bc050bc2648" (UID: "46027d41-98ad-409d-a66f-4bc050bc2648"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:45.348946 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.348920 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46027d41-98ad-409d-a66f-4bc050bc2648-istio-data" pod "46027d41-98ad-409d-a66f-4bc050bc2648" (UID: "46027d41-98ad-409d-a66f-4bc050bc2648"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:45.352696 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.352668 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46027d41-98ad-409d-a66f-4bc050bc2648-istio-podinfo" pod "46027d41-98ad-409d-a66f-4bc050bc2648" (UID: "46027d41-98ad-409d-a66f-4bc050bc2648"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:12:45.352810 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.352700 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46027d41-98ad-409d-a66f-4bc050bc2648-istio-envoy" pod "46027d41-98ad-409d-a66f-4bc050bc2648" (UID: "46027d41-98ad-409d-a66f-4bc050bc2648"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:45.352810 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.352744 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46027d41-98ad-409d-a66f-4bc050bc2648-kube-api-access-6nnf2" pod "46027d41-98ad-409d-a66f-4bc050bc2648" (UID: "46027d41-98ad-409d-a66f-4bc050bc2648"). InnerVolumeSpecName "kube-api-access-6nnf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:12:45.353870 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.353852 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46027d41-98ad-409d-a66f-4bc050bc2648-istio-token" pod "46027d41-98ad-409d-a66f-4bc050bc2648" (UID: "46027d41-98ad-409d-a66f-4bc050bc2648"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:12:45.449598 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.449501 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6e1c3c74-139f-4299-97c4-4609e0b9e030-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e1c3c74-139f-4299-97c4-4609e0b9e030-istio-podinfo\") pod \"6e1c3c74-139f-4299-97c4-4609e0b9e030\" (UID: \"6e1c3c74-139f-4299-97c4-4609e0b9e030\") " Oct 30 17:12:45.449598 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.449586 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e1c3c74-139f-4299-97c4-4609e0b9e030-istio-token\" (UniqueName: \"kubernetes.io/projected/6e1c3c74-139f-4299-97c4-4609e0b9e030-istio-token\") pod \"6e1c3c74-139f-4299-97c4-4609e0b9e030\" (UID: \"6e1c3c74-139f-4299-97c4-4609e0b9e030\") " Oct 30 17:12:45.449869 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.449621 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e1c3c74-139f-4299-97c4-4609e0b9e030-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e1c3c74-139f-4299-97c4-4609e0b9e030-istio-data\") pod \"6e1c3c74-139f-4299-97c4-4609e0b9e030\" (UID: \"6e1c3c74-139f-4299-97c4-4609e0b9e030\") " Oct 30 17:12:45.449869 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.449676 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da926b76-e144-4a03-bb81-67d2effff0a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da926b76-e144-4a03-bb81-67d2effff0a6-istio-envoy\") pod \"da926b76-e144-4a03-bb81-67d2effff0a6\" (UID: \"da926b76-e144-4a03-bb81-67d2effff0a6\") " Oct 30 17:12:45.449869 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.449750 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e1c3c74-139f-4299-97c4-4609e0b9e030-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e1c3c74-139f-4299-97c4-4609e0b9e030-credential-socket\") pod \"6e1c3c74-139f-4299-97c4-4609e0b9e030\" (UID: \"6e1c3c74-139f-4299-97c4-4609e0b9e030\") " Oct 30 17:12:45.449869 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.449796 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da926b76-e144-4a03-bb81-67d2effff0a6-istio-token\" (UniqueName: \"kubernetes.io/projected/da926b76-e144-4a03-bb81-67d2effff0a6-istio-token\") pod \"da926b76-e144-4a03-bb81-67d2effff0a6\" (UID: \"da926b76-e144-4a03-bb81-67d2effff0a6\") " Oct 30 17:12:45.450080 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.449898 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c9c903cd-d636-4311-9afe-e51e4cc3db8e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9c903cd-d636-4311-9afe-e51e4cc3db8e-istio-podinfo\") pod \"c9c903cd-d636-4311-9afe-e51e4cc3db8e\" (UID: \"c9c903cd-d636-4311-9afe-e51e4cc3db8e\") " Oct 30 17:12:45.450080 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.449967 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9c903cd-d636-4311-9afe-e51e4cc3db8e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9c903cd-d636-4311-9afe-e51e4cc3db8e-workload-socket\") pod \"c9c903cd-d636-4311-9afe-e51e4cc3db8e\" (UID: \"c9c903cd-d636-4311-9afe-e51e4cc3db8e\") " Oct 30 17:12:45.450080 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.450009 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da926b76-e144-4a03-bb81-67d2effff0a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da926b76-e144-4a03-bb81-67d2effff0a6-credential-socket\") pod \"da926b76-e144-4a03-bb81-67d2effff0a6\" (UID: \"da926b76-e144-4a03-bb81-67d2effff0a6\") " Oct 30 17:12:45.450080 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.450057 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9c903cd-d636-4311-9afe-e51e4cc3db8e-kube-api-access-zd42w\" (UniqueName: \"kubernetes.io/projected/c9c903cd-d636-4311-9afe-e51e4cc3db8e-kube-api-access-zd42w\") pod \"c9c903cd-d636-4311-9afe-e51e4cc3db8e\" (UID: \"c9c903cd-d636-4311-9afe-e51e4cc3db8e\") " Oct 30 17:12:45.450291 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.450086 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da926b76-e144-4a03-bb81-67d2effff0a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da926b76-e144-4a03-bb81-67d2effff0a6-istio-data\") pod \"da926b76-e144-4a03-bb81-67d2effff0a6\" (UID: \"da926b76-e144-4a03-bb81-67d2effff0a6\") " Oct 30 17:12:45.450291 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.450123 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e1c3c74-139f-4299-97c4-4609e0b9e030-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e1c3c74-139f-4299-97c4-4609e0b9e030-istio-envoy\") pod \"6e1c3c74-139f-4299-97c4-4609e0b9e030\" (UID: \"6e1c3c74-139f-4299-97c4-4609e0b9e030\") " Oct 30 17:12:45.450291 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.450163 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9c903cd-d636-4311-9afe-e51e4cc3db8e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9c903cd-d636-4311-9afe-e51e4cc3db8e-istiod-ca-cert\") pod \"c9c903cd-d636-4311-9afe-e51e4cc3db8e\" (UID: \"c9c903cd-d636-4311-9afe-e51e4cc3db8e\") " Oct 30 17:12:45.450291 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.450233 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da926b76-e144-4a03-bb81-67d2effff0a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da926b76-e144-4a03-bb81-67d2effff0a6-workload-certs\") pod \"da926b76-e144-4a03-bb81-67d2effff0a6\" (UID: \"da926b76-e144-4a03-bb81-67d2effff0a6\") " Oct 30 17:12:45.450540 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.450344 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da926b76-e144-4a03-bb81-67d2effff0a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da926b76-e144-4a03-bb81-67d2effff0a6-workload-socket\") pod \"da926b76-e144-4a03-bb81-67d2effff0a6\" (UID: \"da926b76-e144-4a03-bb81-67d2effff0a6\") " Oct 30 17:12:45.450540 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.450404 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da926b76-e144-4a03-bb81-67d2effff0a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da926b76-e144-4a03-bb81-67d2effff0a6-istio-podinfo\") pod \"da926b76-e144-4a03-bb81-67d2effff0a6\" (UID: \"da926b76-e144-4a03-bb81-67d2effff0a6\") " Oct 30 17:12:45.450540 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.450458 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da926b76-e144-4a03-bb81-67d2effff0a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da926b76-e144-4a03-bb81-67d2effff0a6-istiod-ca-cert\") pod \"da926b76-e144-4a03-bb81-67d2effff0a6\" (UID: \"da926b76-e144-4a03-bb81-67d2effff0a6\") " Oct 30 17:12:45.450540 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.450514 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9c903cd-d636-4311-9afe-e51e4cc3db8e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9c903cd-d636-4311-9afe-e51e4cc3db8e-istio-data\") pod \"c9c903cd-d636-4311-9afe-e51e4cc3db8e\" (UID: \"c9c903cd-d636-4311-9afe-e51e4cc3db8e\") " Oct 30 17:12:45.450748 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.450544 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9c903cd-d636-4311-9afe-e51e4cc3db8e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9c903cd-d636-4311-9afe-e51e4cc3db8e-credential-socket\") pod \"c9c903cd-d636-4311-9afe-e51e4cc3db8e\" (UID: \"c9c903cd-d636-4311-9afe-e51e4cc3db8e\") " Oct 30 17:12:45.450748 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.450627 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da926b76-e144-4a03-bb81-67d2effff0a6-kube-api-access-p4wb5\" (UniqueName: \"kubernetes.io/projected/da926b76-e144-4a03-bb81-67d2effff0a6-kube-api-access-p4wb5\") pod \"da926b76-e144-4a03-bb81-67d2effff0a6\" (UID: \"da926b76-e144-4a03-bb81-67d2effff0a6\") " Oct 30 17:12:45.450748 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.450675 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e1c3c74-139f-4299-97c4-4609e0b9e030-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e1c3c74-139f-4299-97c4-4609e0b9e030-workload-socket\") pod \"6e1c3c74-139f-4299-97c4-4609e0b9e030\" (UID: \"6e1c3c74-139f-4299-97c4-4609e0b9e030\") " Oct 30 17:12:45.450748 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.450722 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9c903cd-d636-4311-9afe-e51e4cc3db8e-istio-token\" (UniqueName: \"kubernetes.io/projected/c9c903cd-d636-4311-9afe-e51e4cc3db8e-istio-token\") pod \"c9c903cd-d636-4311-9afe-e51e4cc3db8e\" (UID: \"c9c903cd-d636-4311-9afe-e51e4cc3db8e\") " Oct 30 17:12:45.450968 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.450762 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9c903cd-d636-4311-9afe-e51e4cc3db8e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9c903cd-d636-4311-9afe-e51e4cc3db8e-istio-envoy\") pod \"c9c903cd-d636-4311-9afe-e51e4cc3db8e\" (UID: \"c9c903cd-d636-4311-9afe-e51e4cc3db8e\") " Oct 30 17:12:45.450968 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.450841 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e1c3c74-139f-4299-97c4-4609e0b9e030-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e1c3c74-139f-4299-97c4-4609e0b9e030-workload-certs\") pod \"6e1c3c74-139f-4299-97c4-4609e0b9e030\" (UID: \"6e1c3c74-139f-4299-97c4-4609e0b9e030\") " Oct 30 17:12:45.450968 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.450936 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9c903cd-d636-4311-9afe-e51e4cc3db8e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9c903cd-d636-4311-9afe-e51e4cc3db8e-workload-certs\") pod \"c9c903cd-d636-4311-9afe-e51e4cc3db8e\" (UID: \"c9c903cd-d636-4311-9afe-e51e4cc3db8e\") " Oct 30 17:12:45.451128 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.450977 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e1c3c74-139f-4299-97c4-4609e0b9e030-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e1c3c74-139f-4299-97c4-4609e0b9e030-istiod-ca-cert\") pod \"6e1c3c74-139f-4299-97c4-4609e0b9e030\" (UID: \"6e1c3c74-139f-4299-97c4-4609e0b9e030\") " Oct 30 17:12:45.451128 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.451025 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e1c3c74-139f-4299-97c4-4609e0b9e030-kube-api-access-gsb9w\" (UniqueName: \"kubernetes.io/projected/6e1c3c74-139f-4299-97c4-4609e0b9e030-kube-api-access-gsb9w\") pod \"6e1c3c74-139f-4299-97c4-4609e0b9e030\" (UID: \"6e1c3c74-139f-4299-97c4-4609e0b9e030\") " Oct 30 17:12:45.451966 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.451929 4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46027d41-98ad-409d-a66f-4bc050bc2648-istio-data\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.451966 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.451961 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nnf2\" (UniqueName: \"kubernetes.io/projected/46027d41-98ad-409d-a66f-4bc050bc2648-kube-api-access-6nnf2\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.452148 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.451976 4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46027d41-98ad-409d-a66f-4bc050bc2648-istio-envoy\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.452148 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.451992 4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46027d41-98ad-409d-a66f-4bc050bc2648-istio-token\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.452148 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.452006 4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46027d41-98ad-409d-a66f-4bc050bc2648-workload-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.452148 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.452019 4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46027d41-98ad-409d-a66f-4bc050bc2648-istio-podinfo\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.452485 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.452370 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da926b76-e144-4a03-bb81-67d2effff0a6-workload-certs" pod "da926b76-e144-4a03-bb81-67d2effff0a6" (UID: "da926b76-e144-4a03-bb81-67d2effff0a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:45.453066 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.453037 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da926b76-e144-4a03-bb81-67d2effff0a6-workload-socket" pod "da926b76-e144-4a03-bb81-67d2effff0a6" (UID: "da926b76-e144-4a03-bb81-67d2effff0a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:45.453208 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.453152 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9c903cd-d636-4311-9afe-e51e4cc3db8e-credential-socket" pod "c9c903cd-d636-4311-9afe-e51e4cc3db8e" (UID: "c9c903cd-d636-4311-9afe-e51e4cc3db8e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:45.453427 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.453398 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9c903cd-d636-4311-9afe-e51e4cc3db8e-istio-data" pod "c9c903cd-d636-4311-9afe-e51e4cc3db8e" (UID: "c9c903cd-d636-4311-9afe-e51e4cc3db8e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:45.453535 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.453463 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da926b76-e144-4a03-bb81-67d2effff0a6-istiod-ca-cert" pod "da926b76-e144-4a03-bb81-67d2effff0a6" (UID: "da926b76-e144-4a03-bb81-67d2effff0a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:12:45.453680 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.453657 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9c903cd-d636-4311-9afe-e51e4cc3db8e-workload-socket" pod "c9c903cd-d636-4311-9afe-e51e4cc3db8e" (UID: "c9c903cd-d636-4311-9afe-e51e4cc3db8e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:45.454109 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.453970 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9c903cd-d636-4311-9afe-e51e4cc3db8e-workload-certs" pod "c9c903cd-d636-4311-9afe-e51e4cc3db8e" (UID: "c9c903cd-d636-4311-9afe-e51e4cc3db8e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:45.454109 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.454024 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e1c3c74-139f-4299-97c4-4609e0b9e030-workload-certs" pod "6e1c3c74-139f-4299-97c4-4609e0b9e030" (UID: "6e1c3c74-139f-4299-97c4-4609e0b9e030"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:45.454573 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.454454 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e1c3c74-139f-4299-97c4-4609e0b9e030-istiod-ca-cert" pod "6e1c3c74-139f-4299-97c4-4609e0b9e030" (UID: "6e1c3c74-139f-4299-97c4-4609e0b9e030"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:12:45.455161 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.454845 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da926b76-e144-4a03-bb81-67d2effff0a6-istio-data" pod "da926b76-e144-4a03-bb81-67d2effff0a6" (UID: "da926b76-e144-4a03-bb81-67d2effff0a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:45.455670 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.455226 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e1c3c74-139f-4299-97c4-4609e0b9e030-workload-socket" pod "6e1c3c74-139f-4299-97c4-4609e0b9e030" (UID: "6e1c3c74-139f-4299-97c4-4609e0b9e030"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:45.455670 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.455260 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e1c3c74-139f-4299-97c4-4609e0b9e030-istio-data" pod "6e1c3c74-139f-4299-97c4-4609e0b9e030" (UID: "6e1c3c74-139f-4299-97c4-4609e0b9e030"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:45.455861 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.455835 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e1c3c74-139f-4299-97c4-4609e0b9e030-credential-socket" pod "6e1c3c74-139f-4299-97c4-4609e0b9e030" (UID: "6e1c3c74-139f-4299-97c4-4609e0b9e030"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:45.455935 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.455890 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da926b76-e144-4a03-bb81-67d2effff0a6-credential-socket" pod "da926b76-e144-4a03-bb81-67d2effff0a6" (UID: "da926b76-e144-4a03-bb81-67d2effff0a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:45.456039 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.456020 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9c903cd-d636-4311-9afe-e51e4cc3db8e-istiod-ca-cert" pod "c9c903cd-d636-4311-9afe-e51e4cc3db8e" (UID: "c9c903cd-d636-4311-9afe-e51e4cc3db8e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:12:45.457538 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.457500 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9c903cd-d636-4311-9afe-e51e4cc3db8e-istio-token" pod "c9c903cd-d636-4311-9afe-e51e4cc3db8e" (UID: "c9c903cd-d636-4311-9afe-e51e4cc3db8e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:12:45.458355 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.458300 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da926b76-e144-4a03-bb81-67d2effff0a6-istio-podinfo" pod "da926b76-e144-4a03-bb81-67d2effff0a6" (UID: "da926b76-e144-4a03-bb81-67d2effff0a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:12:45.459081 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.459053 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9c903cd-d636-4311-9afe-e51e4cc3db8e-istio-podinfo" pod "c9c903cd-d636-4311-9afe-e51e4cc3db8e" (UID: "c9c903cd-d636-4311-9afe-e51e4cc3db8e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:12:45.460074 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.460048 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e1c3c74-139f-4299-97c4-4609e0b9e030-kube-api-access-gsb9w" pod "6e1c3c74-139f-4299-97c4-4609e0b9e030" (UID: "6e1c3c74-139f-4299-97c4-4609e0b9e030"). InnerVolumeSpecName "kube-api-access-gsb9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:12:45.462698 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.462596 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e1c3c74-139f-4299-97c4-4609e0b9e030-istio-envoy" pod "6e1c3c74-139f-4299-97c4-4609e0b9e030" (UID: "6e1c3c74-139f-4299-97c4-4609e0b9e030"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:45.462698 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.462637 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9c903cd-d636-4311-9afe-e51e4cc3db8e-istio-envoy" pod "c9c903cd-d636-4311-9afe-e51e4cc3db8e" (UID: "c9c903cd-d636-4311-9afe-e51e4cc3db8e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:45.462698 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.462657 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da926b76-e144-4a03-bb81-67d2effff0a6-istio-envoy" pod "da926b76-e144-4a03-bb81-67d2effff0a6" (UID: "da926b76-e144-4a03-bb81-67d2effff0a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:45.462698 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.462664 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da926b76-e144-4a03-bb81-67d2effff0a6-istio-token" pod "da926b76-e144-4a03-bb81-67d2effff0a6" (UID: "da926b76-e144-4a03-bb81-67d2effff0a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:12:45.462698 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.462667 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da926b76-e144-4a03-bb81-67d2effff0a6-kube-api-access-p4wb5" pod "da926b76-e144-4a03-bb81-67d2effff0a6" (UID: "da926b76-e144-4a03-bb81-67d2effff0a6"). InnerVolumeSpecName "kube-api-access-p4wb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:12:45.463031 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.462783 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e1c3c74-139f-4299-97c4-4609e0b9e030-istio-token" pod "6e1c3c74-139f-4299-97c4-4609e0b9e030" (UID: "6e1c3c74-139f-4299-97c4-4609e0b9e030"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:12:45.463031 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.462796 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6e1c3c74-139f-4299-97c4-4609e0b9e030-istio-podinfo" pod "6e1c3c74-139f-4299-97c4-4609e0b9e030" (UID: "6e1c3c74-139f-4299-97c4-4609e0b9e030"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:12:45.464136 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.464112 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9c903cd-d636-4311-9afe-e51e4cc3db8e-kube-api-access-zd42w" pod "c9c903cd-d636-4311-9afe-e51e4cc3db8e" (UID: "c9c903cd-d636-4311-9afe-e51e4cc3db8e"). InnerVolumeSpecName "kube-api-access-zd42w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:12:45.553758 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.553723 4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9c903cd-d636-4311-9afe-e51e4cc3db8e-istio-token\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.553758 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.553750 4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9c903cd-d636-4311-9afe-e51e4cc3db8e-istio-envoy\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.553758 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.553763 4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e1c3c74-139f-4299-97c4-4609e0b9e030-workload-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.553981 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.553774 4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9c903cd-d636-4311-9afe-e51e4cc3db8e-workload-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.553981 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.553783 4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e1c3c74-139f-4299-97c4-4609e0b9e030-istiod-ca-cert\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.553981 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.553792 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsb9w\" (UniqueName: \"kubernetes.io/projected/6e1c3c74-139f-4299-97c4-4609e0b9e030-kube-api-access-gsb9w\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.553981 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.553801 4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e1c3c74-139f-4299-97c4-4609e0b9e030-istio-podinfo\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.553981 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.553809 4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e1c3c74-139f-4299-97c4-4609e0b9e030-istio-token\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.553981 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.553817 4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e1c3c74-139f-4299-97c4-4609e0b9e030-istio-data\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.553981 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.553825 4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da926b76-e144-4a03-bb81-67d2effff0a6-istio-envoy\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.553981 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.553832 4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e1c3c74-139f-4299-97c4-4609e0b9e030-credential-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.553981 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.553845 4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da926b76-e144-4a03-bb81-67d2effff0a6-istio-token\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.553981 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.553857 4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9c903cd-d636-4311-9afe-e51e4cc3db8e-istio-podinfo\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.553981 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.553864 4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9c903cd-d636-4311-9afe-e51e4cc3db8e-workload-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.553981 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.553873 4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da926b76-e144-4a03-bb81-67d2effff0a6-credential-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.553981 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.553880 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zd42w\" (UniqueName: \"kubernetes.io/projected/c9c903cd-d636-4311-9afe-e51e4cc3db8e-kube-api-access-zd42w\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.553981 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.553888 4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da926b76-e144-4a03-bb81-67d2effff0a6-istio-data\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.553981 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.553898 4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e1c3c74-139f-4299-97c4-4609e0b9e030-istio-envoy\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.553981 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.553905 4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9c903cd-d636-4311-9afe-e51e4cc3db8e-istiod-ca-cert\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.553981 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.553912 4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da926b76-e144-4a03-bb81-67d2effff0a6-workload-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.553981 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.553926 4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da926b76-e144-4a03-bb81-67d2effff0a6-workload-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.553981 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.553937 4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da926b76-e144-4a03-bb81-67d2effff0a6-istio-podinfo\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.553981 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.553945 4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da926b76-e144-4a03-bb81-67d2effff0a6-istiod-ca-cert\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.553981 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.553955 4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9c903cd-d636-4311-9afe-e51e4cc3db8e-istio-data\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.553981 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.553962 4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9c903cd-d636-4311-9afe-e51e4cc3db8e-credential-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.553981 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.553970 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4wb5\" (UniqueName: \"kubernetes.io/projected/da926b76-e144-4a03-bb81-67d2effff0a6-kube-api-access-p4wb5\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.553981 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:45.553978 4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e1c3c74-139f-4299-97c4-4609e0b9e030-workload-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:12:46.093691 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:46.093660 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6" Oct 30 17:12:46.093691 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:46.093676 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6" event={"ID":"c9c903cd-d636-4311-9afe-e51e4cc3db8e","Type":"ContainerDied","Data":"ea8640a8778e159d35d77c7a15914642bc337acabb3917faca104500d40b670f"} Oct 30 17:12:46.094221 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:46.093746 4522 scope.go:122] "RemoveContainer" containerID="73b47ad2ff23e446c1c18cea703cd9198beb3a34408083ae602202c9a326d39f" Oct 30 17:12:46.095444 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:46.095425 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m" Oct 30 17:12:46.095557 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:46.095425 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m" event={"ID":"da926b76-e144-4a03-bb81-67d2effff0a6","Type":"ContainerDied","Data":"d8bc17c707aef38605fea8e2f91bef9539f70780f0b1e9138292d7e70cf2b14b"} Oct 30 17:12:46.097060 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:46.097035 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq" Oct 30 17:12:46.097268 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:46.097080 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq" event={"ID":"46027d41-98ad-409d-a66f-4bc050bc2648","Type":"ContainerDied","Data":"2a7121ed0a6a5f33732d3d22cccfe3924f233937acd8da6258605605060aad6b"} Oct 30 17:12:46.101347 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:46.098759 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5" event={"ID":"6e1c3c74-139f-4299-97c4-4609e0b9e030","Type":"ContainerDied","Data":"809bd884c62ac8f3741c174e59ef033bbf03155b7e44a392e31e2b814a3bf5c1"} Oct 30 17:12:46.101347 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:46.098777 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5" Oct 30 17:12:46.109331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:46.109284 4522 scope.go:122] "RemoveContainer" containerID="b10a9c1634fcda6bb2882eb293c4a023244cb460c922442454335a65a65d90c1" Oct 30 17:12:46.121378 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:46.121353 4522 scope.go:122] "RemoveContainer" containerID="c09766a2cb5828462c8d758dc680c4b58133409e57b861a19da7f333f2604862" Oct 30 17:12:46.131755 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:46.131738 4522 scope.go:122] "RemoveContainer" containerID="a60d5b915fec64c6758b8ef572674b5753e55fc85993d77537df3276db0a3c87" Oct 30 17:12:46.191504 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:46.191424 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6"] Oct 30 17:12:46.201027 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:46.200996 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-2c9c6"] Oct 30 17:12:46.287736 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:46.287354 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5"] Oct 30 17:12:46.291699 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:46.291674 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-gbbz5"] Oct 30 17:12:46.328346 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:46.328301 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m"] Oct 30 17:12:46.335118 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:46.335086 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-5n98m"] Oct 30 17:12:46.375050 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:46.375018 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq"] Oct 30 17:12:46.381118 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:46.381092 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-b5qrq"] Oct 30 17:12:48.147172 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:48.147127 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46027d41-98ad-409d-a66f-4bc050bc2648" path="/var/lib/kubelet/pods/46027d41-98ad-409d-a66f-4bc050bc2648/volumes" Oct 30 17:12:48.147906 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:48.147883 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e1c3c74-139f-4299-97c4-4609e0b9e030" path="/var/lib/kubelet/pods/6e1c3c74-139f-4299-97c4-4609e0b9e030/volumes" Oct 30 17:12:48.148452 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:48.148432 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9c903cd-d636-4311-9afe-e51e4cc3db8e" path="/var/lib/kubelet/pods/c9c903cd-d636-4311-9afe-e51e4cc3db8e/volumes" Oct 30 17:12:48.149291 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:48.149268 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da926b76-e144-4a03-bb81-67d2effff0a6" path="/var/lib/kubelet/pods/da926b76-e144-4a03-bb81-67d2effff0a6/volumes" Oct 30 17:12:49.429704 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:49.429673 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4c2e4-vjl42"] Oct 30 17:12:49.430437 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:49.430417 4522 cpu_manager.go:414] "containerMap: removing container" podUID="c9c903cd-d636-4311-9afe-e51e4cc3db8e" containerName="istio-proxy" Oct 30 17:12:49.430437 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:49.430437 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9c903cd-d636-4311-9afe-e51e4cc3db8e" containerName="istio-proxy" Oct 30 17:12:49.430596 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:49.430468 4522 cpu_manager.go:414] "containerMap: removing container" podUID="46027d41-98ad-409d-a66f-4bc050bc2648" containerName="istio-proxy" Oct 30 17:12:49.430596 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:49.430473 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46027d41-98ad-409d-a66f-4bc050bc2648" containerName="istio-proxy" Oct 30 17:12:49.430596 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:49.430494 4522 cpu_manager.go:414] "containerMap: removing container" podUID="da926b76-e144-4a03-bb81-67d2effff0a6" containerName="istio-proxy" Oct 30 17:12:49.430596 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:49.430501 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da926b76-e144-4a03-bb81-67d2effff0a6" containerName="istio-proxy" Oct 30 17:12:49.430596 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:49.430519 4522 cpu_manager.go:414] "containerMap: removing container" podUID="6e1c3c74-139f-4299-97c4-4609e0b9e030" containerName="istio-proxy" Oct 30 17:12:49.430596 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:49.430524 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e1c3c74-139f-4299-97c4-4609e0b9e030" containerName="istio-proxy" Oct 30 17:12:49.430789 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:49.430648 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e1c3c74-139f-4299-97c4-4609e0b9e030" containerName="istio-proxy" Oct 30 17:12:49.430789 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:49.430662 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9c903cd-d636-4311-9afe-e51e4cc3db8e" containerName="istio-proxy" Oct 30 17:12:49.430789 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:49.430672 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="46027d41-98ad-409d-a66f-4bc050bc2648" containerName="istio-proxy" Oct 30 17:12:49.430789 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:49.430681 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="da926b76-e144-4a03-bb81-67d2effff0a6" containerName="istio-proxy" Oct 30 17:12:49.434833 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:49.434813 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4c2e4-vjl42" Oct 30 17:12:49.456722 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:49.456692 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4c2e4-vjl42"] Oct 30 17:12:49.468530 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:49.465186 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:12:49.468530 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:49.465264 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:12:49.522361 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:49.522317 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsq4g\" (UniqueName: \"kubernetes.io/projected/013f626d-4c83-4506-9867-874cbafd85ab-kube-api-access-hsq4g\") pod \"maas-api-revoke-keys-e2e-api-b-c4c2e4-vjl42\" (UID: \"013f626d-4c83-4506-9867-874cbafd85ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4c2e4-vjl42" Oct 30 17:12:49.522524 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:49.522387 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/013f626d-4c83-4506-9867-874cbafd85ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c4c2e4-vjl42\" (UID: \"013f626d-4c83-4506-9867-874cbafd85ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4c2e4-vjl42" Oct 30 17:12:49.522618 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:49.522548 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/013f626d-4c83-4506-9867-874cbafd85ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c4c2e4-vjl42\" (UID: \"013f626d-4c83-4506-9867-874cbafd85ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4c2e4-vjl42" Oct 30 17:12:49.624040 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:49.624004 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsq4g\" (UniqueName: \"kubernetes.io/projected/013f626d-4c83-4506-9867-874cbafd85ab-kube-api-access-hsq4g\") pod \"maas-api-revoke-keys-e2e-api-b-c4c2e4-vjl42\" (UID: \"013f626d-4c83-4506-9867-874cbafd85ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4c2e4-vjl42" Oct 30 17:12:49.624208 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:49.624050 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/013f626d-4c83-4506-9867-874cbafd85ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c4c2e4-vjl42\" (UID: \"013f626d-4c83-4506-9867-874cbafd85ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4c2e4-vjl42" Oct 30 17:12:49.624208 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:49.624112 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/013f626d-4c83-4506-9867-874cbafd85ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c4c2e4-vjl42\" (UID: \"013f626d-4c83-4506-9867-874cbafd85ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4c2e4-vjl42" Oct 30 17:12:49.624816 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:49.624789 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/013f626d-4c83-4506-9867-874cbafd85ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c4c2e4-vjl42\" (UID: \"013f626d-4c83-4506-9867-874cbafd85ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4c2e4-vjl42" Oct 30 17:12:49.626692 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:49.626670 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/013f626d-4c83-4506-9867-874cbafd85ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c4c2e4-vjl42\" (UID: \"013f626d-4c83-4506-9867-874cbafd85ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4c2e4-vjl42" Oct 30 17:12:49.650622 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:49.650591 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsq4g\" (UniqueName: \"kubernetes.io/projected/013f626d-4c83-4506-9867-874cbafd85ab-kube-api-access-hsq4g\") pod \"maas-api-revoke-keys-e2e-api-b-c4c2e4-vjl42\" (UID: \"013f626d-4c83-4506-9867-874cbafd85ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4c2e4-vjl42" Oct 30 17:12:49.754593 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:49.754547 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4c2e4-vjl42" Oct 30 17:12:50.065493 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:12:50.065459 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod013f626d_4c83_4506_9867_874cbafd85ab.slice/crio-1fb5187bb78030c80052dcc06befb06c4f848fa35b80531c982a7a53cc5d9b5c WatchSource:0}: Error finding container 1fb5187bb78030c80052dcc06befb06c4f848fa35b80531c982a7a53cc5d9b5c: Status 404 returned error can't find the container with id 1fb5187bb78030c80052dcc06befb06c4f848fa35b80531c982a7a53cc5d9b5c Oct 30 17:12:50.066985 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:50.066957 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4c2e4-vjl42"] Oct 30 17:12:50.091546 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:50.091518 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b2c3fb-7c87c57c7c-pz9ls" Oct 30 17:12:50.159171 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:50.159134 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4c2e4-vjl42" event={"ID":"013f626d-4c83-4506-9867-874cbafd85ab","Type":"ContainerStarted","Data":"1fb5187bb78030c80052dcc06befb06c4f848fa35b80531c982a7a53cc5d9b5c"} Oct 30 17:12:51.172264 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:51.172224 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4c2e4-vjl42" event={"ID":"013f626d-4c83-4506-9867-874cbafd85ab","Type":"ContainerStarted","Data":"9fe1a813d428b2732acbf2b35e49c247f2ff770795be0bcb825862e14242c7a4"} Oct 30 17:12:51.218613 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:51.218561 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4c2e4-vjl42" podStartSLOduration=2.21854761 podStartE2EDuration="2.21854761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:12:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:51.218466834 +0000 UTC m=+1545.219221758" watchObservedRunningTime="2026-10-30 17:12:51.21854761 +0000 UTC m=+1545.219302511" Oct 30 17:12:52.181957 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:52.181925 4522 generic.go:356] "Generic (PLEG): container finished" podID="013f626d-4c83-4506-9867-874cbafd85ab" containerID="9fe1a813d428b2732acbf2b35e49c247f2ff770795be0bcb825862e14242c7a4" exitCode=0 Oct 30 17:12:52.182531 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:52.182016 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4c2e4-vjl42" event={"ID":"013f626d-4c83-4506-9867-874cbafd85ab","Type":"ContainerDied","Data":"9fe1a813d428b2732acbf2b35e49c247f2ff770795be0bcb825862e14242c7a4"} Oct 30 17:12:53.375383 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:53.375358 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4c2e4-vjl42" Oct 30 17:12:53.505821 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:53.505789 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/013f626d-4c83-4506-9867-874cbafd85ab-kube-api-access-hsq4g\" (UniqueName: \"kubernetes.io/projected/013f626d-4c83-4506-9867-874cbafd85ab-kube-api-access-hsq4g\") pod \"013f626d-4c83-4506-9867-874cbafd85ab\" (UID: \"013f626d-4c83-4506-9867-874cbafd85ab\") " Oct 30 17:12:53.506005 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:53.505991 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/013f626d-4c83-4506-9867-874cbafd85ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/013f626d-4c83-4506-9867-874cbafd85ab-maas-api-xks-ca\") pod \"013f626d-4c83-4506-9867-874cbafd85ab\" (UID: \"013f626d-4c83-4506-9867-874cbafd85ab\") " Oct 30 17:12:53.506069 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:53.506057 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/013f626d-4c83-4506-9867-874cbafd85ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/013f626d-4c83-4506-9867-874cbafd85ab-maas-api-openshift-ca\") pod \"013f626d-4c83-4506-9867-874cbafd85ab\" (UID: \"013f626d-4c83-4506-9867-874cbafd85ab\") " Oct 30 17:12:53.506466 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:53.506437 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/013f626d-4c83-4506-9867-874cbafd85ab-maas-api-openshift-ca" pod "013f626d-4c83-4506-9867-874cbafd85ab" (UID: "013f626d-4c83-4506-9867-874cbafd85ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:12:53.506600 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:53.506585 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/013f626d-4c83-4506-9867-874cbafd85ab-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:12:53.507975 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:53.507952 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/013f626d-4c83-4506-9867-874cbafd85ab-kube-api-access-hsq4g" pod "013f626d-4c83-4506-9867-874cbafd85ab" (UID: "013f626d-4c83-4506-9867-874cbafd85ab"). InnerVolumeSpecName "kube-api-access-hsq4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:12:53.508062 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:53.508033 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/013f626d-4c83-4506-9867-874cbafd85ab-maas-api-xks-ca" pod "013f626d-4c83-4506-9867-874cbafd85ab" (UID: "013f626d-4c83-4506-9867-874cbafd85ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:12:53.607708 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:53.607678 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/013f626d-4c83-4506-9867-874cbafd85ab-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:12:53.607708 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:53.607706 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsq4g\" (UniqueName: \"kubernetes.io/projected/013f626d-4c83-4506-9867-874cbafd85ab-kube-api-access-hsq4g\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:12:54.199674 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:54.199648 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4c2e4-vjl42" Oct 30 17:12:54.199861 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:54.199675 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4c2e4-vjl42" event={"ID":"013f626d-4c83-4506-9867-874cbafd85ab","Type":"ContainerDied","Data":"1fb5187bb78030c80052dcc06befb06c4f848fa35b80531c982a7a53cc5d9b5c"} Oct 30 17:12:54.199861 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:54.199718 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1fb5187bb78030c80052dcc06befb06c4f848fa35b80531c982a7a53cc5d9b5c" Oct 30 17:12:59.530854 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:12:59.530811 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c4c2e4-6f66b94d58-6xb5n"] Oct 30 17:13:00.430879 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:00.430841 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ebeb6e1-qfdc7"] Oct 30 17:13:00.435205 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:00.435177 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ebeb6e1-qfdc7"] Oct 30 17:13:02.147993 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:02.147954 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18d1b373-ad7a-466e-9a6c-ed7f557c386a" path="/var/lib/kubelet/pods/18d1b373-ad7a-466e-9a6c-ed7f557c386a/volumes" Oct 30 17:13:04.532160 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:04.532119 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-c4c2e4-6f66b94d58-6xb5n" podUID="bff503f8-b6d2-41d4-a8c4-883a3afa38a1" containerName="payload-processing" containerID="cri-o://b54bde885f10a4861270a34e8860cf90aa8ec9105bb946239bf9a86a13ffaad3" gracePeriod=25 Oct 30 17:13:04.837436 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:04.837411 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c4c2e4-6f66b94d58-6xb5n" Oct 30 17:13:04.886192 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:04.886158 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bff503f8-b6d2-41d4-a8c4-883a3afa38a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bff503f8-b6d2-41d4-a8c4-883a3afa38a1-plugins-config-volume\") pod \"bff503f8-b6d2-41d4-a8c4-883a3afa38a1\" (UID: \"bff503f8-b6d2-41d4-a8c4-883a3afa38a1\") " Oct 30 17:13:04.886410 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:04.886349 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bff503f8-b6d2-41d4-a8c4-883a3afa38a1-kube-api-access-44mkv\" (UniqueName: \"kubernetes.io/projected/bff503f8-b6d2-41d4-a8c4-883a3afa38a1-kube-api-access-44mkv\") pod \"bff503f8-b6d2-41d4-a8c4-883a3afa38a1\" (UID: \"bff503f8-b6d2-41d4-a8c4-883a3afa38a1\") " Oct 30 17:13:04.886652 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:04.886619 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bff503f8-b6d2-41d4-a8c4-883a3afa38a1-plugins-config-volume" pod "bff503f8-b6d2-41d4-a8c4-883a3afa38a1" (UID: "bff503f8-b6d2-41d4-a8c4-883a3afa38a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:04.886992 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:04.886971 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bff503f8-b6d2-41d4-a8c4-883a3afa38a1-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:13:04.888568 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:04.888545 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bff503f8-b6d2-41d4-a8c4-883a3afa38a1-kube-api-access-44mkv" pod "bff503f8-b6d2-41d4-a8c4-883a3afa38a1" (UID: "bff503f8-b6d2-41d4-a8c4-883a3afa38a1"). InnerVolumeSpecName "kube-api-access-44mkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:04.988812 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:04.988781 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44mkv\" (UniqueName: \"kubernetes.io/projected/bff503f8-b6d2-41d4-a8c4-883a3afa38a1-kube-api-access-44mkv\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:13:05.309902 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:05.309869 4522 generic.go:356] "Generic (PLEG): container finished" podID="bff503f8-b6d2-41d4-a8c4-883a3afa38a1" containerID="b54bde885f10a4861270a34e8860cf90aa8ec9105bb946239bf9a86a13ffaad3" exitCode=0 Oct 30 17:13:05.310088 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:05.309909 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c4c2e4-6f66b94d58-6xb5n" Oct 30 17:13:05.310088 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:05.309950 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c4c2e4-6f66b94d58-6xb5n" event={"ID":"bff503f8-b6d2-41d4-a8c4-883a3afa38a1","Type":"ContainerDied","Data":"b54bde885f10a4861270a34e8860cf90aa8ec9105bb946239bf9a86a13ffaad3"} Oct 30 17:13:05.310088 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:05.309999 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c4c2e4-6f66b94d58-6xb5n" event={"ID":"bff503f8-b6d2-41d4-a8c4-883a3afa38a1","Type":"ContainerDied","Data":"f5d6054415017f2bf4129df4479ed32e5ec57d974d67a467d73f9b2cf3b2e296"} Oct 30 17:13:05.310088 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:05.310019 4522 scope.go:122] "RemoveContainer" containerID="b54bde885f10a4861270a34e8860cf90aa8ec9105bb946239bf9a86a13ffaad3" Oct 30 17:13:05.322046 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:05.322027 4522 scope.go:122] "RemoveContainer" containerID="b54bde885f10a4861270a34e8860cf90aa8ec9105bb946239bf9a86a13ffaad3" Oct 30 17:13:05.322336 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:13:05.322301 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b54bde885f10a4861270a34e8860cf90aa8ec9105bb946239bf9a86a13ffaad3\": container with ID starting with b54bde885f10a4861270a34e8860cf90aa8ec9105bb946239bf9a86a13ffaad3 not found: ID does not exist" containerID="b54bde885f10a4861270a34e8860cf90aa8ec9105bb946239bf9a86a13ffaad3" Oct 30 17:13:05.322388 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:05.322346 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b54bde885f10a4861270a34e8860cf90aa8ec9105bb946239bf9a86a13ffaad3"} err="failed to get container status \"b54bde885f10a4861270a34e8860cf90aa8ec9105bb946239bf9a86a13ffaad3\": rpc error: code = NotFound desc = could not find container \"b54bde885f10a4861270a34e8860cf90aa8ec9105bb946239bf9a86a13ffaad3\": container with ID starting with b54bde885f10a4861270a34e8860cf90aa8ec9105bb946239bf9a86a13ffaad3 not found: ID does not exist" Oct 30 17:13:05.374069 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:05.374028 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c4c2e4-6f66b94d58-6xb5n"] Oct 30 17:13:05.388454 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:05.388422 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c4c2e4-6f66b94d58-6xb5n"] Oct 30 17:13:06.149207 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:06.149177 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bff503f8-b6d2-41d4-a8c4-883a3afa38a1" path="/var/lib/kubelet/pods/bff503f8-b6d2-41d4-a8c4-883a3afa38a1/volumes" Oct 30 17:13:08.073703 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:08.073672 4522 scope.go:122] "RemoveContainer" containerID="10129b75a3b759ac5a0f505882112e1359ffcbcf6ed1159dce7e4dd5b824f666" Oct 30 17:13:08.084187 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:08.084164 4522 scope.go:122] "RemoveContainer" containerID="c0b5faac0c01ceb47c78d6f3d72d9a9a222a2edc3bca5f5834424336402ce6bc" Oct 30 17:13:08.093701 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:08.093680 4522 scope.go:122] "RemoveContainer" containerID="5cfac171006fcc5c15d23fb5642ea21810ea480d4c6572b094edbdfee563c594" Oct 30 17:13:08.114166 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:08.114141 4522 scope.go:122] "RemoveContainer" containerID="946f7c57908d8b1da989603bfbd684b6df04387bbfb4438fdc88492781057bdb" Oct 30 17:13:08.131516 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:08.129586 4522 scope.go:122] "RemoveContainer" containerID="b4bde40ea0b334f350514a55a89c0c946e3f36eaf6a8ac209c37562c76092778" Oct 30 17:13:08.147600 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:08.147577 4522 scope.go:122] "RemoveContainer" containerID="524be0d6eaaf4bda7a7ee742c80b4e590e0e35f698b79a0770180be1c66987a1" Oct 30 17:13:09.990089 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:09.990050 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc"] Oct 30 17:13:09.990659 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:09.990263 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc" podUID="570bc5c9-330c-402e-b262-16b3be6340e7" containerName="istio-proxy" containerID="cri-o://c4291b8c23114bb9edea91f8b1eaf0eda4f75862b2ce722eafd724879aabf378" gracePeriod=30 Oct 30 17:13:12.963706 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:13:12.963668 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:13:15.296484 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:15.296407 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc" Oct 30 17:13:15.398562 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:15.398529 4522 generic.go:356] "Generic (PLEG): container finished" podID="570bc5c9-330c-402e-b262-16b3be6340e7" containerID="c4291b8c23114bb9edea91f8b1eaf0eda4f75862b2ce722eafd724879aabf378" exitCode=0 Oct 30 17:13:15.398769 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:15.398581 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc" Oct 30 17:13:15.398769 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:15.398618 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc" event={"ID":"570bc5c9-330c-402e-b262-16b3be6340e7","Type":"ContainerDied","Data":"c4291b8c23114bb9edea91f8b1eaf0eda4f75862b2ce722eafd724879aabf378"} Oct 30 17:13:15.398769 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:15.398672 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc" event={"ID":"570bc5c9-330c-402e-b262-16b3be6340e7","Type":"ContainerDied","Data":"168d1157b2d86cad74e65d178fff3547e4ff9b1ca26c626bfe1c9b466d73b3e0"} Oct 30 17:13:15.398769 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:15.398698 4522 scope.go:122] "RemoveContainer" containerID="c4291b8c23114bb9edea91f8b1eaf0eda4f75862b2ce722eafd724879aabf378" Oct 30 17:13:15.409442 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:15.409417 4522 scope.go:122] "RemoveContainer" containerID="c4291b8c23114bb9edea91f8b1eaf0eda4f75862b2ce722eafd724879aabf378" Oct 30 17:13:15.409728 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:13:15.409699 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4291b8c23114bb9edea91f8b1eaf0eda4f75862b2ce722eafd724879aabf378\": container with ID starting with c4291b8c23114bb9edea91f8b1eaf0eda4f75862b2ce722eafd724879aabf378 not found: ID does not exist" containerID="c4291b8c23114bb9edea91f8b1eaf0eda4f75862b2ce722eafd724879aabf378" Oct 30 17:13:15.409779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:15.409737 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4291b8c23114bb9edea91f8b1eaf0eda4f75862b2ce722eafd724879aabf378"} err="failed to get container status \"c4291b8c23114bb9edea91f8b1eaf0eda4f75862b2ce722eafd724879aabf378\": rpc error: code = NotFound desc = could not find container \"c4291b8c23114bb9edea91f8b1eaf0eda4f75862b2ce722eafd724879aabf378\": container with ID starting with c4291b8c23114bb9edea91f8b1eaf0eda4f75862b2ce722eafd724879aabf378 not found: ID does not exist" Oct 30 17:13:15.468185 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:15.468156 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/570bc5c9-330c-402e-b262-16b3be6340e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/570bc5c9-330c-402e-b262-16b3be6340e7-workload-socket\") pod \"570bc5c9-330c-402e-b262-16b3be6340e7\" (UID: \"570bc5c9-330c-402e-b262-16b3be6340e7\") " Oct 30 17:13:15.468371 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:15.468210 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/570bc5c9-330c-402e-b262-16b3be6340e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/570bc5c9-330c-402e-b262-16b3be6340e7-workload-certs\") pod \"570bc5c9-330c-402e-b262-16b3be6340e7\" (UID: \"570bc5c9-330c-402e-b262-16b3be6340e7\") " Oct 30 17:13:15.468371 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:15.468347 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/570bc5c9-330c-402e-b262-16b3be6340e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/570bc5c9-330c-402e-b262-16b3be6340e7-istio-podinfo\") pod \"570bc5c9-330c-402e-b262-16b3be6340e7\" (UID: \"570bc5c9-330c-402e-b262-16b3be6340e7\") " Oct 30 17:13:15.468464 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:15.468394 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/570bc5c9-330c-402e-b262-16b3be6340e7-istio-token\" (UniqueName: \"kubernetes.io/projected/570bc5c9-330c-402e-b262-16b3be6340e7-istio-token\") pod \"570bc5c9-330c-402e-b262-16b3be6340e7\" (UID: \"570bc5c9-330c-402e-b262-16b3be6340e7\") " Oct 30 17:13:15.468464 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:15.468422 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/570bc5c9-330c-402e-b262-16b3be6340e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/570bc5c9-330c-402e-b262-16b3be6340e7-istio-data\") pod \"570bc5c9-330c-402e-b262-16b3be6340e7\" (UID: \"570bc5c9-330c-402e-b262-16b3be6340e7\") " Oct 30 17:13:15.468538 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:15.468469 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/570bc5c9-330c-402e-b262-16b3be6340e7-workload-socket" pod "570bc5c9-330c-402e-b262-16b3be6340e7" (UID: "570bc5c9-330c-402e-b262-16b3be6340e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:13:15.468574 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:15.468538 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/570bc5c9-330c-402e-b262-16b3be6340e7-workload-certs" pod "570bc5c9-330c-402e-b262-16b3be6340e7" (UID: "570bc5c9-330c-402e-b262-16b3be6340e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:13:15.468611 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:15.468572 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/570bc5c9-330c-402e-b262-16b3be6340e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/570bc5c9-330c-402e-b262-16b3be6340e7-credential-socket\") pod \"570bc5c9-330c-402e-b262-16b3be6340e7\" (UID: \"570bc5c9-330c-402e-b262-16b3be6340e7\") " Oct 30 17:13:15.468690 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:15.468674 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/570bc5c9-330c-402e-b262-16b3be6340e7-kube-api-access-bz2bx\" (UniqueName: \"kubernetes.io/projected/570bc5c9-330c-402e-b262-16b3be6340e7-kube-api-access-bz2bx\") pod \"570bc5c9-330c-402e-b262-16b3be6340e7\" (UID: \"570bc5c9-330c-402e-b262-16b3be6340e7\") " Oct 30 17:13:15.468752 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:15.468738 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/570bc5c9-330c-402e-b262-16b3be6340e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/570bc5c9-330c-402e-b262-16b3be6340e7-istio-envoy\") pod \"570bc5c9-330c-402e-b262-16b3be6340e7\" (UID: \"570bc5c9-330c-402e-b262-16b3be6340e7\") " Oct 30 17:13:15.468813 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:15.468778 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/570bc5c9-330c-402e-b262-16b3be6340e7-istio-data" pod "570bc5c9-330c-402e-b262-16b3be6340e7" (UID: "570bc5c9-330c-402e-b262-16b3be6340e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:13:15.468813 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:15.468796 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/570bc5c9-330c-402e-b262-16b3be6340e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/570bc5c9-330c-402e-b262-16b3be6340e7-istiod-ca-cert\") pod \"570bc5c9-330c-402e-b262-16b3be6340e7\" (UID: \"570bc5c9-330c-402e-b262-16b3be6340e7\") " Oct 30 17:13:15.469003 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:15.468807 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/570bc5c9-330c-402e-b262-16b3be6340e7-credential-socket" pod "570bc5c9-330c-402e-b262-16b3be6340e7" (UID: "570bc5c9-330c-402e-b262-16b3be6340e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:13:15.469225 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:15.469199 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/570bc5c9-330c-402e-b262-16b3be6340e7-istiod-ca-cert" pod "570bc5c9-330c-402e-b262-16b3be6340e7" (UID: "570bc5c9-330c-402e-b262-16b3be6340e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:15.469593 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:15.469569 4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/570bc5c9-330c-402e-b262-16b3be6340e7-istiod-ca-cert\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:13:15.469684 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:15.469599 4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/570bc5c9-330c-402e-b262-16b3be6340e7-workload-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:13:15.469684 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:15.469617 4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/570bc5c9-330c-402e-b262-16b3be6340e7-workload-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:13:15.469684 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:15.469632 4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/570bc5c9-330c-402e-b262-16b3be6340e7-istio-data\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:13:15.469684 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:15.469644 4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/570bc5c9-330c-402e-b262-16b3be6340e7-credential-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:13:15.470742 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:15.470715 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/570bc5c9-330c-402e-b262-16b3be6340e7-istio-podinfo" pod "570bc5c9-330c-402e-b262-16b3be6340e7" (UID: "570bc5c9-330c-402e-b262-16b3be6340e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:13:15.470852 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:15.470792 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/570bc5c9-330c-402e-b262-16b3be6340e7-istio-token" pod "570bc5c9-330c-402e-b262-16b3be6340e7" (UID: "570bc5c9-330c-402e-b262-16b3be6340e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:15.470852 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:15.470815 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/570bc5c9-330c-402e-b262-16b3be6340e7-kube-api-access-bz2bx" pod "570bc5c9-330c-402e-b262-16b3be6340e7" (UID: "570bc5c9-330c-402e-b262-16b3be6340e7"). InnerVolumeSpecName "kube-api-access-bz2bx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:15.470982 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:15.470967 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/570bc5c9-330c-402e-b262-16b3be6340e7-istio-envoy" pod "570bc5c9-330c-402e-b262-16b3be6340e7" (UID: "570bc5c9-330c-402e-b262-16b3be6340e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:13:15.571548 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:15.571464 4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/570bc5c9-330c-402e-b262-16b3be6340e7-istio-podinfo\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:13:15.571548 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:15.571492 4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/570bc5c9-330c-402e-b262-16b3be6340e7-istio-token\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:13:15.571548 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:15.571503 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bz2bx\" (UniqueName: \"kubernetes.io/projected/570bc5c9-330c-402e-b262-16b3be6340e7-kube-api-access-bz2bx\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:13:15.571548 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:15.571512 4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/570bc5c9-330c-402e-b262-16b3be6340e7-istio-envoy\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:13:15.756164 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:15.756123 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc"] Oct 30 17:13:15.770363 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:15.770329 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-n52tc"] Oct 30 17:13:16.147444 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:16.147415 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="570bc5c9-330c-402e-b262-16b3be6340e7" path="/var/lib/kubelet/pods/570bc5c9-330c-402e-b262-16b3be6340e7/volumes" Oct 30 17:13:17.242611 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:17.242571 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-362bbf-pbh42"] Oct 30 17:13:17.257587 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:17.257554 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-362bbf-pbh42"] Oct 30 17:13:18.147022 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:18.146992 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19794b16-e65f-452d-b69b-15435649d07e" path="/var/lib/kubelet/pods/19794b16-e65f-452d-b69b-15435649d07e/volumes" Oct 30 17:13:25.272184 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:25.271970 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m"] Oct 30 17:13:25.273049 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:25.273014 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m" podUID="4d683a0f-9de0-46dc-9772-d0915b0cb985" containerName="istio-proxy" containerID="cri-o://d8434a2a67396bb270b71f833213bbe6b7fb4eacbbbd26781a03bce6f5eacbf7" gracePeriod=30 Oct 30 17:13:27.693778 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:27.693740 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d003a5-5cc689c7b6-z4b8q"] Oct 30 17:13:27.694969 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:27.694942 4522 cpu_manager.go:414] "containerMap: removing container" podUID="570bc5c9-330c-402e-b262-16b3be6340e7" containerName="istio-proxy" Oct 30 17:13:27.695101 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:27.694968 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="570bc5c9-330c-402e-b262-16b3be6340e7" containerName="istio-proxy" Oct 30 17:13:27.695101 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:27.695001 4522 cpu_manager.go:414] "containerMap: removing container" podUID="013f626d-4c83-4506-9867-874cbafd85ab" containerName="revoke-keys" Oct 30 17:13:27.695101 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:27.695008 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="013f626d-4c83-4506-9867-874cbafd85ab" containerName="revoke-keys" Oct 30 17:13:27.695101 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:27.695043 4522 cpu_manager.go:414] "containerMap: removing container" podUID="bff503f8-b6d2-41d4-a8c4-883a3afa38a1" containerName="payload-processing" Oct 30 17:13:27.695101 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:27.695050 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bff503f8-b6d2-41d4-a8c4-883a3afa38a1" containerName="payload-processing" Oct 30 17:13:27.695376 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:27.695237 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="013f626d-4c83-4506-9867-874cbafd85ab" containerName="revoke-keys" Oct 30 17:13:27.695376 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:27.695255 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="bff503f8-b6d2-41d4-a8c4-883a3afa38a1" containerName="payload-processing" Oct 30 17:13:27.695376 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:27.695275 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="570bc5c9-330c-402e-b262-16b3be6340e7" containerName="istio-proxy" Oct 30 17:13:27.698982 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:27.698960 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d003a5-5cc689c7b6-z4b8q" Oct 30 17:13:27.721750 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:27.721682 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d003a5-5cc689c7b6-z4b8q"] Oct 30 17:13:27.732377 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:27.730973 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-d003a5\"" Oct 30 17:13:27.732377 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:27.731593 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-d003a5-dockercfg-5blbx\"" Oct 30 17:13:27.764113 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:27.764078 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc96q\" (UniqueName: \"kubernetes.io/projected/8f8c9da3-072b-4c71-8cd9-be0dd174e99d-kube-api-access-sc96q\") pod \"payload-processing-e2e-inf-a-d003a5-5cc689c7b6-z4b8q\" (UID: \"8f8c9da3-072b-4c71-8cd9-be0dd174e99d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d003a5-5cc689c7b6-z4b8q" Oct 30 17:13:27.764362 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:27.764148 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f8c9da3-072b-4c71-8cd9-be0dd174e99d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d003a5-5cc689c7b6-z4b8q\" (UID: \"8f8c9da3-072b-4c71-8cd9-be0dd174e99d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d003a5-5cc689c7b6-z4b8q" Oct 30 17:13:27.865992 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:27.865954 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f8c9da3-072b-4c71-8cd9-be0dd174e99d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d003a5-5cc689c7b6-z4b8q\" (UID: \"8f8c9da3-072b-4c71-8cd9-be0dd174e99d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d003a5-5cc689c7b6-z4b8q" Oct 30 17:13:27.866230 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:27.866216 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc96q\" (UniqueName: \"kubernetes.io/projected/8f8c9da3-072b-4c71-8cd9-be0dd174e99d-kube-api-access-sc96q\") pod \"payload-processing-e2e-inf-a-d003a5-5cc689c7b6-z4b8q\" (UID: \"8f8c9da3-072b-4c71-8cd9-be0dd174e99d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d003a5-5cc689c7b6-z4b8q" Oct 30 17:13:27.866866 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:27.866834 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f8c9da3-072b-4c71-8cd9-be0dd174e99d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d003a5-5cc689c7b6-z4b8q\" (UID: \"8f8c9da3-072b-4c71-8cd9-be0dd174e99d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d003a5-5cc689c7b6-z4b8q" Oct 30 17:13:27.903254 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:27.903223 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc96q\" (UniqueName: \"kubernetes.io/projected/8f8c9da3-072b-4c71-8cd9-be0dd174e99d-kube-api-access-sc96q\") pod \"payload-processing-e2e-inf-a-d003a5-5cc689c7b6-z4b8q\" (UID: \"8f8c9da3-072b-4c71-8cd9-be0dd174e99d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d003a5-5cc689c7b6-z4b8q" Oct 30 17:13:28.027577 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:28.027538 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d003a5-5cc689c7b6-z4b8q" Oct 30 17:13:28.448924 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:13:28.448888 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f8c9da3_072b_4c71_8cd9_be0dd174e99d.slice/crio-359c4a92f67f080e8144a397ff536807f8ad98c28965e8aff3f7a5ed541478a6 WatchSource:0}: Error finding container 359c4a92f67f080e8144a397ff536807f8ad98c28965e8aff3f7a5ed541478a6: Status 404 returned error can't find the container with id 359c4a92f67f080e8144a397ff536807f8ad98c28965e8aff3f7a5ed541478a6 Oct 30 17:13:28.450877 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:28.450849 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d003a5-5cc689c7b6-z4b8q"] Oct 30 17:13:28.619597 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:28.619555 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d003a5-5cc689c7b6-z4b8q" event={"ID":"8f8c9da3-072b-4c71-8cd9-be0dd174e99d","Type":"ContainerStarted","Data":"a8d72af0c8590b93f2e8150276dced3eeefa36c637b9fe8a35bac273dec88400"} Oct 30 17:13:28.619776 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:28.619616 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d003a5-5cc689c7b6-z4b8q" event={"ID":"8f8c9da3-072b-4c71-8cd9-be0dd174e99d","Type":"ContainerStarted","Data":"359c4a92f67f080e8144a397ff536807f8ad98c28965e8aff3f7a5ed541478a6"} Oct 30 17:13:28.619776 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:28.619640 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d003a5-5cc689c7b6-z4b8q" Oct 30 17:13:28.673530 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:28.673469 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-d003a5-5cc689c7b6-z4b8q" podStartSLOduration=1.67345185 podStartE2EDuration="1.67345185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:13:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:13:28.672331425 +0000 UTC m=+1582.673086328" watchObservedRunningTime="2026-10-30 17:13:28.67345185 +0000 UTC m=+1582.674206751" Oct 30 17:13:30.537850 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:30.537817 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m" Oct 30 17:13:30.634875 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:30.634792 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4d683a0f-9de0-46dc-9772-d0915b0cb985-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d683a0f-9de0-46dc-9772-d0915b0cb985-istio-podinfo\") pod \"4d683a0f-9de0-46dc-9772-d0915b0cb985\" (UID: \"4d683a0f-9de0-46dc-9772-d0915b0cb985\") " Oct 30 17:13:30.634875 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:30.634853 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d683a0f-9de0-46dc-9772-d0915b0cb985-kube-api-access-pvsr9\" (UniqueName: \"kubernetes.io/projected/4d683a0f-9de0-46dc-9772-d0915b0cb985-kube-api-access-pvsr9\") pod \"4d683a0f-9de0-46dc-9772-d0915b0cb985\" (UID: \"4d683a0f-9de0-46dc-9772-d0915b0cb985\") " Oct 30 17:13:30.635055 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:30.634949 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d683a0f-9de0-46dc-9772-d0915b0cb985-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d683a0f-9de0-46dc-9772-d0915b0cb985-istio-envoy\") pod \"4d683a0f-9de0-46dc-9772-d0915b0cb985\" (UID: \"4d683a0f-9de0-46dc-9772-d0915b0cb985\") " Oct 30 17:13:30.635055 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:30.635020 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d683a0f-9de0-46dc-9772-d0915b0cb985-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d683a0f-9de0-46dc-9772-d0915b0cb985-istio-data\") pod \"4d683a0f-9de0-46dc-9772-d0915b0cb985\" (UID: \"4d683a0f-9de0-46dc-9772-d0915b0cb985\") " Oct 30 17:13:30.635131 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:30.635070 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d683a0f-9de0-46dc-9772-d0915b0cb985-istio-token\" (UniqueName: \"kubernetes.io/projected/4d683a0f-9de0-46dc-9772-d0915b0cb985-istio-token\") pod \"4d683a0f-9de0-46dc-9772-d0915b0cb985\" (UID: \"4d683a0f-9de0-46dc-9772-d0915b0cb985\") " Oct 30 17:13:30.635131 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:30.635116 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d683a0f-9de0-46dc-9772-d0915b0cb985-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d683a0f-9de0-46dc-9772-d0915b0cb985-workload-certs\") pod \"4d683a0f-9de0-46dc-9772-d0915b0cb985\" (UID: \"4d683a0f-9de0-46dc-9772-d0915b0cb985\") " Oct 30 17:13:30.635272 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:30.635254 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d683a0f-9de0-46dc-9772-d0915b0cb985-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d683a0f-9de0-46dc-9772-d0915b0cb985-credential-socket\") pod \"4d683a0f-9de0-46dc-9772-d0915b0cb985\" (UID: \"4d683a0f-9de0-46dc-9772-d0915b0cb985\") " Oct 30 17:13:30.635986 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:30.635939 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d683a0f-9de0-46dc-9772-d0915b0cb985-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d683a0f-9de0-46dc-9772-d0915b0cb985-workload-socket\") pod \"4d683a0f-9de0-46dc-9772-d0915b0cb985\" (UID: \"4d683a0f-9de0-46dc-9772-d0915b0cb985\") " Oct 30 17:13:30.636112 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:30.636050 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d683a0f-9de0-46dc-9772-d0915b0cb985-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d683a0f-9de0-46dc-9772-d0915b0cb985-istiod-ca-cert\") pod \"4d683a0f-9de0-46dc-9772-d0915b0cb985\" (UID: \"4d683a0f-9de0-46dc-9772-d0915b0cb985\") " Oct 30 17:13:30.636192 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:30.636140 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d683a0f-9de0-46dc-9772-d0915b0cb985-istio-data" pod "4d683a0f-9de0-46dc-9772-d0915b0cb985" (UID: "4d683a0f-9de0-46dc-9772-d0915b0cb985"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:13:30.636368 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:30.636343 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d683a0f-9de0-46dc-9772-d0915b0cb985-credential-socket" pod "4d683a0f-9de0-46dc-9772-d0915b0cb985" (UID: "4d683a0f-9de0-46dc-9772-d0915b0cb985"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:13:30.636499 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:30.636377 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d683a0f-9de0-46dc-9772-d0915b0cb985-workload-certs" pod "4d683a0f-9de0-46dc-9772-d0915b0cb985" (UID: "4d683a0f-9de0-46dc-9772-d0915b0cb985"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:13:30.636631 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:30.636612 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d683a0f-9de0-46dc-9772-d0915b0cb985-workload-socket" pod "4d683a0f-9de0-46dc-9772-d0915b0cb985" (UID: "4d683a0f-9de0-46dc-9772-d0915b0cb985"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:13:30.637035 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:30.637010 4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d683a0f-9de0-46dc-9772-d0915b0cb985-istio-data\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:13:30.637129 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:30.637042 4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d683a0f-9de0-46dc-9772-d0915b0cb985-workload-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:13:30.637129 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:30.637055 4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d683a0f-9de0-46dc-9772-d0915b0cb985-credential-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:13:30.637129 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:30.637070 4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d683a0f-9de0-46dc-9772-d0915b0cb985-workload-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:13:30.638083 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:30.638056 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d683a0f-9de0-46dc-9772-d0915b0cb985-istiod-ca-cert" pod "4d683a0f-9de0-46dc-9772-d0915b0cb985" (UID: "4d683a0f-9de0-46dc-9772-d0915b0cb985"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:30.638202 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:30.638124 4522 generic.go:356] "Generic (PLEG): container finished" podID="4d683a0f-9de0-46dc-9772-d0915b0cb985" containerID="d8434a2a67396bb270b71f833213bbe6b7fb4eacbbbd26781a03bce6f5eacbf7" exitCode=0 Oct 30 17:13:30.638269 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:30.638231 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m" event={"ID":"4d683a0f-9de0-46dc-9772-d0915b0cb985","Type":"ContainerDied","Data":"d8434a2a67396bb270b71f833213bbe6b7fb4eacbbbd26781a03bce6f5eacbf7"} Oct 30 17:13:30.638343 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:30.638292 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m" event={"ID":"4d683a0f-9de0-46dc-9772-d0915b0cb985","Type":"ContainerDied","Data":"71b12b9d9a1d05168da6f3cb56c4dc052ec1d898e0aa2c1506b194408c3df704"} Oct 30 17:13:30.638343 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:30.638337 4522 scope.go:122] "RemoveContainer" containerID="d8434a2a67396bb270b71f833213bbe6b7fb4eacbbbd26781a03bce6f5eacbf7" Oct 30 17:13:30.638535 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:30.638520 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m" Oct 30 17:13:30.638957 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:30.638933 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d683a0f-9de0-46dc-9772-d0915b0cb985-kube-api-access-pvsr9" pod "4d683a0f-9de0-46dc-9772-d0915b0cb985" (UID: "4d683a0f-9de0-46dc-9772-d0915b0cb985"). InnerVolumeSpecName "kube-api-access-pvsr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:30.639607 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:30.639584 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d683a0f-9de0-46dc-9772-d0915b0cb985-istio-token" pod "4d683a0f-9de0-46dc-9772-d0915b0cb985" (UID: "4d683a0f-9de0-46dc-9772-d0915b0cb985"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:30.639837 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:30.639817 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d683a0f-9de0-46dc-9772-d0915b0cb985-istio-envoy" pod "4d683a0f-9de0-46dc-9772-d0915b0cb985" (UID: "4d683a0f-9de0-46dc-9772-d0915b0cb985"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:13:30.640681 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:30.640652 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d683a0f-9de0-46dc-9772-d0915b0cb985-istio-podinfo" pod "4d683a0f-9de0-46dc-9772-d0915b0cb985" (UID: "4d683a0f-9de0-46dc-9772-d0915b0cb985"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:13:30.690165 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:30.690134 4522 scope.go:122] "RemoveContainer" containerID="35d32d97fd6f7931d29ae91feb87af869106f6d3cf72391bf7f35b37dbd5e81a" Oct 30 17:13:30.701605 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:30.701580 4522 scope.go:122] "RemoveContainer" containerID="d8434a2a67396bb270b71f833213bbe6b7fb4eacbbbd26781a03bce6f5eacbf7" Oct 30 17:13:30.701938 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:13:30.701915 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8434a2a67396bb270b71f833213bbe6b7fb4eacbbbd26781a03bce6f5eacbf7\": container with ID starting with d8434a2a67396bb270b71f833213bbe6b7fb4eacbbbd26781a03bce6f5eacbf7 not found: ID does not exist" containerID="d8434a2a67396bb270b71f833213bbe6b7fb4eacbbbd26781a03bce6f5eacbf7" Oct 30 17:13:30.702016 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:30.701951 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8434a2a67396bb270b71f833213bbe6b7fb4eacbbbd26781a03bce6f5eacbf7"} err="failed to get container status \"d8434a2a67396bb270b71f833213bbe6b7fb4eacbbbd26781a03bce6f5eacbf7\": rpc error: code = NotFound desc = could not find container \"d8434a2a67396bb270b71f833213bbe6b7fb4eacbbbd26781a03bce6f5eacbf7\": container with ID starting with d8434a2a67396bb270b71f833213bbe6b7fb4eacbbbd26781a03bce6f5eacbf7 not found: ID does not exist" Oct 30 17:13:30.702016 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:30.701973 4522 scope.go:122] "RemoveContainer" containerID="35d32d97fd6f7931d29ae91feb87af869106f6d3cf72391bf7f35b37dbd5e81a" Oct 30 17:13:30.702284 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:13:30.702259 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35d32d97fd6f7931d29ae91feb87af869106f6d3cf72391bf7f35b37dbd5e81a\": container with ID starting with 35d32d97fd6f7931d29ae91feb87af869106f6d3cf72391bf7f35b37dbd5e81a not found: ID does not exist" containerID="35d32d97fd6f7931d29ae91feb87af869106f6d3cf72391bf7f35b37dbd5e81a" Oct 30 17:13:30.702348 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:30.702296 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35d32d97fd6f7931d29ae91feb87af869106f6d3cf72391bf7f35b37dbd5e81a"} err="failed to get container status \"35d32d97fd6f7931d29ae91feb87af869106f6d3cf72391bf7f35b37dbd5e81a\": rpc error: code = NotFound desc = could not find container \"35d32d97fd6f7931d29ae91feb87af869106f6d3cf72391bf7f35b37dbd5e81a\": container with ID starting with 35d32d97fd6f7931d29ae91feb87af869106f6d3cf72391bf7f35b37dbd5e81a not found: ID does not exist" Oct 30 17:13:30.738918 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:30.738877 4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d683a0f-9de0-46dc-9772-d0915b0cb985-istio-podinfo\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:13:30.738918 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:30.738911 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvsr9\" (UniqueName: \"kubernetes.io/projected/4d683a0f-9de0-46dc-9772-d0915b0cb985-kube-api-access-pvsr9\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:13:30.738918 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:30.738922 4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d683a0f-9de0-46dc-9772-d0915b0cb985-istio-envoy\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:13:30.739152 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:30.738931 4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d683a0f-9de0-46dc-9772-d0915b0cb985-istio-token\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:13:30.739152 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:30.738941 4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d683a0f-9de0-46dc-9772-d0915b0cb985-istiod-ca-cert\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:13:30.991463 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:30.991431 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m"] Oct 30 17:13:30.998061 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:30.998038 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-362bbf-openshift-default-7cccfb5bd5-nrg7m"] Oct 30 17:13:31.694203 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:31.694170 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-432680-s6l4m"] Oct 30 17:13:31.694908 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:31.694892 4522 cpu_manager.go:414] "containerMap: removing container" podUID="4d683a0f-9de0-46dc-9772-d0915b0cb985" containerName="istio-proxy" Oct 30 17:13:31.694951 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:31.694910 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d683a0f-9de0-46dc-9772-d0915b0cb985" containerName="istio-proxy" Oct 30 17:13:31.694951 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:31.694929 4522 cpu_manager.go:414] "containerMap: removing container" podUID="4d683a0f-9de0-46dc-9772-d0915b0cb985" containerName="istio-proxy" Oct 30 17:13:31.694951 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:31.694934 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d683a0f-9de0-46dc-9772-d0915b0cb985" containerName="istio-proxy" Oct 30 17:13:31.695059 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:31.695049 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d683a0f-9de0-46dc-9772-d0915b0cb985" containerName="istio-proxy" Oct 30 17:13:31.700197 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:31.700180 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-432680-s6l4m" Oct 30 17:13:31.716504 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:31.716475 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-432680-s6l4m"] Oct 30 17:13:31.717661 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:31.717639 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:13:31.725942 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:31.725922 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:13:31.857995 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:31.857960 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff5bs\" (UniqueName: \"kubernetes.io/projected/85597425-1abd-470a-b407-3c095a55bef9-kube-api-access-ff5bs\") pod \"maas-api-revoke-keys-e2e-api-a-432680-s6l4m\" (UID: \"85597425-1abd-470a-b407-3c095a55bef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-432680-s6l4m" Oct 30 17:13:31.858188 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:31.858084 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85597425-1abd-470a-b407-3c095a55bef9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-432680-s6l4m\" (UID: \"85597425-1abd-470a-b407-3c095a55bef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-432680-s6l4m" Oct 30 17:13:31.858297 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:31.858270 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85597425-1abd-470a-b407-3c095a55bef9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-432680-s6l4m\" (UID: \"85597425-1abd-470a-b407-3c095a55bef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-432680-s6l4m" Oct 30 17:13:31.959428 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:31.959391 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85597425-1abd-470a-b407-3c095a55bef9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-432680-s6l4m\" (UID: \"85597425-1abd-470a-b407-3c095a55bef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-432680-s6l4m" Oct 30 17:13:31.959641 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:31.959576 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85597425-1abd-470a-b407-3c095a55bef9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-432680-s6l4m\" (UID: \"85597425-1abd-470a-b407-3c095a55bef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-432680-s6l4m" Oct 30 17:13:31.959765 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:31.959750 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff5bs\" (UniqueName: \"kubernetes.io/projected/85597425-1abd-470a-b407-3c095a55bef9-kube-api-access-ff5bs\") pod \"maas-api-revoke-keys-e2e-api-a-432680-s6l4m\" (UID: \"85597425-1abd-470a-b407-3c095a55bef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-432680-s6l4m" Oct 30 17:13:31.960064 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:31.960039 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85597425-1abd-470a-b407-3c095a55bef9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-432680-s6l4m\" (UID: \"85597425-1abd-470a-b407-3c095a55bef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-432680-s6l4m" Oct 30 17:13:31.961961 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:31.961940 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85597425-1abd-470a-b407-3c095a55bef9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-432680-s6l4m\" (UID: \"85597425-1abd-470a-b407-3c095a55bef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-432680-s6l4m" Oct 30 17:13:31.986008 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:31.985979 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff5bs\" (UniqueName: \"kubernetes.io/projected/85597425-1abd-470a-b407-3c095a55bef9-kube-api-access-ff5bs\") pod \"maas-api-revoke-keys-e2e-api-a-432680-s6l4m\" (UID: \"85597425-1abd-470a-b407-3c095a55bef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-432680-s6l4m" Oct 30 17:13:32.015901 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:32.015870 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-432680-s6l4m" Oct 30 17:13:32.152621 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:32.152584 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d683a0f-9de0-46dc-9772-d0915b0cb985" path="/var/lib/kubelet/pods/4d683a0f-9de0-46dc-9772-d0915b0cb985/volumes" Oct 30 17:13:32.331274 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:13:32.331238 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85597425_1abd_470a_b407_3c095a55bef9.slice/crio-45c80e88e5b7c591fa9e90914228277c5cf854d92d9bb019c52138c8165b70b8 WatchSource:0}: Error finding container 45c80e88e5b7c591fa9e90914228277c5cf854d92d9bb019c52138c8165b70b8: Status 404 returned error can't find the container with id 45c80e88e5b7c591fa9e90914228277c5cf854d92d9bb019c52138c8165b70b8 Oct 30 17:13:32.333103 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:32.333072 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-432680-s6l4m"] Oct 30 17:13:32.658363 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:32.658321 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-432680-s6l4m" event={"ID":"85597425-1abd-470a-b407-3c095a55bef9","Type":"ContainerStarted","Data":"e6257cf785e7c6eb273e6c8c1cd921ca9a9973b8a084bdb970d8bb683d301e6d"} Oct 30 17:13:32.658548 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:32.658414 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-432680-s6l4m" event={"ID":"85597425-1abd-470a-b407-3c095a55bef9","Type":"ContainerStarted","Data":"45c80e88e5b7c591fa9e90914228277c5cf854d92d9bb019c52138c8165b70b8"} Oct 30 17:13:32.746128 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:32.746072 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-432680-s6l4m" podStartSLOduration=1.746058426 podStartE2EDuration="1.746058426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:13:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:13:32.745025014 +0000 UTC m=+1586.745779914" watchObservedRunningTime="2026-10-30 17:13:32.746058426 +0000 UTC m=+1586.746813327" Oct 30 17:13:33.672768 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:33.672736 4522 generic.go:356] "Generic (PLEG): container finished" podID="85597425-1abd-470a-b407-3c095a55bef9" containerID="e6257cf785e7c6eb273e6c8c1cd921ca9a9973b8a084bdb970d8bb683d301e6d" exitCode=0 Oct 30 17:13:33.672948 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:33.672776 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-432680-s6l4m" event={"ID":"85597425-1abd-470a-b407-3c095a55bef9","Type":"ContainerDied","Data":"e6257cf785e7c6eb273e6c8c1cd921ca9a9973b8a084bdb970d8bb683d301e6d"} Oct 30 17:13:33.931603 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:33.931526 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f42baf-58fcd7df8-kshhs"] Oct 30 17:13:33.932370 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:33.932354 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d683a0f-9de0-46dc-9772-d0915b0cb985" containerName="istio-proxy" Oct 30 17:13:33.936357 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:33.936337 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f42baf-58fcd7df8-kshhs" Oct 30 17:13:33.954409 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:33.954370 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-f42baf\"" Oct 30 17:13:33.957874 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:33.957850 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f42baf-58fcd7df8-kshhs"] Oct 30 17:13:33.999385 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:33.999330 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f42baf-6c555f88f9-k8szt"] Oct 30 17:13:34.003917 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:34.003896 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f42baf-6c555f88f9-k8szt" Oct 30 17:13:34.021439 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:34.021402 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-f42baf-dockercfg-pdcpf\"" Oct 30 17:13:34.025567 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:34.025545 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f42baf-6c555f88f9-k8szt"] Oct 30 17:13:34.143332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:34.142537 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdct4\" (UniqueName: \"kubernetes.io/projected/46c462ae-1044-4037-939b-43ad0906630d-kube-api-access-fdct4\") pod \"payload-pre-processing-e2e-inf-a-f42baf-6c555f88f9-k8szt\" (UID: \"46c462ae-1044-4037-939b-43ad0906630d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f42baf-6c555f88f9-k8szt" Oct 30 17:13:34.143332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:34.142808 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46c462ae-1044-4037-939b-43ad0906630d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f42baf-6c555f88f9-k8szt\" (UID: \"46c462ae-1044-4037-939b-43ad0906630d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f42baf-6c555f88f9-k8szt" Oct 30 17:13:34.143332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:34.142870 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrbzd\" (UniqueName: \"kubernetes.io/projected/e9629e12-7b8a-4e59-acb9-8e7476994826-kube-api-access-hrbzd\") pod \"payload-processing-e2e-inf-a-f42baf-58fcd7df8-kshhs\" (UID: \"e9629e12-7b8a-4e59-acb9-8e7476994826\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f42baf-58fcd7df8-kshhs" Oct 30 17:13:34.143332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:34.142923 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9629e12-7b8a-4e59-acb9-8e7476994826-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f42baf-58fcd7df8-kshhs\" (UID: \"e9629e12-7b8a-4e59-acb9-8e7476994826\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f42baf-58fcd7df8-kshhs" Oct 30 17:13:34.245818 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:34.245480 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46c462ae-1044-4037-939b-43ad0906630d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f42baf-6c555f88f9-k8szt\" (UID: \"46c462ae-1044-4037-939b-43ad0906630d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f42baf-6c555f88f9-k8szt" Oct 30 17:13:34.245818 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:34.245549 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46c462ae-1044-4037-939b-43ad0906630d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f42baf-6c555f88f9-k8szt\" (UID: \"46c462ae-1044-4037-939b-43ad0906630d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f42baf-6c555f88f9-k8szt" Oct 30 17:13:34.245818 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:34.245596 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrbzd\" (UniqueName: \"kubernetes.io/projected/e9629e12-7b8a-4e59-acb9-8e7476994826-kube-api-access-hrbzd\") pod \"payload-processing-e2e-inf-a-f42baf-58fcd7df8-kshhs\" (UID: \"e9629e12-7b8a-4e59-acb9-8e7476994826\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f42baf-58fcd7df8-kshhs" Oct 30 17:13:34.245818 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:34.245628 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9629e12-7b8a-4e59-acb9-8e7476994826-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f42baf-58fcd7df8-kshhs\" (UID: \"e9629e12-7b8a-4e59-acb9-8e7476994826\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f42baf-58fcd7df8-kshhs" Oct 30 17:13:34.245818 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:34.245757 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdct4\" (UniqueName: \"kubernetes.io/projected/46c462ae-1044-4037-939b-43ad0906630d-kube-api-access-fdct4\") pod \"payload-pre-processing-e2e-inf-a-f42baf-6c555f88f9-k8szt\" (UID: \"46c462ae-1044-4037-939b-43ad0906630d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f42baf-6c555f88f9-k8szt" Oct 30 17:13:34.247720 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:34.247148 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9629e12-7b8a-4e59-acb9-8e7476994826-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f42baf-58fcd7df8-kshhs\" (UID: \"e9629e12-7b8a-4e59-acb9-8e7476994826\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f42baf-58fcd7df8-kshhs" Oct 30 17:13:34.291401 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:34.273218 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrbzd\" (UniqueName: \"kubernetes.io/projected/e9629e12-7b8a-4e59-acb9-8e7476994826-kube-api-access-hrbzd\") pod \"payload-processing-e2e-inf-a-f42baf-58fcd7df8-kshhs\" (UID: \"e9629e12-7b8a-4e59-acb9-8e7476994826\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f42baf-58fcd7df8-kshhs" Oct 30 17:13:34.294352 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:34.294285 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdct4\" (UniqueName: \"kubernetes.io/projected/46c462ae-1044-4037-939b-43ad0906630d-kube-api-access-fdct4\") pod \"payload-pre-processing-e2e-inf-a-f42baf-6c555f88f9-k8szt\" (UID: \"46c462ae-1044-4037-939b-43ad0906630d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f42baf-6c555f88f9-k8szt" Oct 30 17:13:34.319050 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:34.319020 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f42baf-6c555f88f9-k8szt" Oct 30 17:13:34.550736 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:34.550660 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f42baf-58fcd7df8-kshhs" Oct 30 17:13:34.640357 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:13:34.640327 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46c462ae_1044_4037_939b_43ad0906630d.slice/crio-a80ffd6513c902d82a591996c25b9d6755f6e8d84eada3c04cf9d085b846c596 WatchSource:0}: Error finding container a80ffd6513c902d82a591996c25b9d6755f6e8d84eada3c04cf9d085b846c596: Status 404 returned error can't find the container with id a80ffd6513c902d82a591996c25b9d6755f6e8d84eada3c04cf9d085b846c596 Oct 30 17:13:34.641877 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:34.641851 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f42baf-6c555f88f9-k8szt"] Oct 30 17:13:34.686847 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:34.686797 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f42baf-6c555f88f9-k8szt" event={"ID":"46c462ae-1044-4037-939b-43ad0906630d","Type":"ContainerStarted","Data":"a80ffd6513c902d82a591996c25b9d6755f6e8d84eada3c04cf9d085b846c596"} Oct 30 17:13:34.835483 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:34.835457 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-432680-s6l4m" Oct 30 17:13:34.852692 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:34.852662 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85597425-1abd-470a-b407-3c095a55bef9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85597425-1abd-470a-b407-3c095a55bef9-maas-api-xks-ca\") pod \"85597425-1abd-470a-b407-3c095a55bef9\" (UID: \"85597425-1abd-470a-b407-3c095a55bef9\") " Oct 30 17:13:34.852827 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:34.852731 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85597425-1abd-470a-b407-3c095a55bef9-kube-api-access-ff5bs\" (UniqueName: \"kubernetes.io/projected/85597425-1abd-470a-b407-3c095a55bef9-kube-api-access-ff5bs\") pod \"85597425-1abd-470a-b407-3c095a55bef9\" (UID: \"85597425-1abd-470a-b407-3c095a55bef9\") " Oct 30 17:13:34.853038 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:34.853017 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85597425-1abd-470a-b407-3c095a55bef9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85597425-1abd-470a-b407-3c095a55bef9-maas-api-openshift-ca\") pod \"85597425-1abd-470a-b407-3c095a55bef9\" (UID: \"85597425-1abd-470a-b407-3c095a55bef9\") " Oct 30 17:13:34.853467 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:34.853439 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85597425-1abd-470a-b407-3c095a55bef9-maas-api-openshift-ca" pod "85597425-1abd-470a-b407-3c095a55bef9" (UID: "85597425-1abd-470a-b407-3c095a55bef9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:34.853676 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:34.853654 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85597425-1abd-470a-b407-3c095a55bef9-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:13:34.855050 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:34.854927 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85597425-1abd-470a-b407-3c095a55bef9-maas-api-xks-ca" pod "85597425-1abd-470a-b407-3c095a55bef9" (UID: "85597425-1abd-470a-b407-3c095a55bef9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:13:34.855191 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:34.855167 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85597425-1abd-470a-b407-3c095a55bef9-kube-api-access-ff5bs" pod "85597425-1abd-470a-b407-3c095a55bef9" (UID: "85597425-1abd-470a-b407-3c095a55bef9"). InnerVolumeSpecName "kube-api-access-ff5bs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:34.859458 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:13:34.859432 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9629e12_7b8a_4e59_acb9_8e7476994826.slice/crio-5be3a18557a5f04f323b0e7e7e1fa8af0937d0aa333b14f07c0127ed1a88a607 WatchSource:0}: Error finding container 5be3a18557a5f04f323b0e7e7e1fa8af0937d0aa333b14f07c0127ed1a88a607: Status 404 returned error can't find the container with id 5be3a18557a5f04f323b0e7e7e1fa8af0937d0aa333b14f07c0127ed1a88a607 Oct 30 17:13:34.861705 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:34.861683 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f42baf-58fcd7df8-kshhs"] Oct 30 17:13:34.955258 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:34.955224 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85597425-1abd-470a-b407-3c095a55bef9-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:13:34.955258 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:34.955255 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ff5bs\" (UniqueName: \"kubernetes.io/projected/85597425-1abd-470a-b407-3c095a55bef9-kube-api-access-ff5bs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:13:35.709872 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:35.709833 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f42baf-6c555f88f9-k8szt" event={"ID":"46c462ae-1044-4037-939b-43ad0906630d","Type":"ContainerStarted","Data":"708c7b66d9f4c6246fcc1c976856835d6200f2ab9683119c0813ff60c3b2348a"} Oct 30 17:13:35.710231 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:35.710020 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f42baf-6c555f88f9-k8szt" Oct 30 17:13:35.712204 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:35.712170 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f42baf-58fcd7df8-kshhs" event={"ID":"e9629e12-7b8a-4e59-acb9-8e7476994826","Type":"ContainerStarted","Data":"aebbd3d5e02197db7c1da22c7e704f05a5d794038600d9d84a395bcbb5b1e83f"} Oct 30 17:13:35.712374 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:35.712225 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f42baf-58fcd7df8-kshhs" event={"ID":"e9629e12-7b8a-4e59-acb9-8e7476994826","Type":"ContainerStarted","Data":"5be3a18557a5f04f323b0e7e7e1fa8af0937d0aa333b14f07c0127ed1a88a607"} Oct 30 17:13:35.712374 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:35.712254 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-f42baf-58fcd7df8-kshhs" Oct 30 17:13:35.719340 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:35.719295 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-432680-s6l4m" Oct 30 17:13:35.719453 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:35.719343 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-432680-s6l4m" event={"ID":"85597425-1abd-470a-b407-3c095a55bef9","Type":"ContainerDied","Data":"45c80e88e5b7c591fa9e90914228277c5cf854d92d9bb019c52138c8165b70b8"} Oct 30 17:13:35.719453 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:35.719380 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45c80e88e5b7c591fa9e90914228277c5cf854d92d9bb019c52138c8165b70b8" Oct 30 17:13:35.757233 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:35.757175 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f42baf-6c555f88f9-k8szt" podStartSLOduration=2.757163482 podStartE2EDuration="2.757163482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:13:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:13:35.756566044 +0000 UTC m=+1589.757320957" watchObservedRunningTime="2026-10-30 17:13:35.757163482 +0000 UTC m=+1589.757918407" Oct 30 17:13:35.803974 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:35.803925 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-f42baf-58fcd7df8-kshhs" podStartSLOduration=2.803911655 podStartE2EDuration="2.803911655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:13:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:13:35.80214427 +0000 UTC m=+1589.802899172" watchObservedRunningTime="2026-10-30 17:13:35.803911655 +0000 UTC m=+1589.804666556" Oct 30 17:13:39.631342 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:39.631294 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d003a5-5cc689c7b6-z4b8q" Oct 30 17:13:40.361454 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:40.361417 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97"] Oct 30 17:13:40.362212 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:40.362191 4522 cpu_manager.go:414] "containerMap: removing container" podUID="85597425-1abd-470a-b407-3c095a55bef9" containerName="revoke-keys" Oct 30 17:13:40.362348 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:40.362213 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85597425-1abd-470a-b407-3c095a55bef9" containerName="revoke-keys" Oct 30 17:13:40.362470 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:40.362456 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="85597425-1abd-470a-b407-3c095a55bef9" containerName="revoke-keys" Oct 30 17:13:40.366462 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:40.366442 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97" Oct 30 17:13:40.384142 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:40.384116 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-728ad7-openshift-default-dockercfg-hsckd\"" Oct 30 17:13:40.403235 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:40.403197 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97"] Oct 30 17:13:40.443820 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:40.443781 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed5776a0-2912-446d-a83f-8c84a75e5cbf-credential-socket\") pod \"e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97\" (UID: \"ed5776a0-2912-446d-a83f-8c84a75e5cbf\") " pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97" Oct 30 17:13:40.443998 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:40.443928 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed5776a0-2912-446d-a83f-8c84a75e5cbf-istio-podinfo\") pod \"e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97\" (UID: \"ed5776a0-2912-446d-a83f-8c84a75e5cbf\") " pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97" Oct 30 17:13:40.443998 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:40.443966 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed5776a0-2912-446d-a83f-8c84a75e5cbf-istio-data\") pod \"e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97\" (UID: \"ed5776a0-2912-446d-a83f-8c84a75e5cbf\") " pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97" Oct 30 17:13:40.444124 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:40.444062 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed5776a0-2912-446d-a83f-8c84a75e5cbf-workload-socket\") pod \"e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97\" (UID: \"ed5776a0-2912-446d-a83f-8c84a75e5cbf\") " pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97" Oct 30 17:13:40.444179 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:40.444123 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h6wg\" (UniqueName: \"kubernetes.io/projected/ed5776a0-2912-446d-a83f-8c84a75e5cbf-kube-api-access-2h6wg\") pod \"e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97\" (UID: \"ed5776a0-2912-446d-a83f-8c84a75e5cbf\") " pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97" Oct 30 17:13:40.444237 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:40.444194 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed5776a0-2912-446d-a83f-8c84a75e5cbf-istiod-ca-cert\") pod \"e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97\" (UID: \"ed5776a0-2912-446d-a83f-8c84a75e5cbf\") " pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97" Oct 30 17:13:40.444319 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:40.444293 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed5776a0-2912-446d-a83f-8c84a75e5cbf-istio-token\") pod \"e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97\" (UID: \"ed5776a0-2912-446d-a83f-8c84a75e5cbf\") " pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97" Oct 30 17:13:40.444362 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:40.444343 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed5776a0-2912-446d-a83f-8c84a75e5cbf-workload-certs\") pod \"e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97\" (UID: \"ed5776a0-2912-446d-a83f-8c84a75e5cbf\") " pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97" Oct 30 17:13:40.444396 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:40.444366 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed5776a0-2912-446d-a83f-8c84a75e5cbf-istio-envoy\") pod \"e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97\" (UID: \"ed5776a0-2912-446d-a83f-8c84a75e5cbf\") " pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97" Oct 30 17:13:40.545489 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:40.545448 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed5776a0-2912-446d-a83f-8c84a75e5cbf-workload-certs\") pod \"e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97\" (UID: \"ed5776a0-2912-446d-a83f-8c84a75e5cbf\") " pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97" Oct 30 17:13:40.545489 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:40.545484 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed5776a0-2912-446d-a83f-8c84a75e5cbf-istio-envoy\") pod \"e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97\" (UID: \"ed5776a0-2912-446d-a83f-8c84a75e5cbf\") " pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97" Oct 30 17:13:40.545756 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:40.545552 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed5776a0-2912-446d-a83f-8c84a75e5cbf-credential-socket\") pod \"e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97\" (UID: \"ed5776a0-2912-446d-a83f-8c84a75e5cbf\") " pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97" Oct 30 17:13:40.545756 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:40.545633 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed5776a0-2912-446d-a83f-8c84a75e5cbf-istio-podinfo\") pod \"e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97\" (UID: \"ed5776a0-2912-446d-a83f-8c84a75e5cbf\") " pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97" Oct 30 17:13:40.545756 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:40.545660 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed5776a0-2912-446d-a83f-8c84a75e5cbf-istio-data\") pod \"e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97\" (UID: \"ed5776a0-2912-446d-a83f-8c84a75e5cbf\") " pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97" Oct 30 17:13:40.545756 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:40.545713 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed5776a0-2912-446d-a83f-8c84a75e5cbf-workload-socket\") pod \"e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97\" (UID: \"ed5776a0-2912-446d-a83f-8c84a75e5cbf\") " pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97" Oct 30 17:13:40.545989 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:40.545761 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h6wg\" (UniqueName: \"kubernetes.io/projected/ed5776a0-2912-446d-a83f-8c84a75e5cbf-kube-api-access-2h6wg\") pod \"e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97\" (UID: \"ed5776a0-2912-446d-a83f-8c84a75e5cbf\") " pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97" Oct 30 17:13:40.545989 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:40.545830 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed5776a0-2912-446d-a83f-8c84a75e5cbf-istiod-ca-cert\") pod \"e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97\" (UID: \"ed5776a0-2912-446d-a83f-8c84a75e5cbf\") " pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97" Oct 30 17:13:40.545989 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:40.545919 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed5776a0-2912-446d-a83f-8c84a75e5cbf-workload-certs\") pod \"e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97\" (UID: \"ed5776a0-2912-446d-a83f-8c84a75e5cbf\") " pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97" Oct 30 17:13:40.545989 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:40.545948 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed5776a0-2912-446d-a83f-8c84a75e5cbf-istio-token\") pod \"e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97\" (UID: \"ed5776a0-2912-446d-a83f-8c84a75e5cbf\") " pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97" Oct 30 17:13:40.546216 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:40.546018 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed5776a0-2912-446d-a83f-8c84a75e5cbf-credential-socket\") pod \"e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97\" (UID: \"ed5776a0-2912-446d-a83f-8c84a75e5cbf\") " pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97" Oct 30 17:13:40.546216 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:40.546153 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed5776a0-2912-446d-a83f-8c84a75e5cbf-workload-socket\") pod \"e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97\" (UID: \"ed5776a0-2912-446d-a83f-8c84a75e5cbf\") " pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97" Oct 30 17:13:40.546553 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:40.546491 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed5776a0-2912-446d-a83f-8c84a75e5cbf-istio-data\") pod \"e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97\" (UID: \"ed5776a0-2912-446d-a83f-8c84a75e5cbf\") " pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97" Oct 30 17:13:40.546638 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:40.546579 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed5776a0-2912-446d-a83f-8c84a75e5cbf-istiod-ca-cert\") pod \"e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97\" (UID: \"ed5776a0-2912-446d-a83f-8c84a75e5cbf\") " pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97" Oct 30 17:13:40.548052 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:40.548030 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed5776a0-2912-446d-a83f-8c84a75e5cbf-istio-envoy\") pod \"e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97\" (UID: \"ed5776a0-2912-446d-a83f-8c84a75e5cbf\") " pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97" Oct 30 17:13:40.548152 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:40.548100 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed5776a0-2912-446d-a83f-8c84a75e5cbf-istio-podinfo\") pod \"e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97\" (UID: \"ed5776a0-2912-446d-a83f-8c84a75e5cbf\") " pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97" Oct 30 17:13:40.572086 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:40.572051 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h6wg\" (UniqueName: \"kubernetes.io/projected/ed5776a0-2912-446d-a83f-8c84a75e5cbf-kube-api-access-2h6wg\") pod \"e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97\" (UID: \"ed5776a0-2912-446d-a83f-8c84a75e5cbf\") " pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97" Oct 30 17:13:40.572194 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:40.572107 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed5776a0-2912-446d-a83f-8c84a75e5cbf-istio-token\") pod \"e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97\" (UID: \"ed5776a0-2912-446d-a83f-8c84a75e5cbf\") " pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97" Oct 30 17:13:40.680608 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:40.680509 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97" Oct 30 17:13:41.004346 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:13:41.004299 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded5776a0_2912_446d_a83f_8c84a75e5cbf.slice/crio-c68a3b6a9856b64756962b910ab54ad60a434add1939f49c917f8dcdf73a2b03 WatchSource:0}: Error finding container c68a3b6a9856b64756962b910ab54ad60a434add1939f49c917f8dcdf73a2b03: Status 404 returned error can't find the container with id c68a3b6a9856b64756962b910ab54ad60a434add1939f49c917f8dcdf73a2b03 Oct 30 17:13:41.006767 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:41.006741 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97"] Oct 30 17:13:41.007249 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:41.007219 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:13:41.007297 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:41.007283 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:13:41.007348 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:41.007328 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:13:41.775442 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:41.775407 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97" event={"ID":"ed5776a0-2912-446d-a83f-8c84a75e5cbf","Type":"ContainerStarted","Data":"3745dd62c7c27502c535aa6beaeac2478495fdc937d908cd054412b9f4b5d220"} Oct 30 17:13:41.775857 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:41.775466 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97" event={"ID":"ed5776a0-2912-446d-a83f-8c84a75e5cbf","Type":"ContainerStarted","Data":"c68a3b6a9856b64756962b910ab54ad60a434add1939f49c917f8dcdf73a2b03"} Oct 30 17:13:41.861895 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:41.861830 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97" podStartSLOduration=1.861815901 podStartE2EDuration="1.861815901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:13:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:13:41.844057186 +0000 UTC m=+1595.844812088" watchObservedRunningTime="2026-10-30 17:13:41.861815901 +0000 UTC m=+1595.862570802" Oct 30 17:13:42.681861 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:42.681827 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97" Oct 30 17:13:42.684670 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:42.684640 4522 patch_prober.go:29] interesting pod/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 30 17:13:42.684932 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:42.684904 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97" podUID="ed5776a0-2912-446d-a83f-8c84a75e5cbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 30 17:13:42.950350 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:13:42.950231 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:13:43.033477 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:43.033443 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12388952-l92fj"] Oct 30 17:13:43.047657 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:43.047628 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12388952-l92fj"] Oct 30 17:13:43.681134 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:43.681094 4522 patch_prober.go:29] interesting pod/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 30 17:13:43.681363 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:43.681160 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97" podUID="ed5776a0-2912-446d-a83f-8c84a75e5cbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 30 17:13:44.146955 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:44.146921 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb824f5f-750a-41f1-ba75-51de3cfa276c" path="/var/lib/kubelet/pods/bb824f5f-750a-41f1-ba75-51de3cfa276c/volumes" Oct 30 17:13:44.681148 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:44.681113 4522 patch_prober.go:29] interesting pod/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 30 17:13:44.681361 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:44.681184 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97" podUID="ed5776a0-2912-446d-a83f-8c84a75e5cbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 30 17:13:45.681323 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:45.681235 4522 patch_prober.go:29] interesting pod/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 30 17:13:45.681905 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:45.681345 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97" podUID="ed5776a0-2912-446d-a83f-8c84a75e5cbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 30 17:13:46.680948 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:46.680912 4522 patch_prober.go:29] interesting pod/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 30 17:13:46.681111 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:46.680964 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97" podUID="ed5776a0-2912-446d-a83f-8c84a75e5cbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 30 17:13:46.729696 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:46.729661 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f42baf-6c555f88f9-k8szt" Oct 30 17:13:46.730112 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:46.730092 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-f42baf-58fcd7df8-kshhs" Oct 30 17:13:47.777280 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:47.776631 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97" Oct 30 17:13:47.778237 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:47.778202 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97" Oct 30 17:13:47.779176 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:47.779156 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97" Oct 30 17:13:54.483882 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:54.483845 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8496cdb9-67f54"] Oct 30 17:13:54.489756 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:54.489722 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8496cdb9-67f54" Oct 30 17:13:54.511696 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:54.511658 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8496cdb9-67f54"] Oct 30 17:13:54.520359 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:54.520293 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:13:54.520559 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:54.520442 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:13:54.641208 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:54.641171 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33afccfc-04f5-4b07-86d0-228ab10bedd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8496cdb9-67f54\" (UID: \"33afccfc-04f5-4b07-86d0-228ab10bedd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8496cdb9-67f54" Oct 30 17:13:54.641208 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:54.641209 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33afccfc-04f5-4b07-86d0-228ab10bedd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8496cdb9-67f54\" (UID: \"33afccfc-04f5-4b07-86d0-228ab10bedd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8496cdb9-67f54" Oct 30 17:13:54.641483 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:54.641243 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8kkr\" (UniqueName: \"kubernetes.io/projected/33afccfc-04f5-4b07-86d0-228ab10bedd2-kube-api-access-x8kkr\") pod \"maas-api-revoke-sub-8496cdb9-67f54\" (UID: \"33afccfc-04f5-4b07-86d0-228ab10bedd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8496cdb9-67f54" Oct 30 17:13:54.742681 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:54.742598 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33afccfc-04f5-4b07-86d0-228ab10bedd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8496cdb9-67f54\" (UID: \"33afccfc-04f5-4b07-86d0-228ab10bedd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8496cdb9-67f54" Oct 30 17:13:54.742681 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:54.742636 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33afccfc-04f5-4b07-86d0-228ab10bedd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8496cdb9-67f54\" (UID: \"33afccfc-04f5-4b07-86d0-228ab10bedd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8496cdb9-67f54" Oct 30 17:13:54.742873 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:54.742771 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8kkr\" (UniqueName: \"kubernetes.io/projected/33afccfc-04f5-4b07-86d0-228ab10bedd2-kube-api-access-x8kkr\") pod \"maas-api-revoke-sub-8496cdb9-67f54\" (UID: \"33afccfc-04f5-4b07-86d0-228ab10bedd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8496cdb9-67f54" Oct 30 17:13:54.743367 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:54.743350 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33afccfc-04f5-4b07-86d0-228ab10bedd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8496cdb9-67f54\" (UID: \"33afccfc-04f5-4b07-86d0-228ab10bedd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8496cdb9-67f54" Oct 30 17:13:54.745119 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:54.745094 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33afccfc-04f5-4b07-86d0-228ab10bedd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8496cdb9-67f54\" (UID: \"33afccfc-04f5-4b07-86d0-228ab10bedd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8496cdb9-67f54" Oct 30 17:13:54.767866 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:54.767842 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8kkr\" (UniqueName: \"kubernetes.io/projected/33afccfc-04f5-4b07-86d0-228ab10bedd2-kube-api-access-x8kkr\") pod \"maas-api-revoke-sub-8496cdb9-67f54\" (UID: \"33afccfc-04f5-4b07-86d0-228ab10bedd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8496cdb9-67f54" Oct 30 17:13:54.805974 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:54.805926 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8496cdb9-67f54" Oct 30 17:13:55.114337 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:13:55.114278 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33afccfc_04f5_4b07_86d0_228ab10bedd2.slice/crio-eeea178e72b78287612cffbd2bad3f98fec66ca3a535710cc9401b6dc5242326 WatchSource:0}: Error finding container eeea178e72b78287612cffbd2bad3f98fec66ca3a535710cc9401b6dc5242326: Status 404 returned error can't find the container with id eeea178e72b78287612cffbd2bad3f98fec66ca3a535710cc9401b6dc5242326 Oct 30 17:13:55.118022 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:55.117990 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8496cdb9-67f54"] Oct 30 17:13:55.919415 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:55.919372 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8496cdb9-67f54" event={"ID":"33afccfc-04f5-4b07-86d0-228ab10bedd2","Type":"ContainerStarted","Data":"91671f414fcdef316b94568815a631fa9abf542b37a15f6552d7a4556e7be1f0"} Oct 30 17:13:55.919792 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:55.919457 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8496cdb9-67f54" event={"ID":"33afccfc-04f5-4b07-86d0-228ab10bedd2","Type":"ContainerStarted","Data":"eeea178e72b78287612cffbd2bad3f98fec66ca3a535710cc9401b6dc5242326"} Oct 30 17:13:55.964136 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:55.964083 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8496cdb9-67f54" podStartSLOduration=1.964069853 podStartE2EDuration="1.964069853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:13:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:13:55.963234335 +0000 UTC m=+1609.963989254" watchObservedRunningTime="2026-10-30 17:13:55.964069853 +0000 UTC m=+1609.964824753" Oct 30 17:13:56.928895 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:56.928857 4522 generic.go:356] "Generic (PLEG): container finished" podID="33afccfc-04f5-4b07-86d0-228ab10bedd2" containerID="91671f414fcdef316b94568815a631fa9abf542b37a15f6552d7a4556e7be1f0" exitCode=0 Oct 30 17:13:56.929404 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:56.928947 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8496cdb9-67f54" event={"ID":"33afccfc-04f5-4b07-86d0-228ab10bedd2","Type":"ContainerDied","Data":"91671f414fcdef316b94568815a631fa9abf542b37a15f6552d7a4556e7be1f0"} Oct 30 17:13:58.098071 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:58.098043 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8496cdb9-67f54" Oct 30 17:13:58.196607 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:58.196523 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33afccfc-04f5-4b07-86d0-228ab10bedd2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33afccfc-04f5-4b07-86d0-228ab10bedd2-maas-api-xks-ca\") pod \"33afccfc-04f5-4b07-86d0-228ab10bedd2\" (UID: \"33afccfc-04f5-4b07-86d0-228ab10bedd2\") " Oct 30 17:13:58.196607 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:58.196598 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33afccfc-04f5-4b07-86d0-228ab10bedd2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33afccfc-04f5-4b07-86d0-228ab10bedd2-maas-api-openshift-ca\") pod \"33afccfc-04f5-4b07-86d0-228ab10bedd2\" (UID: \"33afccfc-04f5-4b07-86d0-228ab10bedd2\") " Oct 30 17:13:58.196833 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:58.196707 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33afccfc-04f5-4b07-86d0-228ab10bedd2-kube-api-access-x8kkr\" (UniqueName: \"kubernetes.io/projected/33afccfc-04f5-4b07-86d0-228ab10bedd2-kube-api-access-x8kkr\") pod \"33afccfc-04f5-4b07-86d0-228ab10bedd2\" (UID: \"33afccfc-04f5-4b07-86d0-228ab10bedd2\") " Oct 30 17:13:58.197066 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:58.197031 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33afccfc-04f5-4b07-86d0-228ab10bedd2-maas-api-openshift-ca" pod "33afccfc-04f5-4b07-86d0-228ab10bedd2" (UID: "33afccfc-04f5-4b07-86d0-228ab10bedd2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:58.197901 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:58.197684 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33afccfc-04f5-4b07-86d0-228ab10bedd2-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:13:58.199074 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:58.199047 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33afccfc-04f5-4b07-86d0-228ab10bedd2-maas-api-xks-ca" pod "33afccfc-04f5-4b07-86d0-228ab10bedd2" (UID: "33afccfc-04f5-4b07-86d0-228ab10bedd2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:13:58.199478 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:58.199454 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33afccfc-04f5-4b07-86d0-228ab10bedd2-kube-api-access-x8kkr" pod "33afccfc-04f5-4b07-86d0-228ab10bedd2" (UID: "33afccfc-04f5-4b07-86d0-228ab10bedd2"). InnerVolumeSpecName "kube-api-access-x8kkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:58.298717 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:58.298684 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8kkr\" (UniqueName: \"kubernetes.io/projected/33afccfc-04f5-4b07-86d0-228ab10bedd2-kube-api-access-x8kkr\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:13:58.298717 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:58.298714 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33afccfc-04f5-4b07-86d0-228ab10bedd2-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:13:58.945283 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:58.945252 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8496cdb9-67f54" Oct 30 17:13:58.945507 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:58.945283 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8496cdb9-67f54" event={"ID":"33afccfc-04f5-4b07-86d0-228ab10bedd2","Type":"ContainerDied","Data":"eeea178e72b78287612cffbd2bad3f98fec66ca3a535710cc9401b6dc5242326"} Oct 30 17:13:58.945507 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:13:58.945374 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eeea178e72b78287612cffbd2bad3f98fec66ca3a535710cc9401b6dc5242326" Oct 30 17:14:12.603399 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:12.603356 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-78c9c2-777cdc8975-7qp6m"] Oct 30 17:14:12.685341 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:12.665690 4522 cpu_manager.go:414] "containerMap: removing container" podUID="33afccfc-04f5-4b07-86d0-228ab10bedd2" containerName="revoke-keys" Oct 30 17:14:12.689342 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:12.685581 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33afccfc-04f5-4b07-86d0-228ab10bedd2" containerName="revoke-keys" Oct 30 17:14:12.718341 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:12.697641 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="33afccfc-04f5-4b07-86d0-228ab10bedd2" containerName="revoke-keys" Oct 30 17:14:12.774331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:12.768552 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-78c9c2-777cdc8975-7qp6m"] Oct 30 17:14:12.778330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:12.774613 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-78c9c2-6b9c4447c4-p5z99"] Oct 30 17:14:12.871462 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:12.777367 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-78c9c2-777cdc8975-7qp6m" Oct 30 17:14:12.984933 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:12.860527 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90cb278c-f9d6-4948-9843-0330b688521a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-78c9c2-777cdc8975-7qp6m\" (UID: \"90cb278c-f9d6-4948-9843-0330b688521a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-78c9c2-777cdc8975-7qp6m" Oct 30 17:14:12.985290 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:12.985259 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kpsq\" (UniqueName: \"kubernetes.io/projected/90cb278c-f9d6-4948-9843-0330b688521a-kube-api-access-6kpsq\") pod \"payload-pre-processing-e2e-shared-a-w5-78c9c2-777cdc8975-7qp6m\" (UID: \"90cb278c-f9d6-4948-9843-0330b688521a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-78c9c2-777cdc8975-7qp6m" Oct 30 17:14:12.985628 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:12.949623 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-78c9c2-6b9c4447c4-p5z99" Oct 30 17:14:12.986450 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:12.949538 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-78c9c2-6b9c4447c4-p5z99"] Oct 30 17:14:13.077533 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:13.077498 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-78c9c2\"" Oct 30 17:14:13.081000 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:14:13.079760 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:14:13.081152 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:13.080529 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-78c9c2-dockercfg-nt455\"" Oct 30 17:14:13.087990 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:13.087962 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90cb278c-f9d6-4948-9843-0330b688521a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-78c9c2-777cdc8975-7qp6m\" (UID: \"90cb278c-f9d6-4948-9843-0330b688521a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-78c9c2-777cdc8975-7qp6m" Oct 30 17:14:13.088137 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:13.088033 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kpsq\" (UniqueName: \"kubernetes.io/projected/90cb278c-f9d6-4948-9843-0330b688521a-kube-api-access-6kpsq\") pod \"payload-pre-processing-e2e-shared-a-w5-78c9c2-777cdc8975-7qp6m\" (UID: \"90cb278c-f9d6-4948-9843-0330b688521a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-78c9c2-777cdc8975-7qp6m" Oct 30 17:14:13.088137 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:13.088067 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbzk5\" (UniqueName: \"kubernetes.io/projected/63ee8702-b07d-4e00-81d4-bffadca6608d-kube-api-access-qbzk5\") pod \"payload-processing-e2e-shared-a-w5-78c9c2-6b9c4447c4-p5z99\" (UID: \"63ee8702-b07d-4e00-81d4-bffadca6608d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-78c9c2-6b9c4447c4-p5z99" Oct 30 17:14:13.088220 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:13.088192 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63ee8702-b07d-4e00-81d4-bffadca6608d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-78c9c2-6b9c4447c4-p5z99\" (UID: \"63ee8702-b07d-4e00-81d4-bffadca6608d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-78c9c2-6b9c4447c4-p5z99" Oct 30 17:14:13.089061 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:13.089033 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90cb278c-f9d6-4948-9843-0330b688521a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-78c9c2-777cdc8975-7qp6m\" (UID: \"90cb278c-f9d6-4948-9843-0330b688521a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-78c9c2-777cdc8975-7qp6m" Oct 30 17:14:13.178928 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:13.178901 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kpsq\" (UniqueName: \"kubernetes.io/projected/90cb278c-f9d6-4948-9843-0330b688521a-kube-api-access-6kpsq\") pod \"payload-pre-processing-e2e-shared-a-w5-78c9c2-777cdc8975-7qp6m\" (UID: \"90cb278c-f9d6-4948-9843-0330b688521a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-78c9c2-777cdc8975-7qp6m" Oct 30 17:14:13.189981 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:13.189938 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbzk5\" (UniqueName: \"kubernetes.io/projected/63ee8702-b07d-4e00-81d4-bffadca6608d-kube-api-access-qbzk5\") pod \"payload-processing-e2e-shared-a-w5-78c9c2-6b9c4447c4-p5z99\" (UID: \"63ee8702-b07d-4e00-81d4-bffadca6608d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-78c9c2-6b9c4447c4-p5z99" Oct 30 17:14:13.190175 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:13.190102 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63ee8702-b07d-4e00-81d4-bffadca6608d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-78c9c2-6b9c4447c4-p5z99\" (UID: \"63ee8702-b07d-4e00-81d4-bffadca6608d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-78c9c2-6b9c4447c4-p5z99" Oct 30 17:14:13.214331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:13.191849 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63ee8702-b07d-4e00-81d4-bffadca6608d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-78c9c2-6b9c4447c4-p5z99\" (UID: \"63ee8702-b07d-4e00-81d4-bffadca6608d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-78c9c2-6b9c4447c4-p5z99" Oct 30 17:14:13.281532 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:13.281499 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbzk5\" (UniqueName: \"kubernetes.io/projected/63ee8702-b07d-4e00-81d4-bffadca6608d-kube-api-access-qbzk5\") pod \"payload-processing-e2e-shared-a-w5-78c9c2-6b9c4447c4-p5z99\" (UID: \"63ee8702-b07d-4e00-81d4-bffadca6608d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-78c9c2-6b9c4447c4-p5z99" Oct 30 17:14:13.477888 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:13.477590 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-78c9c2-6b9c4447c4-p5z99" Oct 30 17:14:13.478370 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:13.478346 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-78c9c2-777cdc8975-7qp6m" Oct 30 17:14:14.316097 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:14.286802 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq" podUID="f18303fb-6d27-4ff1-9cc9-ed032e43d098" containerName="main" probeResult="failure" output="Get \"https://10.134.0.120:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:14:14.639408 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:14.639257 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-78c9c2-6b9c4447c4-p5z99"] Oct 30 17:14:14.769463 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:14:14.769419 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90cb278c_f9d6_4948_9843_0330b688521a.slice/crio-b075e29656963aa8384f69b24c261b1ec8ede8bc885d27535d126472e7421c38 WatchSource:0}: Error finding container b075e29656963aa8384f69b24c261b1ec8ede8bc885d27535d126472e7421c38: Status 404 returned error can't find the container with id b075e29656963aa8384f69b24c261b1ec8ede8bc885d27535d126472e7421c38 Oct 30 17:14:14.772068 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:14.772037 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-78c9c2-777cdc8975-7qp6m"] Oct 30 17:14:15.598918 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:15.598874 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-78c9c2-777cdc8975-7qp6m" event={"ID":"90cb278c-f9d6-4948-9843-0330b688521a","Type":"ContainerStarted","Data":"2cd513a1c4dbe4618ea797e265570f43d628568fd7b64a7816c5368b256d12a2"} Oct 30 17:14:15.599368 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:15.598943 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-78c9c2-777cdc8975-7qp6m" Oct 30 17:14:15.599368 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:15.598967 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-78c9c2-777cdc8975-7qp6m" event={"ID":"90cb278c-f9d6-4948-9843-0330b688521a","Type":"ContainerStarted","Data":"b075e29656963aa8384f69b24c261b1ec8ede8bc885d27535d126472e7421c38"} Oct 30 17:14:15.600708 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:15.600682 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-78c9c2-6b9c4447c4-p5z99" event={"ID":"63ee8702-b07d-4e00-81d4-bffadca6608d","Type":"ContainerStarted","Data":"fe602c2e074fc9de9203620088c8a6c36983991f0872c2359eba459749d5edc3"} Oct 30 17:14:15.600832 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:15.600733 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-78c9c2-6b9c4447c4-p5z99" event={"ID":"63ee8702-b07d-4e00-81d4-bffadca6608d","Type":"ContainerStarted","Data":"802f14b19138c3bd2017ebf0fac71c5db3831bcc3dfeaa0d2155e9797da4225d"} Oct 30 17:14:15.600832 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:15.600765 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-78c9c2-6b9c4447c4-p5z99" Oct 30 17:14:15.652697 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:15.652644 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-78c9c2-777cdc8975-7qp6m" podStartSLOduration=3.652629236 podStartE2EDuration="3.652629236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:14:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:14:15.648347315 +0000 UTC m=+1629.649102215" watchObservedRunningTime="2026-10-30 17:14:15.652629236 +0000 UTC m=+1629.653384136" Oct 30 17:14:15.699318 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:15.699263 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-78c9c2-6b9c4447c4-p5z99" podStartSLOduration=3.699249479 podStartE2EDuration="3.699249479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:14:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:14:15.696725906 +0000 UTC m=+1629.697480807" watchObservedRunningTime="2026-10-30 17:14:15.699249479 +0000 UTC m=+1629.700004379" Oct 30 17:14:18.531179 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:18.531143 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4c2e4-vjl42"] Oct 30 17:14:18.542801 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:18.542774 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4c2e4-vjl42"] Oct 30 17:14:20.147201 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:20.147164 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="013f626d-4c83-4506-9867-874cbafd85ab" path="/var/lib/kubelet/pods/013f626d-4c83-4506-9867-874cbafd85ab/volumes" Oct 30 17:14:21.858582 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:21.858528 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2d01ec-785fcdd98-cz2mk"] Oct 30 17:14:21.939331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:21.919777 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2d01ec-785fcdd98-cz2mk" Oct 30 17:14:21.960353 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:21.941021 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-2d01ec\"" Oct 30 17:14:21.960598 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:21.956487 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-2d01ec-dockercfg-nqvkh\"" Oct 30 17:14:21.968747 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:21.968715 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2d01ec-785fcdd98-cz2mk"] Oct 30 17:14:22.014085 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:22.014047 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7778b16-47c7-4892-a0e3-368175354cf4-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2d01ec-785fcdd98-cz2mk\" (UID: \"e7778b16-47c7-4892-a0e3-368175354cf4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2d01ec-785fcdd98-cz2mk" Oct 30 17:14:22.014548 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:22.014352 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n44nj\" (UniqueName: \"kubernetes.io/projected/e7778b16-47c7-4892-a0e3-368175354cf4-kube-api-access-n44nj\") pod \"payload-processing-e2e-inf-a-2d01ec-785fcdd98-cz2mk\" (UID: \"e7778b16-47c7-4892-a0e3-368175354cf4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2d01ec-785fcdd98-cz2mk" Oct 30 17:14:22.116162 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:22.116069 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7778b16-47c7-4892-a0e3-368175354cf4-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2d01ec-785fcdd98-cz2mk\" (UID: \"e7778b16-47c7-4892-a0e3-368175354cf4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2d01ec-785fcdd98-cz2mk" Oct 30 17:14:22.116336 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:22.116266 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n44nj\" (UniqueName: \"kubernetes.io/projected/e7778b16-47c7-4892-a0e3-368175354cf4-kube-api-access-n44nj\") pod \"payload-processing-e2e-inf-a-2d01ec-785fcdd98-cz2mk\" (UID: \"e7778b16-47c7-4892-a0e3-368175354cf4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2d01ec-785fcdd98-cz2mk" Oct 30 17:14:22.116863 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:22.116839 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7778b16-47c7-4892-a0e3-368175354cf4-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2d01ec-785fcdd98-cz2mk\" (UID: \"e7778b16-47c7-4892-a0e3-368175354cf4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2d01ec-785fcdd98-cz2mk" Oct 30 17:14:22.146043 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:22.146007 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n44nj\" (UniqueName: \"kubernetes.io/projected/e7778b16-47c7-4892-a0e3-368175354cf4-kube-api-access-n44nj\") pod \"payload-processing-e2e-inf-a-2d01ec-785fcdd98-cz2mk\" (UID: \"e7778b16-47c7-4892-a0e3-368175354cf4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2d01ec-785fcdd98-cz2mk" Oct 30 17:14:22.255002 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:22.254961 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2d01ec-785fcdd98-cz2mk" Oct 30 17:14:22.580593 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:14:22.580560 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7778b16_47c7_4892_a0e3_368175354cf4.slice/crio-c8477b25dfff824d89aa79bbcb293c2d4c759dd3a883754882eb9fa135b9f42c WatchSource:0}: Error finding container c8477b25dfff824d89aa79bbcb293c2d4c759dd3a883754882eb9fa135b9f42c: Status 404 returned error can't find the container with id c8477b25dfff824d89aa79bbcb293c2d4c759dd3a883754882eb9fa135b9f42c Oct 30 17:14:22.581820 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:22.581796 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2d01ec-785fcdd98-cz2mk"] Oct 30 17:14:22.660159 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:22.660127 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2d01ec-785fcdd98-cz2mk" event={"ID":"e7778b16-47c7-4892-a0e3-368175354cf4","Type":"ContainerStarted","Data":"c8477b25dfff824d89aa79bbcb293c2d4c759dd3a883754882eb9fa135b9f42c"} Oct 30 17:14:23.670372 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:23.670331 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2d01ec-785fcdd98-cz2mk" event={"ID":"e7778b16-47c7-4892-a0e3-368175354cf4","Type":"ContainerStarted","Data":"9ce7ea259599b11a545e1623e1ddbe10fcb1ff962a67b0d1c8135440bcc38287"} Oct 30 17:14:23.670760 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:23.670426 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2d01ec-785fcdd98-cz2mk" Oct 30 17:14:23.717979 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:23.717929 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-2d01ec-785fcdd98-cz2mk" podStartSLOduration=2.717915532 podStartE2EDuration="2.717915532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:14:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:14:23.714472011 +0000 UTC m=+1637.715226947" watchObservedRunningTime="2026-10-30 17:14:23.717915532 +0000 UTC m=+1637.718670433" Oct 30 17:14:26.611426 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:26.611395 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-78c9c2-777cdc8975-7qp6m" Oct 30 17:14:26.611839 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:26.611464 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-78c9c2-6b9c4447c4-p5z99" Oct 30 17:14:27.357836 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:27.357797 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h"] Oct 30 17:14:27.358117 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:27.358051 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h" podUID="6ef91484-ab4c-4b90-9802-0ead2c544bf3" containerName="istio-proxy" containerID="cri-o://f0790e925b052e5a80917a8801576521214dc41e97bdf1a8c08f62309ce4de04" gracePeriod=30 Oct 30 17:14:32.654269 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:32.654243 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h" Oct 30 17:14:32.691230 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:32.691199 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ef91484-ab4c-4b90-9802-0ead2c544bf3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ef91484-ab4c-4b90-9802-0ead2c544bf3-istio-envoy\") pod \"6ef91484-ab4c-4b90-9802-0ead2c544bf3\" (UID: \"6ef91484-ab4c-4b90-9802-0ead2c544bf3\") " Oct 30 17:14:32.691423 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:32.691252 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ef91484-ab4c-4b90-9802-0ead2c544bf3-kube-api-access-qmw8n\" (UniqueName: \"kubernetes.io/projected/6ef91484-ab4c-4b90-9802-0ead2c544bf3-kube-api-access-qmw8n\") pod \"6ef91484-ab4c-4b90-9802-0ead2c544bf3\" (UID: \"6ef91484-ab4c-4b90-9802-0ead2c544bf3\") " Oct 30 17:14:32.691423 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:32.691297 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ef91484-ab4c-4b90-9802-0ead2c544bf3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ef91484-ab4c-4b90-9802-0ead2c544bf3-istio-podinfo\") pod \"6ef91484-ab4c-4b90-9802-0ead2c544bf3\" (UID: \"6ef91484-ab4c-4b90-9802-0ead2c544bf3\") " Oct 30 17:14:32.691423 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:32.691346 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ef91484-ab4c-4b90-9802-0ead2c544bf3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ef91484-ab4c-4b90-9802-0ead2c544bf3-workload-socket\") pod \"6ef91484-ab4c-4b90-9802-0ead2c544bf3\" (UID: \"6ef91484-ab4c-4b90-9802-0ead2c544bf3\") " Oct 30 17:14:32.691423 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:32.691390 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ef91484-ab4c-4b90-9802-0ead2c544bf3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ef91484-ab4c-4b90-9802-0ead2c544bf3-credential-socket\") pod \"6ef91484-ab4c-4b90-9802-0ead2c544bf3\" (UID: \"6ef91484-ab4c-4b90-9802-0ead2c544bf3\") " Oct 30 17:14:32.691923 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:32.691455 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ef91484-ab4c-4b90-9802-0ead2c544bf3-istio-token\" (UniqueName: \"kubernetes.io/projected/6ef91484-ab4c-4b90-9802-0ead2c544bf3-istio-token\") pod \"6ef91484-ab4c-4b90-9802-0ead2c544bf3\" (UID: \"6ef91484-ab4c-4b90-9802-0ead2c544bf3\") " Oct 30 17:14:32.691923 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:32.691493 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ef91484-ab4c-4b90-9802-0ead2c544bf3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ef91484-ab4c-4b90-9802-0ead2c544bf3-workload-certs\") pod \"6ef91484-ab4c-4b90-9802-0ead2c544bf3\" (UID: \"6ef91484-ab4c-4b90-9802-0ead2c544bf3\") " Oct 30 17:14:32.691923 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:32.691536 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ef91484-ab4c-4b90-9802-0ead2c544bf3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ef91484-ab4c-4b90-9802-0ead2c544bf3-istiod-ca-cert\") pod \"6ef91484-ab4c-4b90-9802-0ead2c544bf3\" (UID: \"6ef91484-ab4c-4b90-9802-0ead2c544bf3\") " Oct 30 17:14:32.691923 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:32.691636 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ef91484-ab4c-4b90-9802-0ead2c544bf3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ef91484-ab4c-4b90-9802-0ead2c544bf3-istio-data\") pod \"6ef91484-ab4c-4b90-9802-0ead2c544bf3\" (UID: \"6ef91484-ab4c-4b90-9802-0ead2c544bf3\") " Oct 30 17:14:32.692117 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:32.691941 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ef91484-ab4c-4b90-9802-0ead2c544bf3-workload-certs" pod "6ef91484-ab4c-4b90-9802-0ead2c544bf3" (UID: "6ef91484-ab4c-4b90-9802-0ead2c544bf3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:32.693009 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:32.692270 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ef91484-ab4c-4b90-9802-0ead2c544bf3-credential-socket" pod "6ef91484-ab4c-4b90-9802-0ead2c544bf3" (UID: "6ef91484-ab4c-4b90-9802-0ead2c544bf3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:32.693009 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:32.692513 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ef91484-ab4c-4b90-9802-0ead2c544bf3-istio-data" pod "6ef91484-ab4c-4b90-9802-0ead2c544bf3" (UID: "6ef91484-ab4c-4b90-9802-0ead2c544bf3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:32.693009 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:32.692540 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ef91484-ab4c-4b90-9802-0ead2c544bf3-workload-socket" pod "6ef91484-ab4c-4b90-9802-0ead2c544bf3" (UID: "6ef91484-ab4c-4b90-9802-0ead2c544bf3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:32.693009 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:32.692614 4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ef91484-ab4c-4b90-9802-0ead2c544bf3-credential-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:14:32.693009 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:32.692631 4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ef91484-ab4c-4b90-9802-0ead2c544bf3-workload-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:14:32.693009 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:32.692704 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ef91484-ab4c-4b90-9802-0ead2c544bf3-istiod-ca-cert" pod "6ef91484-ab4c-4b90-9802-0ead2c544bf3" (UID: "6ef91484-ab4c-4b90-9802-0ead2c544bf3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:14:32.694043 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:32.694018 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ef91484-ab4c-4b90-9802-0ead2c544bf3-istio-envoy" pod "6ef91484-ab4c-4b90-9802-0ead2c544bf3" (UID: "6ef91484-ab4c-4b90-9802-0ead2c544bf3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:32.694043 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:32.694026 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ef91484-ab4c-4b90-9802-0ead2c544bf3-istio-podinfo" pod "6ef91484-ab4c-4b90-9802-0ead2c544bf3" (UID: "6ef91484-ab4c-4b90-9802-0ead2c544bf3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:14:32.694166 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:32.694083 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ef91484-ab4c-4b90-9802-0ead2c544bf3-istio-token" pod "6ef91484-ab4c-4b90-9802-0ead2c544bf3" (UID: "6ef91484-ab4c-4b90-9802-0ead2c544bf3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:14:32.694547 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:32.694528 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ef91484-ab4c-4b90-9802-0ead2c544bf3-kube-api-access-qmw8n" pod "6ef91484-ab4c-4b90-9802-0ead2c544bf3" (UID: "6ef91484-ab4c-4b90-9802-0ead2c544bf3"). InnerVolumeSpecName "kube-api-access-qmw8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:14:32.750140 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:32.750104 4522 generic.go:356] "Generic (PLEG): container finished" podID="6ef91484-ab4c-4b90-9802-0ead2c544bf3" containerID="f0790e925b052e5a80917a8801576521214dc41e97bdf1a8c08f62309ce4de04" exitCode=0 Oct 30 17:14:32.750372 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:32.750171 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h" Oct 30 17:14:32.750463 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:32.750398 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h" event={"ID":"6ef91484-ab4c-4b90-9802-0ead2c544bf3","Type":"ContainerDied","Data":"f0790e925b052e5a80917a8801576521214dc41e97bdf1a8c08f62309ce4de04"} Oct 30 17:14:32.750463 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:32.750442 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h" event={"ID":"6ef91484-ab4c-4b90-9802-0ead2c544bf3","Type":"ContainerDied","Data":"0d9123e14519b5b9479d2b7b27098f572848e7b273dc64f8f60bfcd443bae472"} Oct 30 17:14:32.750586 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:32.750467 4522 scope.go:122] "RemoveContainer" containerID="f0790e925b052e5a80917a8801576521214dc41e97bdf1a8c08f62309ce4de04" Oct 30 17:14:32.767007 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:32.766979 4522 scope.go:122] "RemoveContainer" containerID="f0790e925b052e5a80917a8801576521214dc41e97bdf1a8c08f62309ce4de04" Oct 30 17:14:32.767574 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:14:32.767533 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0790e925b052e5a80917a8801576521214dc41e97bdf1a8c08f62309ce4de04\": container with ID starting with f0790e925b052e5a80917a8801576521214dc41e97bdf1a8c08f62309ce4de04 not found: ID does not exist" containerID="f0790e925b052e5a80917a8801576521214dc41e97bdf1a8c08f62309ce4de04" Oct 30 17:14:32.767709 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:32.767588 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0790e925b052e5a80917a8801576521214dc41e97bdf1a8c08f62309ce4de04"} err="failed to get container status \"f0790e925b052e5a80917a8801576521214dc41e97bdf1a8c08f62309ce4de04\": rpc error: code = NotFound desc = could not find container \"f0790e925b052e5a80917a8801576521214dc41e97bdf1a8c08f62309ce4de04\": container with ID starting with f0790e925b052e5a80917a8801576521214dc41e97bdf1a8c08f62309ce4de04 not found: ID does not exist" Oct 30 17:14:32.794506 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:32.794470 4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ef91484-ab4c-4b90-9802-0ead2c544bf3-istio-envoy\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:14:32.794506 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:32.794507 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmw8n\" (UniqueName: \"kubernetes.io/projected/6ef91484-ab4c-4b90-9802-0ead2c544bf3-kube-api-access-qmw8n\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:14:32.794781 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:32.794524 4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ef91484-ab4c-4b90-9802-0ead2c544bf3-istio-podinfo\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:14:32.794781 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:32.794538 4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ef91484-ab4c-4b90-9802-0ead2c544bf3-workload-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:14:32.794781 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:32.794552 4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ef91484-ab4c-4b90-9802-0ead2c544bf3-istio-token\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:14:32.794781 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:32.794565 4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ef91484-ab4c-4b90-9802-0ead2c544bf3-istiod-ca-cert\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:14:32.794781 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:32.794578 4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ef91484-ab4c-4b90-9802-0ead2c544bf3-istio-data\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:14:32.817281 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:32.817253 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h"] Oct 30 17:14:32.823996 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:32.823969 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-bb84h"] Oct 30 17:14:34.147090 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:34.147061 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ef91484-ab4c-4b90-9802-0ead2c544bf3" path="/var/lib/kubelet/pods/6ef91484-ab4c-4b90-9802-0ead2c544bf3/volumes" Oct 30 17:14:34.522195 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:34.522162 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7c52f8-c64c6769f-7m7qv"] Oct 30 17:14:34.523196 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:34.523175 4522 cpu_manager.go:414] "containerMap: removing container" podUID="6ef91484-ab4c-4b90-9802-0ead2c544bf3" containerName="istio-proxy" Oct 30 17:14:34.523273 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:34.523200 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ef91484-ab4c-4b90-9802-0ead2c544bf3" containerName="istio-proxy" Oct 30 17:14:34.523415 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:34.523403 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ef91484-ab4c-4b90-9802-0ead2c544bf3" containerName="istio-proxy" Oct 30 17:14:34.529214 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:34.529185 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7c52f8-c64c6769f-7m7qv" Oct 30 17:14:34.561125 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:34.561094 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-7c52f8-dockercfg-8hct9\"" Oct 30 17:14:34.561269 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:34.561094 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-7c52f8\"" Oct 30 17:14:34.568132 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:34.568104 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7c52f8-c64c6769f-7m7qv"] Oct 30 17:14:34.630827 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:34.630771 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n9n4\" (UniqueName: \"kubernetes.io/projected/3abcffab-3269-43fd-8d19-5fecae1eb0c1-kube-api-access-2n9n4\") pod \"payload-pre-processing-e2e-inf-a-7c52f8-c64c6769f-7m7qv\" (UID: \"3abcffab-3269-43fd-8d19-5fecae1eb0c1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7c52f8-c64c6769f-7m7qv" Oct 30 17:14:34.631077 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:34.630977 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3abcffab-3269-43fd-8d19-5fecae1eb0c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7c52f8-c64c6769f-7m7qv\" (UID: \"3abcffab-3269-43fd-8d19-5fecae1eb0c1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7c52f8-c64c6769f-7m7qv" Oct 30 17:14:34.646224 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:34.644574 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7c52f8-cf4f98854-2v744"] Oct 30 17:14:34.649704 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:34.649680 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7c52f8-cf4f98854-2v744" Oct 30 17:14:34.682939 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:34.682280 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7c52f8-cf4f98854-2v744"] Oct 30 17:14:34.721392 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:34.721355 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2d01ec-785fcdd98-cz2mk" Oct 30 17:14:34.732802 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:34.732771 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bb27ee5-8209-41d1-bf37-b868a1dcc6a5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7c52f8-cf4f98854-2v744\" (UID: \"7bb27ee5-8209-41d1-bf37-b868a1dcc6a5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7c52f8-cf4f98854-2v744" Oct 30 17:14:34.732978 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:34.732873 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3abcffab-3269-43fd-8d19-5fecae1eb0c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7c52f8-c64c6769f-7m7qv\" (UID: \"3abcffab-3269-43fd-8d19-5fecae1eb0c1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7c52f8-c64c6769f-7m7qv" Oct 30 17:14:34.733056 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:34.733040 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv2bs\" (UniqueName: \"kubernetes.io/projected/7bb27ee5-8209-41d1-bf37-b868a1dcc6a5-kube-api-access-jv2bs\") pod \"payload-processing-e2e-inf-a-7c52f8-cf4f98854-2v744\" (UID: \"7bb27ee5-8209-41d1-bf37-b868a1dcc6a5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7c52f8-cf4f98854-2v744" Oct 30 17:14:34.733138 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:34.733125 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n9n4\" (UniqueName: \"kubernetes.io/projected/3abcffab-3269-43fd-8d19-5fecae1eb0c1-kube-api-access-2n9n4\") pod \"payload-pre-processing-e2e-inf-a-7c52f8-c64c6769f-7m7qv\" (UID: \"3abcffab-3269-43fd-8d19-5fecae1eb0c1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7c52f8-c64c6769f-7m7qv" Oct 30 17:14:34.734434 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:34.734399 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3abcffab-3269-43fd-8d19-5fecae1eb0c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7c52f8-c64c6769f-7m7qv\" (UID: \"3abcffab-3269-43fd-8d19-5fecae1eb0c1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7c52f8-c64c6769f-7m7qv" Oct 30 17:14:34.762532 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:34.762498 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n9n4\" (UniqueName: \"kubernetes.io/projected/3abcffab-3269-43fd-8d19-5fecae1eb0c1-kube-api-access-2n9n4\") pod \"payload-pre-processing-e2e-inf-a-7c52f8-c64c6769f-7m7qv\" (UID: \"3abcffab-3269-43fd-8d19-5fecae1eb0c1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7c52f8-c64c6769f-7m7qv" Oct 30 17:14:34.834228 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:34.834144 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv2bs\" (UniqueName: \"kubernetes.io/projected/7bb27ee5-8209-41d1-bf37-b868a1dcc6a5-kube-api-access-jv2bs\") pod \"payload-processing-e2e-inf-a-7c52f8-cf4f98854-2v744\" (UID: \"7bb27ee5-8209-41d1-bf37-b868a1dcc6a5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7c52f8-cf4f98854-2v744" Oct 30 17:14:34.834406 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:34.834327 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bb27ee5-8209-41d1-bf37-b868a1dcc6a5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7c52f8-cf4f98854-2v744\" (UID: \"7bb27ee5-8209-41d1-bf37-b868a1dcc6a5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7c52f8-cf4f98854-2v744" Oct 30 17:14:34.835355 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:34.835328 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bb27ee5-8209-41d1-bf37-b868a1dcc6a5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7c52f8-cf4f98854-2v744\" (UID: \"7bb27ee5-8209-41d1-bf37-b868a1dcc6a5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7c52f8-cf4f98854-2v744" Oct 30 17:14:34.845467 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:34.845444 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7c52f8-c64c6769f-7m7qv" Oct 30 17:14:34.910584 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:34.907807 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv2bs\" (UniqueName: \"kubernetes.io/projected/7bb27ee5-8209-41d1-bf37-b868a1dcc6a5-kube-api-access-jv2bs\") pod \"payload-processing-e2e-inf-a-7c52f8-cf4f98854-2v744\" (UID: \"7bb27ee5-8209-41d1-bf37-b868a1dcc6a5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7c52f8-cf4f98854-2v744" Oct 30 17:14:34.985600 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:34.971023 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7c52f8-cf4f98854-2v744" Oct 30 17:14:35.209639 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:14:35.209609 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3abcffab_3269_43fd_8d19_5fecae1eb0c1.slice/crio-a8ea693d497d84f02fb4cd0861e204c0ac18665a97e7af7795a6da1dfd814ce6 WatchSource:0}: Error finding container a8ea693d497d84f02fb4cd0861e204c0ac18665a97e7af7795a6da1dfd814ce6: Status 404 returned error can't find the container with id a8ea693d497d84f02fb4cd0861e204c0ac18665a97e7af7795a6da1dfd814ce6 Oct 30 17:14:35.211776 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:35.211753 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7c52f8-c64c6769f-7m7qv"] Oct 30 17:14:35.352243 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:14:35.352208 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bb27ee5_8209_41d1_bf37_b868a1dcc6a5.slice/crio-e8b6e8725113920cdccc4f3cc26009c426429c5ab2dd2708e20f030ed8900890 WatchSource:0}: Error finding container e8b6e8725113920cdccc4f3cc26009c426429c5ab2dd2708e20f030ed8900890: Status 404 returned error can't find the container with id e8b6e8725113920cdccc4f3cc26009c426429c5ab2dd2708e20f030ed8900890 Oct 30 17:14:35.354000 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:35.353978 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7c52f8-cf4f98854-2v744"] Oct 30 17:14:35.778283 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:35.778244 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7c52f8-cf4f98854-2v744" event={"ID":"7bb27ee5-8209-41d1-bf37-b868a1dcc6a5","Type":"ContainerStarted","Data":"d55657195295ec88d79a68b6a1341eeece3f844d66eadd92cde181a4f9292601"} Oct 30 17:14:35.778525 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:35.778340 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7c52f8-cf4f98854-2v744" event={"ID":"7bb27ee5-8209-41d1-bf37-b868a1dcc6a5","Type":"ContainerStarted","Data":"e8b6e8725113920cdccc4f3cc26009c426429c5ab2dd2708e20f030ed8900890"} Oct 30 17:14:35.778525 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:35.778425 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-7c52f8-cf4f98854-2v744" Oct 30 17:14:35.780270 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:35.780242 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7c52f8-c64c6769f-7m7qv" event={"ID":"3abcffab-3269-43fd-8d19-5fecae1eb0c1","Type":"ContainerStarted","Data":"569e9a604533dd6fb01d4644f3034eb5ff191563bb8cd262f6d24a158fd99ab3"} Oct 30 17:14:35.780428 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:35.780289 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7c52f8-c64c6769f-7m7qv" event={"ID":"3abcffab-3269-43fd-8d19-5fecae1eb0c1","Type":"ContainerStarted","Data":"a8ea693d497d84f02fb4cd0861e204c0ac18665a97e7af7795a6da1dfd814ce6"} Oct 30 17:14:35.780428 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:35.780330 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7c52f8-c64c6769f-7m7qv" Oct 30 17:14:35.824316 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:35.824271 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-7c52f8-cf4f98854-2v744" podStartSLOduration=1.824259782 podStartE2EDuration="1.824259782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:14:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:14:35.821668451 +0000 UTC m=+1649.822423353" watchObservedRunningTime="2026-10-30 17:14:35.824259782 +0000 UTC m=+1649.825014722" Oct 30 17:14:35.866148 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:35.866101 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7c52f8-c64c6769f-7m7qv" podStartSLOduration=1.866089062 podStartE2EDuration="1.866089062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:14:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:14:35.865967978 +0000 UTC m=+1649.866722906" watchObservedRunningTime="2026-10-30 17:14:35.866089062 +0000 UTC m=+1649.866843963" Oct 30 17:14:37.123552 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:37.123515 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33b8828f-w7xtc"] Oct 30 17:14:37.128611 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:37.128553 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33b8828f-w7xtc" Oct 30 17:14:37.150363 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:37.149578 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:14:37.150363 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:37.149647 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:14:37.158379 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:37.156684 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33b8828f-w7xtc"] Oct 30 17:14:37.269754 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:37.269722 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td2rm\" (UniqueName: \"kubernetes.io/projected/70ccb80a-0b1e-4b73-851c-8c5294468c2c-kube-api-access-td2rm\") pod \"maas-api-revoke-sub-33b8828f-w7xtc\" (UID: \"70ccb80a-0b1e-4b73-851c-8c5294468c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33b8828f-w7xtc" Oct 30 17:14:37.269965 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:37.269900 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70ccb80a-0b1e-4b73-851c-8c5294468c2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33b8828f-w7xtc\" (UID: \"70ccb80a-0b1e-4b73-851c-8c5294468c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33b8828f-w7xtc" Oct 30 17:14:37.270260 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:37.270228 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70ccb80a-0b1e-4b73-851c-8c5294468c2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33b8828f-w7xtc\" (UID: \"70ccb80a-0b1e-4b73-851c-8c5294468c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33b8828f-w7xtc" Oct 30 17:14:37.371789 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:37.371736 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td2rm\" (UniqueName: \"kubernetes.io/projected/70ccb80a-0b1e-4b73-851c-8c5294468c2c-kube-api-access-td2rm\") pod \"maas-api-revoke-sub-33b8828f-w7xtc\" (UID: \"70ccb80a-0b1e-4b73-851c-8c5294468c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33b8828f-w7xtc" Oct 30 17:14:37.372036 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:37.371869 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70ccb80a-0b1e-4b73-851c-8c5294468c2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33b8828f-w7xtc\" (UID: \"70ccb80a-0b1e-4b73-851c-8c5294468c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33b8828f-w7xtc" Oct 30 17:14:37.372036 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:37.372014 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70ccb80a-0b1e-4b73-851c-8c5294468c2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33b8828f-w7xtc\" (UID: \"70ccb80a-0b1e-4b73-851c-8c5294468c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33b8828f-w7xtc" Oct 30 17:14:37.373009 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:37.372975 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70ccb80a-0b1e-4b73-851c-8c5294468c2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33b8828f-w7xtc\" (UID: \"70ccb80a-0b1e-4b73-851c-8c5294468c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33b8828f-w7xtc" Oct 30 17:14:37.380534 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:37.379054 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70ccb80a-0b1e-4b73-851c-8c5294468c2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33b8828f-w7xtc\" (UID: \"70ccb80a-0b1e-4b73-851c-8c5294468c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33b8828f-w7xtc" Oct 30 17:14:37.398028 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:37.397998 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td2rm\" (UniqueName: \"kubernetes.io/projected/70ccb80a-0b1e-4b73-851c-8c5294468c2c-kube-api-access-td2rm\") pod \"maas-api-revoke-sub-33b8828f-w7xtc\" (UID: \"70ccb80a-0b1e-4b73-851c-8c5294468c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33b8828f-w7xtc" Oct 30 17:14:37.452555 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:37.452517 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33b8828f-w7xtc" Oct 30 17:14:37.795474 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:37.795445 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33b8828f-w7xtc"] Oct 30 17:14:38.075776 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:38.073772 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-432680-s6l4m"] Oct 30 17:14:38.079932 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:38.079893 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-432680-s6l4m"] Oct 30 17:14:38.148163 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:38.148128 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85597425-1abd-470a-b407-3c095a55bef9" path="/var/lib/kubelet/pods/85597425-1abd-470a-b407-3c095a55bef9/volumes" Oct 30 17:14:38.816280 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:38.816245 4522 generic.go:356] "Generic (PLEG): container finished" podID="70ccb80a-0b1e-4b73-851c-8c5294468c2c" containerID="4cb06a02a69d533e98a56f4837a9754e20888bc0b185861ffa135936a0d6cffe" exitCode=0 Oct 30 17:14:38.816574 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:38.816286 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33b8828f-w7xtc" event={"ID":"70ccb80a-0b1e-4b73-851c-8c5294468c2c","Type":"ContainerDied","Data":"4cb06a02a69d533e98a56f4837a9754e20888bc0b185861ffa135936a0d6cffe"} Oct 30 17:14:38.816574 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:38.816345 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33b8828f-w7xtc" event={"ID":"70ccb80a-0b1e-4b73-851c-8c5294468c2c","Type":"ContainerStarted","Data":"389afe73d60af0ef83265e740b87877065ca2225528162204a80566789cd310f"} Oct 30 17:14:40.006149 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:40.006117 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33b8828f-w7xtc" Oct 30 17:14:40.131097 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:40.131008 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70ccb80a-0b1e-4b73-851c-8c5294468c2c-kube-api-access-td2rm\" (UniqueName: \"kubernetes.io/projected/70ccb80a-0b1e-4b73-851c-8c5294468c2c-kube-api-access-td2rm\") pod \"70ccb80a-0b1e-4b73-851c-8c5294468c2c\" (UID: \"70ccb80a-0b1e-4b73-851c-8c5294468c2c\") " Oct 30 17:14:40.131279 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:40.131169 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70ccb80a-0b1e-4b73-851c-8c5294468c2c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70ccb80a-0b1e-4b73-851c-8c5294468c2c-maas-api-xks-ca\") pod \"70ccb80a-0b1e-4b73-851c-8c5294468c2c\" (UID: \"70ccb80a-0b1e-4b73-851c-8c5294468c2c\") " Oct 30 17:14:40.131279 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:40.131253 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70ccb80a-0b1e-4b73-851c-8c5294468c2c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70ccb80a-0b1e-4b73-851c-8c5294468c2c-maas-api-openshift-ca\") pod \"70ccb80a-0b1e-4b73-851c-8c5294468c2c\" (UID: \"70ccb80a-0b1e-4b73-851c-8c5294468c2c\") " Oct 30 17:14:40.131651 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:40.131624 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70ccb80a-0b1e-4b73-851c-8c5294468c2c-maas-api-openshift-ca" pod "70ccb80a-0b1e-4b73-851c-8c5294468c2c" (UID: "70ccb80a-0b1e-4b73-851c-8c5294468c2c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:14:40.131876 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:40.131857 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70ccb80a-0b1e-4b73-851c-8c5294468c2c-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:14:40.133237 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:40.133212 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70ccb80a-0b1e-4b73-851c-8c5294468c2c-kube-api-access-td2rm" pod "70ccb80a-0b1e-4b73-851c-8c5294468c2c" (UID: "70ccb80a-0b1e-4b73-851c-8c5294468c2c"). InnerVolumeSpecName "kube-api-access-td2rm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:14:40.133349 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:40.133261 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70ccb80a-0b1e-4b73-851c-8c5294468c2c-maas-api-xks-ca" pod "70ccb80a-0b1e-4b73-851c-8c5294468c2c" (UID: "70ccb80a-0b1e-4b73-851c-8c5294468c2c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:14:40.233765 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:40.233734 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70ccb80a-0b1e-4b73-851c-8c5294468c2c-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:14:40.233765 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:40.233763 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td2rm\" (UniqueName: \"kubernetes.io/projected/70ccb80a-0b1e-4b73-851c-8c5294468c2c-kube-api-access-td2rm\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:14:40.847981 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:40.847950 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33b8828f-w7xtc" Oct 30 17:14:40.848172 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:40.847988 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33b8828f-w7xtc" event={"ID":"70ccb80a-0b1e-4b73-851c-8c5294468c2c","Type":"ContainerDied","Data":"389afe73d60af0ef83265e740b87877065ca2225528162204a80566789cd310f"} Oct 30 17:14:40.848172 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:40.848042 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="389afe73d60af0ef83265e740b87877065ca2225528162204a80566789cd310f" Oct 30 17:14:42.949276 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:14:42.949236 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:14:46.803338 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:46.794270 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7c52f8-c64c6769f-7m7qv" Oct 30 17:14:46.803812 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:46.803673 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-7c52f8-cf4f98854-2v744" Oct 30 17:14:47.915128 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:47.915091 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b2c3fb-qs5xm"] Oct 30 17:14:47.915884 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:47.915868 4522 cpu_manager.go:414] "containerMap: removing container" podUID="70ccb80a-0b1e-4b73-851c-8c5294468c2c" containerName="revoke-keys" Oct 30 17:14:47.915884 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:47.915884 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70ccb80a-0b1e-4b73-851c-8c5294468c2c" containerName="revoke-keys" Oct 30 17:14:47.916028 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:47.916018 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="70ccb80a-0b1e-4b73-851c-8c5294468c2c" containerName="revoke-keys" Oct 30 17:14:47.921113 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:47.921094 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b2c3fb-qs5xm" Oct 30 17:14:47.941467 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:47.941439 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b2c3fb-qs5xm"] Oct 30 17:14:47.948615 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:47.948585 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:14:47.948848 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:47.948823 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:14:47.964658 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:47.964619 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/486d41a9-c5ac-43c0-bb8e-5e9032acf5d2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b2c3fb-qs5xm\" (UID: \"486d41a9-c5ac-43c0-bb8e-5e9032acf5d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b2c3fb-qs5xm" Oct 30 17:14:47.964919 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:47.964900 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7r9k\" (UniqueName: \"kubernetes.io/projected/486d41a9-c5ac-43c0-bb8e-5e9032acf5d2-kube-api-access-f7r9k\") pod \"maas-api-revoke-keys-e2e-api-b-b2c3fb-qs5xm\" (UID: \"486d41a9-c5ac-43c0-bb8e-5e9032acf5d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b2c3fb-qs5xm" Oct 30 17:14:47.965078 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:47.965059 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/486d41a9-c5ac-43c0-bb8e-5e9032acf5d2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b2c3fb-qs5xm\" (UID: \"486d41a9-c5ac-43c0-bb8e-5e9032acf5d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b2c3fb-qs5xm" Oct 30 17:14:48.066384 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:48.066349 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7r9k\" (UniqueName: \"kubernetes.io/projected/486d41a9-c5ac-43c0-bb8e-5e9032acf5d2-kube-api-access-f7r9k\") pod \"maas-api-revoke-keys-e2e-api-b-b2c3fb-qs5xm\" (UID: \"486d41a9-c5ac-43c0-bb8e-5e9032acf5d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b2c3fb-qs5xm" Oct 30 17:14:48.066609 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:48.066445 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/486d41a9-c5ac-43c0-bb8e-5e9032acf5d2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b2c3fb-qs5xm\" (UID: \"486d41a9-c5ac-43c0-bb8e-5e9032acf5d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b2c3fb-qs5xm" Oct 30 17:14:48.066609 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:48.066527 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/486d41a9-c5ac-43c0-bb8e-5e9032acf5d2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b2c3fb-qs5xm\" (UID: \"486d41a9-c5ac-43c0-bb8e-5e9032acf5d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b2c3fb-qs5xm" Oct 30 17:14:48.067209 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:48.067183 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/486d41a9-c5ac-43c0-bb8e-5e9032acf5d2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b2c3fb-qs5xm\" (UID: \"486d41a9-c5ac-43c0-bb8e-5e9032acf5d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b2c3fb-qs5xm" Oct 30 17:14:48.068894 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:48.068869 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/486d41a9-c5ac-43c0-bb8e-5e9032acf5d2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b2c3fb-qs5xm\" (UID: \"486d41a9-c5ac-43c0-bb8e-5e9032acf5d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b2c3fb-qs5xm" Oct 30 17:14:48.091183 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:48.091155 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7r9k\" (UniqueName: \"kubernetes.io/projected/486d41a9-c5ac-43c0-bb8e-5e9032acf5d2-kube-api-access-f7r9k\") pod \"maas-api-revoke-keys-e2e-api-b-b2c3fb-qs5xm\" (UID: \"486d41a9-c5ac-43c0-bb8e-5e9032acf5d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b2c3fb-qs5xm" Oct 30 17:14:48.236958 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:48.236923 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b2c3fb-qs5xm" Oct 30 17:14:48.563867 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:14:48.563835 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod486d41a9_c5ac_43c0_bb8e_5e9032acf5d2.slice/crio-9ee229c8df41a045d18a22c70125d63eba45d2941a9960ebd1461edad79772d4 WatchSource:0}: Error finding container 9ee229c8df41a045d18a22c70125d63eba45d2941a9960ebd1461edad79772d4: Status 404 returned error can't find the container with id 9ee229c8df41a045d18a22c70125d63eba45d2941a9960ebd1461edad79772d4 Oct 30 17:14:48.566739 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:48.566708 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b2c3fb-qs5xm"] Oct 30 17:14:48.954249 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:48.954218 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b2c3fb-qs5xm" event={"ID":"486d41a9-c5ac-43c0-bb8e-5e9032acf5d2","Type":"ContainerStarted","Data":"9ee229c8df41a045d18a22c70125d63eba45d2941a9960ebd1461edad79772d4"} Oct 30 17:14:49.175087 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:49.174999 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ebacbb2-27rdm"] Oct 30 17:14:49.180941 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:49.180900 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ebacbb2-27rdm" Oct 30 17:14:49.198006 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:49.197891 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ebacbb2-27rdm"] Oct 30 17:14:49.297061 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:49.297032 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78khw\" (UniqueName: \"kubernetes.io/projected/a1555702-8acd-4384-acb9-94e3f51495de-kube-api-access-78khw\") pod \"maas-api-revoke-sub-9ebacbb2-27rdm\" (UID: \"a1555702-8acd-4384-acb9-94e3f51495de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ebacbb2-27rdm" Oct 30 17:14:49.297274 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:49.297109 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1555702-8acd-4384-acb9-94e3f51495de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ebacbb2-27rdm\" (UID: \"a1555702-8acd-4384-acb9-94e3f51495de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ebacbb2-27rdm" Oct 30 17:14:49.297274 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:49.297195 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1555702-8acd-4384-acb9-94e3f51495de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ebacbb2-27rdm\" (UID: \"a1555702-8acd-4384-acb9-94e3f51495de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ebacbb2-27rdm" Oct 30 17:14:49.398498 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:49.398461 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78khw\" (UniqueName: \"kubernetes.io/projected/a1555702-8acd-4384-acb9-94e3f51495de-kube-api-access-78khw\") pod \"maas-api-revoke-sub-9ebacbb2-27rdm\" (UID: \"a1555702-8acd-4384-acb9-94e3f51495de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ebacbb2-27rdm" Oct 30 17:14:49.398674 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:49.398537 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1555702-8acd-4384-acb9-94e3f51495de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ebacbb2-27rdm\" (UID: \"a1555702-8acd-4384-acb9-94e3f51495de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ebacbb2-27rdm" Oct 30 17:14:49.398674 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:49.398556 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1555702-8acd-4384-acb9-94e3f51495de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ebacbb2-27rdm\" (UID: \"a1555702-8acd-4384-acb9-94e3f51495de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ebacbb2-27rdm" Oct 30 17:14:49.399191 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:49.399164 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1555702-8acd-4384-acb9-94e3f51495de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ebacbb2-27rdm\" (UID: \"a1555702-8acd-4384-acb9-94e3f51495de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ebacbb2-27rdm" Oct 30 17:14:49.401027 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:49.401005 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1555702-8acd-4384-acb9-94e3f51495de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ebacbb2-27rdm\" (UID: \"a1555702-8acd-4384-acb9-94e3f51495de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ebacbb2-27rdm" Oct 30 17:14:49.423290 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:49.423260 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78khw\" (UniqueName: \"kubernetes.io/projected/a1555702-8acd-4384-acb9-94e3f51495de-kube-api-access-78khw\") pod \"maas-api-revoke-sub-9ebacbb2-27rdm\" (UID: \"a1555702-8acd-4384-acb9-94e3f51495de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ebacbb2-27rdm" Oct 30 17:14:49.502909 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:49.502876 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ebacbb2-27rdm" Oct 30 17:14:49.811790 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:14:49.811758 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1555702_8acd_4384_acb9_94e3f51495de.slice/crio-66989b6682c30060167b75dd8b6ca33b8d38b8c6be5ca9572263906577680b9f WatchSource:0}: Error finding container 66989b6682c30060167b75dd8b6ca33b8d38b8c6be5ca9572263906577680b9f: Status 404 returned error can't find the container with id 66989b6682c30060167b75dd8b6ca33b8d38b8c6be5ca9572263906577680b9f Oct 30 17:14:49.813816 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:49.813794 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ebacbb2-27rdm"] Oct 30 17:14:49.966329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:49.966286 4522 generic.go:356] "Generic (PLEG): container finished" podID="486d41a9-c5ac-43c0-bb8e-5e9032acf5d2" containerID="60437ae41bb5026b169ea16b93008db4ac72a1ce6e6dbc9db561917e40c5b72b" exitCode=0 Oct 30 17:14:49.966758 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:49.966333 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b2c3fb-qs5xm" event={"ID":"486d41a9-c5ac-43c0-bb8e-5e9032acf5d2","Type":"ContainerDied","Data":"60437ae41bb5026b169ea16b93008db4ac72a1ce6e6dbc9db561917e40c5b72b"} Oct 30 17:14:49.967951 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:49.967927 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ebacbb2-27rdm" event={"ID":"a1555702-8acd-4384-acb9-94e3f51495de","Type":"ContainerStarted","Data":"66989b6682c30060167b75dd8b6ca33b8d38b8c6be5ca9572263906577680b9f"} Oct 30 17:14:51.000541 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:51.000505 4522 generic.go:356] "Generic (PLEG): container finished" podID="a1555702-8acd-4384-acb9-94e3f51495de" containerID="66e3281cce76250b6de5c4fc4dd447e13e29d387dbda945fc8a43ee9df4209c5" exitCode=0 Oct 30 17:14:51.001030 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:51.000591 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ebacbb2-27rdm" event={"ID":"a1555702-8acd-4384-acb9-94e3f51495de","Type":"ContainerDied","Data":"66e3281cce76250b6de5c4fc4dd447e13e29d387dbda945fc8a43ee9df4209c5"} Oct 30 17:14:51.166213 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:51.166188 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b2c3fb-qs5xm" Oct 30 17:14:51.311773 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:51.306722 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/486d41a9-c5ac-43c0-bb8e-5e9032acf5d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/486d41a9-c5ac-43c0-bb8e-5e9032acf5d2-maas-api-openshift-ca\") pod \"486d41a9-c5ac-43c0-bb8e-5e9032acf5d2\" (UID: \"486d41a9-c5ac-43c0-bb8e-5e9032acf5d2\") " Oct 30 17:14:51.311773 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:51.306912 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/486d41a9-c5ac-43c0-bb8e-5e9032acf5d2-kube-api-access-f7r9k\" (UniqueName: \"kubernetes.io/projected/486d41a9-c5ac-43c0-bb8e-5e9032acf5d2-kube-api-access-f7r9k\") pod \"486d41a9-c5ac-43c0-bb8e-5e9032acf5d2\" (UID: \"486d41a9-c5ac-43c0-bb8e-5e9032acf5d2\") " Oct 30 17:14:51.311773 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:51.307032 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/486d41a9-c5ac-43c0-bb8e-5e9032acf5d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/486d41a9-c5ac-43c0-bb8e-5e9032acf5d2-maas-api-xks-ca\") pod \"486d41a9-c5ac-43c0-bb8e-5e9032acf5d2\" (UID: \"486d41a9-c5ac-43c0-bb8e-5e9032acf5d2\") " Oct 30 17:14:51.311773 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:51.310156 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/486d41a9-c5ac-43c0-bb8e-5e9032acf5d2-maas-api-openshift-ca" pod "486d41a9-c5ac-43c0-bb8e-5e9032acf5d2" (UID: "486d41a9-c5ac-43c0-bb8e-5e9032acf5d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:14:51.340828 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:51.324188 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/486d41a9-c5ac-43c0-bb8e-5e9032acf5d2-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:14:51.340828 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:51.331819 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/486d41a9-c5ac-43c0-bb8e-5e9032acf5d2-kube-api-access-f7r9k" pod "486d41a9-c5ac-43c0-bb8e-5e9032acf5d2" (UID: "486d41a9-c5ac-43c0-bb8e-5e9032acf5d2"). InnerVolumeSpecName "kube-api-access-f7r9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:14:51.341390 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:51.341086 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/486d41a9-c5ac-43c0-bb8e-5e9032acf5d2-maas-api-xks-ca" pod "486d41a9-c5ac-43c0-bb8e-5e9032acf5d2" (UID: "486d41a9-c5ac-43c0-bb8e-5e9032acf5d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:14:51.425717 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:51.425666 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7r9k\" (UniqueName: \"kubernetes.io/projected/486d41a9-c5ac-43c0-bb8e-5e9032acf5d2-kube-api-access-f7r9k\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:14:51.425717 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:51.425722 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/486d41a9-c5ac-43c0-bb8e-5e9032acf5d2-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:14:52.009612 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:52.009576 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b2c3fb-qs5xm" Oct 30 17:14:52.010120 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:52.009615 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b2c3fb-qs5xm" event={"ID":"486d41a9-c5ac-43c0-bb8e-5e9032acf5d2","Type":"ContainerDied","Data":"9ee229c8df41a045d18a22c70125d63eba45d2941a9960ebd1461edad79772d4"} Oct 30 17:14:52.010120 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:52.009705 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ee229c8df41a045d18a22c70125d63eba45d2941a9960ebd1461edad79772d4" Oct 30 17:14:52.174072 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:52.174047 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ebacbb2-27rdm" Oct 30 17:14:52.338659 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:52.338580 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1555702-8acd-4384-acb9-94e3f51495de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1555702-8acd-4384-acb9-94e3f51495de-maas-api-openshift-ca\") pod \"a1555702-8acd-4384-acb9-94e3f51495de\" (UID: \"a1555702-8acd-4384-acb9-94e3f51495de\") " Oct 30 17:14:52.338837 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:52.338677 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1555702-8acd-4384-acb9-94e3f51495de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1555702-8acd-4384-acb9-94e3f51495de-maas-api-xks-ca\") pod \"a1555702-8acd-4384-acb9-94e3f51495de\" (UID: \"a1555702-8acd-4384-acb9-94e3f51495de\") " Oct 30 17:14:52.338837 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:52.338745 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1555702-8acd-4384-acb9-94e3f51495de-kube-api-access-78khw\" (UniqueName: \"kubernetes.io/projected/a1555702-8acd-4384-acb9-94e3f51495de-kube-api-access-78khw\") pod \"a1555702-8acd-4384-acb9-94e3f51495de\" (UID: \"a1555702-8acd-4384-acb9-94e3f51495de\") " Oct 30 17:14:52.339011 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:52.338983 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1555702-8acd-4384-acb9-94e3f51495de-maas-api-openshift-ca" pod "a1555702-8acd-4384-acb9-94e3f51495de" (UID: "a1555702-8acd-4384-acb9-94e3f51495de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:14:52.339225 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:52.339210 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1555702-8acd-4384-acb9-94e3f51495de-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:14:52.340950 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:52.340916 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1555702-8acd-4384-acb9-94e3f51495de-kube-api-access-78khw" pod "a1555702-8acd-4384-acb9-94e3f51495de" (UID: "a1555702-8acd-4384-acb9-94e3f51495de"). InnerVolumeSpecName "kube-api-access-78khw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:14:52.340950 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:52.340939 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1555702-8acd-4384-acb9-94e3f51495de-maas-api-xks-ca" pod "a1555702-8acd-4384-acb9-94e3f51495de" (UID: "a1555702-8acd-4384-acb9-94e3f51495de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:14:52.440695 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:52.440664 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1555702-8acd-4384-acb9-94e3f51495de-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:14:52.440695 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:52.440691 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78khw\" (UniqueName: \"kubernetes.io/projected/a1555702-8acd-4384-acb9-94e3f51495de-kube-api-access-78khw\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:14:52.905744 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:52.905710 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c82666-7867d55d87-89bwv"] Oct 30 17:14:52.906488 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:52.906471 4522 cpu_manager.go:414] "containerMap: removing container" podUID="a1555702-8acd-4384-acb9-94e3f51495de" containerName="revoke-keys" Oct 30 17:14:52.906536 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:52.906489 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1555702-8acd-4384-acb9-94e3f51495de" containerName="revoke-keys" Oct 30 17:14:52.906579 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:52.906539 4522 cpu_manager.go:414] "containerMap: removing container" podUID="486d41a9-c5ac-43c0-bb8e-5e9032acf5d2" containerName="revoke-keys" Oct 30 17:14:52.906579 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:52.906544 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="486d41a9-c5ac-43c0-bb8e-5e9032acf5d2" containerName="revoke-keys" Oct 30 17:14:52.906673 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:52.906663 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="486d41a9-c5ac-43c0-bb8e-5e9032acf5d2" containerName="revoke-keys" Oct 30 17:14:52.906710 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:52.906677 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1555702-8acd-4384-acb9-94e3f51495de" containerName="revoke-keys" Oct 30 17:14:52.910594 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:52.910578 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c82666-7867d55d87-89bwv" Oct 30 17:14:52.927024 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:52.926999 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-c82666\"" Oct 30 17:14:52.927024 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:52.927005 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-c82666-dockercfg-m6hn6\"" Oct 30 17:14:52.932588 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:52.932561 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c82666-7867d55d87-89bwv"] Oct 30 17:14:52.949413 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:52.949383 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1c26ee9-a385-4923-84f6-d1413e4a8302-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c82666-7867d55d87-89bwv\" (UID: \"a1c26ee9-a385-4923-84f6-d1413e4a8302\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c82666-7867d55d87-89bwv" Oct 30 17:14:52.949753 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:52.949728 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5fg8\" (UniqueName: \"kubernetes.io/projected/a1c26ee9-a385-4923-84f6-d1413e4a8302-kube-api-access-m5fg8\") pod \"payload-pre-processing-e2e-inf-b-c82666-7867d55d87-89bwv\" (UID: \"a1c26ee9-a385-4923-84f6-d1413e4a8302\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c82666-7867d55d87-89bwv" Oct 30 17:14:53.012363 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:53.011033 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c82666-6ccbf4bbb4-tw2zk"] Oct 30 17:14:53.082360 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:53.058947 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5fg8\" (UniqueName: \"kubernetes.io/projected/a1c26ee9-a385-4923-84f6-d1413e4a8302-kube-api-access-m5fg8\") pod \"payload-pre-processing-e2e-inf-b-c82666-7867d55d87-89bwv\" (UID: \"a1c26ee9-a385-4923-84f6-d1413e4a8302\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c82666-7867d55d87-89bwv" Oct 30 17:14:53.141789 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:53.141755 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1c26ee9-a385-4923-84f6-d1413e4a8302-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c82666-7867d55d87-89bwv\" (UID: \"a1c26ee9-a385-4923-84f6-d1413e4a8302\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c82666-7867d55d87-89bwv" Oct 30 17:14:53.142037 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:53.142000 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c82666-6ccbf4bbb4-tw2zk" Oct 30 17:14:53.146556 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:53.141921 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c82666-6ccbf4bbb4-tw2zk"] Oct 30 17:14:53.150440 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:53.148484 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1c26ee9-a385-4923-84f6-d1413e4a8302-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c82666-7867d55d87-89bwv\" (UID: \"a1c26ee9-a385-4923-84f6-d1413e4a8302\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c82666-7867d55d87-89bwv" Oct 30 17:14:53.152875 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:53.152801 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ebacbb2-27rdm" event={"ID":"a1555702-8acd-4384-acb9-94e3f51495de","Type":"ContainerDied","Data":"66989b6682c30060167b75dd8b6ca33b8d38b8c6be5ca9572263906577680b9f"} Oct 30 17:14:53.152875 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:53.152826 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ebacbb2-27rdm" Oct 30 17:14:53.152875 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:53.152853 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66989b6682c30060167b75dd8b6ca33b8d38b8c6be5ca9572263906577680b9f" Oct 30 17:14:53.182537 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:53.182512 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5fg8\" (UniqueName: \"kubernetes.io/projected/a1c26ee9-a385-4923-84f6-d1413e4a8302-kube-api-access-m5fg8\") pod \"payload-pre-processing-e2e-inf-b-c82666-7867d55d87-89bwv\" (UID: \"a1c26ee9-a385-4923-84f6-d1413e4a8302\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c82666-7867d55d87-89bwv" Oct 30 17:14:53.225541 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:53.225510 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c82666-7867d55d87-89bwv" Oct 30 17:14:53.246380 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:53.246347 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zh6n\" (UniqueName: \"kubernetes.io/projected/e3f6c9b4-919b-4ea3-9909-1c9824339e55-kube-api-access-6zh6n\") pod \"payload-processing-e2e-inf-b-c82666-6ccbf4bbb4-tw2zk\" (UID: \"e3f6c9b4-919b-4ea3-9909-1c9824339e55\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c82666-6ccbf4bbb4-tw2zk" Oct 30 17:14:53.246612 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:53.246582 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3f6c9b4-919b-4ea3-9909-1c9824339e55-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c82666-6ccbf4bbb4-tw2zk\" (UID: \"e3f6c9b4-919b-4ea3-9909-1c9824339e55\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c82666-6ccbf4bbb4-tw2zk" Oct 30 17:14:53.348171 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:53.348119 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zh6n\" (UniqueName: \"kubernetes.io/projected/e3f6c9b4-919b-4ea3-9909-1c9824339e55-kube-api-access-6zh6n\") pod \"payload-processing-e2e-inf-b-c82666-6ccbf4bbb4-tw2zk\" (UID: \"e3f6c9b4-919b-4ea3-9909-1c9824339e55\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c82666-6ccbf4bbb4-tw2zk" Oct 30 17:14:53.348414 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:53.348295 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3f6c9b4-919b-4ea3-9909-1c9824339e55-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c82666-6ccbf4bbb4-tw2zk\" (UID: \"e3f6c9b4-919b-4ea3-9909-1c9824339e55\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c82666-6ccbf4bbb4-tw2zk" Oct 30 17:14:53.349393 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:53.349361 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3f6c9b4-919b-4ea3-9909-1c9824339e55-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c82666-6ccbf4bbb4-tw2zk\" (UID: \"e3f6c9b4-919b-4ea3-9909-1c9824339e55\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c82666-6ccbf4bbb4-tw2zk" Oct 30 17:14:53.377238 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:53.377205 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zh6n\" (UniqueName: \"kubernetes.io/projected/e3f6c9b4-919b-4ea3-9909-1c9824339e55-kube-api-access-6zh6n\") pod \"payload-processing-e2e-inf-b-c82666-6ccbf4bbb4-tw2zk\" (UID: \"e3f6c9b4-919b-4ea3-9909-1c9824339e55\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c82666-6ccbf4bbb4-tw2zk" Oct 30 17:14:53.482143 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:53.482106 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c82666-6ccbf4bbb4-tw2zk" Oct 30 17:14:53.558233 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:53.558204 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c82666-7867d55d87-89bwv"] Oct 30 17:14:53.827275 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:14:53.827246 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3f6c9b4_919b_4ea3_9909_1c9824339e55.slice/crio-568917d413441288e464c8252646d6a25362de4a42e91b61cc96c46d2e9b9e5e WatchSource:0}: Error finding container 568917d413441288e464c8252646d6a25362de4a42e91b61cc96c46d2e9b9e5e: Status 404 returned error can't find the container with id 568917d413441288e464c8252646d6a25362de4a42e91b61cc96c46d2e9b9e5e Oct 30 17:14:53.829369 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:53.829268 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c82666-6ccbf4bbb4-tw2zk"] Oct 30 17:14:54.175944 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:54.175836 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c82666-7867d55d87-89bwv" event={"ID":"a1c26ee9-a385-4923-84f6-d1413e4a8302","Type":"ContainerStarted","Data":"168a7ce10e5e16b41f4bd3a435a416cdcd91764c29b85043b0b472a6f79d5838"} Oct 30 17:14:54.175944 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:54.175887 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c82666-7867d55d87-89bwv" event={"ID":"a1c26ee9-a385-4923-84f6-d1413e4a8302","Type":"ContainerStarted","Data":"95dc8c3277bbd1243ef37fba4cfb6ab6f27fe6e88f3ebdd918acd6551ef70127"} Oct 30 17:14:54.176514 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:54.176047 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c82666-7867d55d87-89bwv" Oct 30 17:14:54.177762 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:54.177735 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c82666-6ccbf4bbb4-tw2zk" event={"ID":"e3f6c9b4-919b-4ea3-9909-1c9824339e55","Type":"ContainerStarted","Data":"e73ca24f781d3f592f04c4fddf86fda9afb6a621ae62d504b1cab1bae60e76b4"} Oct 30 17:14:54.177868 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:54.177777 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c82666-6ccbf4bbb4-tw2zk" event={"ID":"e3f6c9b4-919b-4ea3-9909-1c9824339e55","Type":"ContainerStarted","Data":"568917d413441288e464c8252646d6a25362de4a42e91b61cc96c46d2e9b9e5e"} Oct 30 17:14:54.177868 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:54.177815 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-c82666-6ccbf4bbb4-tw2zk" Oct 30 17:14:54.226650 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:54.226598 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c82666-7867d55d87-89bwv" podStartSLOduration=2.226583968 podStartE2EDuration="2.226583968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:14:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:14:54.222834907 +0000 UTC m=+1668.223589851" watchObservedRunningTime="2026-10-30 17:14:54.226583968 +0000 UTC m=+1668.227338887" Oct 30 17:14:54.266040 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:54.265992 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-c82666-6ccbf4bbb4-tw2zk" podStartSLOduration=2.265980003 podStartE2EDuration="2.265980003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:14:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:14:54.265325105 +0000 UTC m=+1668.266080006" watchObservedRunningTime="2026-10-30 17:14:54.265980003 +0000 UTC m=+1668.266734958" Oct 30 17:14:55.028329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:55.027293 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd"] Oct 30 17:14:55.052327 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:55.043839 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd" Oct 30 17:14:55.085699 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:55.072583 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-5d67ca-openshift-default-dockercfg-z2w2w\"" Oct 30 17:14:55.099356 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:55.096946 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd"] Oct 30 17:14:55.192893 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:55.192851 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/065164fc-1823-49af-9921-d14d9a94e997-istiod-ca-cert\") pod \"e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd\" (UID: \"065164fc-1823-49af-9921-d14d9a94e997\") " pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd" Oct 30 17:14:55.193499 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:55.192943 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/065164fc-1823-49af-9921-d14d9a94e997-workload-certs\") pod \"e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd\" (UID: \"065164fc-1823-49af-9921-d14d9a94e997\") " pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd" Oct 30 17:14:55.193499 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:55.193338 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/065164fc-1823-49af-9921-d14d9a94e997-credential-socket\") pod \"e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd\" (UID: \"065164fc-1823-49af-9921-d14d9a94e997\") " pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd" Oct 30 17:14:55.195329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:55.194133 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfb27\" (UniqueName: \"kubernetes.io/projected/065164fc-1823-49af-9921-d14d9a94e997-kube-api-access-qfb27\") pod \"e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd\" (UID: \"065164fc-1823-49af-9921-d14d9a94e997\") " pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd" Oct 30 17:14:55.195329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:55.194182 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/065164fc-1823-49af-9921-d14d9a94e997-istio-data\") pod \"e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd\" (UID: \"065164fc-1823-49af-9921-d14d9a94e997\") " pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd" Oct 30 17:14:55.195329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:55.194210 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/065164fc-1823-49af-9921-d14d9a94e997-istio-podinfo\") pod \"e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd\" (UID: \"065164fc-1823-49af-9921-d14d9a94e997\") " pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd" Oct 30 17:14:55.195329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:55.194238 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/065164fc-1823-49af-9921-d14d9a94e997-workload-socket\") pod \"e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd\" (UID: \"065164fc-1823-49af-9921-d14d9a94e997\") " pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd" Oct 30 17:14:55.195329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:55.194264 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/065164fc-1823-49af-9921-d14d9a94e997-istio-envoy\") pod \"e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd\" (UID: \"065164fc-1823-49af-9921-d14d9a94e997\") " pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd" Oct 30 17:14:55.195329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:55.194516 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/065164fc-1823-49af-9921-d14d9a94e997-istio-token\") pod \"e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd\" (UID: \"065164fc-1823-49af-9921-d14d9a94e997\") " pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd" Oct 30 17:14:55.299387 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:55.299018 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/065164fc-1823-49af-9921-d14d9a94e997-istio-token\") pod \"e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd\" (UID: \"065164fc-1823-49af-9921-d14d9a94e997\") " pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd" Oct 30 17:14:55.299387 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:55.299140 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/065164fc-1823-49af-9921-d14d9a94e997-istiod-ca-cert\") pod \"e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd\" (UID: \"065164fc-1823-49af-9921-d14d9a94e997\") " pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd" Oct 30 17:14:55.300979 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:55.300947 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/065164fc-1823-49af-9921-d14d9a94e997-istiod-ca-cert\") pod \"e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd\" (UID: \"065164fc-1823-49af-9921-d14d9a94e997\") " pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd" Oct 30 17:14:55.302472 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:55.302434 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/065164fc-1823-49af-9921-d14d9a94e997-workload-certs\") pod \"e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd\" (UID: \"065164fc-1823-49af-9921-d14d9a94e997\") " pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd" Oct 30 17:14:55.302686 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:55.302667 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/065164fc-1823-49af-9921-d14d9a94e997-credential-socket\") pod \"e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd\" (UID: \"065164fc-1823-49af-9921-d14d9a94e997\") " pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd" Oct 30 17:14:55.302770 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:55.302719 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfb27\" (UniqueName: \"kubernetes.io/projected/065164fc-1823-49af-9921-d14d9a94e997-kube-api-access-qfb27\") pod \"e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd\" (UID: \"065164fc-1823-49af-9921-d14d9a94e997\") " pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd" Oct 30 17:14:55.302893 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:55.302767 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/065164fc-1823-49af-9921-d14d9a94e997-istio-data\") pod \"e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd\" (UID: \"065164fc-1823-49af-9921-d14d9a94e997\") " pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd" Oct 30 17:14:55.302893 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:55.302797 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/065164fc-1823-49af-9921-d14d9a94e997-istio-podinfo\") pod \"e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd\" (UID: \"065164fc-1823-49af-9921-d14d9a94e997\") " pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd" Oct 30 17:14:55.302893 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:55.302830 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/065164fc-1823-49af-9921-d14d9a94e997-workload-socket\") pod \"e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd\" (UID: \"065164fc-1823-49af-9921-d14d9a94e997\") " pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd" Oct 30 17:14:55.302893 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:55.302860 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/065164fc-1823-49af-9921-d14d9a94e997-istio-envoy\") pod \"e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd\" (UID: \"065164fc-1823-49af-9921-d14d9a94e997\") " pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd" Oct 30 17:14:55.303449 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:55.303426 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/065164fc-1823-49af-9921-d14d9a94e997-workload-certs\") pod \"e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd\" (UID: \"065164fc-1823-49af-9921-d14d9a94e997\") " pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd" Oct 30 17:14:55.304042 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:55.304020 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/065164fc-1823-49af-9921-d14d9a94e997-credential-socket\") pod \"e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd\" (UID: \"065164fc-1823-49af-9921-d14d9a94e997\") " pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd" Oct 30 17:14:55.304420 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:55.304396 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/065164fc-1823-49af-9921-d14d9a94e997-workload-socket\") pod \"e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd\" (UID: \"065164fc-1823-49af-9921-d14d9a94e997\") " pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd" Oct 30 17:14:55.304718 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:55.304698 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/065164fc-1823-49af-9921-d14d9a94e997-istio-data\") pod \"e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd\" (UID: \"065164fc-1823-49af-9921-d14d9a94e997\") " pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd" Oct 30 17:14:55.307483 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:55.307463 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/065164fc-1823-49af-9921-d14d9a94e997-istio-envoy\") pod \"e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd\" (UID: \"065164fc-1823-49af-9921-d14d9a94e997\") " pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd" Oct 30 17:14:55.308141 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:55.308115 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/065164fc-1823-49af-9921-d14d9a94e997-istio-podinfo\") pod \"e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd\" (UID: \"065164fc-1823-49af-9921-d14d9a94e997\") " pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd" Oct 30 17:14:55.324865 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:55.324814 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/065164fc-1823-49af-9921-d14d9a94e997-istio-token\") pod \"e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd\" (UID: \"065164fc-1823-49af-9921-d14d9a94e997\") " pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd" Oct 30 17:14:55.328635 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:55.328609 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfb27\" (UniqueName: \"kubernetes.io/projected/065164fc-1823-49af-9921-d14d9a94e997-kube-api-access-qfb27\") pod \"e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd\" (UID: \"065164fc-1823-49af-9921-d14d9a94e997\") " pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd" Oct 30 17:14:55.395733 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:55.395685 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd" Oct 30 17:14:55.766755 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:14:55.766720 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod065164fc_1823_49af_9921_d14d9a94e997.slice/crio-b40e3ac0fe5bb7b9672ec6d437fda14a018a35f3b911b9b8cc4ed5e4116d4216 WatchSource:0}: Error finding container b40e3ac0fe5bb7b9672ec6d437fda14a018a35f3b911b9b8cc4ed5e4116d4216: Status 404 returned error can't find the container with id b40e3ac0fe5bb7b9672ec6d437fda14a018a35f3b911b9b8cc4ed5e4116d4216 Oct 30 17:14:55.768080 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:55.768055 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd"] Oct 30 17:14:55.770723 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:55.770684 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:14:55.770809 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:55.770792 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:14:55.770850 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:55.770839 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:14:56.200792 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:56.200705 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd" event={"ID":"065164fc-1823-49af-9921-d14d9a94e997","Type":"ContainerStarted","Data":"290cce3c9c779f0915bd8a6795025cd875f0cd66d2d3588424cfd0536ee49648"} Oct 30 17:14:56.200792 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:56.200767 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd" event={"ID":"065164fc-1823-49af-9921-d14d9a94e997","Type":"ContainerStarted","Data":"b40e3ac0fe5bb7b9672ec6d437fda14a018a35f3b911b9b8cc4ed5e4116d4216"} Oct 30 17:14:56.252781 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:56.252730 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd" podStartSLOduration=2.25271599 podStartE2EDuration="2.25271599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:14:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:14:56.248836753 +0000 UTC m=+1670.249591678" watchObservedRunningTime="2026-10-30 17:14:56.25271599 +0000 UTC m=+1670.253470890" Oct 30 17:14:56.396513 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:56.396468 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd" Oct 30 17:14:56.402784 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:56.402757 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd" Oct 30 17:14:57.225220 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:57.224487 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd" Oct 30 17:14:57.228573 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:57.227623 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd" Oct 30 17:14:58.205331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:58.195822 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b2c3fb-5f877958dd-99vkh"] Oct 30 17:14:58.275143 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:14:58.275103 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b2c3fb-7698878959-chvds"] Oct 30 17:15:00.155681 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.155594 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889675-xhg9d"] Oct 30 17:15:00.160756 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.160727 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889675-crbqg"] Oct 30 17:15:00.160900 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.160848 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889675-xhg9d" Oct 30 17:15:00.165488 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.165468 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889675-crbqg" Oct 30 17:15:00.166878 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.166852 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889675-dbh98"] Oct 30 17:15:00.172100 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.172075 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889675-dbh98" Oct 30 17:15:00.173575 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.173548 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f42baf-29889675-2mg4p"] Oct 30 17:15:00.202554 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.192251 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:15:00.237278 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.237247 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889675-xhg9d"] Oct 30 17:15:00.237483 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.237294 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889675-crbqg"] Oct 30 17:15:00.237483 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.237353 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889675-dbh98"] Oct 30 17:15:00.237483 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.237454 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f42baf-29889675-2mg4p" Oct 30 17:15:00.239880 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.239858 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f42baf-29889675-2mg4p"] Oct 30 17:15:00.267211 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.266300 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889675-fdxml"] Oct 30 17:15:00.307063 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.304875 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8ds5\" (UniqueName: \"kubernetes.io/projected/5802774a-44fd-4660-b00e-8d979654b196-kube-api-access-n8ds5\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889675-crbqg\" (UID: \"5802774a-44fd-4660-b00e-8d979654b196\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889675-crbqg" Oct 30 17:15:00.307063 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.304929 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78db2ae0-039b-4edc-8fdb-c4513041d366-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889675-dbh98\" (UID: \"78db2ae0-039b-4edc-8fdb-c4513041d366\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889675-dbh98" Oct 30 17:15:00.307063 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.304959 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vqsr\" (UniqueName: \"kubernetes.io/projected/78db2ae0-039b-4edc-8fdb-c4513041d366-kube-api-access-5vqsr\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889675-dbh98\" (UID: \"78db2ae0-039b-4edc-8fdb-c4513041d366\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889675-dbh98" Oct 30 17:15:00.307063 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.305023 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdwlk\" (UniqueName: \"kubernetes.io/projected/4e7e3bf8-d7a4-4839-bcf4-5161e1f998c9-kube-api-access-vdwlk\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889675-xhg9d\" (UID: \"4e7e3bf8-d7a4-4839-bcf4-5161e1f998c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889675-xhg9d" Oct 30 17:15:00.307063 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.305146 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e7e3bf8-d7a4-4839-bcf4-5161e1f998c9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889675-xhg9d\" (UID: \"4e7e3bf8-d7a4-4839-bcf4-5161e1f998c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889675-xhg9d" Oct 30 17:15:00.307063 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.305550 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5802774a-44fd-4660-b00e-8d979654b196-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889675-crbqg\" (UID: \"5802774a-44fd-4660-b00e-8d979654b196\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889675-crbqg" Oct 30 17:15:00.313778 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.313616 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889675-rbwqn"] Oct 30 17:15:00.314123 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.313752 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889675-fdxml" Oct 30 17:15:00.320730 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.320706 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b2c3fb-29889675-rckqx"] Oct 30 17:15:00.320889 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.320816 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889675-rbwqn" Oct 30 17:15:00.326794 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.326763 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889675-fdxml"] Oct 30 17:15:00.326919 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.326816 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889675-lcm6b"] Oct 30 17:15:00.326988 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.326941 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b2c3fb-29889675-rckqx" Oct 30 17:15:00.332279 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.332253 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-728ad7-29889675-hgghk"] Oct 30 17:15:00.334031 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.334008 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889675-lcm6b" Oct 30 17:15:00.342678 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.342650 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-728ad7-29889675-hgghk" Oct 30 17:15:00.357835 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.357541 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b2c3fb-29889675-rckqx"] Oct 30 17:15:00.364387 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.364359 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889675-rbwqn"] Oct 30 17:15:00.371706 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.371678 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39fecf-29889675-jshxf"] Oct 30 17:15:00.377725 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.377666 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39fecf-29889675-jshxf" Oct 30 17:15:00.378072 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.378048 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889675-s24lj"] Oct 30 17:15:00.384669 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.384647 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889675-lcm6b"] Oct 30 17:15:00.384810 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.384751 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889675-s24lj" Oct 30 17:15:00.389097 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.389056 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-728ad7-29889675-hgghk"] Oct 30 17:15:00.399983 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.399958 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889675-s24lj"] Oct 30 17:15:00.404155 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.404126 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39fecf-29889675-jshxf"] Oct 30 17:15:00.406750 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.406701 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdwlk\" (UniqueName: \"kubernetes.io/projected/4e7e3bf8-d7a4-4839-bcf4-5161e1f998c9-kube-api-access-vdwlk\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889675-xhg9d\" (UID: \"4e7e3bf8-d7a4-4839-bcf4-5161e1f998c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889675-xhg9d" Oct 30 17:15:00.406891 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.406871 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpm2j\" (UniqueName: \"kubernetes.io/projected/c53d8512-95ac-4d60-aaa6-0d74b95e0cf8-kube-api-access-kpm2j\") pod \"maas-api-key-cleanup-e2e-inf-a-f42baf-29889675-2mg4p\" (UID: \"c53d8512-95ac-4d60-aaa6-0d74b95e0cf8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f42baf-29889675-2mg4p" Oct 30 17:15:00.407000 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.406979 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e7e3bf8-d7a4-4839-bcf4-5161e1f998c9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889675-xhg9d\" (UID: \"4e7e3bf8-d7a4-4839-bcf4-5161e1f998c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889675-xhg9d" Oct 30 17:15:00.407268 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.407251 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c53d8512-95ac-4d60-aaa6-0d74b95e0cf8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-f42baf-29889675-2mg4p\" (UID: \"c53d8512-95ac-4d60-aaa6-0d74b95e0cf8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f42baf-29889675-2mg4p" Oct 30 17:15:00.407400 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.407379 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5802774a-44fd-4660-b00e-8d979654b196-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889675-crbqg\" (UID: \"5802774a-44fd-4660-b00e-8d979654b196\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889675-crbqg" Oct 30 17:15:00.407467 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.407441 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8ds5\" (UniqueName: \"kubernetes.io/projected/5802774a-44fd-4660-b00e-8d979654b196-kube-api-access-n8ds5\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889675-crbqg\" (UID: \"5802774a-44fd-4660-b00e-8d979654b196\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889675-crbqg" Oct 30 17:15:00.407520 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.407477 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78db2ae0-039b-4edc-8fdb-c4513041d366-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889675-dbh98\" (UID: \"78db2ae0-039b-4edc-8fdb-c4513041d366\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889675-dbh98" Oct 30 17:15:00.407581 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.407544 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vqsr\" (UniqueName: \"kubernetes.io/projected/78db2ae0-039b-4edc-8fdb-c4513041d366-kube-api-access-5vqsr\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889675-dbh98\" (UID: \"78db2ae0-039b-4edc-8fdb-c4513041d366\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889675-dbh98" Oct 30 17:15:00.407916 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.407877 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e7e3bf8-d7a4-4839-bcf4-5161e1f998c9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889675-xhg9d\" (UID: \"4e7e3bf8-d7a4-4839-bcf4-5161e1f998c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889675-xhg9d" Oct 30 17:15:00.408426 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.408402 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78db2ae0-039b-4edc-8fdb-c4513041d366-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889675-dbh98\" (UID: \"78db2ae0-039b-4edc-8fdb-c4513041d366\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889675-dbh98" Oct 30 17:15:00.408771 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.408749 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5802774a-44fd-4660-b00e-8d979654b196-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889675-crbqg\" (UID: \"5802774a-44fd-4660-b00e-8d979654b196\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889675-crbqg" Oct 30 17:15:00.433260 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.433219 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vqsr\" (UniqueName: \"kubernetes.io/projected/78db2ae0-039b-4edc-8fdb-c4513041d366-kube-api-access-5vqsr\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889675-dbh98\" (UID: \"78db2ae0-039b-4edc-8fdb-c4513041d366\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889675-dbh98" Oct 30 17:15:00.433433 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.433379 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdwlk\" (UniqueName: \"kubernetes.io/projected/4e7e3bf8-d7a4-4839-bcf4-5161e1f998c9-kube-api-access-vdwlk\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889675-xhg9d\" (UID: \"4e7e3bf8-d7a4-4839-bcf4-5161e1f998c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889675-xhg9d" Oct 30 17:15:00.433433 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.433389 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8ds5\" (UniqueName: \"kubernetes.io/projected/5802774a-44fd-4660-b00e-8d979654b196-kube-api-access-n8ds5\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889675-crbqg\" (UID: \"5802774a-44fd-4660-b00e-8d979654b196\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889675-crbqg" Oct 30 17:15:00.494374 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.494343 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889675-xhg9d" Oct 30 17:15:00.507255 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.507224 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889675-crbqg" Oct 30 17:15:00.508698 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.508675 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ea441b9-186a-4e43-b2c7-37052bd5dcd7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889675-fdxml\" (UID: \"1ea441b9-186a-4e43-b2c7-37052bd5dcd7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889675-fdxml" Oct 30 17:15:00.508897 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.508877 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-777l6\" (UniqueName: \"kubernetes.io/projected/79a79c82-3142-4035-a072-004b80dfe24d-kube-api-access-777l6\") pod \"maas-api-key-cleanup-e2e-worker-w5-86f50a-29889675-s24lj\" (UID: \"79a79c82-3142-4035-a072-004b80dfe24d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889675-s24lj" Oct 30 17:15:00.508981 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.508910 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5hmw\" (UniqueName: \"kubernetes.io/projected/ad604ad3-0fd0-4544-af4d-e5061bd561e3-kube-api-access-h5hmw\") pod \"maas-api-key-cleanup-e2e-api-b-b2c3fb-29889675-rckqx\" (UID: \"ad604ad3-0fd0-4544-af4d-e5061bd561e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b2c3fb-29889675-rckqx" Oct 30 17:15:00.509038 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.509013 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khpmd\" (UniqueName: \"kubernetes.io/projected/f5227ac7-04e8-4758-a1ed-1b907783431e-kube-api-access-khpmd\") pod \"maas-api-key-cleanup-e2e-inf-b-39fecf-29889675-jshxf\" (UID: \"f5227ac7-04e8-4758-a1ed-1b907783431e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39fecf-29889675-jshxf" Oct 30 17:15:00.509134 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.509116 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w25vf\" (UniqueName: \"kubernetes.io/projected/417f0b10-2a38-4757-8563-d47b75088b4d-kube-api-access-w25vf\") pod \"maas-api-key-cleanup-e2e-inf-a-728ad7-29889675-hgghk\" (UID: \"417f0b10-2a38-4757-8563-d47b75088b4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-728ad7-29889675-hgghk" Oct 30 17:15:00.509242 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.509227 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c53d8512-95ac-4d60-aaa6-0d74b95e0cf8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-f42baf-29889675-2mg4p\" (UID: \"c53d8512-95ac-4d60-aaa6-0d74b95e0cf8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f42baf-29889675-2mg4p" Oct 30 17:15:00.509317 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.509262 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm9nj\" (UniqueName: \"kubernetes.io/projected/1ea441b9-186a-4e43-b2c7-37052bd5dcd7-kube-api-access-bm9nj\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889675-fdxml\" (UID: \"1ea441b9-186a-4e43-b2c7-37052bd5dcd7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889675-fdxml" Oct 30 17:15:00.509410 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.509396 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d21c30e-1815-436a-92b5-4cb61b88ea19-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889675-lcm6b\" (UID: \"5d21c30e-1815-436a-92b5-4cb61b88ea19\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889675-lcm6b" Oct 30 17:15:00.509561 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.509547 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79a79c82-3142-4035-a072-004b80dfe24d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-86f50a-29889675-s24lj\" (UID: \"79a79c82-3142-4035-a072-004b80dfe24d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889675-s24lj" Oct 30 17:15:00.509646 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.509592 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/171da860-d46a-420a-88a3-423d6b58c86b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889675-rbwqn\" (UID: \"171da860-d46a-420a-88a3-423d6b58c86b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889675-rbwqn" Oct 30 17:15:00.509762 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.509731 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/417f0b10-2a38-4757-8563-d47b75088b4d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-728ad7-29889675-hgghk\" (UID: \"417f0b10-2a38-4757-8563-d47b75088b4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-728ad7-29889675-hgghk" Oct 30 17:15:00.509826 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.509786 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k24s\" (UniqueName: \"kubernetes.io/projected/171da860-d46a-420a-88a3-423d6b58c86b-kube-api-access-9k24s\") pod \"maas-api-key-cleanup-29889675-rbwqn\" (UID: \"171da860-d46a-420a-88a3-423d6b58c86b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889675-rbwqn" Oct 30 17:15:00.509869 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.509828 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpm2j\" (UniqueName: \"kubernetes.io/projected/c53d8512-95ac-4d60-aaa6-0d74b95e0cf8-kube-api-access-kpm2j\") pod \"maas-api-key-cleanup-e2e-inf-a-f42baf-29889675-2mg4p\" (UID: \"c53d8512-95ac-4d60-aaa6-0d74b95e0cf8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f42baf-29889675-2mg4p" Oct 30 17:15:00.510100 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.510075 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c53d8512-95ac-4d60-aaa6-0d74b95e0cf8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-f42baf-29889675-2mg4p\" (UID: \"c53d8512-95ac-4d60-aaa6-0d74b95e0cf8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f42baf-29889675-2mg4p" Oct 30 17:15:00.510439 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.510220 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5227ac7-04e8-4758-a1ed-1b907783431e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-39fecf-29889675-jshxf\" (UID: \"f5227ac7-04e8-4758-a1ed-1b907783431e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39fecf-29889675-jshxf" Oct 30 17:15:00.510439 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.510262 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad604ad3-0fd0-4544-af4d-e5061bd561e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-b2c3fb-29889675-rckqx\" (UID: \"ad604ad3-0fd0-4544-af4d-e5061bd561e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b2c3fb-29889675-rckqx" Oct 30 17:15:00.510439 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.510300 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdm4x\" (UniqueName: \"kubernetes.io/projected/5d21c30e-1815-436a-92b5-4cb61b88ea19-kube-api-access-fdm4x\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889675-lcm6b\" (UID: \"5d21c30e-1815-436a-92b5-4cb61b88ea19\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889675-lcm6b" Oct 30 17:15:00.535633 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.535571 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpm2j\" (UniqueName: \"kubernetes.io/projected/c53d8512-95ac-4d60-aaa6-0d74b95e0cf8-kube-api-access-kpm2j\") pod \"maas-api-key-cleanup-e2e-inf-a-f42baf-29889675-2mg4p\" (UID: \"c53d8512-95ac-4d60-aaa6-0d74b95e0cf8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f42baf-29889675-2mg4p" Oct 30 17:15:00.542547 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.542367 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889675-dbh98" Oct 30 17:15:00.579907 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.579883 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f42baf-29889675-2mg4p" Oct 30 17:15:00.614942 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.611942 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79a79c82-3142-4035-a072-004b80dfe24d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-86f50a-29889675-s24lj\" (UID: \"79a79c82-3142-4035-a072-004b80dfe24d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889675-s24lj" Oct 30 17:15:00.614942 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.612005 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/171da860-d46a-420a-88a3-423d6b58c86b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889675-rbwqn\" (UID: \"171da860-d46a-420a-88a3-423d6b58c86b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889675-rbwqn" Oct 30 17:15:00.614942 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.612038 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/417f0b10-2a38-4757-8563-d47b75088b4d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-728ad7-29889675-hgghk\" (UID: \"417f0b10-2a38-4757-8563-d47b75088b4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-728ad7-29889675-hgghk" Oct 30 17:15:00.614942 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.612067 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k24s\" (UniqueName: \"kubernetes.io/projected/171da860-d46a-420a-88a3-423d6b58c86b-kube-api-access-9k24s\") pod \"maas-api-key-cleanup-29889675-rbwqn\" (UID: \"171da860-d46a-420a-88a3-423d6b58c86b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889675-rbwqn" Oct 30 17:15:00.614942 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.612175 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5227ac7-04e8-4758-a1ed-1b907783431e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-39fecf-29889675-jshxf\" (UID: \"f5227ac7-04e8-4758-a1ed-1b907783431e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39fecf-29889675-jshxf" Oct 30 17:15:00.614942 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.612206 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad604ad3-0fd0-4544-af4d-e5061bd561e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-b2c3fb-29889675-rckqx\" (UID: \"ad604ad3-0fd0-4544-af4d-e5061bd561e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b2c3fb-29889675-rckqx" Oct 30 17:15:00.614942 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.612248 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdm4x\" (UniqueName: \"kubernetes.io/projected/5d21c30e-1815-436a-92b5-4cb61b88ea19-kube-api-access-fdm4x\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889675-lcm6b\" (UID: \"5d21c30e-1815-436a-92b5-4cb61b88ea19\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889675-lcm6b" Oct 30 17:15:00.614942 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.612321 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ea441b9-186a-4e43-b2c7-37052bd5dcd7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889675-fdxml\" (UID: \"1ea441b9-186a-4e43-b2c7-37052bd5dcd7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889675-fdxml" Oct 30 17:15:00.614942 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.612370 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-777l6\" (UniqueName: \"kubernetes.io/projected/79a79c82-3142-4035-a072-004b80dfe24d-kube-api-access-777l6\") pod \"maas-api-key-cleanup-e2e-worker-w5-86f50a-29889675-s24lj\" (UID: \"79a79c82-3142-4035-a072-004b80dfe24d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889675-s24lj" Oct 30 17:15:00.614942 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.612395 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5hmw\" (UniqueName: \"kubernetes.io/projected/ad604ad3-0fd0-4544-af4d-e5061bd561e3-kube-api-access-h5hmw\") pod \"maas-api-key-cleanup-e2e-api-b-b2c3fb-29889675-rckqx\" (UID: \"ad604ad3-0fd0-4544-af4d-e5061bd561e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b2c3fb-29889675-rckqx" Oct 30 17:15:00.614942 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.612420 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khpmd\" (UniqueName: \"kubernetes.io/projected/f5227ac7-04e8-4758-a1ed-1b907783431e-kube-api-access-khpmd\") pod \"maas-api-key-cleanup-e2e-inf-b-39fecf-29889675-jshxf\" (UID: \"f5227ac7-04e8-4758-a1ed-1b907783431e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39fecf-29889675-jshxf" Oct 30 17:15:00.614942 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.612489 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w25vf\" (UniqueName: \"kubernetes.io/projected/417f0b10-2a38-4757-8563-d47b75088b4d-kube-api-access-w25vf\") pod \"maas-api-key-cleanup-e2e-inf-a-728ad7-29889675-hgghk\" (UID: \"417f0b10-2a38-4757-8563-d47b75088b4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-728ad7-29889675-hgghk" Oct 30 17:15:00.614942 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.612559 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm9nj\" (UniqueName: \"kubernetes.io/projected/1ea441b9-186a-4e43-b2c7-37052bd5dcd7-kube-api-access-bm9nj\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889675-fdxml\" (UID: \"1ea441b9-186a-4e43-b2c7-37052bd5dcd7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889675-fdxml" Oct 30 17:15:00.614942 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.612626 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d21c30e-1815-436a-92b5-4cb61b88ea19-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889675-lcm6b\" (UID: \"5d21c30e-1815-436a-92b5-4cb61b88ea19\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889675-lcm6b" Oct 30 17:15:00.614942 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.613533 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d21c30e-1815-436a-92b5-4cb61b88ea19-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889675-lcm6b\" (UID: \"5d21c30e-1815-436a-92b5-4cb61b88ea19\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889675-lcm6b" Oct 30 17:15:00.614942 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.614189 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79a79c82-3142-4035-a072-004b80dfe24d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-86f50a-29889675-s24lj\" (UID: \"79a79c82-3142-4035-a072-004b80dfe24d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889675-s24lj" Oct 30 17:15:00.614942 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.614819 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/171da860-d46a-420a-88a3-423d6b58c86b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889675-rbwqn\" (UID: \"171da860-d46a-420a-88a3-423d6b58c86b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889675-rbwqn" Oct 30 17:15:00.616101 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.615577 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/417f0b10-2a38-4757-8563-d47b75088b4d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-728ad7-29889675-hgghk\" (UID: \"417f0b10-2a38-4757-8563-d47b75088b4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-728ad7-29889675-hgghk" Oct 30 17:15:00.620283 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.616487 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5227ac7-04e8-4758-a1ed-1b907783431e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-39fecf-29889675-jshxf\" (UID: \"f5227ac7-04e8-4758-a1ed-1b907783431e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39fecf-29889675-jshxf" Oct 30 17:15:00.620283 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.617611 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad604ad3-0fd0-4544-af4d-e5061bd561e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-b2c3fb-29889675-rckqx\" (UID: \"ad604ad3-0fd0-4544-af4d-e5061bd561e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b2c3fb-29889675-rckqx" Oct 30 17:15:00.620283 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.618536 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ea441b9-186a-4e43-b2c7-37052bd5dcd7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889675-fdxml\" (UID: \"1ea441b9-186a-4e43-b2c7-37052bd5dcd7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889675-fdxml" Oct 30 17:15:00.642863 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.642830 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k24s\" (UniqueName: \"kubernetes.io/projected/171da860-d46a-420a-88a3-423d6b58c86b-kube-api-access-9k24s\") pod \"maas-api-key-cleanup-29889675-rbwqn\" (UID: \"171da860-d46a-420a-88a3-423d6b58c86b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889675-rbwqn" Oct 30 17:15:00.645377 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.644892 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdm4x\" (UniqueName: \"kubernetes.io/projected/5d21c30e-1815-436a-92b5-4cb61b88ea19-kube-api-access-fdm4x\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889675-lcm6b\" (UID: \"5d21c30e-1815-436a-92b5-4cb61b88ea19\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889675-lcm6b" Oct 30 17:15:00.648866 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.648773 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm9nj\" (UniqueName: \"kubernetes.io/projected/1ea441b9-186a-4e43-b2c7-37052bd5dcd7-kube-api-access-bm9nj\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889675-fdxml\" (UID: \"1ea441b9-186a-4e43-b2c7-37052bd5dcd7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889675-fdxml" Oct 30 17:15:00.667401 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.667093 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889675-rbwqn" Oct 30 17:15:00.683723 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.683666 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khpmd\" (UniqueName: \"kubernetes.io/projected/f5227ac7-04e8-4758-a1ed-1b907783431e-kube-api-access-khpmd\") pod \"maas-api-key-cleanup-e2e-inf-b-39fecf-29889675-jshxf\" (UID: \"f5227ac7-04e8-4758-a1ed-1b907783431e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39fecf-29889675-jshxf" Oct 30 17:15:00.684499 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.684445 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-777l6\" (UniqueName: \"kubernetes.io/projected/79a79c82-3142-4035-a072-004b80dfe24d-kube-api-access-777l6\") pod \"maas-api-key-cleanup-e2e-worker-w5-86f50a-29889675-s24lj\" (UID: \"79a79c82-3142-4035-a072-004b80dfe24d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889675-s24lj" Oct 30 17:15:00.687162 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.687137 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5hmw\" (UniqueName: \"kubernetes.io/projected/ad604ad3-0fd0-4544-af4d-e5061bd561e3-kube-api-access-h5hmw\") pod \"maas-api-key-cleanup-e2e-api-b-b2c3fb-29889675-rckqx\" (UID: \"ad604ad3-0fd0-4544-af4d-e5061bd561e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b2c3fb-29889675-rckqx" Oct 30 17:15:00.687274 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.687195 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w25vf\" (UniqueName: \"kubernetes.io/projected/417f0b10-2a38-4757-8563-d47b75088b4d-kube-api-access-w25vf\") pod \"maas-api-key-cleanup-e2e-inf-a-728ad7-29889675-hgghk\" (UID: \"417f0b10-2a38-4757-8563-d47b75088b4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-728ad7-29889675-hgghk" Oct 30 17:15:00.689946 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.689762 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889675-lcm6b" Oct 30 17:15:00.699725 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.698545 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-728ad7-29889675-hgghk" Oct 30 17:15:00.708613 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.708587 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889675-s24lj" Oct 30 17:15:00.716534 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.716509 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39fecf-29889675-jshxf" Oct 30 17:15:00.829279 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:15:00.829235 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e7e3bf8_d7a4_4839_bcf4_5161e1f998c9.slice/crio-ac1652d784e7300fe4f2e76fef2905b01949c71aa7be9f55b8a03a6a76c6e6a7 WatchSource:0}: Error finding container ac1652d784e7300fe4f2e76fef2905b01949c71aa7be9f55b8a03a6a76c6e6a7: Status 404 returned error can't find the container with id ac1652d784e7300fe4f2e76fef2905b01949c71aa7be9f55b8a03a6a76c6e6a7 Oct 30 17:15:00.833423 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.832481 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889675-xhg9d"] Oct 30 17:15:00.939154 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.939075 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889675-fdxml" Oct 30 17:15:00.977331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:00.977141 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b2c3fb-29889675-rckqx" Oct 30 17:15:01.028544 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:01.028513 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889675-rbwqn"] Oct 30 17:15:01.067910 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:01.067864 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889675-crbqg"] Oct 30 17:15:01.110410 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:15:01.110367 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod171da860_d46a_420a_88a3_423d6b58c86b.slice/crio-342d9b3678f6e1526dbd59bfe60230f2279a8e44ac09a9ec4e4f165239b6d018 WatchSource:0}: Error finding container 342d9b3678f6e1526dbd59bfe60230f2279a8e44ac09a9ec4e4f165239b6d018: Status 404 returned error can't find the container with id 342d9b3678f6e1526dbd59bfe60230f2279a8e44ac09a9ec4e4f165239b6d018 Oct 30 17:15:01.111097 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:15:01.111061 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5802774a_44fd_4660_b00e_8d979654b196.slice/crio-cb4d989a8fcd17b30410ba4abb35e4f3f15c46ccab00dd50910575cd358bfa75 WatchSource:0}: Error finding container cb4d989a8fcd17b30410ba4abb35e4f3f15c46ccab00dd50910575cd358bfa75: Status 404 returned error can't find the container with id cb4d989a8fcd17b30410ba4abb35e4f3f15c46ccab00dd50910575cd358bfa75 Oct 30 17:15:01.233719 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:01.163855 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889675-dbh98"] Oct 30 17:15:01.242213 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:01.242184 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f42baf-29889675-2mg4p"] Oct 30 17:15:01.250706 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:01.250684 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39fecf-29889675-jshxf"] Oct 30 17:15:01.252129 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:15:01.252096 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc53d8512_95ac_4d60_aaa6_0d74b95e0cf8.slice/crio-3347383a5aac7e68a780853b84a6ff903a110f88cca50853357a5ba528a0ed2f WatchSource:0}: Error finding container 3347383a5aac7e68a780853b84a6ff903a110f88cca50853357a5ba528a0ed2f: Status 404 returned error can't find the container with id 3347383a5aac7e68a780853b84a6ff903a110f88cca50853357a5ba528a0ed2f Oct 30 17:15:01.345819 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:01.345785 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889675-fdxml"] Oct 30 17:15:01.370183 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:01.370150 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889675-lcm6b"] Oct 30 17:15:01.400943 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:01.400803 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-728ad7-29889675-hgghk"] Oct 30 17:15:01.410055 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:01.410017 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889675-s24lj"] Oct 30 17:15:01.413427 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:01.412880 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889675-crbqg" event={"ID":"5802774a-44fd-4660-b00e-8d979654b196","Type":"ContainerStarted","Data":"cb4d989a8fcd17b30410ba4abb35e4f3f15c46ccab00dd50910575cd358bfa75"} Oct 30 17:15:01.414971 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:01.414904 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889675-lcm6b" event={"ID":"5d21c30e-1815-436a-92b5-4cb61b88ea19","Type":"ContainerStarted","Data":"f83a7ac167fbb0b7bd4db1978f088e96dac54fe49b87686ae8dae544bfdd6202"} Oct 30 17:15:01.416455 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:01.416389 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f42baf-29889675-2mg4p" event={"ID":"c53d8512-95ac-4d60-aaa6-0d74b95e0cf8","Type":"ContainerStarted","Data":"3347383a5aac7e68a780853b84a6ff903a110f88cca50853357a5ba528a0ed2f"} Oct 30 17:15:01.418628 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:01.418246 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889675-xhg9d" event={"ID":"4e7e3bf8-d7a4-4839-bcf4-5161e1f998c9","Type":"ContainerStarted","Data":"ac1652d784e7300fe4f2e76fef2905b01949c71aa7be9f55b8a03a6a76c6e6a7"} Oct 30 17:15:01.418877 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:01.418856 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-728ad7-xvnc7"] Oct 30 17:15:01.425241 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:01.425112 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-728ad7-29889675-hgghk" event={"ID":"417f0b10-2a38-4757-8563-d47b75088b4d","Type":"ContainerStarted","Data":"2cd00dd25202a3484cc54c19ce2e8c05a273560267e5664cd646c7cba5ab983d"} Oct 30 17:15:01.425241 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:01.425142 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39fecf-29889675-jshxf" event={"ID":"f5227ac7-04e8-4758-a1ed-1b907783431e","Type":"ContainerStarted","Data":"8fea04062e25af688714d860cb9e7498a79907ddd755d11439d758853ceebeab"} Oct 30 17:15:01.425241 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:01.425166 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889675-dbh98" event={"ID":"78db2ae0-039b-4edc-8fdb-c4513041d366","Type":"ContainerStarted","Data":"7d64a3b39799decc2dd3ef9ebd7ff7aae72288209f5ca20362547df417a891dc"} Oct 30 17:15:01.425241 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:01.425190 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889675-fdxml" event={"ID":"1ea441b9-186a-4e43-b2c7-37052bd5dcd7","Type":"ContainerStarted","Data":"ce8cc35a4a56bee6fc44abf612feab68019a1d9b9f48e6c7df1ba359914e91a6"} Oct 30 17:15:01.425241 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:01.425211 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889675-rbwqn" event={"ID":"171da860-d46a-420a-88a3-423d6b58c86b","Type":"ContainerStarted","Data":"342d9b3678f6e1526dbd59bfe60230f2279a8e44ac09a9ec4e4f165239b6d018"} Oct 30 17:15:01.425241 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:01.425230 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-728ad7-xvnc7" Oct 30 17:15:01.441460 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:01.441371 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-728ad7-xvnc7"] Oct 30 17:15:01.492915 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:01.492883 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b2c3fb-29889675-rckqx"] Oct 30 17:15:01.520093 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:15:01.519278 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad604ad3_0fd0_4544_af4d_e5061bd561e3.slice/crio-49e0a28087326f9871783df694581479a27776b452bdd55df2656b76bbe589ce WatchSource:0}: Error finding container 49e0a28087326f9871783df694581479a27776b452bdd55df2656b76bbe589ce: Status 404 returned error can't find the container with id 49e0a28087326f9871783df694581479a27776b452bdd55df2656b76bbe589ce Oct 30 17:15:01.520093 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:01.519562 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:15:01.545109 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:01.545079 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/931cb7fe-faf6-4181-bee3-b3d1e3e61679-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-728ad7-xvnc7\" (UID: \"931cb7fe-faf6-4181-bee3-b3d1e3e61679\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-728ad7-xvnc7" Oct 30 17:15:01.545416 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:01.545380 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh7fp\" (UniqueName: \"kubernetes.io/projected/931cb7fe-faf6-4181-bee3-b3d1e3e61679-kube-api-access-zh7fp\") pod \"maas-api-revoke-keys-e2e-inf-a-728ad7-xvnc7\" (UID: \"931cb7fe-faf6-4181-bee3-b3d1e3e61679\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-728ad7-xvnc7" Oct 30 17:15:01.545596 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:01.545576 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/931cb7fe-faf6-4181-bee3-b3d1e3e61679-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-728ad7-xvnc7\" (UID: \"931cb7fe-faf6-4181-bee3-b3d1e3e61679\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-728ad7-xvnc7" Oct 30 17:15:01.646796 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:01.646768 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/931cb7fe-faf6-4181-bee3-b3d1e3e61679-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-728ad7-xvnc7\" (UID: \"931cb7fe-faf6-4181-bee3-b3d1e3e61679\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-728ad7-xvnc7" Oct 30 17:15:01.646924 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:01.646828 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh7fp\" (UniqueName: \"kubernetes.io/projected/931cb7fe-faf6-4181-bee3-b3d1e3e61679-kube-api-access-zh7fp\") pod \"maas-api-revoke-keys-e2e-inf-a-728ad7-xvnc7\" (UID: \"931cb7fe-faf6-4181-bee3-b3d1e3e61679\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-728ad7-xvnc7" Oct 30 17:15:01.646985 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:01.646953 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/931cb7fe-faf6-4181-bee3-b3d1e3e61679-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-728ad7-xvnc7\" (UID: \"931cb7fe-faf6-4181-bee3-b3d1e3e61679\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-728ad7-xvnc7" Oct 30 17:15:01.647607 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:01.647585 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/931cb7fe-faf6-4181-bee3-b3d1e3e61679-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-728ad7-xvnc7\" (UID: \"931cb7fe-faf6-4181-bee3-b3d1e3e61679\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-728ad7-xvnc7" Oct 30 17:15:01.649428 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:01.649407 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/931cb7fe-faf6-4181-bee3-b3d1e3e61679-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-728ad7-xvnc7\" (UID: \"931cb7fe-faf6-4181-bee3-b3d1e3e61679\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-728ad7-xvnc7" Oct 30 17:15:01.671868 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:01.671835 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh7fp\" (UniqueName: \"kubernetes.io/projected/931cb7fe-faf6-4181-bee3-b3d1e3e61679-kube-api-access-zh7fp\") pod \"maas-api-revoke-keys-e2e-inf-a-728ad7-xvnc7\" (UID: \"931cb7fe-faf6-4181-bee3-b3d1e3e61679\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-728ad7-xvnc7" Oct 30 17:15:01.934065 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:01.934040 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-728ad7-xvnc7" Oct 30 17:15:02.245105 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:15:02.245072 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod931cb7fe_faf6_4181_bee3_b3d1e3e61679.slice/crio-318c2b6f77c48d1b0648627194d001c90e3b9ade23082cf361315a69eae2582e WatchSource:0}: Error finding container 318c2b6f77c48d1b0648627194d001c90e3b9ade23082cf361315a69eae2582e: Status 404 returned error can't find the container with id 318c2b6f77c48d1b0648627194d001c90e3b9ade23082cf361315a69eae2582e Oct 30 17:15:02.247618 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:02.247590 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-728ad7-xvnc7"] Oct 30 17:15:02.435164 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:02.435132 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889675-lcm6b" event={"ID":"5d21c30e-1815-436a-92b5-4cb61b88ea19","Type":"ContainerStarted","Data":"4fd7d038bdbc41c03867752b50d5100d8be74beaeff1d777b5dd4e61b0958876"} Oct 30 17:15:02.436944 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:02.436913 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f42baf-29889675-2mg4p" event={"ID":"c53d8512-95ac-4d60-aaa6-0d74b95e0cf8","Type":"ContainerStarted","Data":"7e668c2652adf88762a3f9b4497469b452cd7bbf514377e96cdd16d25d4fca1d"} Oct 30 17:15:02.438476 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:02.438450 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b2c3fb-29889675-rckqx" event={"ID":"ad604ad3-0fd0-4544-af4d-e5061bd561e3","Type":"ContainerStarted","Data":"a5549e755a9f28f3c1e06381a9b304317e3495f11c7b020c2d3171bd88e15d2d"} Oct 30 17:15:02.438605 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:02.438498 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b2c3fb-29889675-rckqx" event={"ID":"ad604ad3-0fd0-4544-af4d-e5061bd561e3","Type":"ContainerStarted","Data":"49e0a28087326f9871783df694581479a27776b452bdd55df2656b76bbe589ce"} Oct 30 17:15:02.439991 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:02.439970 4522 generic.go:356] "Generic (PLEG): container finished" podID="4e7e3bf8-d7a4-4839-bcf4-5161e1f998c9" containerID="8fa0ba800fdaa20cea206006270561dcf1e822ab7efb1aeade1a14ecc0bf283b" exitCode=60 Oct 30 17:15:02.440109 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:02.440002 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889675-xhg9d" event={"ID":"4e7e3bf8-d7a4-4839-bcf4-5161e1f998c9","Type":"ContainerDied","Data":"8fa0ba800fdaa20cea206006270561dcf1e822ab7efb1aeade1a14ecc0bf283b"} Oct 30 17:15:02.440321 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:02.440289 4522 scope.go:122] "RemoveContainer" containerID="8fa0ba800fdaa20cea206006270561dcf1e822ab7efb1aeade1a14ecc0bf283b" Oct 30 17:15:02.441951 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:02.441920 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-728ad7-29889675-hgghk" event={"ID":"417f0b10-2a38-4757-8563-d47b75088b4d","Type":"ContainerStarted","Data":"7cd890c66cb3981a9ac0db067cff528580759573ca00ecd9941b9480638f55cc"} Oct 30 17:15:02.444944 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:02.444921 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39fecf-29889675-jshxf" event={"ID":"f5227ac7-04e8-4758-a1ed-1b907783431e","Type":"ContainerStarted","Data":"818bf54cc5abe8d8faf727bd5a77cb44694d9c23cc42ae2291b8f5ec3d3c711c"} Oct 30 17:15:02.446614 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:02.446590 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889675-dbh98" event={"ID":"78db2ae0-039b-4edc-8fdb-c4513041d366","Type":"ContainerStarted","Data":"9a9bb25d1146ed3bacaf51cd0c912ffc6242dcb575404b8e862fce36b605c3c5"} Oct 30 17:15:02.448733 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:02.448711 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889675-fdxml" event={"ID":"1ea441b9-186a-4e43-b2c7-37052bd5dcd7","Type":"ContainerStarted","Data":"3685f0fd3104829e21923c3eaa530915bd2e5d7e330b318e0222050d93dedf11"} Oct 30 17:15:02.450115 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:02.450091 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-728ad7-xvnc7" event={"ID":"931cb7fe-faf6-4181-bee3-b3d1e3e61679","Type":"ContainerStarted","Data":"318c2b6f77c48d1b0648627194d001c90e3b9ade23082cf361315a69eae2582e"} Oct 30 17:15:02.451979 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:02.451959 4522 generic.go:356] "Generic (PLEG): container finished" podID="171da860-d46a-420a-88a3-423d6b58c86b" containerID="fa1b358e843e77702fd19d4e93328357eb08e820c99e34baee77b860910ee80d" exitCode=60 Oct 30 17:15:02.452130 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:02.451985 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889675-rbwqn" event={"ID":"171da860-d46a-420a-88a3-423d6b58c86b","Type":"ContainerDied","Data":"fa1b358e843e77702fd19d4e93328357eb08e820c99e34baee77b860910ee80d"} Oct 30 17:15:02.452268 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:02.452245 4522 scope.go:122] "RemoveContainer" containerID="fa1b358e843e77702fd19d4e93328357eb08e820c99e34baee77b860910ee80d" Oct 30 17:15:02.464068 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:02.464040 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889675-s24lj" event={"ID":"79a79c82-3142-4035-a072-004b80dfe24d","Type":"ContainerStarted","Data":"2f561325bded613db2f7e48509109d42221dd7e401e1760f6cdf3f3cd9e3acef"} Oct 30 17:15:02.464186 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:02.464098 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889675-s24lj" event={"ID":"79a79c82-3142-4035-a072-004b80dfe24d","Type":"ContainerStarted","Data":"a4aa0ad532b576b241769fd8be93d7e0c88e37d3cb3bd7a4316e173f669df52f"} Oct 30 17:15:02.466455 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:02.466430 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889675-crbqg" event={"ID":"5802774a-44fd-4660-b00e-8d979654b196","Type":"ContainerStarted","Data":"1d42be34b47784e6ad750812395535aa9ae0ef7b2c80f49f35c0cdd99ff5ce16"} Oct 30 17:15:02.479282 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:02.479239 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889675-lcm6b" podStartSLOduration=2.479224641 podStartE2EDuration="2.479224641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:02.478391332 +0000 UTC m=+1676.479146234" watchObservedRunningTime="2026-10-30 17:15:02.479224641 +0000 UTC m=+1676.479979541" Oct 30 17:15:02.540819 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:02.540768 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-728ad7-29889675-hgghk" podStartSLOduration=2.540752172 podStartE2EDuration="2.540752172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:02.521244602 +0000 UTC m=+1676.521999524" watchObservedRunningTime="2026-10-30 17:15:02.540752172 +0000 UTC m=+1676.541507115" Oct 30 17:15:02.570384 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:02.570333 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f42baf-29889675-2mg4p" podStartSLOduration=2.570318961 podStartE2EDuration="2.570318961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:02.567504181 +0000 UTC m=+1676.568259083" watchObservedRunningTime="2026-10-30 17:15:02.570318961 +0000 UTC m=+1676.571073855" Oct 30 17:15:02.662238 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:02.662137 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39fecf-29889675-jshxf" podStartSLOduration=2.66211622 podStartE2EDuration="2.66211622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:02.65745113 +0000 UTC m=+1676.658206033" watchObservedRunningTime="2026-10-30 17:15:02.66211622 +0000 UTC m=+1676.662871125" Oct 30 17:15:02.707928 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:02.707878 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889675-fdxml" podStartSLOduration=2.70786309 podStartE2EDuration="2.70786309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:02.70638856 +0000 UTC m=+1676.707143462" watchObservedRunningTime="2026-10-30 17:15:02.70786309 +0000 UTC m=+1676.708617991" Oct 30 17:15:02.757704 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:02.757654 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b2c3fb-29889675-rckqx" podStartSLOduration=2.757638317 podStartE2EDuration="2.757638317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:02.753888938 +0000 UTC m=+1676.754643839" watchObservedRunningTime="2026-10-30 17:15:02.757638317 +0000 UTC m=+1676.758393219" Oct 30 17:15:02.811631 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:02.811572 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889675-dbh98" podStartSLOduration=2.811551748 podStartE2EDuration="2.811551748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:02.803140529 +0000 UTC m=+1676.803895430" watchObservedRunningTime="2026-10-30 17:15:02.811551748 +0000 UTC m=+1676.812306653" Oct 30 17:15:02.850611 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:02.850562 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889675-crbqg" podStartSLOduration=2.850547198 podStartE2EDuration="2.850547198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:02.850049245 +0000 UTC m=+1676.850804147" watchObservedRunningTime="2026-10-30 17:15:02.850547198 +0000 UTC m=+1676.851302099" Oct 30 17:15:02.919056 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:02.918965 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889675-s24lj" podStartSLOduration=2.918951635 podStartE2EDuration="2.918951635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:02.916149389 +0000 UTC m=+1676.916904290" watchObservedRunningTime="2026-10-30 17:15:02.918951635 +0000 UTC m=+1676.919706530" Oct 30 17:15:03.206492 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:03.206432 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-b2c3fb-5f877958dd-99vkh" podUID="42e57fa2-479b-4655-9ce4-c49ee4757cc4" containerName="payload-processing" containerID="cri-o://bed897987805e59b26129f538252a1ab876ddf29ab8d61933a84d09d7eaad5b0" gracePeriod=25 Oct 30 17:15:03.280410 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:03.280367 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b2c3fb-7698878959-chvds" podUID="90af553e-d49b-40d0-b182-8177a514e1eb" containerName="payload-pre-processing" containerID="cri-o://19aa7d1ddbbea0692fdd28ce84ae0399cc3513ede37d257eb2fe0b427915c624" gracePeriod=25 Oct 30 17:15:03.485515 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:03.485482 4522 generic.go:356] "Generic (PLEG): container finished" podID="931cb7fe-faf6-4181-bee3-b3d1e3e61679" containerID="fb8d4d3acf269d2b84411b262a8ec1655ae4632c28030b5cdc397f4bc75cff6a" exitCode=0 Oct 30 17:15:03.485722 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:03.485588 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-728ad7-xvnc7" event={"ID":"931cb7fe-faf6-4181-bee3-b3d1e3e61679","Type":"ContainerDied","Data":"fb8d4d3acf269d2b84411b262a8ec1655ae4632c28030b5cdc397f4bc75cff6a"} Oct 30 17:15:03.489393 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:03.489322 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889675-rbwqn" event={"ID":"171da860-d46a-420a-88a3-423d6b58c86b","Type":"ContainerStarted","Data":"381b0e1200fdb2b7b341c0a81ae472d6a4714b6a8fd8ca4049541cec20a440a1"} Oct 30 17:15:03.491483 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:03.491457 4522 generic.go:356] "Generic (PLEG): container finished" podID="79a79c82-3142-4035-a072-004b80dfe24d" containerID="2f561325bded613db2f7e48509109d42221dd7e401e1760f6cdf3f3cd9e3acef" exitCode=60 Oct 30 17:15:03.491637 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:03.491540 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889675-s24lj" event={"ID":"79a79c82-3142-4035-a072-004b80dfe24d","Type":"ContainerDied","Data":"2f561325bded613db2f7e48509109d42221dd7e401e1760f6cdf3f3cd9e3acef"} Oct 30 17:15:03.491962 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:03.491945 4522 scope.go:122] "RemoveContainer" containerID="2f561325bded613db2f7e48509109d42221dd7e401e1760f6cdf3f3cd9e3acef" Oct 30 17:15:03.505914 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:03.505891 4522 generic.go:356] "Generic (PLEG): container finished" podID="5802774a-44fd-4660-b00e-8d979654b196" containerID="1d42be34b47784e6ad750812395535aa9ae0ef7b2c80f49f35c0cdd99ff5ce16" exitCode=60 Oct 30 17:15:03.506056 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:03.506038 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889675-crbqg" event={"ID":"5802774a-44fd-4660-b00e-8d979654b196","Type":"ContainerDied","Data":"1d42be34b47784e6ad750812395535aa9ae0ef7b2c80f49f35c0cdd99ff5ce16"} Oct 30 17:15:03.506329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:03.506294 4522 scope.go:122] "RemoveContainer" containerID="1d42be34b47784e6ad750812395535aa9ae0ef7b2c80f49f35c0cdd99ff5ce16" Oct 30 17:15:03.512952 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:03.509585 4522 generic.go:356] "Generic (PLEG): container finished" podID="5d21c30e-1815-436a-92b5-4cb61b88ea19" containerID="4fd7d038bdbc41c03867752b50d5100d8be74beaeff1d777b5dd4e61b0958876" exitCode=60 Oct 30 17:15:03.512952 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:03.509691 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889675-lcm6b" event={"ID":"5d21c30e-1815-436a-92b5-4cb61b88ea19","Type":"ContainerDied","Data":"4fd7d038bdbc41c03867752b50d5100d8be74beaeff1d777b5dd4e61b0958876"} Oct 30 17:15:03.512952 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:03.510067 4522 scope.go:122] "RemoveContainer" containerID="4fd7d038bdbc41c03867752b50d5100d8be74beaeff1d777b5dd4e61b0958876" Oct 30 17:15:03.512952 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:03.512399 4522 generic.go:356] "Generic (PLEG): container finished" podID="90af553e-d49b-40d0-b182-8177a514e1eb" containerID="19aa7d1ddbbea0692fdd28ce84ae0399cc3513ede37d257eb2fe0b427915c624" exitCode=0 Oct 30 17:15:03.512952 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:03.512570 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b2c3fb-7698878959-chvds" event={"ID":"90af553e-d49b-40d0-b182-8177a514e1eb","Type":"ContainerDied","Data":"19aa7d1ddbbea0692fdd28ce84ae0399cc3513ede37d257eb2fe0b427915c624"} Oct 30 17:15:03.524366 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:03.519346 4522 generic.go:356] "Generic (PLEG): container finished" podID="c53d8512-95ac-4d60-aaa6-0d74b95e0cf8" containerID="7e668c2652adf88762a3f9b4497469b452cd7bbf514377e96cdd16d25d4fca1d" exitCode=60 Oct 30 17:15:03.524366 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:03.519773 4522 scope.go:122] "RemoveContainer" containerID="7e668c2652adf88762a3f9b4497469b452cd7bbf514377e96cdd16d25d4fca1d" Oct 30 17:15:03.524366 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:03.519818 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f42baf-29889675-2mg4p" event={"ID":"c53d8512-95ac-4d60-aaa6-0d74b95e0cf8","Type":"ContainerDied","Data":"7e668c2652adf88762a3f9b4497469b452cd7bbf514377e96cdd16d25d4fca1d"} Oct 30 17:15:03.547193 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:03.547167 4522 generic.go:356] "Generic (PLEG): container finished" podID="ad604ad3-0fd0-4544-af4d-e5061bd561e3" containerID="a5549e755a9f28f3c1e06381a9b304317e3495f11c7b020c2d3171bd88e15d2d" exitCode=60 Oct 30 17:15:03.547350 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:03.547285 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b2c3fb-29889675-rckqx" event={"ID":"ad604ad3-0fd0-4544-af4d-e5061bd561e3","Type":"ContainerDied","Data":"a5549e755a9f28f3c1e06381a9b304317e3495f11c7b020c2d3171bd88e15d2d"} Oct 30 17:15:03.547838 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:03.547816 4522 scope.go:122] "RemoveContainer" containerID="a5549e755a9f28f3c1e06381a9b304317e3495f11c7b020c2d3171bd88e15d2d" Oct 30 17:15:03.552920 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:03.552890 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889675-xhg9d" event={"ID":"4e7e3bf8-d7a4-4839-bcf4-5161e1f998c9","Type":"ContainerStarted","Data":"fb4f4c31a35039169f64e40d8e050702257f0faf60eb9c0be49dbb062bcb5339"} Oct 30 17:15:03.566467 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:03.562116 4522 generic.go:356] "Generic (PLEG): container finished" podID="417f0b10-2a38-4757-8563-d47b75088b4d" containerID="7cd890c66cb3981a9ac0db067cff528580759573ca00ecd9941b9480638f55cc" exitCode=60 Oct 30 17:15:03.566467 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:03.562197 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-728ad7-29889675-hgghk" event={"ID":"417f0b10-2a38-4757-8563-d47b75088b4d","Type":"ContainerDied","Data":"7cd890c66cb3981a9ac0db067cff528580759573ca00ecd9941b9480638f55cc"} Oct 30 17:15:03.566467 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:03.562731 4522 scope.go:122] "RemoveContainer" containerID="7cd890c66cb3981a9ac0db067cff528580759573ca00ecd9941b9480638f55cc" Oct 30 17:15:03.566467 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:03.565283 4522 generic.go:356] "Generic (PLEG): container finished" podID="f5227ac7-04e8-4758-a1ed-1b907783431e" containerID="818bf54cc5abe8d8faf727bd5a77cb44694d9c23cc42ae2291b8f5ec3d3c711c" exitCode=60 Oct 30 17:15:03.566467 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:03.565385 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39fecf-29889675-jshxf" event={"ID":"f5227ac7-04e8-4758-a1ed-1b907783431e","Type":"ContainerDied","Data":"818bf54cc5abe8d8faf727bd5a77cb44694d9c23cc42ae2291b8f5ec3d3c711c"} Oct 30 17:15:03.566467 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:03.566011 4522 scope.go:122] "RemoveContainer" containerID="818bf54cc5abe8d8faf727bd5a77cb44694d9c23cc42ae2291b8f5ec3d3c711c" Oct 30 17:15:03.568533 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:03.568511 4522 generic.go:356] "Generic (PLEG): container finished" podID="78db2ae0-039b-4edc-8fdb-c4513041d366" containerID="9a9bb25d1146ed3bacaf51cd0c912ffc6242dcb575404b8e862fce36b605c3c5" exitCode=60 Oct 30 17:15:03.568641 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:03.568600 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889675-dbh98" event={"ID":"78db2ae0-039b-4edc-8fdb-c4513041d366","Type":"ContainerDied","Data":"9a9bb25d1146ed3bacaf51cd0c912ffc6242dcb575404b8e862fce36b605c3c5"} Oct 30 17:15:03.568771 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:03.568756 4522 scope.go:122] "RemoveContainer" containerID="9a9bb25d1146ed3bacaf51cd0c912ffc6242dcb575404b8e862fce36b605c3c5" Oct 30 17:15:03.570106 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:03.570084 4522 generic.go:356] "Generic (PLEG): container finished" podID="1ea441b9-186a-4e43-b2c7-37052bd5dcd7" containerID="3685f0fd3104829e21923c3eaa530915bd2e5d7e330b318e0222050d93dedf11" exitCode=60 Oct 30 17:15:03.570217 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:03.570114 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889675-fdxml" event={"ID":"1ea441b9-186a-4e43-b2c7-37052bd5dcd7","Type":"ContainerDied","Data":"3685f0fd3104829e21923c3eaa530915bd2e5d7e330b318e0222050d93dedf11"} Oct 30 17:15:03.570376 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:03.570359 4522 scope.go:122] "RemoveContainer" containerID="3685f0fd3104829e21923c3eaa530915bd2e5d7e330b318e0222050d93dedf11" Oct 30 17:15:03.571994 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:03.571971 4522 generic.go:356] "Generic (PLEG): container finished" podID="42e57fa2-479b-4655-9ce4-c49ee4757cc4" containerID="bed897987805e59b26129f538252a1ab876ddf29ab8d61933a84d09d7eaad5b0" exitCode=0 Oct 30 17:15:03.572081 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:03.572023 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b2c3fb-5f877958dd-99vkh" event={"ID":"42e57fa2-479b-4655-9ce4-c49ee4757cc4","Type":"ContainerDied","Data":"bed897987805e59b26129f538252a1ab876ddf29ab8d61933a84d09d7eaad5b0"} Oct 30 17:15:03.672558 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:03.672507 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889675-rbwqn" podStartSLOduration=3.672493106 podStartE2EDuration="3.672493106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:03.669029362 +0000 UTC m=+1677.669784282" watchObservedRunningTime="2026-10-30 17:15:03.672493106 +0000 UTC m=+1677.673248003" Oct 30 17:15:03.809427 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:03.809362 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889675-xhg9d" podStartSLOduration=3.809339446 podStartE2EDuration="3.809339446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:03.800805381 +0000 UTC m=+1677.801560402" watchObservedRunningTime="2026-10-30 17:15:03.809339446 +0000 UTC m=+1677.810094348" Oct 30 17:15:03.979371 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:03.979301 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b2c3fb-5f877958dd-99vkh" Oct 30 17:15:03.986253 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:03.986215 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b2c3fb-7698878959-chvds" Oct 30 17:15:04.084165 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:04.084058 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90af553e-d49b-40d0-b182-8177a514e1eb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90af553e-d49b-40d0-b182-8177a514e1eb-plugins-config-volume\") pod \"90af553e-d49b-40d0-b182-8177a514e1eb\" (UID: \"90af553e-d49b-40d0-b182-8177a514e1eb\") " Oct 30 17:15:04.084349 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:04.084172 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42e57fa2-479b-4655-9ce4-c49ee4757cc4-kube-api-access-bqcvj\" (UniqueName: \"kubernetes.io/projected/42e57fa2-479b-4655-9ce4-c49ee4757cc4-kube-api-access-bqcvj\") pod \"42e57fa2-479b-4655-9ce4-c49ee4757cc4\" (UID: \"42e57fa2-479b-4655-9ce4-c49ee4757cc4\") " Oct 30 17:15:04.084442 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:04.084421 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42e57fa2-479b-4655-9ce4-c49ee4757cc4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42e57fa2-479b-4655-9ce4-c49ee4757cc4-plugins-config-volume\") pod \"42e57fa2-479b-4655-9ce4-c49ee4757cc4\" (UID: \"42e57fa2-479b-4655-9ce4-c49ee4757cc4\") " Oct 30 17:15:04.084533 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:04.084490 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90af553e-d49b-40d0-b182-8177a514e1eb-kube-api-access-gzd6g\" (UniqueName: \"kubernetes.io/projected/90af553e-d49b-40d0-b182-8177a514e1eb-kube-api-access-gzd6g\") pod \"90af553e-d49b-40d0-b182-8177a514e1eb\" (UID: \"90af553e-d49b-40d0-b182-8177a514e1eb\") " Oct 30 17:15:04.084670 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:04.084572 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90af553e-d49b-40d0-b182-8177a514e1eb-plugins-config-volume" pod "90af553e-d49b-40d0-b182-8177a514e1eb" (UID: "90af553e-d49b-40d0-b182-8177a514e1eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:04.085091 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:04.085048 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42e57fa2-479b-4655-9ce4-c49ee4757cc4-plugins-config-volume" pod "42e57fa2-479b-4655-9ce4-c49ee4757cc4" (UID: "42e57fa2-479b-4655-9ce4-c49ee4757cc4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:04.085298 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:04.085271 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42e57fa2-479b-4655-9ce4-c49ee4757cc4-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:15:04.085435 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:04.085300 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90af553e-d49b-40d0-b182-8177a514e1eb-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:15:04.194936 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:04.194897 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42e57fa2-479b-4655-9ce4-c49ee4757cc4-kube-api-access-bqcvj" pod "42e57fa2-479b-4655-9ce4-c49ee4757cc4" (UID: "42e57fa2-479b-4655-9ce4-c49ee4757cc4"). InnerVolumeSpecName "kube-api-access-bqcvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:04.195753 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:04.195716 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90af553e-d49b-40d0-b182-8177a514e1eb-kube-api-access-gzd6g" pod "90af553e-d49b-40d0-b182-8177a514e1eb" (UID: "90af553e-d49b-40d0-b182-8177a514e1eb"). InnerVolumeSpecName "kube-api-access-gzd6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:04.231825 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:04.231793 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5d67ca-6c5bcd5578-n5lgb"] Oct 30 17:15:04.232799 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:04.232780 4522 cpu_manager.go:414] "containerMap: removing container" podUID="42e57fa2-479b-4655-9ce4-c49ee4757cc4" containerName="payload-processing" Oct 30 17:15:04.232945 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:04.232923 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42e57fa2-479b-4655-9ce4-c49ee4757cc4" containerName="payload-processing" Oct 30 17:15:04.233025 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:04.232964 4522 cpu_manager.go:414] "containerMap: removing container" podUID="90af553e-d49b-40d0-b182-8177a514e1eb" containerName="payload-pre-processing" Oct 30 17:15:04.233025 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:04.232975 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90af553e-d49b-40d0-b182-8177a514e1eb" containerName="payload-pre-processing" Oct 30 17:15:04.233248 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:04.233233 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="90af553e-d49b-40d0-b182-8177a514e1eb" containerName="payload-pre-processing" Oct 30 17:15:04.233367 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:04.233271 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="42e57fa2-479b-4655-9ce4-c49ee4757cc4" containerName="payload-processing" Oct 30 17:15:04.240078 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:04.240055 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5d67ca-6c5bcd5578-n5lgb" Oct 30 17:15:04.277014 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:04.276976 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5d67ca-6c5bcd5578-n5lgb"] Oct 30 17:15:04.321620 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:04.301891 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjgkr\" (UniqueName: \"kubernetes.io/projected/75475e2a-de0d-429f-afc2-34714ef410a1-kube-api-access-jjgkr\") pod \"payload-processing-e2e-inf-b-5d67ca-6c5bcd5578-n5lgb\" (UID: \"75475e2a-de0d-429f-afc2-34714ef410a1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5d67ca-6c5bcd5578-n5lgb" Oct 30 17:15:04.321620 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:04.302026 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75475e2a-de0d-429f-afc2-34714ef410a1-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5d67ca-6c5bcd5578-n5lgb\" (UID: \"75475e2a-de0d-429f-afc2-34714ef410a1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5d67ca-6c5bcd5578-n5lgb" Oct 30 17:15:04.321620 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:04.302380 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqcvj\" (UniqueName: \"kubernetes.io/projected/42e57fa2-479b-4655-9ce4-c49ee4757cc4-kube-api-access-bqcvj\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:15:04.321620 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:04.302421 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzd6g\" (UniqueName: \"kubernetes.io/projected/90af553e-d49b-40d0-b182-8177a514e1eb-kube-api-access-gzd6g\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:15:04.405300 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:04.355577 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-5d67ca\"" Oct 30 17:15:04.405300 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:04.362670 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-5d67ca-dockercfg-qs6qg\"" Oct 30 17:15:04.412682 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:04.412644 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjgkr\" (UniqueName: \"kubernetes.io/projected/75475e2a-de0d-429f-afc2-34714ef410a1-kube-api-access-jjgkr\") pod \"payload-processing-e2e-inf-b-5d67ca-6c5bcd5578-n5lgb\" (UID: \"75475e2a-de0d-429f-afc2-34714ef410a1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5d67ca-6c5bcd5578-n5lgb" Oct 30 17:15:04.415168 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:04.415139 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75475e2a-de0d-429f-afc2-34714ef410a1-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5d67ca-6c5bcd5578-n5lgb\" (UID: \"75475e2a-de0d-429f-afc2-34714ef410a1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5d67ca-6c5bcd5578-n5lgb" Oct 30 17:15:04.416494 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:04.416461 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75475e2a-de0d-429f-afc2-34714ef410a1-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5d67ca-6c5bcd5578-n5lgb\" (UID: \"75475e2a-de0d-429f-afc2-34714ef410a1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5d67ca-6c5bcd5578-n5lgb" Oct 30 17:15:04.496162 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:04.495936 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjgkr\" (UniqueName: \"kubernetes.io/projected/75475e2a-de0d-429f-afc2-34714ef410a1-kube-api-access-jjgkr\") pod \"payload-processing-e2e-inf-b-5d67ca-6c5bcd5578-n5lgb\" (UID: \"75475e2a-de0d-429f-afc2-34714ef410a1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5d67ca-6c5bcd5578-n5lgb" Oct 30 17:15:04.577014 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:04.576984 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5d67ca-6c5bcd5578-n5lgb" Oct 30 17:15:04.583818 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:04.583789 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889675-s24lj" event={"ID":"79a79c82-3142-4035-a072-004b80dfe24d","Type":"ContainerStarted","Data":"0311a7d0d184580d6b21fa6bcfe15b844ae8912c29b62ad4c204d5db6e288e02"} Oct 30 17:15:04.586063 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:04.586028 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889675-crbqg" event={"ID":"5802774a-44fd-4660-b00e-8d979654b196","Type":"ContainerStarted","Data":"9c71566bdd00b201399a17ee2dc929e295cf9fa3061eab999f3ce6966ccb90d8"} Oct 30 17:15:04.588088 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:04.588062 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889675-lcm6b" event={"ID":"5d21c30e-1815-436a-92b5-4cb61b88ea19","Type":"ContainerStarted","Data":"0c2da43e32750ff2478ac633e9556d33ab432cdd72510c88a6b02072b7ece5df"} Oct 30 17:15:04.589644 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:04.589626 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b2c3fb-7698878959-chvds" Oct 30 17:15:04.589726 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:04.589660 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b2c3fb-7698878959-chvds" event={"ID":"90af553e-d49b-40d0-b182-8177a514e1eb","Type":"ContainerDied","Data":"88f375b05345e088b063c7e9d7ef1cd96d9dff07933468bba6c56f7a48c70d3d"} Oct 30 17:15:04.589726 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:04.589711 4522 scope.go:122] "RemoveContainer" containerID="19aa7d1ddbbea0692fdd28ce84ae0399cc3513ede37d257eb2fe0b427915c624" Oct 30 17:15:04.595663 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:04.595513 4522 generic.go:356] "Generic (PLEG): container finished" podID="4e7e3bf8-d7a4-4839-bcf4-5161e1f998c9" containerID="fb4f4c31a35039169f64e40d8e050702257f0faf60eb9c0be49dbb062bcb5339" exitCode=60 Oct 30 17:15:04.595663 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:04.595620 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889675-xhg9d" event={"ID":"4e7e3bf8-d7a4-4839-bcf4-5161e1f998c9","Type":"ContainerDied","Data":"fb4f4c31a35039169f64e40d8e050702257f0faf60eb9c0be49dbb062bcb5339"} Oct 30 17:15:04.596052 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:04.596030 4522 scope.go:122] "RemoveContainer" containerID="fb4f4c31a35039169f64e40d8e050702257f0faf60eb9c0be49dbb062bcb5339" Oct 30 17:15:04.596654 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:15:04.596447 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889675-xhg9d_odh-ai-gateway-infra(4e7e3bf8-d7a4-4839-bcf4-5161e1f998c9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889675-xhg9d" podUID="4e7e3bf8-d7a4-4839-bcf4-5161e1f998c9" Oct 30 17:15:04.600079 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:04.599472 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889675-dbh98" event={"ID":"78db2ae0-039b-4edc-8fdb-c4513041d366","Type":"ContainerStarted","Data":"048f5bb530e316f8e4a3b65b0b78c304a33dfe7d4f68445eb587753f964784cb"} Oct 30 17:15:04.611482 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:04.611453 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b2c3fb-5f877958dd-99vkh" Oct 30 17:15:04.618546 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:04.618513 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b2c3fb-5f877958dd-99vkh" event={"ID":"42e57fa2-479b-4655-9ce4-c49ee4757cc4","Type":"ContainerDied","Data":"a4728d490b3bca62de83699f05b3694b28570ac0630925da55f662cc1d4c77db"} Oct 30 17:15:04.621928 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:04.621900 4522 generic.go:356] "Generic (PLEG): container finished" podID="171da860-d46a-420a-88a3-423d6b58c86b" containerID="381b0e1200fdb2b7b341c0a81ae472d6a4714b6a8fd8ca4049541cec20a440a1" exitCode=60 Oct 30 17:15:04.623606 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:04.623588 4522 scope.go:122] "RemoveContainer" containerID="381b0e1200fdb2b7b341c0a81ae472d6a4714b6a8fd8ca4049541cec20a440a1" Oct 30 17:15:04.624504 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:15:04.624291 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889675-rbwqn_odh-ai-gateway-infra(171da860-d46a-420a-88a3-423d6b58c86b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889675-rbwqn" podUID="171da860-d46a-420a-88a3-423d6b58c86b" Oct 30 17:15:04.624652 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:04.622345 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889675-rbwqn" event={"ID":"171da860-d46a-420a-88a3-423d6b58c86b","Type":"ContainerDied","Data":"381b0e1200fdb2b7b341c0a81ae472d6a4714b6a8fd8ca4049541cec20a440a1"} Oct 30 17:15:04.666946 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:04.666815 4522 scope.go:122] "RemoveContainer" containerID="8fa0ba800fdaa20cea206006270561dcf1e822ab7efb1aeade1a14ecc0bf283b" Oct 30 17:15:04.700550 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:04.699677 4522 scope.go:122] "RemoveContainer" containerID="bed897987805e59b26129f538252a1ab876ddf29ab8d61933a84d09d7eaad5b0" Oct 30 17:15:04.730006 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:04.729970 4522 scope.go:122] "RemoveContainer" containerID="fa1b358e843e77702fd19d4e93328357eb08e820c99e34baee77b860910ee80d" Oct 30 17:15:04.901679 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:04.901654 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-728ad7-xvnc7" Oct 30 17:15:04.930828 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:04.930798 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/931cb7fe-faf6-4181-bee3-b3d1e3e61679-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/931cb7fe-faf6-4181-bee3-b3d1e3e61679-maas-api-xks-ca\") pod \"931cb7fe-faf6-4181-bee3-b3d1e3e61679\" (UID: \"931cb7fe-faf6-4181-bee3-b3d1e3e61679\") " Oct 30 17:15:04.930986 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:04.930859 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/931cb7fe-faf6-4181-bee3-b3d1e3e61679-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/931cb7fe-faf6-4181-bee3-b3d1e3e61679-maas-api-openshift-ca\") pod \"931cb7fe-faf6-4181-bee3-b3d1e3e61679\" (UID: \"931cb7fe-faf6-4181-bee3-b3d1e3e61679\") " Oct 30 17:15:04.930986 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:04.930892 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/931cb7fe-faf6-4181-bee3-b3d1e3e61679-kube-api-access-zh7fp\" (UniqueName: \"kubernetes.io/projected/931cb7fe-faf6-4181-bee3-b3d1e3e61679-kube-api-access-zh7fp\") pod \"931cb7fe-faf6-4181-bee3-b3d1e3e61679\" (UID: \"931cb7fe-faf6-4181-bee3-b3d1e3e61679\") " Oct 30 17:15:04.931362 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:04.931300 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/931cb7fe-faf6-4181-bee3-b3d1e3e61679-maas-api-openshift-ca" pod "931cb7fe-faf6-4181-bee3-b3d1e3e61679" (UID: "931cb7fe-faf6-4181-bee3-b3d1e3e61679"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:04.931597 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:04.931579 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/931cb7fe-faf6-4181-bee3-b3d1e3e61679-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:15:04.933065 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:04.933039 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/931cb7fe-faf6-4181-bee3-b3d1e3e61679-maas-api-xks-ca" pod "931cb7fe-faf6-4181-bee3-b3d1e3e61679" (UID: "931cb7fe-faf6-4181-bee3-b3d1e3e61679"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:15:04.933184 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:04.933166 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/931cb7fe-faf6-4181-bee3-b3d1e3e61679-kube-api-access-zh7fp" pod "931cb7fe-faf6-4181-bee3-b3d1e3e61679" (UID: "931cb7fe-faf6-4181-bee3-b3d1e3e61679"). InnerVolumeSpecName "kube-api-access-zh7fp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:04.944095 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:04.944066 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b2c3fb-5f877958dd-99vkh"] Oct 30 17:15:04.949143 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:04.949118 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b2c3fb-5f877958dd-99vkh"] Oct 30 17:15:04.953625 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:04.953601 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5d67ca-6c5bcd5578-n5lgb"] Oct 30 17:15:04.975453 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:15:04.975424 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75475e2a_de0d_429f_afc2_34714ef410a1.slice/crio-57e908bf7bb4bd3bcac9165520307a1a5a88893c9afbe54a0b63442af4bce08f WatchSource:0}: Error finding container 57e908bf7bb4bd3bcac9165520307a1a5a88893c9afbe54a0b63442af4bce08f: Status 404 returned error can't find the container with id 57e908bf7bb4bd3bcac9165520307a1a5a88893c9afbe54a0b63442af4bce08f Oct 30 17:15:04.985192 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:04.985170 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b2c3fb-7698878959-chvds"] Oct 30 17:15:04.989654 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:04.989631 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b2c3fb-7698878959-chvds"] Oct 30 17:15:05.032930 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:05.032902 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/931cb7fe-faf6-4181-bee3-b3d1e3e61679-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:15:05.032930 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:05.032929 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh7fp\" (UniqueName: \"kubernetes.io/projected/931cb7fe-faf6-4181-bee3-b3d1e3e61679-kube-api-access-zh7fp\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:15:05.195594 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:05.194530 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c82666-7867d55d87-89bwv" Oct 30 17:15:05.196525 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:05.196504 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-c82666-6ccbf4bbb4-tw2zk" Oct 30 17:15:05.636817 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:05.636781 4522 generic.go:356] "Generic (PLEG): container finished" podID="c53d8512-95ac-4d60-aaa6-0d74b95e0cf8" containerID="056c84715eb606c8c3fb26ad219701f1a64f638fdc3c0ea0fd630a297f650237" exitCode=60 Oct 30 17:15:05.637206 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:05.636887 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f42baf-29889675-2mg4p" event={"ID":"c53d8512-95ac-4d60-aaa6-0d74b95e0cf8","Type":"ContainerDied","Data":"056c84715eb606c8c3fb26ad219701f1a64f638fdc3c0ea0fd630a297f650237"} Oct 30 17:15:05.637206 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:05.636946 4522 scope.go:122] "RemoveContainer" containerID="7e668c2652adf88762a3f9b4497469b452cd7bbf514377e96cdd16d25d4fca1d" Oct 30 17:15:05.637510 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:05.637489 4522 scope.go:122] "RemoveContainer" containerID="056c84715eb606c8c3fb26ad219701f1a64f638fdc3c0ea0fd630a297f650237" Oct 30 17:15:05.637965 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:15:05.637934 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-f42baf-29889675-2mg4p_odh-ai-gateway-infra(c53d8512-95ac-4d60-aaa6-0d74b95e0cf8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f42baf-29889675-2mg4p" podUID="c53d8512-95ac-4d60-aaa6-0d74b95e0cf8" Oct 30 17:15:05.639616 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:05.639596 4522 generic.go:356] "Generic (PLEG): container finished" podID="ad604ad3-0fd0-4544-af4d-e5061bd561e3" containerID="7f730ede20c0cf383b713f150c509035cbf7296c4b8b715a32d7f9a3b89f5b8f" exitCode=60 Oct 30 17:15:05.639694 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:05.639679 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b2c3fb-29889675-rckqx" event={"ID":"ad604ad3-0fd0-4544-af4d-e5061bd561e3","Type":"ContainerDied","Data":"7f730ede20c0cf383b713f150c509035cbf7296c4b8b715a32d7f9a3b89f5b8f"} Oct 30 17:15:05.640039 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:05.640021 4522 scope.go:122] "RemoveContainer" containerID="7f730ede20c0cf383b713f150c509035cbf7296c4b8b715a32d7f9a3b89f5b8f" Oct 30 17:15:05.640416 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:15:05.640382 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-b2c3fb-29889675-rckqx_odh-ai-gateway-infra(ad604ad3-0fd0-4544-af4d-e5061bd561e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b2c3fb-29889675-rckqx" podUID="ad604ad3-0fd0-4544-af4d-e5061bd561e3" Oct 30 17:15:05.641998 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:05.641977 4522 scope.go:122] "RemoveContainer" containerID="fb4f4c31a35039169f64e40d8e050702257f0faf60eb9c0be49dbb062bcb5339" Oct 30 17:15:05.642651 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:15:05.642622 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889675-xhg9d_odh-ai-gateway-infra(4e7e3bf8-d7a4-4839-bcf4-5161e1f998c9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889675-xhg9d" podUID="4e7e3bf8-d7a4-4839-bcf4-5161e1f998c9" Oct 30 17:15:05.643878 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:05.643855 4522 generic.go:356] "Generic (PLEG): container finished" podID="417f0b10-2a38-4757-8563-d47b75088b4d" containerID="d5be21be86435e2d970f039039f886115377dbf3ac717c5da031e4a2d8266325" exitCode=60 Oct 30 17:15:05.643972 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:05.643936 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-728ad7-29889675-hgghk" event={"ID":"417f0b10-2a38-4757-8563-d47b75088b4d","Type":"ContainerDied","Data":"d5be21be86435e2d970f039039f886115377dbf3ac717c5da031e4a2d8266325"} Oct 30 17:15:05.644179 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:05.644162 4522 scope.go:122] "RemoveContainer" containerID="d5be21be86435e2d970f039039f886115377dbf3ac717c5da031e4a2d8266325" Oct 30 17:15:05.644587 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:15:05.644563 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-728ad7-29889675-hgghk_odh-ai-gateway-infra(417f0b10-2a38-4757-8563-d47b75088b4d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-728ad7-29889675-hgghk" podUID="417f0b10-2a38-4757-8563-d47b75088b4d" Oct 30 17:15:05.646433 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:05.646409 4522 generic.go:356] "Generic (PLEG): container finished" podID="f5227ac7-04e8-4758-a1ed-1b907783431e" containerID="fa203612fef197fc913f066787e63253f4d6e290ce374538704d16eeee9e7d9e" exitCode=60 Oct 30 17:15:05.646550 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:05.646500 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39fecf-29889675-jshxf" event={"ID":"f5227ac7-04e8-4758-a1ed-1b907783431e","Type":"ContainerDied","Data":"fa203612fef197fc913f066787e63253f4d6e290ce374538704d16eeee9e7d9e"} Oct 30 17:15:05.646914 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:05.646896 4522 scope.go:122] "RemoveContainer" containerID="fa203612fef197fc913f066787e63253f4d6e290ce374538704d16eeee9e7d9e" Oct 30 17:15:05.647139 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:15:05.647121 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-39fecf-29889675-jshxf_odh-ai-gateway-infra(f5227ac7-04e8-4758-a1ed-1b907783431e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39fecf-29889675-jshxf" podUID="f5227ac7-04e8-4758-a1ed-1b907783431e" Oct 30 17:15:05.649377 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:05.649355 4522 generic.go:356] "Generic (PLEG): container finished" podID="78db2ae0-039b-4edc-8fdb-c4513041d366" containerID="048f5bb530e316f8e4a3b65b0b78c304a33dfe7d4f68445eb587753f964784cb" exitCode=60 Oct 30 17:15:05.649478 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:05.649427 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889675-dbh98" event={"ID":"78db2ae0-039b-4edc-8fdb-c4513041d366","Type":"ContainerDied","Data":"048f5bb530e316f8e4a3b65b0b78c304a33dfe7d4f68445eb587753f964784cb"} Oct 30 17:15:05.649624 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:05.649608 4522 scope.go:122] "RemoveContainer" containerID="048f5bb530e316f8e4a3b65b0b78c304a33dfe7d4f68445eb587753f964784cb" Oct 30 17:15:05.649857 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:15:05.649838 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889675-dbh98_odh-ai-gateway-infra(78db2ae0-039b-4edc-8fdb-c4513041d366)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889675-dbh98" podUID="78db2ae0-039b-4edc-8fdb-c4513041d366" Oct 30 17:15:05.651661 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:05.651644 4522 generic.go:356] "Generic (PLEG): container finished" podID="1ea441b9-186a-4e43-b2c7-37052bd5dcd7" containerID="eb81caca77b39e990e61f509f2c027f1419eaeabcedcd0e60ead3c974fbfa97e" exitCode=60 Oct 30 17:15:05.651747 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:05.651705 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889675-fdxml" event={"ID":"1ea441b9-186a-4e43-b2c7-37052bd5dcd7","Type":"ContainerDied","Data":"eb81caca77b39e990e61f509f2c027f1419eaeabcedcd0e60ead3c974fbfa97e"} Oct 30 17:15:05.651948 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:05.651929 4522 scope.go:122] "RemoveContainer" containerID="eb81caca77b39e990e61f509f2c027f1419eaeabcedcd0e60ead3c974fbfa97e" Oct 30 17:15:05.652187 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:15:05.652164 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889675-fdxml_odh-ai-gateway-infra(1ea441b9-186a-4e43-b2c7-37052bd5dcd7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889675-fdxml" podUID="1ea441b9-186a-4e43-b2c7-37052bd5dcd7" Oct 30 17:15:05.654651 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:05.654629 4522 scope.go:122] "RemoveContainer" containerID="a5549e755a9f28f3c1e06381a9b304317e3495f11c7b020c2d3171bd88e15d2d" Oct 30 17:15:05.654770 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:05.654755 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-728ad7-xvnc7" Oct 30 17:15:05.654839 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:05.654774 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-728ad7-xvnc7" event={"ID":"931cb7fe-faf6-4181-bee3-b3d1e3e61679","Type":"ContainerDied","Data":"318c2b6f77c48d1b0648627194d001c90e3b9ade23082cf361315a69eae2582e"} Oct 30 17:15:05.654839 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:05.654811 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="318c2b6f77c48d1b0648627194d001c90e3b9ade23082cf361315a69eae2582e" Oct 30 17:15:05.657277 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:05.657237 4522 scope.go:122] "RemoveContainer" containerID="381b0e1200fdb2b7b341c0a81ae472d6a4714b6a8fd8ca4049541cec20a440a1" Oct 30 17:15:05.657546 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:15:05.657523 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889675-rbwqn_odh-ai-gateway-infra(171da860-d46a-420a-88a3-423d6b58c86b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889675-rbwqn" podUID="171da860-d46a-420a-88a3-423d6b58c86b" Oct 30 17:15:05.659123 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:05.659100 4522 generic.go:356] "Generic (PLEG): container finished" podID="79a79c82-3142-4035-a072-004b80dfe24d" containerID="0311a7d0d184580d6b21fa6bcfe15b844ae8912c29b62ad4c204d5db6e288e02" exitCode=60 Oct 30 17:15:05.659213 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:05.659177 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889675-s24lj" event={"ID":"79a79c82-3142-4035-a072-004b80dfe24d","Type":"ContainerDied","Data":"0311a7d0d184580d6b21fa6bcfe15b844ae8912c29b62ad4c204d5db6e288e02"} Oct 30 17:15:05.659436 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:05.659419 4522 scope.go:122] "RemoveContainer" containerID="0311a7d0d184580d6b21fa6bcfe15b844ae8912c29b62ad4c204d5db6e288e02" Oct 30 17:15:05.659694 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:15:05.659673 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-86f50a-29889675-s24lj_odh-ai-gateway-infra(79a79c82-3142-4035-a072-004b80dfe24d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889675-s24lj" podUID="79a79c82-3142-4035-a072-004b80dfe24d" Oct 30 17:15:05.661973 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:05.661915 4522 generic.go:356] "Generic (PLEG): container finished" podID="5802774a-44fd-4660-b00e-8d979654b196" containerID="9c71566bdd00b201399a17ee2dc929e295cf9fa3061eab999f3ce6966ccb90d8" exitCode=60 Oct 30 17:15:05.662352 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:05.662301 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889675-crbqg" event={"ID":"5802774a-44fd-4660-b00e-8d979654b196","Type":"ContainerDied","Data":"9c71566bdd00b201399a17ee2dc929e295cf9fa3061eab999f3ce6966ccb90d8"} Oct 30 17:15:05.662756 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:05.662734 4522 scope.go:122] "RemoveContainer" containerID="9c71566bdd00b201399a17ee2dc929e295cf9fa3061eab999f3ce6966ccb90d8" Oct 30 17:15:05.663061 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:15:05.663033 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889675-crbqg_odh-ai-gateway-infra(5802774a-44fd-4660-b00e-8d979654b196)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889675-crbqg" podUID="5802774a-44fd-4660-b00e-8d979654b196" Oct 30 17:15:05.665846 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:05.665828 4522 generic.go:356] "Generic (PLEG): container finished" podID="5d21c30e-1815-436a-92b5-4cb61b88ea19" containerID="0c2da43e32750ff2478ac633e9556d33ab432cdd72510c88a6b02072b7ece5df" exitCode=60 Oct 30 17:15:05.665938 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:05.665915 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889675-lcm6b" event={"ID":"5d21c30e-1815-436a-92b5-4cb61b88ea19","Type":"ContainerDied","Data":"0c2da43e32750ff2478ac633e9556d33ab432cdd72510c88a6b02072b7ece5df"} Oct 30 17:15:05.666117 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:05.666103 4522 scope.go:122] "RemoveContainer" containerID="0c2da43e32750ff2478ac633e9556d33ab432cdd72510c88a6b02072b7ece5df" Oct 30 17:15:05.666434 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:15:05.666410 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889675-lcm6b_odh-ai-gateway-infra(5d21c30e-1815-436a-92b5-4cb61b88ea19)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889675-lcm6b" podUID="5d21c30e-1815-436a-92b5-4cb61b88ea19" Oct 30 17:15:05.670354 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:05.670096 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5d67ca-6c5bcd5578-n5lgb" event={"ID":"75475e2a-de0d-429f-afc2-34714ef410a1","Type":"ContainerStarted","Data":"104af81e4d837c87be3d47899e54a5a4cb672cb1a368f644e7ddf2c66027feed"} Oct 30 17:15:05.670354 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:05.670145 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5d67ca-6c5bcd5578-n5lgb" event={"ID":"75475e2a-de0d-429f-afc2-34714ef410a1","Type":"ContainerStarted","Data":"57e908bf7bb4bd3bcac9165520307a1a5a88893c9afbe54a0b63442af4bce08f"} Oct 30 17:15:05.671092 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:05.671072 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-5d67ca-6c5bcd5578-n5lgb" Oct 30 17:15:05.675090 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:05.675071 4522 scope.go:122] "RemoveContainer" containerID="7cd890c66cb3981a9ac0db067cff528580759573ca00ecd9941b9480638f55cc" Oct 30 17:15:05.697124 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:05.696969 4522 scope.go:122] "RemoveContainer" containerID="818bf54cc5abe8d8faf727bd5a77cb44694d9c23cc42ae2291b8f5ec3d3c711c" Oct 30 17:15:05.713826 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:05.713803 4522 scope.go:122] "RemoveContainer" containerID="9a9bb25d1146ed3bacaf51cd0c912ffc6242dcb575404b8e862fce36b605c3c5" Oct 30 17:15:05.729576 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:05.729553 4522 scope.go:122] "RemoveContainer" containerID="3685f0fd3104829e21923c3eaa530915bd2e5d7e330b318e0222050d93dedf11" Oct 30 17:15:05.744418 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:05.744391 4522 scope.go:122] "RemoveContainer" containerID="2f561325bded613db2f7e48509109d42221dd7e401e1760f6cdf3f3cd9e3acef" Oct 30 17:15:05.758732 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:05.758710 4522 scope.go:122] "RemoveContainer" containerID="1d42be34b47784e6ad750812395535aa9ae0ef7b2c80f49f35c0cdd99ff5ce16" Oct 30 17:15:05.773597 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:05.773572 4522 scope.go:122] "RemoveContainer" containerID="4fd7d038bdbc41c03867752b50d5100d8be74beaeff1d777b5dd4e61b0958876" Oct 30 17:15:05.813079 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:05.813023 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-5d67ca-6c5bcd5578-n5lgb" podStartSLOduration=1.813009198 podStartE2EDuration="1.813009198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:05.809487954 +0000 UTC m=+1679.810242858" watchObservedRunningTime="2026-10-30 17:15:05.813009198 +0000 UTC m=+1679.813764098" Oct 30 17:15:06.162032 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:06.161957 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42e57fa2-479b-4655-9ce4-c49ee4757cc4" path="/var/lib/kubelet/pods/42e57fa2-479b-4655-9ce4-c49ee4757cc4/volumes" Oct 30 17:15:06.162475 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:06.162454 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90af553e-d49b-40d0-b182-8177a514e1eb" path="/var/lib/kubelet/pods/90af553e-d49b-40d0-b182-8177a514e1eb/volumes" Oct 30 17:15:06.684907 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:06.684881 4522 scope.go:122] "RemoveContainer" containerID="056c84715eb606c8c3fb26ad219701f1a64f638fdc3c0ea0fd630a297f650237" Oct 30 17:15:06.685807 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:15:06.685777 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-f42baf-29889675-2mg4p_odh-ai-gateway-infra(c53d8512-95ac-4d60-aaa6-0d74b95e0cf8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f42baf-29889675-2mg4p" podUID="c53d8512-95ac-4d60-aaa6-0d74b95e0cf8" Oct 30 17:15:06.691360 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:06.688399 4522 scope.go:122] "RemoveContainer" containerID="7f730ede20c0cf383b713f150c509035cbf7296c4b8b715a32d7f9a3b89f5b8f" Oct 30 17:15:06.691360 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:15:06.688910 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-b2c3fb-29889675-rckqx_odh-ai-gateway-infra(ad604ad3-0fd0-4544-af4d-e5061bd561e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b2c3fb-29889675-rckqx" podUID="ad604ad3-0fd0-4544-af4d-e5061bd561e3" Oct 30 17:15:06.692391 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:06.692346 4522 scope.go:122] "RemoveContainer" containerID="d5be21be86435e2d970f039039f886115377dbf3ac717c5da031e4a2d8266325" Oct 30 17:15:06.692714 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:15:06.692689 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-728ad7-29889675-hgghk_odh-ai-gateway-infra(417f0b10-2a38-4757-8563-d47b75088b4d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-728ad7-29889675-hgghk" podUID="417f0b10-2a38-4757-8563-d47b75088b4d" Oct 30 17:15:06.696084 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:06.696060 4522 scope.go:122] "RemoveContainer" containerID="fa203612fef197fc913f066787e63253f4d6e290ce374538704d16eeee9e7d9e" Oct 30 17:15:06.696495 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:15:06.696455 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-39fecf-29889675-jshxf_odh-ai-gateway-infra(f5227ac7-04e8-4758-a1ed-1b907783431e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39fecf-29889675-jshxf" podUID="f5227ac7-04e8-4758-a1ed-1b907783431e" Oct 30 17:15:06.698431 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:06.698411 4522 scope.go:122] "RemoveContainer" containerID="048f5bb530e316f8e4a3b65b0b78c304a33dfe7d4f68445eb587753f964784cb" Oct 30 17:15:06.698897 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:15:06.698869 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889675-dbh98_odh-ai-gateway-infra(78db2ae0-039b-4edc-8fdb-c4513041d366)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889675-dbh98" podUID="78db2ae0-039b-4edc-8fdb-c4513041d366" Oct 30 17:15:06.704833 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:06.704808 4522 scope.go:122] "RemoveContainer" containerID="eb81caca77b39e990e61f509f2c027f1419eaeabcedcd0e60ead3c974fbfa97e" Oct 30 17:15:06.705274 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:15:06.705250 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889675-fdxml_odh-ai-gateway-infra(1ea441b9-186a-4e43-b2c7-37052bd5dcd7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889675-fdxml" podUID="1ea441b9-186a-4e43-b2c7-37052bd5dcd7" Oct 30 17:15:06.707251 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:06.707229 4522 scope.go:122] "RemoveContainer" containerID="0311a7d0d184580d6b21fa6bcfe15b844ae8912c29b62ad4c204d5db6e288e02" Oct 30 17:15:06.707881 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:15:06.707856 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-86f50a-29889675-s24lj_odh-ai-gateway-infra(79a79c82-3142-4035-a072-004b80dfe24d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889675-s24lj" podUID="79a79c82-3142-4035-a072-004b80dfe24d" Oct 30 17:15:06.709925 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:06.709906 4522 scope.go:122] "RemoveContainer" containerID="9c71566bdd00b201399a17ee2dc929e295cf9fa3061eab999f3ce6966ccb90d8" Oct 30 17:15:06.710533 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:15:06.710509 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889675-crbqg_odh-ai-gateway-infra(5802774a-44fd-4660-b00e-8d979654b196)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889675-crbqg" podUID="5802774a-44fd-4660-b00e-8d979654b196" Oct 30 17:15:06.712534 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:06.712513 4522 scope.go:122] "RemoveContainer" containerID="0c2da43e32750ff2478ac633e9556d33ab432cdd72510c88a6b02072b7ece5df" Oct 30 17:15:06.713043 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:15:06.713020 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889675-lcm6b_odh-ai-gateway-infra(5d21c30e-1815-436a-92b5-4cb61b88ea19)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889675-lcm6b" podUID="5d21c30e-1815-436a-92b5-4cb61b88ea19" Oct 30 17:15:08.270016 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:08.269983 4522 scope.go:122] "RemoveContainer" containerID="955e9113a96933e27a89161b0ddae216db049944af6d8c48c6066a8f63b968a4" Oct 30 17:15:11.593401 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:11.593362 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-728ad7-66cb7bbbd6-dwrgr"] Oct 30 17:15:11.631132 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:11.631100 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-728ad7-7b4857f67-lf57w"] Oct 30 17:15:11.888160 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:11.888041 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv"] Oct 30 17:15:11.889077 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:11.889058 4522 cpu_manager.go:414] "containerMap: removing container" podUID="931cb7fe-faf6-4181-bee3-b3d1e3e61679" containerName="revoke-keys" Oct 30 17:15:11.889077 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:11.889078 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="931cb7fe-faf6-4181-bee3-b3d1e3e61679" containerName="revoke-keys" Oct 30 17:15:11.889236 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:11.889226 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="931cb7fe-faf6-4181-bee3-b3d1e3e61679" containerName="revoke-keys" Oct 30 17:15:11.896282 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:11.896255 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv" Oct 30 17:15:11.913588 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:11.913560 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-f42baf-openshift-default-dockercfg-njzvm\"" Oct 30 17:15:11.916197 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:11.916168 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv"] Oct 30 17:15:12.055336 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:12.055281 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d6e738a-984b-463c-acbd-e2b38e345f6d-istio-envoy\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv\" (UID: \"5d6e738a-984b-463c-acbd-e2b38e345f6d\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv" Oct 30 17:15:12.055523 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:12.055338 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d6e738a-984b-463c-acbd-e2b38e345f6d-workload-certs\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv\" (UID: \"5d6e738a-984b-463c-acbd-e2b38e345f6d\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv" Oct 30 17:15:12.055523 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:12.055457 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d6e738a-984b-463c-acbd-e2b38e345f6d-istio-podinfo\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv\" (UID: \"5d6e738a-984b-463c-acbd-e2b38e345f6d\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv" Oct 30 17:15:12.055523 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:12.055502 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d6e738a-984b-463c-acbd-e2b38e345f6d-istio-token\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv\" (UID: \"5d6e738a-984b-463c-acbd-e2b38e345f6d\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv" Oct 30 17:15:12.055751 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:12.055725 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phcz8\" (UniqueName: \"kubernetes.io/projected/5d6e738a-984b-463c-acbd-e2b38e345f6d-kube-api-access-phcz8\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv\" (UID: \"5d6e738a-984b-463c-acbd-e2b38e345f6d\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv" Oct 30 17:15:12.055886 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:12.055867 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6e738a-984b-463c-acbd-e2b38e345f6d-workload-socket\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv\" (UID: \"5d6e738a-984b-463c-acbd-e2b38e345f6d\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv" Oct 30 17:15:12.055967 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:12.055954 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d6e738a-984b-463c-acbd-e2b38e345f6d-istio-data\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv\" (UID: \"5d6e738a-984b-463c-acbd-e2b38e345f6d\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv" Oct 30 17:15:12.056028 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:12.055974 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d6e738a-984b-463c-acbd-e2b38e345f6d-istiod-ca-cert\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv\" (UID: \"5d6e738a-984b-463c-acbd-e2b38e345f6d\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv" Oct 30 17:15:12.056028 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:12.055992 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6e738a-984b-463c-acbd-e2b38e345f6d-credential-socket\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv\" (UID: \"5d6e738a-984b-463c-acbd-e2b38e345f6d\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv" Oct 30 17:15:12.157272 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:12.157185 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d6e738a-984b-463c-acbd-e2b38e345f6d-istio-envoy\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv\" (UID: \"5d6e738a-984b-463c-acbd-e2b38e345f6d\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv" Oct 30 17:15:12.157272 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:12.157233 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d6e738a-984b-463c-acbd-e2b38e345f6d-workload-certs\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv\" (UID: \"5d6e738a-984b-463c-acbd-e2b38e345f6d\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv" Oct 30 17:15:12.157547 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:12.157340 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d6e738a-984b-463c-acbd-e2b38e345f6d-istio-podinfo\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv\" (UID: \"5d6e738a-984b-463c-acbd-e2b38e345f6d\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv" Oct 30 17:15:12.157547 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:12.157390 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d6e738a-984b-463c-acbd-e2b38e345f6d-istio-token\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv\" (UID: \"5d6e738a-984b-463c-acbd-e2b38e345f6d\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv" Oct 30 17:15:12.157671 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:12.157553 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phcz8\" (UniqueName: \"kubernetes.io/projected/5d6e738a-984b-463c-acbd-e2b38e345f6d-kube-api-access-phcz8\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv\" (UID: \"5d6e738a-984b-463c-acbd-e2b38e345f6d\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv" Oct 30 17:15:12.157671 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:12.157656 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6e738a-984b-463c-acbd-e2b38e345f6d-workload-socket\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv\" (UID: \"5d6e738a-984b-463c-acbd-e2b38e345f6d\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv" Oct 30 17:15:12.157781 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:12.157697 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d6e738a-984b-463c-acbd-e2b38e345f6d-workload-certs\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv\" (UID: \"5d6e738a-984b-463c-acbd-e2b38e345f6d\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv" Oct 30 17:15:12.157781 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:12.157731 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d6e738a-984b-463c-acbd-e2b38e345f6d-istio-data\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv\" (UID: \"5d6e738a-984b-463c-acbd-e2b38e345f6d\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv" Oct 30 17:15:12.157781 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:12.157755 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d6e738a-984b-463c-acbd-e2b38e345f6d-istiod-ca-cert\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv\" (UID: \"5d6e738a-984b-463c-acbd-e2b38e345f6d\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv" Oct 30 17:15:12.157781 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:12.157779 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6e738a-984b-463c-acbd-e2b38e345f6d-credential-socket\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv\" (UID: \"5d6e738a-984b-463c-acbd-e2b38e345f6d\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv" Oct 30 17:15:12.158051 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:12.158003 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6e738a-984b-463c-acbd-e2b38e345f6d-workload-socket\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv\" (UID: \"5d6e738a-984b-463c-acbd-e2b38e345f6d\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv" Oct 30 17:15:12.158129 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:12.158083 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d6e738a-984b-463c-acbd-e2b38e345f6d-istio-data\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv\" (UID: \"5d6e738a-984b-463c-acbd-e2b38e345f6d\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv" Oct 30 17:15:12.158188 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:12.158170 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6e738a-984b-463c-acbd-e2b38e345f6d-credential-socket\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv\" (UID: \"5d6e738a-984b-463c-acbd-e2b38e345f6d\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv" Oct 30 17:15:12.158476 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:12.158454 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d6e738a-984b-463c-acbd-e2b38e345f6d-istiod-ca-cert\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv\" (UID: \"5d6e738a-984b-463c-acbd-e2b38e345f6d\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv" Oct 30 17:15:12.159635 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:12.159613 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d6e738a-984b-463c-acbd-e2b38e345f6d-istio-envoy\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv\" (UID: \"5d6e738a-984b-463c-acbd-e2b38e345f6d\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv" Oct 30 17:15:12.160083 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:12.160063 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d6e738a-984b-463c-acbd-e2b38e345f6d-istio-podinfo\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv\" (UID: \"5d6e738a-984b-463c-acbd-e2b38e345f6d\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv" Oct 30 17:15:12.182616 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:12.182591 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d6e738a-984b-463c-acbd-e2b38e345f6d-istio-token\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv\" (UID: \"5d6e738a-984b-463c-acbd-e2b38e345f6d\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv" Oct 30 17:15:12.182981 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:12.182957 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phcz8\" (UniqueName: \"kubernetes.io/projected/5d6e738a-984b-463c-acbd-e2b38e345f6d-kube-api-access-phcz8\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv\" (UID: \"5d6e738a-984b-463c-acbd-e2b38e345f6d\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv" Oct 30 17:15:12.213852 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:12.213828 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv" Oct 30 17:15:12.562298 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:12.562267 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv"] Oct 30 17:15:12.569538 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:12.569495 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:15:12.569650 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:12.569589 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:15:12.569650 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:12.569633 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:15:12.786008 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:12.785955 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv" event={"ID":"5d6e738a-984b-463c-acbd-e2b38e345f6d","Type":"ContainerStarted","Data":"5e5da11a1b7cd82130b6dbe321235e4b8a130a3d4c2bce8d77ab30f878040097"} Oct 30 17:15:12.786395 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:12.786040 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv" event={"ID":"5d6e738a-984b-463c-acbd-e2b38e345f6d","Type":"ContainerStarted","Data":"692bf5f4ecaf6df6db79c010a14ee70c3beef567d53c8ab50b6576c1e5385ea9"} Oct 30 17:15:12.967240 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:15:12.967203 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:15:13.213950 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:13.213912 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv" Oct 30 17:15:13.215366 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:13.215289 4522 patch_prober.go:29] interesting pod/e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Oct 30 17:15:13.215366 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:13.215345 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv" podUID="5d6e738a-984b-463c-acbd-e2b38e345f6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Oct 30 17:15:14.137502 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:14.137467 4522 scope.go:122] "RemoveContainer" containerID="fb4f4c31a35039169f64e40d8e050702257f0faf60eb9c0be49dbb062bcb5339" Oct 30 17:15:14.137977 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:14.137625 4522 scope.go:122] "RemoveContainer" containerID="381b0e1200fdb2b7b341c0a81ae472d6a4714b6a8fd8ca4049541cec20a440a1" Oct 30 17:15:14.217459 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:14.217423 4522 patch_prober.go:29] interesting pod/e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Oct 30 17:15:14.217621 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:14.217485 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv" podUID="5d6e738a-984b-463c-acbd-e2b38e345f6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Oct 30 17:15:14.833453 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:14.833417 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889675-xhg9d" event={"ID":"4e7e3bf8-d7a4-4839-bcf4-5161e1f998c9","Type":"ContainerStarted","Data":"563518bd0705ee43dca31ab995898e647ee8e6252e80fcfce7b9d4f1547d44e3"} Oct 30 17:15:14.835362 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:14.835334 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889675-rbwqn" event={"ID":"171da860-d46a-420a-88a3-423d6b58c86b","Type":"ContainerStarted","Data":"32d8dd646f5db30c2442ade1ecf77db16ad4643cfd1317519d6dea50dce989f9"} Oct 30 17:15:14.882638 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:14.882587 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv" podStartSLOduration=3.882572514 podStartE2EDuration="3.882572514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:12.847232422 +0000 UTC m=+1686.847987384" watchObservedRunningTime="2026-10-30 17:15:14.882572514 +0000 UTC m=+1688.883327414" Oct 30 17:15:15.220232 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:15.219792 4522 patch_prober.go:29] interesting pod/e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Oct 30 17:15:15.220232 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:15.219900 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv" podUID="5d6e738a-984b-463c-acbd-e2b38e345f6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Oct 30 17:15:15.846033 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:15.846003 4522 generic.go:356] "Generic (PLEG): container finished" podID="4e7e3bf8-d7a4-4839-bcf4-5161e1f998c9" containerID="563518bd0705ee43dca31ab995898e647ee8e6252e80fcfce7b9d4f1547d44e3" exitCode=60 Oct 30 17:15:15.846179 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:15.846073 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889675-xhg9d" event={"ID":"4e7e3bf8-d7a4-4839-bcf4-5161e1f998c9","Type":"ContainerDied","Data":"563518bd0705ee43dca31ab995898e647ee8e6252e80fcfce7b9d4f1547d44e3"} Oct 30 17:15:15.846179 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:15.846135 4522 scope.go:122] "RemoveContainer" containerID="fb4f4c31a35039169f64e40d8e050702257f0faf60eb9c0be49dbb062bcb5339" Oct 30 17:15:15.846624 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:15.846606 4522 scope.go:122] "RemoveContainer" containerID="563518bd0705ee43dca31ab995898e647ee8e6252e80fcfce7b9d4f1547d44e3" Oct 30 17:15:15.847056 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:15:15.847036 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889675-xhg9d_odh-ai-gateway-infra(4e7e3bf8-d7a4-4839-bcf4-5161e1f998c9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889675-xhg9d" podUID="4e7e3bf8-d7a4-4839-bcf4-5161e1f998c9" Oct 30 17:15:15.848371 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:15.848353 4522 generic.go:356] "Generic (PLEG): container finished" podID="171da860-d46a-420a-88a3-423d6b58c86b" containerID="32d8dd646f5db30c2442ade1ecf77db16ad4643cfd1317519d6dea50dce989f9" exitCode=60 Oct 30 17:15:15.848453 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:15.848422 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889675-rbwqn" event={"ID":"171da860-d46a-420a-88a3-423d6b58c86b","Type":"ContainerDied","Data":"32d8dd646f5db30c2442ade1ecf77db16ad4643cfd1317519d6dea50dce989f9"} Oct 30 17:15:15.848784 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:15.848770 4522 scope.go:122] "RemoveContainer" containerID="32d8dd646f5db30c2442ade1ecf77db16ad4643cfd1317519d6dea50dce989f9" Oct 30 17:15:15.849104 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:15:15.849079 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889675-rbwqn_odh-ai-gateway-infra(171da860-d46a-420a-88a3-423d6b58c86b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889675-rbwqn" podUID="171da860-d46a-420a-88a3-423d6b58c86b" Oct 30 17:15:15.860352 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:15.860332 4522 scope.go:122] "RemoveContainer" containerID="381b0e1200fdb2b7b341c0a81ae472d6a4714b6a8fd8ca4049541cec20a440a1" Oct 30 17:15:15.923293 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:15.923256 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889675-xhg9d"] Oct 30 17:15:15.947871 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:15.947841 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889675-rbwqn"] Oct 30 17:15:16.143694 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:16.143615 4522 scope.go:122] "RemoveContainer" containerID="fa203612fef197fc913f066787e63253f4d6e290ce374538704d16eeee9e7d9e" Oct 30 17:15:16.143969 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:16.143946 4522 scope.go:122] "RemoveContainer" containerID="9c71566bdd00b201399a17ee2dc929e295cf9fa3061eab999f3ce6966ccb90d8" Oct 30 17:15:16.219702 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:16.219666 4522 patch_prober.go:29] interesting pod/e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Oct 30 17:15:16.219886 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:16.219727 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv" podUID="5d6e738a-984b-463c-acbd-e2b38e345f6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Oct 30 17:15:16.616447 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:16.611424 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-728ad7-66cb7bbbd6-dwrgr" podUID="3fe580bc-2cda-4b54-8b89-61ec5b147ab8" containerName="payload-processing" containerID="cri-o://b0f57b65d2cf0f86cf533b56d7049e9f9231f5ce7d8c4733857010950cc7ffce" gracePeriod=25 Oct 30 17:15:16.631919 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:16.631866 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-728ad7-7b4857f67-lf57w" podUID="58921afe-0009-4d33-b402-0c708e26fe57" containerName="payload-pre-processing" containerID="cri-o://8948cacb23212173411a211fac0d68f5028f574dfb06f4d1034d8865b52fc210" gracePeriod=25 Oct 30 17:15:16.861644 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:16.861616 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39fecf-29889675-jshxf" event={"ID":"f5227ac7-04e8-4758-a1ed-1b907783431e","Type":"ContainerStarted","Data":"9cc047b35d8fe4ccc256b8b90bc35910275153951b26f5cd552907a7c4f56bed"} Oct 30 17:15:16.865601 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:16.865568 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889675-crbqg" event={"ID":"5802774a-44fd-4660-b00e-8d979654b196","Type":"ContainerStarted","Data":"4f94b415b2b721482229b0e76a1691ed78ce308e3cdef466348926b11a9ee470"} Oct 30 17:15:17.143076 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:17.143050 4522 scope.go:122] "RemoveContainer" containerID="056c84715eb606c8c3fb26ad219701f1a64f638fdc3c0ea0fd630a297f650237" Oct 30 17:15:17.147579 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:17.147378 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889675-xhg9d" Oct 30 17:15:17.225377 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:17.224915 4522 patch_prober.go:29] interesting pod/e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Oct 30 17:15:17.225377 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:17.224970 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv" podUID="5d6e738a-984b-463c-acbd-e2b38e345f6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Oct 30 17:15:17.246229 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:17.246142 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e7e3bf8-d7a4-4839-bcf4-5161e1f998c9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e7e3bf8-d7a4-4839-bcf4-5161e1f998c9-maas-api-service-ca\") pod \"4e7e3bf8-d7a4-4839-bcf4-5161e1f998c9\" (UID: \"4e7e3bf8-d7a4-4839-bcf4-5161e1f998c9\") " Oct 30 17:15:17.246547 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:17.246506 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e7e3bf8-d7a4-4839-bcf4-5161e1f998c9-kube-api-access-vdwlk\" (UniqueName: \"kubernetes.io/projected/4e7e3bf8-d7a4-4839-bcf4-5161e1f998c9-kube-api-access-vdwlk\") pod \"4e7e3bf8-d7a4-4839-bcf4-5161e1f998c9\" (UID: \"4e7e3bf8-d7a4-4839-bcf4-5161e1f998c9\") " Oct 30 17:15:17.248849 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:17.246714 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e7e3bf8-d7a4-4839-bcf4-5161e1f998c9-maas-api-service-ca" pod "4e7e3bf8-d7a4-4839-bcf4-5161e1f998c9" (UID: "4e7e3bf8-d7a4-4839-bcf4-5161e1f998c9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:17.249590 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:17.249530 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e7e3bf8-d7a4-4839-bcf4-5161e1f998c9-kube-api-access-vdwlk" pod "4e7e3bf8-d7a4-4839-bcf4-5161e1f998c9" (UID: "4e7e3bf8-d7a4-4839-bcf4-5161e1f998c9"). InnerVolumeSpecName "kube-api-access-vdwlk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:17.255300 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:17.255277 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889675-rbwqn" Oct 30 17:15:17.349237 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:17.349201 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/171da860-d46a-420a-88a3-423d6b58c86b-kube-api-access-9k24s\" (UniqueName: \"kubernetes.io/projected/171da860-d46a-420a-88a3-423d6b58c86b-kube-api-access-9k24s\") pod \"171da860-d46a-420a-88a3-423d6b58c86b\" (UID: \"171da860-d46a-420a-88a3-423d6b58c86b\") " Oct 30 17:15:17.349440 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:17.349364 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/171da860-d46a-420a-88a3-423d6b58c86b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/171da860-d46a-420a-88a3-423d6b58c86b-maas-api-service-ca\") pod \"171da860-d46a-420a-88a3-423d6b58c86b\" (UID: \"171da860-d46a-420a-88a3-423d6b58c86b\") " Oct 30 17:15:17.349898 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:17.349867 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/171da860-d46a-420a-88a3-423d6b58c86b-maas-api-service-ca" pod "171da860-d46a-420a-88a3-423d6b58c86b" (UID: "171da860-d46a-420a-88a3-423d6b58c86b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:17.350242 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:17.350218 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdwlk\" (UniqueName: \"kubernetes.io/projected/4e7e3bf8-d7a4-4839-bcf4-5161e1f998c9-kube-api-access-vdwlk\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:15:17.350370 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:17.350247 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/171da860-d46a-420a-88a3-423d6b58c86b-maas-api-service-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:15:17.350370 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:17.350265 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e7e3bf8-d7a4-4839-bcf4-5161e1f998c9-maas-api-service-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:15:17.418842 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:17.418806 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/171da860-d46a-420a-88a3-423d6b58c86b-kube-api-access-9k24s" pod "171da860-d46a-420a-88a3-423d6b58c86b" (UID: "171da860-d46a-420a-88a3-423d6b58c86b"). InnerVolumeSpecName "kube-api-access-9k24s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:17.452279 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:17.452251 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9k24s\" (UniqueName: \"kubernetes.io/projected/171da860-d46a-420a-88a3-423d6b58c86b-kube-api-access-9k24s\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:15:17.724082 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:17.723889 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-5d67ca-6c5bcd5578-n5lgb" Oct 30 17:15:17.839617 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:17.839519 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-728ad7-66cb7bbbd6-dwrgr" podUID="3fe580bc-2cda-4b54-8b89-61ec5b147ab8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:15:17.840409 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:17.840375 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-728ad7-7b4857f67-lf57w" podUID="58921afe-0009-4d33-b402-0c708e26fe57" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:15:17.875559 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:17.875524 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889675-rbwqn" Oct 30 17:15:17.875559 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:17.875555 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889675-rbwqn" event={"ID":"171da860-d46a-420a-88a3-423d6b58c86b","Type":"ContainerDied","Data":"342d9b3678f6e1526dbd59bfe60230f2279a8e44ac09a9ec4e4f165239b6d018"} Oct 30 17:15:17.875806 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:17.875616 4522 scope.go:122] "RemoveContainer" containerID="32d8dd646f5db30c2442ade1ecf77db16ad4643cfd1317519d6dea50dce989f9" Oct 30 17:15:17.877810 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:17.877786 4522 generic.go:356] "Generic (PLEG): container finished" podID="5802774a-44fd-4660-b00e-8d979654b196" containerID="4f94b415b2b721482229b0e76a1691ed78ce308e3cdef466348926b11a9ee470" exitCode=60 Oct 30 17:15:17.877925 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:17.877823 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889675-crbqg" event={"ID":"5802774a-44fd-4660-b00e-8d979654b196","Type":"ContainerDied","Data":"4f94b415b2b721482229b0e76a1691ed78ce308e3cdef466348926b11a9ee470"} Oct 30 17:15:17.878587 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:17.878551 4522 scope.go:122] "RemoveContainer" containerID="4f94b415b2b721482229b0e76a1691ed78ce308e3cdef466348926b11a9ee470" Oct 30 17:15:17.878987 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:15:17.878961 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889675-crbqg_odh-ai-gateway-infra(5802774a-44fd-4660-b00e-8d979654b196)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889675-crbqg" podUID="5802774a-44fd-4660-b00e-8d979654b196" Oct 30 17:15:17.884481 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:17.884448 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f42baf-29889675-2mg4p" event={"ID":"c53d8512-95ac-4d60-aaa6-0d74b95e0cf8","Type":"ContainerStarted","Data":"e0ebeae55304dc8cefca3cd1db0783696e417c8bc867a9e54c32d2f22342d476"} Oct 30 17:15:17.888087 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:17.888059 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889675-xhg9d" Oct 30 17:15:17.888211 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:17.888082 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889675-xhg9d" event={"ID":"4e7e3bf8-d7a4-4839-bcf4-5161e1f998c9","Type":"ContainerDied","Data":"ac1652d784e7300fe4f2e76fef2905b01949c71aa7be9f55b8a03a6a76c6e6a7"} Oct 30 17:15:17.890115 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:17.890102 4522 scope.go:122] "RemoveContainer" containerID="9c71566bdd00b201399a17ee2dc929e295cf9fa3061eab999f3ce6966ccb90d8" Oct 30 17:15:17.890544 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:17.890525 4522 generic.go:356] "Generic (PLEG): container finished" podID="f5227ac7-04e8-4758-a1ed-1b907783431e" containerID="9cc047b35d8fe4ccc256b8b90bc35910275153951b26f5cd552907a7c4f56bed" exitCode=60 Oct 30 17:15:17.890688 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:17.890563 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39fecf-29889675-jshxf" event={"ID":"f5227ac7-04e8-4758-a1ed-1b907783431e","Type":"ContainerDied","Data":"9cc047b35d8fe4ccc256b8b90bc35910275153951b26f5cd552907a7c4f56bed"} Oct 30 17:15:17.890964 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:17.890950 4522 scope.go:122] "RemoveContainer" containerID="9cc047b35d8fe4ccc256b8b90bc35910275153951b26f5cd552907a7c4f56bed" Oct 30 17:15:17.891265 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:15:17.891247 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-39fecf-29889675-jshxf_odh-ai-gateway-infra(f5227ac7-04e8-4758-a1ed-1b907783431e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39fecf-29889675-jshxf" podUID="f5227ac7-04e8-4758-a1ed-1b907783431e" Oct 30 17:15:17.933444 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:17.933403 4522 scope.go:122] "RemoveContainer" containerID="563518bd0705ee43dca31ab995898e647ee8e6252e80fcfce7b9d4f1547d44e3" Oct 30 17:15:17.947985 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:17.947965 4522 scope.go:122] "RemoveContainer" containerID="fa203612fef197fc913f066787e63253f4d6e290ce374538704d16eeee9e7d9e" Oct 30 17:15:18.026219 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:18.026188 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889675-rbwqn"] Oct 30 17:15:18.031297 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:18.031271 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889675-rbwqn"] Oct 30 17:15:18.064910 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:18.064870 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39fecf-29889675-jshxf"] Oct 30 17:15:18.127230 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:18.127148 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889675-crbqg"] Oct 30 17:15:18.137673 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:18.137640 4522 scope.go:122] "RemoveContainer" containerID="eb81caca77b39e990e61f509f2c027f1419eaeabcedcd0e60ead3c974fbfa97e" Oct 30 17:15:18.138060 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:18.138030 4522 scope.go:122] "RemoveContainer" containerID="7f730ede20c0cf383b713f150c509035cbf7296c4b8b715a32d7f9a3b89f5b8f" Oct 30 17:15:18.138215 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:18.138197 4522 scope.go:122] "RemoveContainer" containerID="d5be21be86435e2d970f039039f886115377dbf3ac717c5da031e4a2d8266325" Oct 30 17:15:18.152038 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:18.152010 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="171da860-d46a-420a-88a3-423d6b58c86b" path="/var/lib/kubelet/pods/171da860-d46a-420a-88a3-423d6b58c86b/volumes" Oct 30 17:15:18.212516 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:18.212072 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889675-xhg9d"] Oct 30 17:15:18.239606 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:18.239574 4522 patch_prober.go:29] interesting pod/e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Oct 30 17:15:18.239928 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:18.239865 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv" podUID="5d6e738a-984b-463c-acbd-e2b38e345f6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Oct 30 17:15:18.262350 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:18.261043 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889675-xhg9d"] Oct 30 17:15:18.907488 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:18.907455 4522 generic.go:356] "Generic (PLEG): container finished" podID="c53d8512-95ac-4d60-aaa6-0d74b95e0cf8" containerID="e0ebeae55304dc8cefca3cd1db0783696e417c8bc867a9e54c32d2f22342d476" exitCode=60 Oct 30 17:15:18.907908 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:18.907553 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f42baf-29889675-2mg4p" event={"ID":"c53d8512-95ac-4d60-aaa6-0d74b95e0cf8","Type":"ContainerDied","Data":"e0ebeae55304dc8cefca3cd1db0783696e417c8bc867a9e54c32d2f22342d476"} Oct 30 17:15:18.907908 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:18.907626 4522 scope.go:122] "RemoveContainer" containerID="056c84715eb606c8c3fb26ad219701f1a64f638fdc3c0ea0fd630a297f650237" Oct 30 17:15:18.909256 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:18.909229 4522 scope.go:122] "RemoveContainer" containerID="e0ebeae55304dc8cefca3cd1db0783696e417c8bc867a9e54c32d2f22342d476" Oct 30 17:15:18.909728 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:15:18.909697 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-f42baf-29889675-2mg4p_odh-ai-gateway-infra(c53d8512-95ac-4d60-aaa6-0d74b95e0cf8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f42baf-29889675-2mg4p" podUID="c53d8512-95ac-4d60-aaa6-0d74b95e0cf8" Oct 30 17:15:18.917653 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:18.917625 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889675-fdxml" event={"ID":"1ea441b9-186a-4e43-b2c7-37052bd5dcd7","Type":"ContainerStarted","Data":"534978f1c569305db3509f2184be134e9f106fc6d2cb0e68406e1c248bb1bbe9"} Oct 30 17:15:19.145374 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:19.142250 4522 scope.go:122] "RemoveContainer" containerID="0311a7d0d184580d6b21fa6bcfe15b844ae8912c29b62ad4c204d5db6e288e02" Oct 30 17:15:19.145374 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:19.143779 4522 scope.go:122] "RemoveContainer" containerID="0c2da43e32750ff2478ac633e9556d33ab432cdd72510c88a6b02072b7ece5df" Oct 30 17:15:19.145374 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:19.143965 4522 scope.go:122] "RemoveContainer" containerID="048f5bb530e316f8e4a3b65b0b78c304a33dfe7d4f68445eb587753f964784cb" Oct 30 17:15:19.145374 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:19.144619 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889675-crbqg" Oct 30 17:15:19.153406 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:19.153380 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f42baf-29889675-2mg4p"] Oct 30 17:15:19.186505 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:19.181060 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5802774a-44fd-4660-b00e-8d979654b196-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5802774a-44fd-4660-b00e-8d979654b196-maas-api-service-ca\") pod \"5802774a-44fd-4660-b00e-8d979654b196\" (UID: \"5802774a-44fd-4660-b00e-8d979654b196\") " Oct 30 17:15:19.186505 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:19.181173 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5802774a-44fd-4660-b00e-8d979654b196-kube-api-access-n8ds5\" (UniqueName: \"kubernetes.io/projected/5802774a-44fd-4660-b00e-8d979654b196-kube-api-access-n8ds5\") pod \"5802774a-44fd-4660-b00e-8d979654b196\" (UID: \"5802774a-44fd-4660-b00e-8d979654b196\") " Oct 30 17:15:19.186505 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:19.182588 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5802774a-44fd-4660-b00e-8d979654b196-maas-api-service-ca" pod "5802774a-44fd-4660-b00e-8d979654b196" (UID: "5802774a-44fd-4660-b00e-8d979654b196"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:19.194176 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:19.194132 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5802774a-44fd-4660-b00e-8d979654b196-kube-api-access-n8ds5" pod "5802774a-44fd-4660-b00e-8d979654b196" (UID: "5802774a-44fd-4660-b00e-8d979654b196"). InnerVolumeSpecName "kube-api-access-n8ds5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:19.229941 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:19.229919 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv" Oct 30 17:15:19.231020 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:19.230990 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv" Oct 30 17:15:19.232182 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:19.232159 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv" Oct 30 17:15:19.257675 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:19.257647 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39fecf-29889675-jshxf" Oct 30 17:15:19.287913 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:19.283478 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5227ac7-04e8-4758-a1ed-1b907783431e-kube-api-access-khpmd\" (UniqueName: \"kubernetes.io/projected/f5227ac7-04e8-4758-a1ed-1b907783431e-kube-api-access-khpmd\") pod \"f5227ac7-04e8-4758-a1ed-1b907783431e\" (UID: \"f5227ac7-04e8-4758-a1ed-1b907783431e\") " Oct 30 17:15:19.287913 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:19.283724 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5227ac7-04e8-4758-a1ed-1b907783431e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5227ac7-04e8-4758-a1ed-1b907783431e-maas-api-service-ca\") pod \"f5227ac7-04e8-4758-a1ed-1b907783431e\" (UID: \"f5227ac7-04e8-4758-a1ed-1b907783431e\") " Oct 30 17:15:19.287913 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:19.284515 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5802774a-44fd-4660-b00e-8d979654b196-maas-api-service-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:15:19.287913 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:19.284537 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8ds5\" (UniqueName: \"kubernetes.io/projected/5802774a-44fd-4660-b00e-8d979654b196-kube-api-access-n8ds5\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:15:19.287913 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:19.285386 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5227ac7-04e8-4758-a1ed-1b907783431e-maas-api-service-ca" pod "f5227ac7-04e8-4758-a1ed-1b907783431e" (UID: "f5227ac7-04e8-4758-a1ed-1b907783431e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:19.287913 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:19.287181 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5227ac7-04e8-4758-a1ed-1b907783431e-kube-api-access-khpmd" pod "f5227ac7-04e8-4758-a1ed-1b907783431e" (UID: "f5227ac7-04e8-4758-a1ed-1b907783431e"). InnerVolumeSpecName "kube-api-access-khpmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:19.386569 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:19.386485 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khpmd\" (UniqueName: \"kubernetes.io/projected/f5227ac7-04e8-4758-a1ed-1b907783431e-kube-api-access-khpmd\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:15:19.386569 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:19.386512 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5227ac7-04e8-4758-a1ed-1b907783431e-maas-api-service-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:15:19.930883 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:19.930798 4522 generic.go:356] "Generic (PLEG): container finished" podID="417f0b10-2a38-4757-8563-d47b75088b4d" containerID="45498dcf9a05230f8970fa3b864ae72a3202dbad46f44be35c3c840315369fc6" exitCode=60 Oct 30 17:15:19.931370 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:19.930876 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-728ad7-29889675-hgghk" event={"ID":"417f0b10-2a38-4757-8563-d47b75088b4d","Type":"ContainerDied","Data":"45498dcf9a05230f8970fa3b864ae72a3202dbad46f44be35c3c840315369fc6"} Oct 30 17:15:19.931370 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:19.930950 4522 scope.go:122] "RemoveContainer" containerID="d5be21be86435e2d970f039039f886115377dbf3ac717c5da031e4a2d8266325" Oct 30 17:15:19.931512 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:19.931495 4522 scope.go:122] "RemoveContainer" containerID="45498dcf9a05230f8970fa3b864ae72a3202dbad46f44be35c3c840315369fc6" Oct 30 17:15:19.931945 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:15:19.931919 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-728ad7-29889675-hgghk_odh-ai-gateway-infra(417f0b10-2a38-4757-8563-d47b75088b4d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-728ad7-29889675-hgghk" podUID="417f0b10-2a38-4757-8563-d47b75088b4d" Oct 30 17:15:19.932482 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:19.932453 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39fecf-29889675-jshxf" Oct 30 17:15:19.932543 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:19.932469 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39fecf-29889675-jshxf" event={"ID":"f5227ac7-04e8-4758-a1ed-1b907783431e","Type":"ContainerDied","Data":"8fea04062e25af688714d860cb9e7498a79907ddd755d11439d758853ceebeab"} Oct 30 17:15:19.934838 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:19.934816 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889675-dbh98" event={"ID":"78db2ae0-039b-4edc-8fdb-c4513041d366","Type":"ContainerStarted","Data":"3436edf1142de019623429681ce5b3a2eb6d1e84f9ddb7f43f9ad7b544fa03e0"} Oct 30 17:15:19.936785 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:19.936765 4522 generic.go:356] "Generic (PLEG): container finished" podID="1ea441b9-186a-4e43-b2c7-37052bd5dcd7" containerID="534978f1c569305db3509f2184be134e9f106fc6d2cb0e68406e1c248bb1bbe9" exitCode=60 Oct 30 17:15:19.936888 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:19.936830 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889675-fdxml" event={"ID":"1ea441b9-186a-4e43-b2c7-37052bd5dcd7","Type":"ContainerDied","Data":"534978f1c569305db3509f2184be134e9f106fc6d2cb0e68406e1c248bb1bbe9"} Oct 30 17:15:19.937171 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:19.937156 4522 scope.go:122] "RemoveContainer" containerID="534978f1c569305db3509f2184be134e9f106fc6d2cb0e68406e1c248bb1bbe9" Oct 30 17:15:19.937498 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:15:19.937476 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889675-fdxml_odh-ai-gateway-infra(1ea441b9-186a-4e43-b2c7-37052bd5dcd7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889675-fdxml" podUID="1ea441b9-186a-4e43-b2c7-37052bd5dcd7" Oct 30 17:15:19.939135 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:19.939112 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889675-s24lj" event={"ID":"79a79c82-3142-4035-a072-004b80dfe24d","Type":"ContainerStarted","Data":"f323c270e29c07b10ab71fe52d1da87981e840401dfb4b66e98273d75da6686d"} Oct 30 17:15:19.940637 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:19.940617 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889675-crbqg" Oct 30 17:15:19.940720 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:19.940651 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889675-crbqg" event={"ID":"5802774a-44fd-4660-b00e-8d979654b196","Type":"ContainerDied","Data":"cb4d989a8fcd17b30410ba4abb35e4f3f15c46ccab00dd50910575cd358bfa75"} Oct 30 17:15:19.943075 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:19.943054 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889675-lcm6b" event={"ID":"5d21c30e-1815-436a-92b5-4cb61b88ea19","Type":"ContainerStarted","Data":"912455fe151c913ee418158f237ca444e85ce41a6e07b0d98af7ca80985b191f"} Oct 30 17:15:19.947019 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:19.946999 4522 generic.go:356] "Generic (PLEG): container finished" podID="ad604ad3-0fd0-4544-af4d-e5061bd561e3" containerID="3aed68c132526274307d91ee6dc0ce206dff5e505e69ae01dc1c9029aa28e37f" exitCode=60 Oct 30 17:15:19.947104 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:19.947072 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b2c3fb-29889675-rckqx" event={"ID":"ad604ad3-0fd0-4544-af4d-e5061bd561e3","Type":"ContainerDied","Data":"3aed68c132526274307d91ee6dc0ce206dff5e505e69ae01dc1c9029aa28e37f"} Oct 30 17:15:19.947544 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:19.947527 4522 scope.go:122] "RemoveContainer" containerID="3aed68c132526274307d91ee6dc0ce206dff5e505e69ae01dc1c9029aa28e37f" Oct 30 17:15:19.947928 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:15:19.947867 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-b2c3fb-29889675-rckqx_odh-ai-gateway-infra(ad604ad3-0fd0-4544-af4d-e5061bd561e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b2c3fb-29889675-rckqx" podUID="ad604ad3-0fd0-4544-af4d-e5061bd561e3" Oct 30 17:15:19.954992 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:19.954970 4522 scope.go:122] "RemoveContainer" containerID="9cc047b35d8fe4ccc256b8b90bc35910275153951b26f5cd552907a7c4f56bed" Oct 30 17:15:19.966022 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:19.966002 4522 scope.go:122] "RemoveContainer" containerID="eb81caca77b39e990e61f509f2c027f1419eaeabcedcd0e60ead3c974fbfa97e" Oct 30 17:15:19.984421 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:19.984394 4522 scope.go:122] "RemoveContainer" containerID="4f94b415b2b721482229b0e76a1691ed78ce308e3cdef466348926b11a9ee470" Oct 30 17:15:19.995603 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:19.995574 4522 scope.go:122] "RemoveContainer" containerID="7f730ede20c0cf383b713f150c509035cbf7296c4b8b715a32d7f9a3b89f5b8f" Oct 30 17:15:20.072977 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:20.070824 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889675-fdxml"] Oct 30 17:15:20.119287 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:20.119258 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889675-crbqg"] Oct 30 17:15:20.124242 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:20.124216 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889675-crbqg"] Oct 30 17:15:20.136996 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:20.136970 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f42baf-29889675-2mg4p" Oct 30 17:15:20.148507 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:20.148478 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e7e3bf8-d7a4-4839-bcf4-5161e1f998c9" path="/var/lib/kubelet/pods/4e7e3bf8-d7a4-4839-bcf4-5161e1f998c9/volumes" Oct 30 17:15:20.149035 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:20.149021 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5802774a-44fd-4660-b00e-8d979654b196" path="/var/lib/kubelet/pods/5802774a-44fd-4660-b00e-8d979654b196/volumes" Oct 30 17:15:20.200059 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:20.199962 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c53d8512-95ac-4d60-aaa6-0d74b95e0cf8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c53d8512-95ac-4d60-aaa6-0d74b95e0cf8-maas-api-service-ca\") pod \"c53d8512-95ac-4d60-aaa6-0d74b95e0cf8\" (UID: \"c53d8512-95ac-4d60-aaa6-0d74b95e0cf8\") " Oct 30 17:15:20.200223 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:20.200192 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c53d8512-95ac-4d60-aaa6-0d74b95e0cf8-kube-api-access-kpm2j\" (UniqueName: \"kubernetes.io/projected/c53d8512-95ac-4d60-aaa6-0d74b95e0cf8-kube-api-access-kpm2j\") pod \"c53d8512-95ac-4d60-aaa6-0d74b95e0cf8\" (UID: \"c53d8512-95ac-4d60-aaa6-0d74b95e0cf8\") " Oct 30 17:15:20.200413 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:20.200388 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c53d8512-95ac-4d60-aaa6-0d74b95e0cf8-maas-api-service-ca" pod "c53d8512-95ac-4d60-aaa6-0d74b95e0cf8" (UID: "c53d8512-95ac-4d60-aaa6-0d74b95e0cf8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:20.200848 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:20.200823 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c53d8512-95ac-4d60-aaa6-0d74b95e0cf8-maas-api-service-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:15:20.202827 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:20.202801 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c53d8512-95ac-4d60-aaa6-0d74b95e0cf8-kube-api-access-kpm2j" pod "c53d8512-95ac-4d60-aaa6-0d74b95e0cf8" (UID: "c53d8512-95ac-4d60-aaa6-0d74b95e0cf8"). InnerVolumeSpecName "kube-api-access-kpm2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:20.290408 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:20.290375 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39fecf-29889675-jshxf"] Oct 30 17:15:20.301053 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:20.301029 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39fecf-29889675-jshxf"] Oct 30 17:15:20.302084 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:20.302059 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpm2j\" (UniqueName: \"kubernetes.io/projected/c53d8512-95ac-4d60-aaa6-0d74b95e0cf8-kube-api-access-kpm2j\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:15:20.960808 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:20.960781 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f42baf-29889675-2mg4p" Oct 30 17:15:20.961240 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:20.960776 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f42baf-29889675-2mg4p" event={"ID":"c53d8512-95ac-4d60-aaa6-0d74b95e0cf8","Type":"ContainerDied","Data":"3347383a5aac7e68a780853b84a6ff903a110f88cca50853357a5ba528a0ed2f"} Oct 30 17:15:20.961240 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:20.960945 4522 scope.go:122] "RemoveContainer" containerID="e0ebeae55304dc8cefca3cd1db0783696e417c8bc867a9e54c32d2f22342d476" Oct 30 17:15:20.967984 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:20.967962 4522 generic.go:356] "Generic (PLEG): container finished" podID="78db2ae0-039b-4edc-8fdb-c4513041d366" containerID="3436edf1142de019623429681ce5b3a2eb6d1e84f9ddb7f43f9ad7b544fa03e0" exitCode=60 Oct 30 17:15:20.968102 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:20.968014 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889675-dbh98" event={"ID":"78db2ae0-039b-4edc-8fdb-c4513041d366","Type":"ContainerDied","Data":"3436edf1142de019623429681ce5b3a2eb6d1e84f9ddb7f43f9ad7b544fa03e0"} Oct 30 17:15:20.968519 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:20.968499 4522 scope.go:122] "RemoveContainer" containerID="3436edf1142de019623429681ce5b3a2eb6d1e84f9ddb7f43f9ad7b544fa03e0" Oct 30 17:15:20.968959 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:15:20.968934 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889675-dbh98_odh-ai-gateway-infra(78db2ae0-039b-4edc-8fdb-c4513041d366)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889675-dbh98" podUID="78db2ae0-039b-4edc-8fdb-c4513041d366" Oct 30 17:15:20.976218 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:20.976198 4522 scope.go:122] "RemoveContainer" containerID="048f5bb530e316f8e4a3b65b0b78c304a33dfe7d4f68445eb587753f964784cb" Oct 30 17:15:20.976674 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:20.976645 4522 generic.go:356] "Generic (PLEG): container finished" podID="79a79c82-3142-4035-a072-004b80dfe24d" containerID="f323c270e29c07b10ab71fe52d1da87981e840401dfb4b66e98273d75da6686d" exitCode=60 Oct 30 17:15:20.976804 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:20.976718 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889675-s24lj" event={"ID":"79a79c82-3142-4035-a072-004b80dfe24d","Type":"ContainerDied","Data":"f323c270e29c07b10ab71fe52d1da87981e840401dfb4b66e98273d75da6686d"} Oct 30 17:15:20.977341 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:20.977320 4522 scope.go:122] "RemoveContainer" containerID="f323c270e29c07b10ab71fe52d1da87981e840401dfb4b66e98273d75da6686d" Oct 30 17:15:20.977796 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:15:20.977772 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-86f50a-29889675-s24lj_odh-ai-gateway-infra(79a79c82-3142-4035-a072-004b80dfe24d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889675-s24lj" podUID="79a79c82-3142-4035-a072-004b80dfe24d" Oct 30 17:15:20.980452 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:20.980433 4522 generic.go:356] "Generic (PLEG): container finished" podID="5d21c30e-1815-436a-92b5-4cb61b88ea19" containerID="912455fe151c913ee418158f237ca444e85ce41a6e07b0d98af7ca80985b191f" exitCode=60 Oct 30 17:15:20.980568 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:20.980502 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889675-lcm6b" event={"ID":"5d21c30e-1815-436a-92b5-4cb61b88ea19","Type":"ContainerDied","Data":"912455fe151c913ee418158f237ca444e85ce41a6e07b0d98af7ca80985b191f"} Oct 30 17:15:20.981037 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:20.981016 4522 scope.go:122] "RemoveContainer" containerID="912455fe151c913ee418158f237ca444e85ce41a6e07b0d98af7ca80985b191f" Oct 30 17:15:20.981476 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:15:20.981452 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889675-lcm6b_odh-ai-gateway-infra(5d21c30e-1815-436a-92b5-4cb61b88ea19)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889675-lcm6b" podUID="5d21c30e-1815-436a-92b5-4cb61b88ea19" Oct 30 17:15:21.001567 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:21.001540 4522 scope.go:122] "RemoveContainer" containerID="0311a7d0d184580d6b21fa6bcfe15b844ae8912c29b62ad4c204d5db6e288e02" Oct 30 17:15:21.005618 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:21.005594 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-728ad7-29889675-hgghk"] Oct 30 17:15:21.027356 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:21.025426 4522 scope.go:122] "RemoveContainer" containerID="0c2da43e32750ff2478ac633e9556d33ab432cdd72510c88a6b02072b7ece5df" Oct 30 17:15:21.046680 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:21.046649 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b2c3fb-29889675-rckqx"] Oct 30 17:15:21.090387 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:21.090333 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889675-dbh98"] Oct 30 17:15:21.146113 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:21.143795 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f42baf-29889675-2mg4p"] Oct 30 17:15:21.151208 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:21.151132 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f42baf-29889675-2mg4p"] Oct 30 17:15:21.177961 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:21.177812 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889675-s24lj"] Oct 30 17:15:21.210125 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:21.210095 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889675-fdxml" Oct 30 17:15:21.226158 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:21.226125 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889675-lcm6b"] Oct 30 17:15:21.305401 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:21.305374 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-728ad7-29889675-hgghk" Oct 30 17:15:21.315918 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:21.315897 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b2c3fb-29889675-rckqx" Oct 30 17:15:21.319235 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:21.319212 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ea441b9-186a-4e43-b2c7-37052bd5dcd7-kube-api-access-bm9nj\" (UniqueName: \"kubernetes.io/projected/1ea441b9-186a-4e43-b2c7-37052bd5dcd7-kube-api-access-bm9nj\") pod \"1ea441b9-186a-4e43-b2c7-37052bd5dcd7\" (UID: \"1ea441b9-186a-4e43-b2c7-37052bd5dcd7\") " Oct 30 17:15:21.319533 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:21.319516 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ea441b9-186a-4e43-b2c7-37052bd5dcd7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ea441b9-186a-4e43-b2c7-37052bd5dcd7-maas-api-service-ca\") pod \"1ea441b9-186a-4e43-b2c7-37052bd5dcd7\" (UID: \"1ea441b9-186a-4e43-b2c7-37052bd5dcd7\") " Oct 30 17:15:21.319818 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:21.319798 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ea441b9-186a-4e43-b2c7-37052bd5dcd7-maas-api-service-ca" pod "1ea441b9-186a-4e43-b2c7-37052bd5dcd7" (UID: "1ea441b9-186a-4e43-b2c7-37052bd5dcd7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:21.320133 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:21.320113 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ea441b9-186a-4e43-b2c7-37052bd5dcd7-maas-api-service-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:15:21.321663 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:21.321642 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ea441b9-186a-4e43-b2c7-37052bd5dcd7-kube-api-access-bm9nj" pod "1ea441b9-186a-4e43-b2c7-37052bd5dcd7" (UID: "1ea441b9-186a-4e43-b2c7-37052bd5dcd7"). InnerVolumeSpecName "kube-api-access-bm9nj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:21.420896 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:21.420858 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/417f0b10-2a38-4757-8563-d47b75088b4d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/417f0b10-2a38-4757-8563-d47b75088b4d-maas-api-service-ca\") pod \"417f0b10-2a38-4757-8563-d47b75088b4d\" (UID: \"417f0b10-2a38-4757-8563-d47b75088b4d\") " Oct 30 17:15:21.421105 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:21.421047 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/417f0b10-2a38-4757-8563-d47b75088b4d-kube-api-access-w25vf\" (UniqueName: \"kubernetes.io/projected/417f0b10-2a38-4757-8563-d47b75088b4d-kube-api-access-w25vf\") pod \"417f0b10-2a38-4757-8563-d47b75088b4d\" (UID: \"417f0b10-2a38-4757-8563-d47b75088b4d\") " Oct 30 17:15:21.421105 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:21.421095 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad604ad3-0fd0-4544-af4d-e5061bd561e3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad604ad3-0fd0-4544-af4d-e5061bd561e3-maas-api-service-ca\") pod \"ad604ad3-0fd0-4544-af4d-e5061bd561e3\" (UID: \"ad604ad3-0fd0-4544-af4d-e5061bd561e3\") " Oct 30 17:15:21.421415 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:21.421384 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/417f0b10-2a38-4757-8563-d47b75088b4d-maas-api-service-ca" pod "417f0b10-2a38-4757-8563-d47b75088b4d" (UID: "417f0b10-2a38-4757-8563-d47b75088b4d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:21.421415 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:21.421401 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad604ad3-0fd0-4544-af4d-e5061bd561e3-kube-api-access-h5hmw\" (UniqueName: \"kubernetes.io/projected/ad604ad3-0fd0-4544-af4d-e5061bd561e3-kube-api-access-h5hmw\") pod \"ad604ad3-0fd0-4544-af4d-e5061bd561e3\" (UID: \"ad604ad3-0fd0-4544-af4d-e5061bd561e3\") " Oct 30 17:15:21.422205 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:21.422181 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bm9nj\" (UniqueName: \"kubernetes.io/projected/1ea441b9-186a-4e43-b2c7-37052bd5dcd7-kube-api-access-bm9nj\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:15:21.422408 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:21.422382 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/417f0b10-2a38-4757-8563-d47b75088b4d-maas-api-service-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:15:21.422608 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:21.422580 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad604ad3-0fd0-4544-af4d-e5061bd561e3-maas-api-service-ca" pod "ad604ad3-0fd0-4544-af4d-e5061bd561e3" (UID: "ad604ad3-0fd0-4544-af4d-e5061bd561e3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:21.424056 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:21.424031 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/417f0b10-2a38-4757-8563-d47b75088b4d-kube-api-access-w25vf" pod "417f0b10-2a38-4757-8563-d47b75088b4d" (UID: "417f0b10-2a38-4757-8563-d47b75088b4d"). InnerVolumeSpecName "kube-api-access-w25vf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:21.424902 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:21.424878 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad604ad3-0fd0-4544-af4d-e5061bd561e3-kube-api-access-h5hmw" pod "ad604ad3-0fd0-4544-af4d-e5061bd561e3" (UID: "ad604ad3-0fd0-4544-af4d-e5061bd561e3"). InnerVolumeSpecName "kube-api-access-h5hmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:21.524438 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:21.524366 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5hmw\" (UniqueName: \"kubernetes.io/projected/ad604ad3-0fd0-4544-af4d-e5061bd561e3-kube-api-access-h5hmw\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:15:21.524438 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:21.524393 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w25vf\" (UniqueName: \"kubernetes.io/projected/417f0b10-2a38-4757-8563-d47b75088b4d-kube-api-access-w25vf\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:15:21.524438 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:21.524405 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad604ad3-0fd0-4544-af4d-e5061bd561e3-maas-api-service-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:15:21.993067 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:21.993041 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b2c3fb-29889675-rckqx" Oct 30 17:15:21.993564 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:21.993039 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b2c3fb-29889675-rckqx" event={"ID":"ad604ad3-0fd0-4544-af4d-e5061bd561e3","Type":"ContainerDied","Data":"49e0a28087326f9871783df694581479a27776b452bdd55df2656b76bbe589ce"} Oct 30 17:15:21.993564 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:21.993195 4522 scope.go:122] "RemoveContainer" containerID="3aed68c132526274307d91ee6dc0ce206dff5e505e69ae01dc1c9029aa28e37f" Oct 30 17:15:21.994675 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:21.994649 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-728ad7-29889675-hgghk" Oct 30 17:15:21.994675 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:21.994664 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-728ad7-29889675-hgghk" event={"ID":"417f0b10-2a38-4757-8563-d47b75088b4d","Type":"ContainerDied","Data":"2cd00dd25202a3484cc54c19ce2e8c05a273560267e5664cd646c7cba5ab983d"} Oct 30 17:15:21.998025 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:21.997997 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889675-fdxml" Oct 30 17:15:21.998128 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:21.998103 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889675-fdxml" event={"ID":"1ea441b9-186a-4e43-b2c7-37052bd5dcd7","Type":"ContainerDied","Data":"ce8cc35a4a56bee6fc44abf612feab68019a1d9b9f48e6c7df1ba359914e91a6"} Oct 30 17:15:22.008213 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:22.008162 4522 scope.go:122] "RemoveContainer" containerID="45498dcf9a05230f8970fa3b864ae72a3202dbad46f44be35c3c840315369fc6" Oct 30 17:15:22.025690 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:22.025665 4522 scope.go:122] "RemoveContainer" containerID="534978f1c569305db3509f2184be134e9f106fc6d2cb0e68406e1c248bb1bbe9" Oct 30 17:15:22.049046 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:22.048974 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd5f560f-ltrhc"] Oct 30 17:15:22.058359 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:22.057480 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd5f560f-ltrhc"] Oct 30 17:15:22.080346 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:22.080283 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b2c3fb-29889675-rckqx"] Oct 30 17:15:22.099872 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:22.099739 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b2c3fb-29889675-rckqx"] Oct 30 17:15:22.202173 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:22.201702 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e4272a1-630d-41c7-bd48-bc0c58bd54c4" path="/var/lib/kubelet/pods/1e4272a1-630d-41c7-bd48-bc0c58bd54c4/volumes" Oct 30 17:15:22.202418 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:22.202290 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad604ad3-0fd0-4544-af4d-e5061bd561e3" path="/var/lib/kubelet/pods/ad604ad3-0fd0-4544-af4d-e5061bd561e3/volumes" Oct 30 17:15:22.203039 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:22.203015 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c53d8512-95ac-4d60-aaa6-0d74b95e0cf8" path="/var/lib/kubelet/pods/c53d8512-95ac-4d60-aaa6-0d74b95e0cf8/volumes" Oct 30 17:15:22.204087 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:22.204062 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5227ac7-04e8-4758-a1ed-1b907783431e" path="/var/lib/kubelet/pods/f5227ac7-04e8-4758-a1ed-1b907783431e/volumes" Oct 30 17:15:22.298275 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:22.298145 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889675-lcm6b" Oct 30 17:15:22.302364 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:22.302334 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889675-s24lj" Oct 30 17:15:22.302529 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:22.302418 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889675-fdxml"] Oct 30 17:15:22.307479 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:22.307456 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889675-fdxml"] Oct 30 17:15:22.310098 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:22.310076 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889675-dbh98" Oct 30 17:15:22.338516 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:22.338486 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-728ad7-29889675-hgghk"] Oct 30 17:15:22.340838 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:22.340811 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79a79c82-3142-4035-a072-004b80dfe24d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79a79c82-3142-4035-a072-004b80dfe24d-maas-api-service-ca\") pod \"79a79c82-3142-4035-a072-004b80dfe24d\" (UID: \"79a79c82-3142-4035-a072-004b80dfe24d\") " Oct 30 17:15:22.340980 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:22.340876 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79a79c82-3142-4035-a072-004b80dfe24d-kube-api-access-777l6\" (UniqueName: \"kubernetes.io/projected/79a79c82-3142-4035-a072-004b80dfe24d-kube-api-access-777l6\") pod \"79a79c82-3142-4035-a072-004b80dfe24d\" (UID: \"79a79c82-3142-4035-a072-004b80dfe24d\") " Oct 30 17:15:22.340980 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:22.340964 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78db2ae0-039b-4edc-8fdb-c4513041d366-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78db2ae0-039b-4edc-8fdb-c4513041d366-maas-api-service-ca\") pod \"78db2ae0-039b-4edc-8fdb-c4513041d366\" (UID: \"78db2ae0-039b-4edc-8fdb-c4513041d366\") " Oct 30 17:15:22.341085 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:22.341060 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d21c30e-1815-436a-92b5-4cb61b88ea19-kube-api-access-fdm4x\" (UniqueName: \"kubernetes.io/projected/5d21c30e-1815-436a-92b5-4cb61b88ea19-kube-api-access-fdm4x\") pod \"5d21c30e-1815-436a-92b5-4cb61b88ea19\" (UID: \"5d21c30e-1815-436a-92b5-4cb61b88ea19\") " Oct 30 17:15:22.341163 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:22.341140 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79a79c82-3142-4035-a072-004b80dfe24d-maas-api-service-ca" pod "79a79c82-3142-4035-a072-004b80dfe24d" (UID: "79a79c82-3142-4035-a072-004b80dfe24d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:22.341228 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:22.341197 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d21c30e-1815-436a-92b5-4cb61b88ea19-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d21c30e-1815-436a-92b5-4cb61b88ea19-maas-api-service-ca\") pod \"5d21c30e-1815-436a-92b5-4cb61b88ea19\" (UID: \"5d21c30e-1815-436a-92b5-4cb61b88ea19\") " Oct 30 17:15:22.341296 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:22.341256 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78db2ae0-039b-4edc-8fdb-c4513041d366-kube-api-access-5vqsr\" (UniqueName: \"kubernetes.io/projected/78db2ae0-039b-4edc-8fdb-c4513041d366-kube-api-access-5vqsr\") pod \"78db2ae0-039b-4edc-8fdb-c4513041d366\" (UID: \"78db2ae0-039b-4edc-8fdb-c4513041d366\") " Oct 30 17:15:22.341449 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:22.341427 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78db2ae0-039b-4edc-8fdb-c4513041d366-maas-api-service-ca" pod "78db2ae0-039b-4edc-8fdb-c4513041d366" (UID: "78db2ae0-039b-4edc-8fdb-c4513041d366"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:22.341538 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:22.341507 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d21c30e-1815-436a-92b5-4cb61b88ea19-maas-api-service-ca" pod "5d21c30e-1815-436a-92b5-4cb61b88ea19" (UID: "5d21c30e-1815-436a-92b5-4cb61b88ea19"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:22.341938 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:22.341911 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d21c30e-1815-436a-92b5-4cb61b88ea19-maas-api-service-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:15:22.342017 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:22.341955 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79a79c82-3142-4035-a072-004b80dfe24d-maas-api-service-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:15:22.342017 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:22.341965 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78db2ae0-039b-4edc-8fdb-c4513041d366-maas-api-service-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:15:22.343190 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:22.343162 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79a79c82-3142-4035-a072-004b80dfe24d-kube-api-access-777l6" pod "79a79c82-3142-4035-a072-004b80dfe24d" (UID: "79a79c82-3142-4035-a072-004b80dfe24d"). InnerVolumeSpecName "kube-api-access-777l6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:22.343497 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:22.343475 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-728ad7-29889675-hgghk"] Oct 30 17:15:22.343681 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:22.343662 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d21c30e-1815-436a-92b5-4cb61b88ea19-kube-api-access-fdm4x" pod "5d21c30e-1815-436a-92b5-4cb61b88ea19" (UID: "5d21c30e-1815-436a-92b5-4cb61b88ea19"). InnerVolumeSpecName "kube-api-access-fdm4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:22.343758 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:22.343694 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78db2ae0-039b-4edc-8fdb-c4513041d366-kube-api-access-5vqsr" pod "78db2ae0-039b-4edc-8fdb-c4513041d366" (UID: "78db2ae0-039b-4edc-8fdb-c4513041d366"). InnerVolumeSpecName "kube-api-access-5vqsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:22.443841 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:22.443806 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vqsr\" (UniqueName: \"kubernetes.io/projected/78db2ae0-039b-4edc-8fdb-c4513041d366-kube-api-access-5vqsr\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:15:22.443841 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:22.443837 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-777l6\" (UniqueName: \"kubernetes.io/projected/79a79c82-3142-4035-a072-004b80dfe24d-kube-api-access-777l6\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:15:22.443841 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:22.443847 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdm4x\" (UniqueName: \"kubernetes.io/projected/5d21c30e-1815-436a-92b5-4cb61b88ea19-kube-api-access-fdm4x\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:15:23.016368 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:23.016331 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889675-s24lj" event={"ID":"79a79c82-3142-4035-a072-004b80dfe24d","Type":"ContainerDied","Data":"a4aa0ad532b576b241769fd8be93d7e0c88e37d3cb3bd7a4316e173f669df52f"} Oct 30 17:15:23.016851 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:23.016402 4522 scope.go:122] "RemoveContainer" containerID="f323c270e29c07b10ab71fe52d1da87981e840401dfb4b66e98273d75da6686d" Oct 30 17:15:23.016851 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:23.016346 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889675-s24lj" Oct 30 17:15:23.018278 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:23.018259 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889675-lcm6b" Oct 30 17:15:23.018454 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:23.018293 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889675-lcm6b" event={"ID":"5d21c30e-1815-436a-92b5-4cb61b88ea19","Type":"ContainerDied","Data":"f83a7ac167fbb0b7bd4db1978f088e96dac54fe49b87686ae8dae544bfdd6202"} Oct 30 17:15:23.022525 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:23.022497 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889675-dbh98" event={"ID":"78db2ae0-039b-4edc-8fdb-c4513041d366","Type":"ContainerDied","Data":"7d64a3b39799decc2dd3ef9ebd7ff7aae72288209f5ca20362547df417a891dc"} Oct 30 17:15:23.022647 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:23.022512 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889675-dbh98" Oct 30 17:15:23.031394 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:23.031375 4522 scope.go:122] "RemoveContainer" containerID="912455fe151c913ee418158f237ca444e85ce41a6e07b0d98af7ca80985b191f" Oct 30 17:15:23.042015 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:23.041999 4522 scope.go:122] "RemoveContainer" containerID="3436edf1142de019623429681ce5b3a2eb6d1e84f9ddb7f43f9ad7b544fa03e0" Oct 30 17:15:23.091470 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:23.091441 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889675-s24lj"] Oct 30 17:15:23.095900 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:23.095878 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889675-s24lj"] Oct 30 17:15:23.134261 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:23.134229 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889675-lcm6b"] Oct 30 17:15:23.145125 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:23.145096 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889675-lcm6b"] Oct 30 17:15:23.181917 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:23.181881 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889675-dbh98"] Oct 30 17:15:23.188090 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:23.188059 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889675-dbh98"] Oct 30 17:15:24.148215 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:24.148178 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ea441b9-186a-4e43-b2c7-37052bd5dcd7" path="/var/lib/kubelet/pods/1ea441b9-186a-4e43-b2c7-37052bd5dcd7/volumes" Oct 30 17:15:24.148714 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:24.148699 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="417f0b10-2a38-4757-8563-d47b75088b4d" path="/var/lib/kubelet/pods/417f0b10-2a38-4757-8563-d47b75088b4d/volumes" Oct 30 17:15:24.149097 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:24.149084 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d21c30e-1815-436a-92b5-4cb61b88ea19" path="/var/lib/kubelet/pods/5d21c30e-1815-436a-92b5-4cb61b88ea19/volumes" Oct 30 17:15:24.149718 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:24.149701 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78db2ae0-039b-4edc-8fdb-c4513041d366" path="/var/lib/kubelet/pods/78db2ae0-039b-4edc-8fdb-c4513041d366/volumes" Oct 30 17:15:24.150113 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:24.150100 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79a79c82-3142-4035-a072-004b80dfe24d" path="/var/lib/kubelet/pods/79a79c82-3142-4035-a072-004b80dfe24d/volumes" Oct 30 17:15:27.838880 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:27.838844 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-728ad7-7b4857f67-lf57w" podUID="58921afe-0009-4d33-b402-0c708e26fe57" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:15:27.839522 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:27.839500 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-728ad7-66cb7bbbd6-dwrgr" podUID="3fe580bc-2cda-4b54-8b89-61ec5b147ab8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:15:35.239443 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.239402 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-728ad7-xvnc7"] Oct 30 17:15:35.249022 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.248979 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-728ad7-xvnc7"] Oct 30 17:15:35.270291 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.270247 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c82666-56fc4d955c-29j8s"] Oct 30 17:15:35.271393 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.271371 4522 cpu_manager.go:414] "containerMap: removing container" podUID="c53d8512-95ac-4d60-aaa6-0d74b95e0cf8" containerName="cleanup" Oct 30 17:15:35.271575 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.271561 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c53d8512-95ac-4d60-aaa6-0d74b95e0cf8" containerName="cleanup" Oct 30 17:15:35.271673 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.271655 4522 cpu_manager.go:414] "containerMap: removing container" podUID="5d21c30e-1815-436a-92b5-4cb61b88ea19" containerName="cleanup" Oct 30 17:15:35.271673 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.271665 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d21c30e-1815-436a-92b5-4cb61b88ea19" containerName="cleanup" Oct 30 17:15:35.271841 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.271682 4522 cpu_manager.go:414] "containerMap: removing container" podUID="1ea441b9-186a-4e43-b2c7-37052bd5dcd7" containerName="cleanup" Oct 30 17:15:35.271841 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.271690 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ea441b9-186a-4e43-b2c7-37052bd5dcd7" containerName="cleanup" Oct 30 17:15:35.271841 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.271704 4522 cpu_manager.go:414] "containerMap: removing container" podUID="5802774a-44fd-4660-b00e-8d979654b196" containerName="cleanup" Oct 30 17:15:35.271841 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.271712 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5802774a-44fd-4660-b00e-8d979654b196" containerName="cleanup" Oct 30 17:15:35.271841 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.271729 4522 cpu_manager.go:414] "containerMap: removing container" podUID="4e7e3bf8-d7a4-4839-bcf4-5161e1f998c9" containerName="cleanup" Oct 30 17:15:35.271841 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.271737 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e7e3bf8-d7a4-4839-bcf4-5161e1f998c9" containerName="cleanup" Oct 30 17:15:35.271841 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.271747 4522 cpu_manager.go:414] "containerMap: removing container" podUID="ad604ad3-0fd0-4544-af4d-e5061bd561e3" containerName="cleanup" Oct 30 17:15:35.271841 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.271754 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad604ad3-0fd0-4544-af4d-e5061bd561e3" containerName="cleanup" Oct 30 17:15:35.271841 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.271767 4522 cpu_manager.go:414] "containerMap: removing container" podUID="417f0b10-2a38-4757-8563-d47b75088b4d" containerName="cleanup" Oct 30 17:15:35.271841 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.271774 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="417f0b10-2a38-4757-8563-d47b75088b4d" containerName="cleanup" Oct 30 17:15:35.271841 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.271786 4522 cpu_manager.go:414] "containerMap: removing container" podUID="417f0b10-2a38-4757-8563-d47b75088b4d" containerName="cleanup" Oct 30 17:15:35.271841 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.271793 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="417f0b10-2a38-4757-8563-d47b75088b4d" containerName="cleanup" Oct 30 17:15:35.271841 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.271802 4522 cpu_manager.go:414] "containerMap: removing container" podUID="1ea441b9-186a-4e43-b2c7-37052bd5dcd7" containerName="cleanup" Oct 30 17:15:35.271841 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.271811 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ea441b9-186a-4e43-b2c7-37052bd5dcd7" containerName="cleanup" Oct 30 17:15:35.271841 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.271819 4522 cpu_manager.go:414] "containerMap: removing container" podUID="ad604ad3-0fd0-4544-af4d-e5061bd561e3" containerName="cleanup" Oct 30 17:15:35.271841 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.271826 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad604ad3-0fd0-4544-af4d-e5061bd561e3" containerName="cleanup" Oct 30 17:15:35.271841 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.271837 4522 cpu_manager.go:414] "containerMap: removing container" podUID="5802774a-44fd-4660-b00e-8d979654b196" containerName="cleanup" Oct 30 17:15:35.271841 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.271844 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5802774a-44fd-4660-b00e-8d979654b196" containerName="cleanup" Oct 30 17:15:35.272779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.271865 4522 cpu_manager.go:414] "containerMap: removing container" podUID="5d21c30e-1815-436a-92b5-4cb61b88ea19" containerName="cleanup" Oct 30 17:15:35.272779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.271872 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d21c30e-1815-436a-92b5-4cb61b88ea19" containerName="cleanup" Oct 30 17:15:35.272779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.271886 4522 cpu_manager.go:414] "containerMap: removing container" podUID="79a79c82-3142-4035-a072-004b80dfe24d" containerName="cleanup" Oct 30 17:15:35.272779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.271892 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79a79c82-3142-4035-a072-004b80dfe24d" containerName="cleanup" Oct 30 17:15:35.272779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.271906 4522 cpu_manager.go:414] "containerMap: removing container" podUID="79a79c82-3142-4035-a072-004b80dfe24d" containerName="cleanup" Oct 30 17:15:35.272779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.271913 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79a79c82-3142-4035-a072-004b80dfe24d" containerName="cleanup" Oct 30 17:15:35.272779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.271930 4522 cpu_manager.go:414] "containerMap: removing container" podUID="78db2ae0-039b-4edc-8fdb-c4513041d366" containerName="cleanup" Oct 30 17:15:35.272779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.271936 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78db2ae0-039b-4edc-8fdb-c4513041d366" containerName="cleanup" Oct 30 17:15:35.272779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.271945 4522 cpu_manager.go:414] "containerMap: removing container" podUID="171da860-d46a-420a-88a3-423d6b58c86b" containerName="cleanup" Oct 30 17:15:35.272779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.271952 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="171da860-d46a-420a-88a3-423d6b58c86b" containerName="cleanup" Oct 30 17:15:35.272779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.271962 4522 cpu_manager.go:414] "containerMap: removing container" podUID="5802774a-44fd-4660-b00e-8d979654b196" containerName="cleanup" Oct 30 17:15:35.272779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.271970 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5802774a-44fd-4660-b00e-8d979654b196" containerName="cleanup" Oct 30 17:15:35.272779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.271987 4522 cpu_manager.go:414] "containerMap: removing container" podUID="c53d8512-95ac-4d60-aaa6-0d74b95e0cf8" containerName="cleanup" Oct 30 17:15:35.272779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.271994 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c53d8512-95ac-4d60-aaa6-0d74b95e0cf8" containerName="cleanup" Oct 30 17:15:35.272779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.272007 4522 cpu_manager.go:414] "containerMap: removing container" podUID="79a79c82-3142-4035-a072-004b80dfe24d" containerName="cleanup" Oct 30 17:15:35.272779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.272014 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79a79c82-3142-4035-a072-004b80dfe24d" containerName="cleanup" Oct 30 17:15:35.272779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.272024 4522 cpu_manager.go:414] "containerMap: removing container" podUID="f5227ac7-04e8-4758-a1ed-1b907783431e" containerName="cleanup" Oct 30 17:15:35.272779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.272032 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5227ac7-04e8-4758-a1ed-1b907783431e" containerName="cleanup" Oct 30 17:15:35.272779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.272047 4522 cpu_manager.go:414] "containerMap: removing container" podUID="f5227ac7-04e8-4758-a1ed-1b907783431e" containerName="cleanup" Oct 30 17:15:35.272779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.272054 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5227ac7-04e8-4758-a1ed-1b907783431e" containerName="cleanup" Oct 30 17:15:35.272779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.272068 4522 cpu_manager.go:414] "containerMap: removing container" podUID="171da860-d46a-420a-88a3-423d6b58c86b" containerName="cleanup" Oct 30 17:15:35.272779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.272075 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="171da860-d46a-420a-88a3-423d6b58c86b" containerName="cleanup" Oct 30 17:15:35.272779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.272096 4522 cpu_manager.go:414] "containerMap: removing container" podUID="4e7e3bf8-d7a4-4839-bcf4-5161e1f998c9" containerName="cleanup" Oct 30 17:15:35.272779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.272103 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e7e3bf8-d7a4-4839-bcf4-5161e1f998c9" containerName="cleanup" Oct 30 17:15:35.272779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.272120 4522 cpu_manager.go:414] "containerMap: removing container" podUID="78db2ae0-039b-4edc-8fdb-c4513041d366" containerName="cleanup" Oct 30 17:15:35.272779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.272127 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78db2ae0-039b-4edc-8fdb-c4513041d366" containerName="cleanup" Oct 30 17:15:35.272779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.272136 4522 cpu_manager.go:414] "containerMap: removing container" podUID="1ea441b9-186a-4e43-b2c7-37052bd5dcd7" containerName="cleanup" Oct 30 17:15:35.272779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.272143 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ea441b9-186a-4e43-b2c7-37052bd5dcd7" containerName="cleanup" Oct 30 17:15:35.272779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.272180 4522 cpu_manager.go:414] "containerMap: removing container" podUID="f5227ac7-04e8-4758-a1ed-1b907783431e" containerName="cleanup" Oct 30 17:15:35.272779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.272188 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5227ac7-04e8-4758-a1ed-1b907783431e" containerName="cleanup" Oct 30 17:15:35.272779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.272423 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="5802774a-44fd-4660-b00e-8d979654b196" containerName="cleanup" Oct 30 17:15:35.272779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.272439 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e7e3bf8-d7a4-4839-bcf4-5161e1f998c9" containerName="cleanup" Oct 30 17:15:35.272779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.272451 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad604ad3-0fd0-4544-af4d-e5061bd561e3" containerName="cleanup" Oct 30 17:15:35.272779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.272462 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="417f0b10-2a38-4757-8563-d47b75088b4d" containerName="cleanup" Oct 30 17:15:35.272779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.272480 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="417f0b10-2a38-4757-8563-d47b75088b4d" containerName="cleanup" Oct 30 17:15:35.272779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.272490 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ea441b9-186a-4e43-b2c7-37052bd5dcd7" containerName="cleanup" Oct 30 17:15:35.272779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.272499 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad604ad3-0fd0-4544-af4d-e5061bd561e3" containerName="cleanup" Oct 30 17:15:35.272779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.272510 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="5802774a-44fd-4660-b00e-8d979654b196" containerName="cleanup" Oct 30 17:15:35.272779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.272521 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d21c30e-1815-436a-92b5-4cb61b88ea19" containerName="cleanup" Oct 30 17:15:35.272779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.272530 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="79a79c82-3142-4035-a072-004b80dfe24d" containerName="cleanup" Oct 30 17:15:35.272779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.272538 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="79a79c82-3142-4035-a072-004b80dfe24d" containerName="cleanup" Oct 30 17:15:35.272779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.272549 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="78db2ae0-039b-4edc-8fdb-c4513041d366" containerName="cleanup" Oct 30 17:15:35.272779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.272558 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="171da860-d46a-420a-88a3-423d6b58c86b" containerName="cleanup" Oct 30 17:15:35.272779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.272566 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e7e3bf8-d7a4-4839-bcf4-5161e1f998c9" containerName="cleanup" Oct 30 17:15:35.272779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.272574 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="c53d8512-95ac-4d60-aaa6-0d74b95e0cf8" containerName="cleanup" Oct 30 17:15:35.272779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.272581 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="78db2ae0-039b-4edc-8fdb-c4513041d366" containerName="cleanup" Oct 30 17:15:35.272779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.272591 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="79a79c82-3142-4035-a072-004b80dfe24d" containerName="cleanup" Oct 30 17:15:35.272779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.272598 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5227ac7-04e8-4758-a1ed-1b907783431e" containerName="cleanup" Oct 30 17:15:35.272779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.272608 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5227ac7-04e8-4758-a1ed-1b907783431e" containerName="cleanup" Oct 30 17:15:35.272779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.272619 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d21c30e-1815-436a-92b5-4cb61b88ea19" containerName="cleanup" Oct 30 17:15:35.272779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.272628 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="c53d8512-95ac-4d60-aaa6-0d74b95e0cf8" containerName="cleanup" Oct 30 17:15:35.272779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.272638 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="171da860-d46a-420a-88a3-423d6b58c86b" containerName="cleanup" Oct 30 17:15:35.272779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.272646 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="c53d8512-95ac-4d60-aaa6-0d74b95e0cf8" containerName="cleanup" Oct 30 17:15:35.272779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.272656 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ea441b9-186a-4e43-b2c7-37052bd5dcd7" containerName="cleanup" Oct 30 17:15:35.277415 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.277395 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c82666-56fc4d955c-29j8s" Oct 30 17:15:35.302073 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.302039 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-c82666-serving-cert\"" Oct 30 17:15:35.303086 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.303062 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c82666-56fc4d955c-29j8s"] Oct 30 17:15:35.469143 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.469103 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08e720cb-49b1-4520-bd37-3f791fb9c4b2-metrics-certs\") pod \"maas-api-e2e-inf-b-c82666-56fc4d955c-29j8s\" (UID: \"08e720cb-49b1-4520-bd37-3f791fb9c4b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c82666-56fc4d955c-29j8s" Oct 30 17:15:35.469363 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.469234 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr7fg\" (UniqueName: \"kubernetes.io/projected/08e720cb-49b1-4520-bd37-3f791fb9c4b2-kube-api-access-mr7fg\") pod \"maas-api-e2e-inf-b-c82666-56fc4d955c-29j8s\" (UID: \"08e720cb-49b1-4520-bd37-3f791fb9c4b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c82666-56fc4d955c-29j8s" Oct 30 17:15:35.469363 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.469338 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08e720cb-49b1-4520-bd37-3f791fb9c4b2-maas-api-tls\") pod \"maas-api-e2e-inf-b-c82666-56fc4d955c-29j8s\" (UID: \"08e720cb-49b1-4520-bd37-3f791fb9c4b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c82666-56fc4d955c-29j8s" Oct 30 17:15:35.469497 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.469447 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08e720cb-49b1-4520-bd37-3f791fb9c4b2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c82666-56fc4d955c-29j8s\" (UID: \"08e720cb-49b1-4520-bd37-3f791fb9c4b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c82666-56fc4d955c-29j8s" Oct 30 17:15:35.571295 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.571204 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08e720cb-49b1-4520-bd37-3f791fb9c4b2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c82666-56fc4d955c-29j8s\" (UID: \"08e720cb-49b1-4520-bd37-3f791fb9c4b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c82666-56fc4d955c-29j8s" Oct 30 17:15:35.571593 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.571570 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08e720cb-49b1-4520-bd37-3f791fb9c4b2-metrics-certs\") pod \"maas-api-e2e-inf-b-c82666-56fc4d955c-29j8s\" (UID: \"08e720cb-49b1-4520-bd37-3f791fb9c4b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c82666-56fc4d955c-29j8s" Oct 30 17:15:35.571700 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.571683 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr7fg\" (UniqueName: \"kubernetes.io/projected/08e720cb-49b1-4520-bd37-3f791fb9c4b2-kube-api-access-mr7fg\") pod \"maas-api-e2e-inf-b-c82666-56fc4d955c-29j8s\" (UID: \"08e720cb-49b1-4520-bd37-3f791fb9c4b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c82666-56fc4d955c-29j8s" Oct 30 17:15:35.571868 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.571846 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08e720cb-49b1-4520-bd37-3f791fb9c4b2-maas-api-tls\") pod \"maas-api-e2e-inf-b-c82666-56fc4d955c-29j8s\" (UID: \"08e720cb-49b1-4520-bd37-3f791fb9c4b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c82666-56fc4d955c-29j8s" Oct 30 17:15:35.571998 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.571976 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08e720cb-49b1-4520-bd37-3f791fb9c4b2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c82666-56fc4d955c-29j8s\" (UID: \"08e720cb-49b1-4520-bd37-3f791fb9c4b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c82666-56fc4d955c-29j8s" Oct 30 17:15:35.574524 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.574490 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08e720cb-49b1-4520-bd37-3f791fb9c4b2-metrics-certs\") pod \"maas-api-e2e-inf-b-c82666-56fc4d955c-29j8s\" (UID: \"08e720cb-49b1-4520-bd37-3f791fb9c4b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c82666-56fc4d955c-29j8s" Oct 30 17:15:35.574650 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.574579 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08e720cb-49b1-4520-bd37-3f791fb9c4b2-maas-api-tls\") pod \"maas-api-e2e-inf-b-c82666-56fc4d955c-29j8s\" (UID: \"08e720cb-49b1-4520-bd37-3f791fb9c4b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c82666-56fc4d955c-29j8s" Oct 30 17:15:35.599222 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.599168 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr7fg\" (UniqueName: \"kubernetes.io/projected/08e720cb-49b1-4520-bd37-3f791fb9c4b2-kube-api-access-mr7fg\") pod \"maas-api-e2e-inf-b-c82666-56fc4d955c-29j8s\" (UID: \"08e720cb-49b1-4520-bd37-3f791fb9c4b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c82666-56fc4d955c-29j8s" Oct 30 17:15:35.896934 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:35.896846 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c82666-56fc4d955c-29j8s" Oct 30 17:15:36.146816 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:36.146784 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="931cb7fe-faf6-4181-bee3-b3d1e3e61679" path="/var/lib/kubelet/pods/931cb7fe-faf6-4181-bee3-b3d1e3e61679/volumes" Oct 30 17:15:36.212985 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:15:36.212958 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08e720cb_49b1_4520_bd37_3f791fb9c4b2.slice/crio-7d569920b50822779eb4548245af5b7bedf9f01ca2f7a72fc00c77a419eb493a WatchSource:0}: Error finding container 7d569920b50822779eb4548245af5b7bedf9f01ca2f7a72fc00c77a419eb493a: Status 404 returned error can't find the container with id 7d569920b50822779eb4548245af5b7bedf9f01ca2f7a72fc00c77a419eb493a Oct 30 17:15:36.215267 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:36.215239 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c82666-56fc4d955c-29j8s"] Oct 30 17:15:36.982840 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:36.982408 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-728ad7-66cb7bbbd6-dwrgr" Oct 30 17:15:37.016782 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:37.016746 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fe580bc-2cda-4b54-8b89-61ec5b147ab8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fe580bc-2cda-4b54-8b89-61ec5b147ab8-plugins-config-volume\") pod \"3fe580bc-2cda-4b54-8b89-61ec5b147ab8\" (UID: \"3fe580bc-2cda-4b54-8b89-61ec5b147ab8\") " Oct 30 17:15:37.017061 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:37.016865 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fe580bc-2cda-4b54-8b89-61ec5b147ab8-kube-api-access-ttcj7\" (UniqueName: \"kubernetes.io/projected/3fe580bc-2cda-4b54-8b89-61ec5b147ab8-kube-api-access-ttcj7\") pod \"3fe580bc-2cda-4b54-8b89-61ec5b147ab8\" (UID: \"3fe580bc-2cda-4b54-8b89-61ec5b147ab8\") " Oct 30 17:15:37.017846 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:37.017772 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fe580bc-2cda-4b54-8b89-61ec5b147ab8-plugins-config-volume" pod "3fe580bc-2cda-4b54-8b89-61ec5b147ab8" (UID: "3fe580bc-2cda-4b54-8b89-61ec5b147ab8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:37.020654 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:37.020626 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fe580bc-2cda-4b54-8b89-61ec5b147ab8-kube-api-access-ttcj7" pod "3fe580bc-2cda-4b54-8b89-61ec5b147ab8" (UID: "3fe580bc-2cda-4b54-8b89-61ec5b147ab8"). InnerVolumeSpecName "kube-api-access-ttcj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:37.063935 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:37.063908 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-728ad7-7b4857f67-lf57w" Oct 30 17:15:37.118819 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:37.118781 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58921afe-0009-4d33-b402-0c708e26fe57-kube-api-access-cxmpr\" (UniqueName: \"kubernetes.io/projected/58921afe-0009-4d33-b402-0c708e26fe57-kube-api-access-cxmpr\") pod \"58921afe-0009-4d33-b402-0c708e26fe57\" (UID: \"58921afe-0009-4d33-b402-0c708e26fe57\") " Oct 30 17:15:37.118966 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:37.118913 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58921afe-0009-4d33-b402-0c708e26fe57-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58921afe-0009-4d33-b402-0c708e26fe57-plugins-config-volume\") pod \"58921afe-0009-4d33-b402-0c708e26fe57\" (UID: \"58921afe-0009-4d33-b402-0c708e26fe57\") " Oct 30 17:15:37.119265 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:37.119242 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58921afe-0009-4d33-b402-0c708e26fe57-plugins-config-volume" pod "58921afe-0009-4d33-b402-0c708e26fe57" (UID: "58921afe-0009-4d33-b402-0c708e26fe57"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:37.119645 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:37.119625 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58921afe-0009-4d33-b402-0c708e26fe57-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:15:37.119645 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:37.119644 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fe580bc-2cda-4b54-8b89-61ec5b147ab8-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:15:37.119790 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:37.119653 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttcj7\" (UniqueName: \"kubernetes.io/projected/3fe580bc-2cda-4b54-8b89-61ec5b147ab8-kube-api-access-ttcj7\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:15:37.120974 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:37.120954 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58921afe-0009-4d33-b402-0c708e26fe57-kube-api-access-cxmpr" pod "58921afe-0009-4d33-b402-0c708e26fe57" (UID: "58921afe-0009-4d33-b402-0c708e26fe57"). InnerVolumeSpecName "kube-api-access-cxmpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:37.148961 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:37.148923 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c82666-56fc4d955c-29j8s" event={"ID":"08e720cb-49b1-4520-bd37-3f791fb9c4b2","Type":"ContainerStarted","Data":"1cac4d23fa81e3dbbc96f5abceb3e0960d9629f83d6cdacfa6aba24be1592aa8"} Oct 30 17:15:37.149119 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:37.148989 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c82666-56fc4d955c-29j8s" event={"ID":"08e720cb-49b1-4520-bd37-3f791fb9c4b2","Type":"ContainerStarted","Data":"7d569920b50822779eb4548245af5b7bedf9f01ca2f7a72fc00c77a419eb493a"} Oct 30 17:15:37.149119 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:37.149021 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c82666-56fc4d955c-29j8s" Oct 30 17:15:37.150668 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:37.150644 4522 generic.go:356] "Generic (PLEG): container finished" podID="3fe580bc-2cda-4b54-8b89-61ec5b147ab8" containerID="b0f57b65d2cf0f86cf533b56d7049e9f9231f5ce7d8c4733857010950cc7ffce" exitCode=0 Oct 30 17:15:37.150805 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:37.150691 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-728ad7-66cb7bbbd6-dwrgr" Oct 30 17:15:37.150805 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:37.150726 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-728ad7-66cb7bbbd6-dwrgr" event={"ID":"3fe580bc-2cda-4b54-8b89-61ec5b147ab8","Type":"ContainerDied","Data":"b0f57b65d2cf0f86cf533b56d7049e9f9231f5ce7d8c4733857010950cc7ffce"} Oct 30 17:15:37.150805 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:37.150771 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-728ad7-66cb7bbbd6-dwrgr" event={"ID":"3fe580bc-2cda-4b54-8b89-61ec5b147ab8","Type":"ContainerDied","Data":"a0f6d71cfc99bd693ae7d72e349a606d45522a28f7646fcf37ad016a4204b7b2"} Oct 30 17:15:37.150955 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:37.150806 4522 scope.go:122] "RemoveContainer" containerID="b0f57b65d2cf0f86cf533b56d7049e9f9231f5ce7d8c4733857010950cc7ffce" Oct 30 17:15:37.152278 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:37.152256 4522 generic.go:356] "Generic (PLEG): container finished" podID="58921afe-0009-4d33-b402-0c708e26fe57" containerID="8948cacb23212173411a211fac0d68f5028f574dfb06f4d1034d8865b52fc210" exitCode=0 Oct 30 17:15:37.152516 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:37.152341 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-728ad7-7b4857f67-lf57w" Oct 30 17:15:37.152516 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:37.152355 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-728ad7-7b4857f67-lf57w" event={"ID":"58921afe-0009-4d33-b402-0c708e26fe57","Type":"ContainerDied","Data":"8948cacb23212173411a211fac0d68f5028f574dfb06f4d1034d8865b52fc210"} Oct 30 17:15:37.152516 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:37.152399 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-728ad7-7b4857f67-lf57w" event={"ID":"58921afe-0009-4d33-b402-0c708e26fe57","Type":"ContainerDied","Data":"ab12214f76d770e16a77ce6afb111923f1e1df0353e3f80f97bbbb6e0aac9af9"} Oct 30 17:15:37.165769 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:37.165742 4522 scope.go:122] "RemoveContainer" containerID="b0f57b65d2cf0f86cf533b56d7049e9f9231f5ce7d8c4733857010950cc7ffce" Oct 30 17:15:37.166108 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:15:37.166087 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0f57b65d2cf0f86cf533b56d7049e9f9231f5ce7d8c4733857010950cc7ffce\": container with ID starting with b0f57b65d2cf0f86cf533b56d7049e9f9231f5ce7d8c4733857010950cc7ffce not found: ID does not exist" containerID="b0f57b65d2cf0f86cf533b56d7049e9f9231f5ce7d8c4733857010950cc7ffce" Oct 30 17:15:37.166207 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:37.166113 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0f57b65d2cf0f86cf533b56d7049e9f9231f5ce7d8c4733857010950cc7ffce"} err="failed to get container status \"b0f57b65d2cf0f86cf533b56d7049e9f9231f5ce7d8c4733857010950cc7ffce\": rpc error: code = NotFound desc = could not find container \"b0f57b65d2cf0f86cf533b56d7049e9f9231f5ce7d8c4733857010950cc7ffce\": container with ID starting with b0f57b65d2cf0f86cf533b56d7049e9f9231f5ce7d8c4733857010950cc7ffce not found: ID does not exist" Oct 30 17:15:37.166207 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:37.166138 4522 scope.go:122] "RemoveContainer" containerID="8948cacb23212173411a211fac0d68f5028f574dfb06f4d1034d8865b52fc210" Oct 30 17:15:37.179727 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:37.179706 4522 scope.go:122] "RemoveContainer" containerID="8948cacb23212173411a211fac0d68f5028f574dfb06f4d1034d8865b52fc210" Oct 30 17:15:37.180008 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:15:37.179992 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8948cacb23212173411a211fac0d68f5028f574dfb06f4d1034d8865b52fc210\": container with ID starting with 8948cacb23212173411a211fac0d68f5028f574dfb06f4d1034d8865b52fc210 not found: ID does not exist" containerID="8948cacb23212173411a211fac0d68f5028f574dfb06f4d1034d8865b52fc210" Oct 30 17:15:37.180064 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:37.180014 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8948cacb23212173411a211fac0d68f5028f574dfb06f4d1034d8865b52fc210"} err="failed to get container status \"8948cacb23212173411a211fac0d68f5028f574dfb06f4d1034d8865b52fc210\": rpc error: code = NotFound desc = could not find container \"8948cacb23212173411a211fac0d68f5028f574dfb06f4d1034d8865b52fc210\": container with ID starting with 8948cacb23212173411a211fac0d68f5028f574dfb06f4d1034d8865b52fc210 not found: ID does not exist" Oct 30 17:15:37.196129 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:37.196085 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c82666-56fc4d955c-29j8s" podStartSLOduration=1.8418109729999999 podStartE2EDuration="2.196074568s" totalImagesPullingTime="354.263595ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:37.195033399 +0000 UTC m=+1711.195788299" watchObservedRunningTime="2026-10-30 17:15:37.196074568 +0000 UTC m=+1711.196829468" Oct 30 17:15:37.233256 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:37.233172 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxmpr\" (UniqueName: \"kubernetes.io/projected/58921afe-0009-4d33-b402-0c708e26fe57-kube-api-access-cxmpr\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:15:37.293551 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:37.293516 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-728ad7-7b4857f67-lf57w"] Oct 30 17:15:37.363872 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:37.363838 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-728ad7-7b4857f67-lf57w"] Oct 30 17:15:37.419741 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:37.419686 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-728ad7-66cb7bbbd6-dwrgr"] Oct 30 17:15:37.423981 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:37.423951 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-728ad7-66cb7bbbd6-dwrgr"] Oct 30 17:15:38.147946 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:38.147914 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fe580bc-2cda-4b54-8b89-61ec5b147ab8" path="/var/lib/kubelet/pods/3fe580bc-2cda-4b54-8b89-61ec5b147ab8/volumes" Oct 30 17:15:38.148411 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:38.148395 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58921afe-0009-4d33-b402-0c708e26fe57" path="/var/lib/kubelet/pods/58921afe-0009-4d33-b402-0c708e26fe57/volumes" Oct 30 17:15:42.955408 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:15:42.955359 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:15:43.209079 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:43.208993 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c82666-56fc4d955c-29j8s" Oct 30 17:15:43.395087 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:43.391353 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm"] Oct 30 17:15:43.395087 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:43.394178 4522 cpu_manager.go:414] "containerMap: removing container" podUID="3fe580bc-2cda-4b54-8b89-61ec5b147ab8" containerName="payload-processing" Oct 30 17:15:43.395087 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:43.394199 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fe580bc-2cda-4b54-8b89-61ec5b147ab8" containerName="payload-processing" Oct 30 17:15:43.395087 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:43.394265 4522 cpu_manager.go:414] "containerMap: removing container" podUID="78db2ae0-039b-4edc-8fdb-c4513041d366" containerName="cleanup" Oct 30 17:15:43.395087 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:43.394275 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78db2ae0-039b-4edc-8fdb-c4513041d366" containerName="cleanup" Oct 30 17:15:43.395087 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:43.394294 4522 cpu_manager.go:414] "containerMap: removing container" podUID="c53d8512-95ac-4d60-aaa6-0d74b95e0cf8" containerName="cleanup" Oct 30 17:15:43.395087 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:43.394322 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c53d8512-95ac-4d60-aaa6-0d74b95e0cf8" containerName="cleanup" Oct 30 17:15:43.395087 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:43.394354 4522 cpu_manager.go:414] "containerMap: removing container" podUID="171da860-d46a-420a-88a3-423d6b58c86b" containerName="cleanup" Oct 30 17:15:43.395087 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:43.394362 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="171da860-d46a-420a-88a3-423d6b58c86b" containerName="cleanup" Oct 30 17:15:43.395087 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:43.394413 4522 cpu_manager.go:414] "containerMap: removing container" podUID="58921afe-0009-4d33-b402-0c708e26fe57" containerName="payload-pre-processing" Oct 30 17:15:43.395087 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:43.394426 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58921afe-0009-4d33-b402-0c708e26fe57" containerName="payload-pre-processing" Oct 30 17:15:43.395087 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:43.394448 4522 cpu_manager.go:414] "containerMap: removing container" podUID="5d21c30e-1815-436a-92b5-4cb61b88ea19" containerName="cleanup" Oct 30 17:15:43.395087 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:43.394458 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d21c30e-1815-436a-92b5-4cb61b88ea19" containerName="cleanup" Oct 30 17:15:43.395087 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:43.394485 4522 cpu_manager.go:414] "containerMap: removing container" podUID="4e7e3bf8-d7a4-4839-bcf4-5161e1f998c9" containerName="cleanup" Oct 30 17:15:43.395087 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:43.394492 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e7e3bf8-d7a4-4839-bcf4-5161e1f998c9" containerName="cleanup" Oct 30 17:15:43.395087 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:43.394508 4522 cpu_manager.go:414] "containerMap: removing container" podUID="417f0b10-2a38-4757-8563-d47b75088b4d" containerName="cleanup" Oct 30 17:15:43.395087 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:43.394518 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="417f0b10-2a38-4757-8563-d47b75088b4d" containerName="cleanup" Oct 30 17:15:43.395087 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:43.394528 4522 cpu_manager.go:414] "containerMap: removing container" podUID="ad604ad3-0fd0-4544-af4d-e5061bd561e3" containerName="cleanup" Oct 30 17:15:43.395087 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:43.394534 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad604ad3-0fd0-4544-af4d-e5061bd561e3" containerName="cleanup" Oct 30 17:15:43.395087 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:43.394735 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="78db2ae0-039b-4edc-8fdb-c4513041d366" containerName="cleanup" Oct 30 17:15:43.395087 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:43.394756 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5227ac7-04e8-4758-a1ed-1b907783431e" containerName="cleanup" Oct 30 17:15:43.395087 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:43.394775 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="58921afe-0009-4d33-b402-0c708e26fe57" containerName="payload-pre-processing" Oct 30 17:15:43.395087 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:43.394791 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e7e3bf8-d7a4-4839-bcf4-5161e1f998c9" containerName="cleanup" Oct 30 17:15:43.395087 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:43.394812 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="417f0b10-2a38-4757-8563-d47b75088b4d" containerName="cleanup" Oct 30 17:15:43.395087 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:43.394820 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ea441b9-186a-4e43-b2c7-37052bd5dcd7" containerName="cleanup" Oct 30 17:15:43.395087 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:43.394830 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad604ad3-0fd0-4544-af4d-e5061bd561e3" containerName="cleanup" Oct 30 17:15:43.395087 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:43.394837 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="5802774a-44fd-4660-b00e-8d979654b196" containerName="cleanup" Oct 30 17:15:43.395087 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:43.394847 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fe580bc-2cda-4b54-8b89-61ec5b147ab8" containerName="payload-processing" Oct 30 17:15:43.395087 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:43.394857 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d21c30e-1815-436a-92b5-4cb61b88ea19" containerName="cleanup" Oct 30 17:15:43.395087 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:43.394870 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="171da860-d46a-420a-88a3-423d6b58c86b" containerName="cleanup" Oct 30 17:15:43.402753 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:43.402728 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm"] Oct 30 17:15:43.403042 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:43.403024 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm" Oct 30 17:15:43.446670 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:43.446639 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-82243b-openshift-default-dockercfg-c4f5c\"" Oct 30 17:15:43.484135 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:43.484021 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40baffd6-35b6-412d-a7bd-f3155ed306db-istiod-ca-cert\") pod \"e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm\" (UID: \"40baffd6-35b6-412d-a7bd-f3155ed306db\") " pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm" Oct 30 17:15:43.484135 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:43.484093 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40baffd6-35b6-412d-a7bd-f3155ed306db-istio-token\") pod \"e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm\" (UID: \"40baffd6-35b6-412d-a7bd-f3155ed306db\") " pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm" Oct 30 17:15:43.484496 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:43.484294 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40baffd6-35b6-412d-a7bd-f3155ed306db-workload-socket\") pod \"e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm\" (UID: \"40baffd6-35b6-412d-a7bd-f3155ed306db\") " pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm" Oct 30 17:15:43.484617 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:43.484552 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40baffd6-35b6-412d-a7bd-f3155ed306db-workload-certs\") pod \"e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm\" (UID: \"40baffd6-35b6-412d-a7bd-f3155ed306db\") " pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm" Oct 30 17:15:43.484787 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:43.484758 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwwvq\" (UniqueName: \"kubernetes.io/projected/40baffd6-35b6-412d-a7bd-f3155ed306db-kube-api-access-pwwvq\") pod \"e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm\" (UID: \"40baffd6-35b6-412d-a7bd-f3155ed306db\") " pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm" Oct 30 17:15:43.484921 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:43.484898 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40baffd6-35b6-412d-a7bd-f3155ed306db-istio-podinfo\") pod \"e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm\" (UID: \"40baffd6-35b6-412d-a7bd-f3155ed306db\") " pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm" Oct 30 17:15:43.485020 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:43.484991 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40baffd6-35b6-412d-a7bd-f3155ed306db-istio-envoy\") pod \"e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm\" (UID: \"40baffd6-35b6-412d-a7bd-f3155ed306db\") " pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm" Oct 30 17:15:43.485097 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:43.485079 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40baffd6-35b6-412d-a7bd-f3155ed306db-istio-data\") pod \"e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm\" (UID: \"40baffd6-35b6-412d-a7bd-f3155ed306db\") " pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm" Oct 30 17:15:43.485166 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:43.485126 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40baffd6-35b6-412d-a7bd-f3155ed306db-credential-socket\") pod \"e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm\" (UID: \"40baffd6-35b6-412d-a7bd-f3155ed306db\") " pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm" Oct 30 17:15:43.626424 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:43.623986 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40baffd6-35b6-412d-a7bd-f3155ed306db-workload-certs\") pod \"e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm\" (UID: \"40baffd6-35b6-412d-a7bd-f3155ed306db\") " pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm" Oct 30 17:15:43.626424 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:43.624277 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwwvq\" (UniqueName: \"kubernetes.io/projected/40baffd6-35b6-412d-a7bd-f3155ed306db-kube-api-access-pwwvq\") pod \"e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm\" (UID: \"40baffd6-35b6-412d-a7bd-f3155ed306db\") " pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm" Oct 30 17:15:43.626424 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:43.624426 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40baffd6-35b6-412d-a7bd-f3155ed306db-istio-podinfo\") pod \"e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm\" (UID: \"40baffd6-35b6-412d-a7bd-f3155ed306db\") " pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm" Oct 30 17:15:43.626424 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:43.624506 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40baffd6-35b6-412d-a7bd-f3155ed306db-istio-envoy\") pod \"e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm\" (UID: \"40baffd6-35b6-412d-a7bd-f3155ed306db\") " pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm" Oct 30 17:15:43.626424 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:43.624748 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40baffd6-35b6-412d-a7bd-f3155ed306db-istio-data\") pod \"e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm\" (UID: \"40baffd6-35b6-412d-a7bd-f3155ed306db\") " pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm" Oct 30 17:15:43.626424 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:43.624799 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40baffd6-35b6-412d-a7bd-f3155ed306db-credential-socket\") pod \"e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm\" (UID: \"40baffd6-35b6-412d-a7bd-f3155ed306db\") " pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm" Oct 30 17:15:43.626424 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:43.625034 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40baffd6-35b6-412d-a7bd-f3155ed306db-istiod-ca-cert\") pod \"e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm\" (UID: \"40baffd6-35b6-412d-a7bd-f3155ed306db\") " pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm" Oct 30 17:15:43.626424 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:43.625092 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40baffd6-35b6-412d-a7bd-f3155ed306db-istio-token\") pod \"e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm\" (UID: \"40baffd6-35b6-412d-a7bd-f3155ed306db\") " pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm" Oct 30 17:15:43.626424 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:43.625370 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40baffd6-35b6-412d-a7bd-f3155ed306db-workload-socket\") pod \"e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm\" (UID: \"40baffd6-35b6-412d-a7bd-f3155ed306db\") " pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm" Oct 30 17:15:43.627089 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:43.626942 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40baffd6-35b6-412d-a7bd-f3155ed306db-workload-certs\") pod \"e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm\" (UID: \"40baffd6-35b6-412d-a7bd-f3155ed306db\") " pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm" Oct 30 17:15:43.627089 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:43.626939 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40baffd6-35b6-412d-a7bd-f3155ed306db-credential-socket\") pod \"e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm\" (UID: \"40baffd6-35b6-412d-a7bd-f3155ed306db\") " pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm" Oct 30 17:15:43.630515 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:43.627804 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40baffd6-35b6-412d-a7bd-f3155ed306db-istio-data\") pod \"e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm\" (UID: \"40baffd6-35b6-412d-a7bd-f3155ed306db\") " pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm" Oct 30 17:15:43.630515 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:43.628295 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40baffd6-35b6-412d-a7bd-f3155ed306db-istiod-ca-cert\") pod \"e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm\" (UID: \"40baffd6-35b6-412d-a7bd-f3155ed306db\") " pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm" Oct 30 17:15:43.637512 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:43.637475 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40baffd6-35b6-412d-a7bd-f3155ed306db-workload-socket\") pod \"e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm\" (UID: \"40baffd6-35b6-412d-a7bd-f3155ed306db\") " pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm" Oct 30 17:15:43.653993 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:43.653952 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40baffd6-35b6-412d-a7bd-f3155ed306db-istio-podinfo\") pod \"e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm\" (UID: \"40baffd6-35b6-412d-a7bd-f3155ed306db\") " pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm" Oct 30 17:15:43.655880 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:43.655444 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40baffd6-35b6-412d-a7bd-f3155ed306db-istio-envoy\") pod \"e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm\" (UID: \"40baffd6-35b6-412d-a7bd-f3155ed306db\") " pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm" Oct 30 17:15:43.656071 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:43.656026 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40baffd6-35b6-412d-a7bd-f3155ed306db-istio-token\") pod \"e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm\" (UID: \"40baffd6-35b6-412d-a7bd-f3155ed306db\") " pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm" Oct 30 17:15:43.658624 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:43.658599 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwwvq\" (UniqueName: \"kubernetes.io/projected/40baffd6-35b6-412d-a7bd-f3155ed306db-kube-api-access-pwwvq\") pod \"e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm\" (UID: \"40baffd6-35b6-412d-a7bd-f3155ed306db\") " pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm" Oct 30 17:15:43.723584 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:43.723545 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm" Oct 30 17:15:44.070079 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:15:44.068871 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40baffd6_35b6_412d_a7bd_f3155ed306db.slice/crio-3e26e774e771a4a9cbdae1d769844d202187b3583f81ad3661ed9b499af96c72 WatchSource:0}: Error finding container 3e26e774e771a4a9cbdae1d769844d202187b3583f81ad3661ed9b499af96c72: Status 404 returned error can't find the container with id 3e26e774e771a4a9cbdae1d769844d202187b3583f81ad3661ed9b499af96c72 Oct 30 17:15:44.070079 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:44.069892 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm"] Oct 30 17:15:44.073231 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:44.073200 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:15:44.073410 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:44.073268 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:15:44.073410 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:44.073297 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:15:44.316394 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:44.316225 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm" event={"ID":"40baffd6-35b6-412d-a7bd-f3155ed306db","Type":"ContainerStarted","Data":"674dcc578e0637f40a21dfbc09d9216f80795539a9e7fd2fd1c29bf09d61fc05"} Oct 30 17:15:44.316394 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:44.316293 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm" event={"ID":"40baffd6-35b6-412d-a7bd-f3155ed306db","Type":"ContainerStarted","Data":"3e26e774e771a4a9cbdae1d769844d202187b3583f81ad3661ed9b499af96c72"} Oct 30 17:15:44.396145 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:44.396089 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm" podStartSLOduration=1.3960756220000001 podStartE2EDuration="1.396075622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:44.391729244 +0000 UTC m=+1718.392484145" watchObservedRunningTime="2026-10-30 17:15:44.396075622 +0000 UTC m=+1718.396830522" Oct 30 17:15:44.724528 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:44.724491 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm" Oct 30 17:15:45.311399 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:45.311359 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97"] Oct 30 17:15:45.311914 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:45.311564 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97" podUID="ed5776a0-2912-446d-a83f-8c84a75e5cbf" containerName="istio-proxy" containerID="cri-o://3745dd62c7c27502c535aa6beaeac2478495fdc937d908cd054412b9f4b5d220" gracePeriod=30 Oct 30 17:15:45.730645 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:45.730604 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm" Oct 30 17:15:45.992659 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:45.992576 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39fecf-2gx5j"] Oct 30 17:15:45.997663 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:45.997614 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39fecf-2gx5j" Oct 30 17:15:46.014046 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:46.014021 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:15:46.014428 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:46.014381 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:15:46.018733 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:46.018702 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39fecf-2gx5j"] Oct 30 17:15:46.085831 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:46.085796 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkpgn\" (UniqueName: \"kubernetes.io/projected/35e117d2-0492-4c63-8af7-2aaa14e7edb3-kube-api-access-qkpgn\") pod \"maas-api-revoke-keys-e2e-inf-b-39fecf-2gx5j\" (UID: \"35e117d2-0492-4c63-8af7-2aaa14e7edb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39fecf-2gx5j" Oct 30 17:15:46.086065 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:46.086045 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35e117d2-0492-4c63-8af7-2aaa14e7edb3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-39fecf-2gx5j\" (UID: \"35e117d2-0492-4c63-8af7-2aaa14e7edb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39fecf-2gx5j" Oct 30 17:15:46.086117 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:46.086092 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35e117d2-0492-4c63-8af7-2aaa14e7edb3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-39fecf-2gx5j\" (UID: \"35e117d2-0492-4c63-8af7-2aaa14e7edb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39fecf-2gx5j" Oct 30 17:15:46.187572 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:46.187528 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkpgn\" (UniqueName: \"kubernetes.io/projected/35e117d2-0492-4c63-8af7-2aaa14e7edb3-kube-api-access-qkpgn\") pod \"maas-api-revoke-keys-e2e-inf-b-39fecf-2gx5j\" (UID: \"35e117d2-0492-4c63-8af7-2aaa14e7edb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39fecf-2gx5j" Oct 30 17:15:46.187813 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:46.187788 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35e117d2-0492-4c63-8af7-2aaa14e7edb3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-39fecf-2gx5j\" (UID: \"35e117d2-0492-4c63-8af7-2aaa14e7edb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39fecf-2gx5j" Oct 30 17:15:46.187900 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:46.187845 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35e117d2-0492-4c63-8af7-2aaa14e7edb3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-39fecf-2gx5j\" (UID: \"35e117d2-0492-4c63-8af7-2aaa14e7edb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39fecf-2gx5j" Oct 30 17:15:46.188553 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:46.188526 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35e117d2-0492-4c63-8af7-2aaa14e7edb3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-39fecf-2gx5j\" (UID: \"35e117d2-0492-4c63-8af7-2aaa14e7edb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39fecf-2gx5j" Oct 30 17:15:46.190531 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:46.190504 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35e117d2-0492-4c63-8af7-2aaa14e7edb3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-39fecf-2gx5j\" (UID: \"35e117d2-0492-4c63-8af7-2aaa14e7edb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39fecf-2gx5j" Oct 30 17:15:46.213643 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:46.213612 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkpgn\" (UniqueName: \"kubernetes.io/projected/35e117d2-0492-4c63-8af7-2aaa14e7edb3-kube-api-access-qkpgn\") pod \"maas-api-revoke-keys-e2e-inf-b-39fecf-2gx5j\" (UID: \"35e117d2-0492-4c63-8af7-2aaa14e7edb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39fecf-2gx5j" Oct 30 17:15:46.316608 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:46.316525 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39fecf-2gx5j" Oct 30 17:15:46.339074 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:46.339043 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm" Oct 30 17:15:46.339876 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:46.339854 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm" Oct 30 17:15:46.628981 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:15:46.628947 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35e117d2_0492_4c63_8af7_2aaa14e7edb3.slice/crio-46aff29beaab008f35b012a0ca19edf2983b91ede3147de0205ba6b32ca8d37b WatchSource:0}: Error finding container 46aff29beaab008f35b012a0ca19edf2983b91ede3147de0205ba6b32ca8d37b: Status 404 returned error can't find the container with id 46aff29beaab008f35b012a0ca19edf2983b91ede3147de0205ba6b32ca8d37b Oct 30 17:15:46.632438 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:46.632380 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39fecf-2gx5j"] Oct 30 17:15:47.348608 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:47.348572 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39fecf-2gx5j" event={"ID":"35e117d2-0492-4c63-8af7-2aaa14e7edb3","Type":"ContainerStarted","Data":"438806e0d021b193c6e01b30ef5233fa7cfb200b307f4f2e11a61590594e2faa"} Oct 30 17:15:47.349003 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:47.348633 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39fecf-2gx5j" event={"ID":"35e117d2-0492-4c63-8af7-2aaa14e7edb3","Type":"ContainerStarted","Data":"46aff29beaab008f35b012a0ca19edf2983b91ede3147de0205ba6b32ca8d37b"} Oct 30 17:15:47.401737 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:47.401677 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39fecf-2gx5j" podStartSLOduration=2.401661571 podStartE2EDuration="2.401661571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:47.398341312 +0000 UTC m=+1721.399096212" watchObservedRunningTime="2026-10-30 17:15:47.401661571 +0000 UTC m=+1721.402416471" Oct 30 17:15:47.780488 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:47.780442 4522 patch_prober.go:29] interesting pod/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:15:47.780657 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:47.780525 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97" podUID="ed5776a0-2912-446d-a83f-8c84a75e5cbf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:15:48.358637 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:48.358606 4522 generic.go:356] "Generic (PLEG): container finished" podID="35e117d2-0492-4c63-8af7-2aaa14e7edb3" containerID="438806e0d021b193c6e01b30ef5233fa7cfb200b307f4f2e11a61590594e2faa" exitCode=0 Oct 30 17:15:48.359030 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:48.358692 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39fecf-2gx5j" event={"ID":"35e117d2-0492-4c63-8af7-2aaa14e7edb3","Type":"ContainerDied","Data":"438806e0d021b193c6e01b30ef5233fa7cfb200b307f4f2e11a61590594e2faa"} Oct 30 17:15:49.535989 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:49.535961 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39fecf-2gx5j" Oct 30 17:15:49.655668 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:49.655579 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35e117d2-0492-4c63-8af7-2aaa14e7edb3-kube-api-access-qkpgn\" (UniqueName: \"kubernetes.io/projected/35e117d2-0492-4c63-8af7-2aaa14e7edb3-kube-api-access-qkpgn\") pod \"35e117d2-0492-4c63-8af7-2aaa14e7edb3\" (UID: \"35e117d2-0492-4c63-8af7-2aaa14e7edb3\") " Oct 30 17:15:49.655844 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:49.655818 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35e117d2-0492-4c63-8af7-2aaa14e7edb3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35e117d2-0492-4c63-8af7-2aaa14e7edb3-maas-api-xks-ca\") pod \"35e117d2-0492-4c63-8af7-2aaa14e7edb3\" (UID: \"35e117d2-0492-4c63-8af7-2aaa14e7edb3\") " Oct 30 17:15:49.655917 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:49.655879 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35e117d2-0492-4c63-8af7-2aaa14e7edb3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35e117d2-0492-4c63-8af7-2aaa14e7edb3-maas-api-openshift-ca\") pod \"35e117d2-0492-4c63-8af7-2aaa14e7edb3\" (UID: \"35e117d2-0492-4c63-8af7-2aaa14e7edb3\") " Oct 30 17:15:49.656280 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:49.656253 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35e117d2-0492-4c63-8af7-2aaa14e7edb3-maas-api-openshift-ca" pod "35e117d2-0492-4c63-8af7-2aaa14e7edb3" (UID: "35e117d2-0492-4c63-8af7-2aaa14e7edb3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:49.656693 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:49.656674 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35e117d2-0492-4c63-8af7-2aaa14e7edb3-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:15:49.657910 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:49.657882 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35e117d2-0492-4c63-8af7-2aaa14e7edb3-maas-api-xks-ca" pod "35e117d2-0492-4c63-8af7-2aaa14e7edb3" (UID: "35e117d2-0492-4c63-8af7-2aaa14e7edb3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:15:49.658017 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:49.657941 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35e117d2-0492-4c63-8af7-2aaa14e7edb3-kube-api-access-qkpgn" pod "35e117d2-0492-4c63-8af7-2aaa14e7edb3" (UID: "35e117d2-0492-4c63-8af7-2aaa14e7edb3"). InnerVolumeSpecName "kube-api-access-qkpgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:49.758758 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:49.758717 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35e117d2-0492-4c63-8af7-2aaa14e7edb3-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:15:49.758758 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:49.758756 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkpgn\" (UniqueName: \"kubernetes.io/projected/35e117d2-0492-4c63-8af7-2aaa14e7edb3-kube-api-access-qkpgn\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:15:49.878656 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:49.878620 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-82243b-6794bbb67-58ssq"] Oct 30 17:15:49.879563 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:49.879536 4522 cpu_manager.go:414] "containerMap: removing container" podUID="35e117d2-0492-4c63-8af7-2aaa14e7edb3" containerName="revoke-keys" Oct 30 17:15:49.879563 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:49.879560 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35e117d2-0492-4c63-8af7-2aaa14e7edb3" containerName="revoke-keys" Oct 30 17:15:49.879816 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:49.879798 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="35e117d2-0492-4c63-8af7-2aaa14e7edb3" containerName="revoke-keys" Oct 30 17:15:49.883797 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:49.883779 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-82243b-6794bbb67-58ssq" Oct 30 17:15:49.904000 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:49.903974 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-82243b\"" Oct 30 17:15:49.904205 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:49.904181 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-82243b-dockercfg-dq9c8\"" Oct 30 17:15:49.907993 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:49.907928 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-82243b-6794bbb67-58ssq"] Oct 30 17:15:49.949622 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:49.949582 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-82243b-7f4f9546b6-mg8hr"] Oct 30 17:15:49.954406 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:49.954385 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-82243b-7f4f9546b6-mg8hr" Oct 30 17:15:49.974689 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:49.974667 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-82243b-7f4f9546b6-mg8hr"] Oct 30 17:15:50.067360 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:50.067159 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5333c134-0e1a-4096-a6d0-6d54f2373e28-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-82243b-7f4f9546b6-mg8hr\" (UID: \"5333c134-0e1a-4096-a6d0-6d54f2373e28\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-82243b-7f4f9546b6-mg8hr" Oct 30 17:15:50.067561 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:50.067383 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l2tw\" (UniqueName: \"kubernetes.io/projected/5333c134-0e1a-4096-a6d0-6d54f2373e28-kube-api-access-8l2tw\") pod \"payload-processing-e2e-shared-b-w5-82243b-7f4f9546b6-mg8hr\" (UID: \"5333c134-0e1a-4096-a6d0-6d54f2373e28\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-82243b-7f4f9546b6-mg8hr" Oct 30 17:15:50.072653 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:50.067473 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bdabfbc-c999-479c-a44e-d4949bd3c73d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-82243b-6794bbb67-58ssq\" (UID: \"5bdabfbc-c999-479c-a44e-d4949bd3c73d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-82243b-6794bbb67-58ssq" Oct 30 17:15:50.072653 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:50.068770 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6jtl\" (UniqueName: \"kubernetes.io/projected/5bdabfbc-c999-479c-a44e-d4949bd3c73d-kube-api-access-m6jtl\") pod \"payload-pre-processing-e2e-shared-b-w5-82243b-6794bbb67-58ssq\" (UID: \"5bdabfbc-c999-479c-a44e-d4949bd3c73d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-82243b-6794bbb67-58ssq" Oct 30 17:15:50.201268 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:50.170253 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6jtl\" (UniqueName: \"kubernetes.io/projected/5bdabfbc-c999-479c-a44e-d4949bd3c73d-kube-api-access-m6jtl\") pod \"payload-pre-processing-e2e-shared-b-w5-82243b-6794bbb67-58ssq\" (UID: \"5bdabfbc-c999-479c-a44e-d4949bd3c73d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-82243b-6794bbb67-58ssq" Oct 30 17:15:50.201268 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:50.199438 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5333c134-0e1a-4096-a6d0-6d54f2373e28-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-82243b-7f4f9546b6-mg8hr\" (UID: \"5333c134-0e1a-4096-a6d0-6d54f2373e28\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-82243b-7f4f9546b6-mg8hr" Oct 30 17:15:50.201268 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:50.199776 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l2tw\" (UniqueName: \"kubernetes.io/projected/5333c134-0e1a-4096-a6d0-6d54f2373e28-kube-api-access-8l2tw\") pod \"payload-processing-e2e-shared-b-w5-82243b-7f4f9546b6-mg8hr\" (UID: \"5333c134-0e1a-4096-a6d0-6d54f2373e28\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-82243b-7f4f9546b6-mg8hr" Oct 30 17:15:50.201268 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:50.199844 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bdabfbc-c999-479c-a44e-d4949bd3c73d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-82243b-6794bbb67-58ssq\" (UID: \"5bdabfbc-c999-479c-a44e-d4949bd3c73d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-82243b-6794bbb67-58ssq" Oct 30 17:15:50.203289 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:50.203253 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5333c134-0e1a-4096-a6d0-6d54f2373e28-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-82243b-7f4f9546b6-mg8hr\" (UID: \"5333c134-0e1a-4096-a6d0-6d54f2373e28\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-82243b-7f4f9546b6-mg8hr" Oct 30 17:15:50.213846 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:50.211722 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bdabfbc-c999-479c-a44e-d4949bd3c73d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-82243b-6794bbb67-58ssq\" (UID: \"5bdabfbc-c999-479c-a44e-d4949bd3c73d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-82243b-6794bbb67-58ssq" Oct 30 17:15:50.231887 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:50.231860 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l2tw\" (UniqueName: \"kubernetes.io/projected/5333c134-0e1a-4096-a6d0-6d54f2373e28-kube-api-access-8l2tw\") pod \"payload-processing-e2e-shared-b-w5-82243b-7f4f9546b6-mg8hr\" (UID: \"5333c134-0e1a-4096-a6d0-6d54f2373e28\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-82243b-7f4f9546b6-mg8hr" Oct 30 17:15:50.233168 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:50.233147 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6jtl\" (UniqueName: \"kubernetes.io/projected/5bdabfbc-c999-479c-a44e-d4949bd3c73d-kube-api-access-m6jtl\") pod \"payload-pre-processing-e2e-shared-b-w5-82243b-6794bbb67-58ssq\" (UID: \"5bdabfbc-c999-479c-a44e-d4949bd3c73d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-82243b-6794bbb67-58ssq" Oct 30 17:15:50.269484 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:50.269451 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-82243b-7f4f9546b6-mg8hr" Oct 30 17:15:50.390435 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:50.390396 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39fecf-2gx5j" event={"ID":"35e117d2-0492-4c63-8af7-2aaa14e7edb3","Type":"ContainerDied","Data":"46aff29beaab008f35b012a0ca19edf2983b91ede3147de0205ba6b32ca8d37b"} Oct 30 17:15:50.390435 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:50.390436 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39fecf-2gx5j" Oct 30 17:15:50.390639 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:50.390459 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46aff29beaab008f35b012a0ca19edf2983b91ede3147de0205ba6b32ca8d37b" Oct 30 17:15:50.504346 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:50.503766 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-82243b-6794bbb67-58ssq" Oct 30 17:15:50.603219 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:50.603183 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-82243b-7f4f9546b6-mg8hr"] Oct 30 17:15:50.672537 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:50.672506 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97" Oct 30 17:15:50.711653 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:50.711624 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed5776a0-2912-446d-a83f-8c84a75e5cbf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed5776a0-2912-446d-a83f-8c84a75e5cbf-istio-data\") pod \"ed5776a0-2912-446d-a83f-8c84a75e5cbf\" (UID: \"ed5776a0-2912-446d-a83f-8c84a75e5cbf\") " Oct 30 17:15:50.711800 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:50.711690 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed5776a0-2912-446d-a83f-8c84a75e5cbf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed5776a0-2912-446d-a83f-8c84a75e5cbf-istiod-ca-cert\") pod \"ed5776a0-2912-446d-a83f-8c84a75e5cbf\" (UID: \"ed5776a0-2912-446d-a83f-8c84a75e5cbf\") " Oct 30 17:15:50.711800 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:50.711736 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed5776a0-2912-446d-a83f-8c84a75e5cbf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed5776a0-2912-446d-a83f-8c84a75e5cbf-workload-certs\") pod \"ed5776a0-2912-446d-a83f-8c84a75e5cbf\" (UID: \"ed5776a0-2912-446d-a83f-8c84a75e5cbf\") " Oct 30 17:15:50.711932 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:50.711826 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed5776a0-2912-446d-a83f-8c84a75e5cbf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed5776a0-2912-446d-a83f-8c84a75e5cbf-credential-socket\") pod \"ed5776a0-2912-446d-a83f-8c84a75e5cbf\" (UID: \"ed5776a0-2912-446d-a83f-8c84a75e5cbf\") " Oct 30 17:15:50.711932 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:50.711915 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed5776a0-2912-446d-a83f-8c84a75e5cbf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed5776a0-2912-446d-a83f-8c84a75e5cbf-istio-podinfo\") pod \"ed5776a0-2912-446d-a83f-8c84a75e5cbf\" (UID: \"ed5776a0-2912-446d-a83f-8c84a75e5cbf\") " Oct 30 17:15:50.712049 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:50.711956 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed5776a0-2912-446d-a83f-8c84a75e5cbf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed5776a0-2912-446d-a83f-8c84a75e5cbf-workload-socket\") pod \"ed5776a0-2912-446d-a83f-8c84a75e5cbf\" (UID: \"ed5776a0-2912-446d-a83f-8c84a75e5cbf\") " Oct 30 17:15:50.712049 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:50.712043 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed5776a0-2912-446d-a83f-8c84a75e5cbf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed5776a0-2912-446d-a83f-8c84a75e5cbf-istio-envoy\") pod \"ed5776a0-2912-446d-a83f-8c84a75e5cbf\" (UID: \"ed5776a0-2912-446d-a83f-8c84a75e5cbf\") " Oct 30 17:15:50.712172 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:50.712076 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed5776a0-2912-446d-a83f-8c84a75e5cbf-kube-api-access-2h6wg\" (UniqueName: \"kubernetes.io/projected/ed5776a0-2912-446d-a83f-8c84a75e5cbf-kube-api-access-2h6wg\") pod \"ed5776a0-2912-446d-a83f-8c84a75e5cbf\" (UID: \"ed5776a0-2912-446d-a83f-8c84a75e5cbf\") " Oct 30 17:15:50.712172 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:50.712138 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed5776a0-2912-446d-a83f-8c84a75e5cbf-workload-certs" pod "ed5776a0-2912-446d-a83f-8c84a75e5cbf" (UID: "ed5776a0-2912-446d-a83f-8c84a75e5cbf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:50.712288 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:50.712232 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed5776a0-2912-446d-a83f-8c84a75e5cbf-istiod-ca-cert" pod "ed5776a0-2912-446d-a83f-8c84a75e5cbf" (UID: "ed5776a0-2912-446d-a83f-8c84a75e5cbf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:50.712585 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:50.712546 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed5776a0-2912-446d-a83f-8c84a75e5cbf-workload-socket" pod "ed5776a0-2912-446d-a83f-8c84a75e5cbf" (UID: "ed5776a0-2912-446d-a83f-8c84a75e5cbf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:50.712711 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:50.712604 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed5776a0-2912-446d-a83f-8c84a75e5cbf-istio-token\" (UniqueName: \"kubernetes.io/projected/ed5776a0-2912-446d-a83f-8c84a75e5cbf-istio-token\") pod \"ed5776a0-2912-446d-a83f-8c84a75e5cbf\" (UID: \"ed5776a0-2912-446d-a83f-8c84a75e5cbf\") " Oct 30 17:15:50.712711 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:50.712612 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed5776a0-2912-446d-a83f-8c84a75e5cbf-istio-data" pod "ed5776a0-2912-446d-a83f-8c84a75e5cbf" (UID: "ed5776a0-2912-446d-a83f-8c84a75e5cbf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:50.712711 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:50.712688 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed5776a0-2912-446d-a83f-8c84a75e5cbf-credential-socket" pod "ed5776a0-2912-446d-a83f-8c84a75e5cbf" (UID: "ed5776a0-2912-446d-a83f-8c84a75e5cbf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:50.713628 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:50.713604 4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed5776a0-2912-446d-a83f-8c84a75e5cbf-istio-data\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:15:50.713732 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:50.713633 4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed5776a0-2912-446d-a83f-8c84a75e5cbf-istiod-ca-cert\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:15:50.713732 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:50.713648 4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed5776a0-2912-446d-a83f-8c84a75e5cbf-workload-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:15:50.713732 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:50.713661 4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed5776a0-2912-446d-a83f-8c84a75e5cbf-credential-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:15:50.713732 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:50.713673 4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed5776a0-2912-446d-a83f-8c84a75e5cbf-workload-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:15:50.714918 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:50.714861 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed5776a0-2912-446d-a83f-8c84a75e5cbf-kube-api-access-2h6wg" pod "ed5776a0-2912-446d-a83f-8c84a75e5cbf" (UID: "ed5776a0-2912-446d-a83f-8c84a75e5cbf"). InnerVolumeSpecName "kube-api-access-2h6wg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:50.715164 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:50.715131 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed5776a0-2912-446d-a83f-8c84a75e5cbf-istio-token" pod "ed5776a0-2912-446d-a83f-8c84a75e5cbf" (UID: "ed5776a0-2912-446d-a83f-8c84a75e5cbf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:50.715579 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:50.715480 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed5776a0-2912-446d-a83f-8c84a75e5cbf-istio-envoy" pod "ed5776a0-2912-446d-a83f-8c84a75e5cbf" (UID: "ed5776a0-2912-446d-a83f-8c84a75e5cbf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:50.715980 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:50.715960 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed5776a0-2912-446d-a83f-8c84a75e5cbf-istio-podinfo" pod "ed5776a0-2912-446d-a83f-8c84a75e5cbf" (UID: "ed5776a0-2912-446d-a83f-8c84a75e5cbf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:15:50.815333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:50.815218 4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed5776a0-2912-446d-a83f-8c84a75e5cbf-istio-token\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:15:50.815333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:50.815253 4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed5776a0-2912-446d-a83f-8c84a75e5cbf-istio-podinfo\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:15:50.815333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:50.815267 4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed5776a0-2912-446d-a83f-8c84a75e5cbf-istio-envoy\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:15:50.815333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:50.815281 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h6wg\" (UniqueName: \"kubernetes.io/projected/ed5776a0-2912-446d-a83f-8c84a75e5cbf-kube-api-access-2h6wg\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:15:50.854108 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:15:50.854078 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bdabfbc_c999_479c_a44e_d4949bd3c73d.slice/crio-f2c04590fc9684b79f217f9788e7ecac19d14ac60f2c7478bfb1d04488f31ac1 WatchSource:0}: Error finding container f2c04590fc9684b79f217f9788e7ecac19d14ac60f2c7478bfb1d04488f31ac1: Status 404 returned error can't find the container with id f2c04590fc9684b79f217f9788e7ecac19d14ac60f2c7478bfb1d04488f31ac1 Oct 30 17:15:50.856824 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:50.856794 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-82243b-6794bbb67-58ssq"] Oct 30 17:15:51.400559 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:51.400521 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-82243b-7f4f9546b6-mg8hr" event={"ID":"5333c134-0e1a-4096-a6d0-6d54f2373e28","Type":"ContainerStarted","Data":"41112eefe8cb88efafdc400f68940bf161a6d97bb4d068ea8c498160480ed412"} Oct 30 17:15:51.400907 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:51.400573 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-82243b-7f4f9546b6-mg8hr" event={"ID":"5333c134-0e1a-4096-a6d0-6d54f2373e28","Type":"ContainerStarted","Data":"6df712a7483e3401ff802366868309605064ccbc8dfc88fbea0c5fa518b34b2c"} Oct 30 17:15:51.400907 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:51.400612 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-82243b-7f4f9546b6-mg8hr" Oct 30 17:15:51.402301 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:51.402270 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-82243b-6794bbb67-58ssq" event={"ID":"5bdabfbc-c999-479c-a44e-d4949bd3c73d","Type":"ContainerStarted","Data":"1f71acee2445475d0860e021cf9070236c0d3fed188889c790e20f62d175672d"} Oct 30 17:15:51.402445 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:51.402333 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-82243b-6794bbb67-58ssq" event={"ID":"5bdabfbc-c999-479c-a44e-d4949bd3c73d","Type":"ContainerStarted","Data":"f2c04590fc9684b79f217f9788e7ecac19d14ac60f2c7478bfb1d04488f31ac1"} Oct 30 17:15:51.402445 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:51.402364 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-82243b-6794bbb67-58ssq" Oct 30 17:15:51.403652 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:51.403633 4522 generic.go:356] "Generic (PLEG): container finished" podID="ed5776a0-2912-446d-a83f-8c84a75e5cbf" containerID="3745dd62c7c27502c535aa6beaeac2478495fdc937d908cd054412b9f4b5d220" exitCode=0 Oct 30 17:15:51.403739 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:51.403668 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97" Oct 30 17:15:51.403739 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:51.403715 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97" event={"ID":"ed5776a0-2912-446d-a83f-8c84a75e5cbf","Type":"ContainerDied","Data":"3745dd62c7c27502c535aa6beaeac2478495fdc937d908cd054412b9f4b5d220"} Oct 30 17:15:51.403842 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:51.403757 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97" event={"ID":"ed5776a0-2912-446d-a83f-8c84a75e5cbf","Type":"ContainerDied","Data":"c68a3b6a9856b64756962b910ab54ad60a434add1939f49c917f8dcdf73a2b03"} Oct 30 17:15:51.403842 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:51.403780 4522 scope.go:122] "RemoveContainer" containerID="3745dd62c7c27502c535aa6beaeac2478495fdc937d908cd054412b9f4b5d220" Oct 30 17:15:51.416158 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:51.416137 4522 scope.go:122] "RemoveContainer" containerID="3745dd62c7c27502c535aa6beaeac2478495fdc937d908cd054412b9f4b5d220" Oct 30 17:15:51.416477 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:15:51.416450 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3745dd62c7c27502c535aa6beaeac2478495fdc937d908cd054412b9f4b5d220\": container with ID starting with 3745dd62c7c27502c535aa6beaeac2478495fdc937d908cd054412b9f4b5d220 not found: ID does not exist" containerID="3745dd62c7c27502c535aa6beaeac2478495fdc937d908cd054412b9f4b5d220" Oct 30 17:15:51.416534 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:51.416488 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3745dd62c7c27502c535aa6beaeac2478495fdc937d908cd054412b9f4b5d220"} err="failed to get container status \"3745dd62c7c27502c535aa6beaeac2478495fdc937d908cd054412b9f4b5d220\": rpc error: code = NotFound desc = could not find container \"3745dd62c7c27502c535aa6beaeac2478495fdc937d908cd054412b9f4b5d220\": container with ID starting with 3745dd62c7c27502c535aa6beaeac2478495fdc937d908cd054412b9f4b5d220 not found: ID does not exist" Oct 30 17:15:51.449595 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:51.449547 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-82243b-7f4f9546b6-mg8hr" podStartSLOduration=2.449534145 podStartE2EDuration="2.449534145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:51.445351397 +0000 UTC m=+1725.446106294" watchObservedRunningTime="2026-10-30 17:15:51.449534145 +0000 UTC m=+1725.450289046" Oct 30 17:15:51.490298 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:51.490242 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-82243b-6794bbb67-58ssq" podStartSLOduration=2.490226955 podStartE2EDuration="2.490226955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:51.48897018 +0000 UTC m=+1725.489725083" watchObservedRunningTime="2026-10-30 17:15:51.490226955 +0000 UTC m=+1725.490981856" Oct 30 17:15:51.531297 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:51.531267 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97"] Oct 30 17:15:51.545044 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:51.545012 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-psf97"] Oct 30 17:15:52.147353 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:52.147302 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed5776a0-2912-446d-a83f-8c84a75e5cbf" path="/var/lib/kubelet/pods/ed5776a0-2912-446d-a83f-8c84a75e5cbf/volumes" Oct 30 17:15:55.035380 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:55.035342 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06a99010-dwbd7"] Oct 30 17:15:55.048448 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:55.048417 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06a99010-dwbd7"] Oct 30 17:15:55.453808 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:55.453780 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx"] Oct 30 17:15:55.453990 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:55.453969 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx" podUID="8208e132-462b-4b94-9246-e0ecc27a13a9" containerName="istio-proxy" containerID="cri-o://0df7b4963c6c0b0fcb666d87adb6ba20cecd890acf9d8e99d35f023b4f7c449b" gracePeriod=30 Oct 30 17:15:56.147877 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:56.147848 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59fadb0c-7100-4e9d-91a5-959c6e747228" path="/var/lib/kubelet/pods/59fadb0c-7100-4e9d-91a5-959c6e747228/volumes" Oct 30 17:15:59.197779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:59.197742 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz"] Oct 30 17:15:59.198619 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:59.198602 4522 cpu_manager.go:414] "containerMap: removing container" podUID="ed5776a0-2912-446d-a83f-8c84a75e5cbf" containerName="istio-proxy" Oct 30 17:15:59.198664 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:59.198620 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed5776a0-2912-446d-a83f-8c84a75e5cbf" containerName="istio-proxy" Oct 30 17:15:59.198751 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:59.198741 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed5776a0-2912-446d-a83f-8c84a75e5cbf" containerName="istio-proxy" Oct 30 17:15:59.202521 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:59.202505 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz" Oct 30 17:15:59.239505 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:59.239469 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz"] Oct 30 17:15:59.248954 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:59.248908 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-7c52f8-openshift-default-dockercfg-lmxqw\"" Oct 30 17:15:59.373105 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:59.373055 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnfk5\" (UniqueName: \"kubernetes.io/projected/7a845206-e2dc-4996-8e24-4f59233146b2-kube-api-access-mnfk5\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz\" (UID: \"7a845206-e2dc-4996-8e24-4f59233146b2\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz" Oct 30 17:15:59.373270 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:59.373121 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a845206-e2dc-4996-8e24-4f59233146b2-istio-token\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz\" (UID: \"7a845206-e2dc-4996-8e24-4f59233146b2\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz" Oct 30 17:15:59.373270 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:59.373194 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a845206-e2dc-4996-8e24-4f59233146b2-credential-socket\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz\" (UID: \"7a845206-e2dc-4996-8e24-4f59233146b2\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz" Oct 30 17:15:59.373410 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:59.373281 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a845206-e2dc-4996-8e24-4f59233146b2-workload-certs\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz\" (UID: \"7a845206-e2dc-4996-8e24-4f59233146b2\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz" Oct 30 17:15:59.373448 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:59.373414 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a845206-e2dc-4996-8e24-4f59233146b2-istio-data\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz\" (UID: \"7a845206-e2dc-4996-8e24-4f59233146b2\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz" Oct 30 17:15:59.373448 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:59.373440 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a845206-e2dc-4996-8e24-4f59233146b2-istio-envoy\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz\" (UID: \"7a845206-e2dc-4996-8e24-4f59233146b2\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz" Oct 30 17:15:59.373509 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:59.373460 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a845206-e2dc-4996-8e24-4f59233146b2-istio-podinfo\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz\" (UID: \"7a845206-e2dc-4996-8e24-4f59233146b2\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz" Oct 30 17:15:59.373584 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:59.373571 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a845206-e2dc-4996-8e24-4f59233146b2-workload-socket\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz\" (UID: \"7a845206-e2dc-4996-8e24-4f59233146b2\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz" Oct 30 17:15:59.373708 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:59.373683 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a845206-e2dc-4996-8e24-4f59233146b2-istiod-ca-cert\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz\" (UID: \"7a845206-e2dc-4996-8e24-4f59233146b2\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz" Oct 30 17:15:59.475665 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:59.475628 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a845206-e2dc-4996-8e24-4f59233146b2-istio-data\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz\" (UID: \"7a845206-e2dc-4996-8e24-4f59233146b2\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz" Oct 30 17:15:59.475665 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:59.475678 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a845206-e2dc-4996-8e24-4f59233146b2-istio-envoy\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz\" (UID: \"7a845206-e2dc-4996-8e24-4f59233146b2\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz" Oct 30 17:15:59.475937 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:59.475702 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a845206-e2dc-4996-8e24-4f59233146b2-istio-podinfo\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz\" (UID: \"7a845206-e2dc-4996-8e24-4f59233146b2\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz" Oct 30 17:15:59.475937 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:59.475763 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a845206-e2dc-4996-8e24-4f59233146b2-workload-socket\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz\" (UID: \"7a845206-e2dc-4996-8e24-4f59233146b2\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz" Oct 30 17:15:59.475937 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:59.475842 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a845206-e2dc-4996-8e24-4f59233146b2-istiod-ca-cert\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz\" (UID: \"7a845206-e2dc-4996-8e24-4f59233146b2\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz" Oct 30 17:15:59.475937 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:59.475916 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnfk5\" (UniqueName: \"kubernetes.io/projected/7a845206-e2dc-4996-8e24-4f59233146b2-kube-api-access-mnfk5\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz\" (UID: \"7a845206-e2dc-4996-8e24-4f59233146b2\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz" Oct 30 17:15:59.476966 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:59.475948 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a845206-e2dc-4996-8e24-4f59233146b2-istio-token\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz\" (UID: \"7a845206-e2dc-4996-8e24-4f59233146b2\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz" Oct 30 17:15:59.476966 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:59.475982 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a845206-e2dc-4996-8e24-4f59233146b2-credential-socket\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz\" (UID: \"7a845206-e2dc-4996-8e24-4f59233146b2\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz" Oct 30 17:15:59.476966 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:59.476085 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a845206-e2dc-4996-8e24-4f59233146b2-workload-certs\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz\" (UID: \"7a845206-e2dc-4996-8e24-4f59233146b2\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz" Oct 30 17:15:59.476966 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:59.476191 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a845206-e2dc-4996-8e24-4f59233146b2-istio-data\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz\" (UID: \"7a845206-e2dc-4996-8e24-4f59233146b2\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz" Oct 30 17:15:59.476966 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:59.476623 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a845206-e2dc-4996-8e24-4f59233146b2-workload-certs\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz\" (UID: \"7a845206-e2dc-4996-8e24-4f59233146b2\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz" Oct 30 17:15:59.476966 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:59.476898 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a845206-e2dc-4996-8e24-4f59233146b2-istiod-ca-cert\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz\" (UID: \"7a845206-e2dc-4996-8e24-4f59233146b2\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz" Oct 30 17:15:59.477299 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:59.477035 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a845206-e2dc-4996-8e24-4f59233146b2-workload-socket\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz\" (UID: \"7a845206-e2dc-4996-8e24-4f59233146b2\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz" Oct 30 17:15:59.477299 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:59.477156 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a845206-e2dc-4996-8e24-4f59233146b2-credential-socket\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz\" (UID: \"7a845206-e2dc-4996-8e24-4f59233146b2\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz" Oct 30 17:15:59.478567 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:59.478545 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a845206-e2dc-4996-8e24-4f59233146b2-istio-envoy\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz\" (UID: \"7a845206-e2dc-4996-8e24-4f59233146b2\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz" Oct 30 17:15:59.478782 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:59.478757 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a845206-e2dc-4996-8e24-4f59233146b2-istio-podinfo\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz\" (UID: \"7a845206-e2dc-4996-8e24-4f59233146b2\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz" Oct 30 17:15:59.526401 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:59.526365 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a845206-e2dc-4996-8e24-4f59233146b2-istio-token\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz\" (UID: \"7a845206-e2dc-4996-8e24-4f59233146b2\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz" Oct 30 17:15:59.527124 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:59.527104 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnfk5\" (UniqueName: \"kubernetes.io/projected/7a845206-e2dc-4996-8e24-4f59233146b2-kube-api-access-mnfk5\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz\" (UID: \"7a845206-e2dc-4996-8e24-4f59233146b2\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz" Oct 30 17:15:59.820186 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:15:59.820149 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz" Oct 30 17:16:00.154820 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:16:00.154791 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a845206_e2dc_4996_8e24_4f59233146b2.slice/crio-0dac48efbe1758b2066c1f2fec414833373b840ba25d045780314314443ef21d WatchSource:0}: Error finding container 0dac48efbe1758b2066c1f2fec414833373b840ba25d045780314314443ef21d: Status 404 returned error can't find the container with id 0dac48efbe1758b2066c1f2fec414833373b840ba25d045780314314443ef21d Oct 30 17:16:00.157503 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:00.157477 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz"] Oct 30 17:16:00.157691 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:00.157662 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:16:00.157742 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:00.157730 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:16:00.157780 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:00.157768 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:16:00.533956 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:00.533914 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz" event={"ID":"7a845206-e2dc-4996-8e24-4f59233146b2","Type":"ContainerStarted","Data":"ec67c1870c73c264be68b4cac46a1cfd30eece1dc7cc10658f9242901320fde5"} Oct 30 17:16:00.534478 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:00.533985 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz" event={"ID":"7a845206-e2dc-4996-8e24-4f59233146b2","Type":"ContainerStarted","Data":"0dac48efbe1758b2066c1f2fec414833373b840ba25d045780314314443ef21d"} Oct 30 17:16:00.590508 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:00.590447 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz" podStartSLOduration=1.5904299910000002 podStartE2EDuration="1.590429991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:16:00.589923546 +0000 UTC m=+1734.590678494" watchObservedRunningTime="2026-10-30 17:16:00.590429991 +0000 UTC m=+1734.591184893" Oct 30 17:16:00.813342 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:00.813281 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx" Oct 30 17:16:00.820257 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:00.820225 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz" Oct 30 17:16:01.002295 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:01.002265 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8208e132-462b-4b94-9246-e0ecc27a13a9-kube-api-access-74jbd\" (UniqueName: \"kubernetes.io/projected/8208e132-462b-4b94-9246-e0ecc27a13a9-kube-api-access-74jbd\") pod \"8208e132-462b-4b94-9246-e0ecc27a13a9\" (UID: \"8208e132-462b-4b94-9246-e0ecc27a13a9\") " Oct 30 17:16:01.002506 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:01.002300 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8208e132-462b-4b94-9246-e0ecc27a13a9-istio-token\" (UniqueName: \"kubernetes.io/projected/8208e132-462b-4b94-9246-e0ecc27a13a9-istio-token\") pod \"8208e132-462b-4b94-9246-e0ecc27a13a9\" (UID: \"8208e132-462b-4b94-9246-e0ecc27a13a9\") " Oct 30 17:16:01.002506 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:01.002412 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8208e132-462b-4b94-9246-e0ecc27a13a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8208e132-462b-4b94-9246-e0ecc27a13a9-istio-podinfo\") pod \"8208e132-462b-4b94-9246-e0ecc27a13a9\" (UID: \"8208e132-462b-4b94-9246-e0ecc27a13a9\") " Oct 30 17:16:01.002628 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:01.002569 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8208e132-462b-4b94-9246-e0ecc27a13a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8208e132-462b-4b94-9246-e0ecc27a13a9-istio-data\") pod \"8208e132-462b-4b94-9246-e0ecc27a13a9\" (UID: \"8208e132-462b-4b94-9246-e0ecc27a13a9\") " Oct 30 17:16:01.002780 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:01.002761 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8208e132-462b-4b94-9246-e0ecc27a13a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8208e132-462b-4b94-9246-e0ecc27a13a9-credential-socket\") pod \"8208e132-462b-4b94-9246-e0ecc27a13a9\" (UID: \"8208e132-462b-4b94-9246-e0ecc27a13a9\") " Oct 30 17:16:01.002847 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:01.002806 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8208e132-462b-4b94-9246-e0ecc27a13a9-istio-data" pod "8208e132-462b-4b94-9246-e0ecc27a13a9" (UID: "8208e132-462b-4b94-9246-e0ecc27a13a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:01.002999 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:01.002975 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8208e132-462b-4b94-9246-e0ecc27a13a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8208e132-462b-4b94-9246-e0ecc27a13a9-istiod-ca-cert\") pod \"8208e132-462b-4b94-9246-e0ecc27a13a9\" (UID: \"8208e132-462b-4b94-9246-e0ecc27a13a9\") " Oct 30 17:16:01.003138 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:01.003002 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8208e132-462b-4b94-9246-e0ecc27a13a9-credential-socket" pod "8208e132-462b-4b94-9246-e0ecc27a13a9" (UID: "8208e132-462b-4b94-9246-e0ecc27a13a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:01.003138 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:01.003031 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8208e132-462b-4b94-9246-e0ecc27a13a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8208e132-462b-4b94-9246-e0ecc27a13a9-workload-socket\") pod \"8208e132-462b-4b94-9246-e0ecc27a13a9\" (UID: \"8208e132-462b-4b94-9246-e0ecc27a13a9\") " Oct 30 17:16:01.003138 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:01.003072 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8208e132-462b-4b94-9246-e0ecc27a13a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8208e132-462b-4b94-9246-e0ecc27a13a9-istio-envoy\") pod \"8208e132-462b-4b94-9246-e0ecc27a13a9\" (UID: \"8208e132-462b-4b94-9246-e0ecc27a13a9\") " Oct 30 17:16:01.003338 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:01.003159 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8208e132-462b-4b94-9246-e0ecc27a13a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8208e132-462b-4b94-9246-e0ecc27a13a9-workload-certs\") pod \"8208e132-462b-4b94-9246-e0ecc27a13a9\" (UID: \"8208e132-462b-4b94-9246-e0ecc27a13a9\") " Oct 30 17:16:01.003411 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:01.003372 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8208e132-462b-4b94-9246-e0ecc27a13a9-istiod-ca-cert" pod "8208e132-462b-4b94-9246-e0ecc27a13a9" (UID: "8208e132-462b-4b94-9246-e0ecc27a13a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:16:01.003659 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:01.003632 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8208e132-462b-4b94-9246-e0ecc27a13a9-workload-socket" pod "8208e132-462b-4b94-9246-e0ecc27a13a9" (UID: "8208e132-462b-4b94-9246-e0ecc27a13a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:01.003659 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:01.003652 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8208e132-462b-4b94-9246-e0ecc27a13a9-workload-certs" pod "8208e132-462b-4b94-9246-e0ecc27a13a9" (UID: "8208e132-462b-4b94-9246-e0ecc27a13a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:01.003974 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:01.003948 4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8208e132-462b-4b94-9246-e0ecc27a13a9-istio-data\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:16:01.003974 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:01.003970 4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8208e132-462b-4b94-9246-e0ecc27a13a9-credential-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:16:01.004111 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:01.003986 4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8208e132-462b-4b94-9246-e0ecc27a13a9-istiod-ca-cert\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:16:01.004111 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:01.004000 4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8208e132-462b-4b94-9246-e0ecc27a13a9-workload-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:16:01.004111 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:01.004013 4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8208e132-462b-4b94-9246-e0ecc27a13a9-workload-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:16:01.005119 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:01.005089 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8208e132-462b-4b94-9246-e0ecc27a13a9-kube-api-access-74jbd" pod "8208e132-462b-4b94-9246-e0ecc27a13a9" (UID: "8208e132-462b-4b94-9246-e0ecc27a13a9"). InnerVolumeSpecName "kube-api-access-74jbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:01.005212 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:01.005132 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8208e132-462b-4b94-9246-e0ecc27a13a9-istio-token" pod "8208e132-462b-4b94-9246-e0ecc27a13a9" (UID: "8208e132-462b-4b94-9246-e0ecc27a13a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:01.005212 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:01.005145 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8208e132-462b-4b94-9246-e0ecc27a13a9-istio-podinfo" pod "8208e132-462b-4b94-9246-e0ecc27a13a9" (UID: "8208e132-462b-4b94-9246-e0ecc27a13a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:16:01.006161 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:01.006139 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8208e132-462b-4b94-9246-e0ecc27a13a9-istio-envoy" pod "8208e132-462b-4b94-9246-e0ecc27a13a9" (UID: "8208e132-462b-4b94-9246-e0ecc27a13a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:01.117456 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:01.109250 4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8208e132-462b-4b94-9246-e0ecc27a13a9-istio-envoy\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:16:01.117456 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:01.109294 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74jbd\" (UniqueName: \"kubernetes.io/projected/8208e132-462b-4b94-9246-e0ecc27a13a9-kube-api-access-74jbd\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:16:01.117456 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:01.109331 4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8208e132-462b-4b94-9246-e0ecc27a13a9-istio-token\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:16:01.117456 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:01.109345 4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8208e132-462b-4b94-9246-e0ecc27a13a9-istio-podinfo\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:16:01.544341 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:01.544290 4522 generic.go:356] "Generic (PLEG): container finished" podID="8208e132-462b-4b94-9246-e0ecc27a13a9" containerID="0df7b4963c6c0b0fcb666d87adb6ba20cecd890acf9d8e99d35f023b4f7c449b" exitCode=0 Oct 30 17:16:01.544879 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:01.544380 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx" Oct 30 17:16:01.544879 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:01.544381 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx" event={"ID":"8208e132-462b-4b94-9246-e0ecc27a13a9","Type":"ContainerDied","Data":"0df7b4963c6c0b0fcb666d87adb6ba20cecd890acf9d8e99d35f023b4f7c449b"} Oct 30 17:16:01.544879 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:01.544434 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx" event={"ID":"8208e132-462b-4b94-9246-e0ecc27a13a9","Type":"ContainerDied","Data":"0375c114a7ae20f68d88688d6d90f550a0cbe15dbcb56f9cd6e6c0f6f7b94bda"} Oct 30 17:16:01.544879 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:01.544457 4522 scope.go:122] "RemoveContainer" containerID="0df7b4963c6c0b0fcb666d87adb6ba20cecd890acf9d8e99d35f023b4f7c449b" Oct 30 17:16:01.559223 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:01.559192 4522 scope.go:122] "RemoveContainer" containerID="0df7b4963c6c0b0fcb666d87adb6ba20cecd890acf9d8e99d35f023b4f7c449b" Oct 30 17:16:01.559569 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:16:01.559541 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0df7b4963c6c0b0fcb666d87adb6ba20cecd890acf9d8e99d35f023b4f7c449b\": container with ID starting with 0df7b4963c6c0b0fcb666d87adb6ba20cecd890acf9d8e99d35f023b4f7c449b not found: ID does not exist" containerID="0df7b4963c6c0b0fcb666d87adb6ba20cecd890acf9d8e99d35f023b4f7c449b" Oct 30 17:16:01.559635 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:01.559581 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0df7b4963c6c0b0fcb666d87adb6ba20cecd890acf9d8e99d35f023b4f7c449b"} err="failed to get container status \"0df7b4963c6c0b0fcb666d87adb6ba20cecd890acf9d8e99d35f023b4f7c449b\": rpc error: code = NotFound desc = could not find container \"0df7b4963c6c0b0fcb666d87adb6ba20cecd890acf9d8e99d35f023b4f7c449b\": container with ID starting with 0df7b4963c6c0b0fcb666d87adb6ba20cecd890acf9d8e99d35f023b4f7c449b not found: ID does not exist" Oct 30 17:16:01.626923 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:01.626884 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx"] Oct 30 17:16:01.633697 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:01.633660 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4rgfx"] Oct 30 17:16:01.822550 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:01.822477 4522 patch_prober.go:29] interesting pod/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body= Oct 30 17:16:01.822550 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:01.822528 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz" podUID="7a845206-e2dc-4996-8e24-4f59233146b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" Oct 30 17:16:02.148527 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:02.148431 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8208e132-462b-4b94-9246-e0ecc27a13a9" path="/var/lib/kubelet/pods/8208e132-462b-4b94-9246-e0ecc27a13a9/volumes" Oct 30 17:16:02.426385 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:02.425994 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-82243b-6794bbb67-58ssq" Oct 30 17:16:02.426549 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:02.426444 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-82243b-7f4f9546b6-mg8hr" Oct 30 17:16:02.820866 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:02.820830 4522 patch_prober.go:29] interesting pod/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body= Oct 30 17:16:02.821290 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:02.820888 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz" podUID="7a845206-e2dc-4996-8e24-4f59233146b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" Oct 30 17:16:03.821074 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:03.821036 4522 patch_prober.go:29] interesting pod/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body= Oct 30 17:16:03.821582 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:03.821087 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz" podUID="7a845206-e2dc-4996-8e24-4f59233146b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" Oct 30 17:16:04.525854 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:04.525770 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e5ac8f-98c6fb647-mlcx8"] Oct 30 17:16:04.526986 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:04.526955 4522 cpu_manager.go:414] "containerMap: removing container" podUID="8208e132-462b-4b94-9246-e0ecc27a13a9" containerName="istio-proxy" Oct 30 17:16:04.526986 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:04.526983 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8208e132-462b-4b94-9246-e0ecc27a13a9" containerName="istio-proxy" Oct 30 17:16:04.527256 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:04.527241 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="8208e132-462b-4b94-9246-e0ecc27a13a9" containerName="istio-proxy" Oct 30 17:16:04.533282 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:04.533249 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e5ac8f-98c6fb647-mlcx8"] Oct 30 17:16:04.533432 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:04.533373 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e5ac8f-98c6fb647-mlcx8" Oct 30 17:16:04.550564 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:04.550540 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-e5ac8f\"" Oct 30 17:16:04.568339 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:04.568286 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35026397-efc0-4b45-b548-01ce4d2ac26e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e5ac8f-98c6fb647-mlcx8\" (UID: \"35026397-efc0-4b45-b548-01ce4d2ac26e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e5ac8f-98c6fb647-mlcx8" Oct 30 17:16:04.568505 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:04.568416 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc54c\" (UniqueName: \"kubernetes.io/projected/35026397-efc0-4b45-b548-01ce4d2ac26e-kube-api-access-cc54c\") pod \"payload-pre-processing-e2e-inf-b-e5ac8f-98c6fb647-mlcx8\" (UID: \"35026397-efc0-4b45-b548-01ce4d2ac26e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e5ac8f-98c6fb647-mlcx8" Oct 30 17:16:04.674852 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:04.674804 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35026397-efc0-4b45-b548-01ce4d2ac26e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e5ac8f-98c6fb647-mlcx8\" (UID: \"35026397-efc0-4b45-b548-01ce4d2ac26e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e5ac8f-98c6fb647-mlcx8" Oct 30 17:16:04.675028 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:04.674892 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc54c\" (UniqueName: \"kubernetes.io/projected/35026397-efc0-4b45-b548-01ce4d2ac26e-kube-api-access-cc54c\") pod \"payload-pre-processing-e2e-inf-b-e5ac8f-98c6fb647-mlcx8\" (UID: \"35026397-efc0-4b45-b548-01ce4d2ac26e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e5ac8f-98c6fb647-mlcx8" Oct 30 17:16:04.685086 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:04.685055 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35026397-efc0-4b45-b548-01ce4d2ac26e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e5ac8f-98c6fb647-mlcx8\" (UID: \"35026397-efc0-4b45-b548-01ce4d2ac26e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e5ac8f-98c6fb647-mlcx8" Oct 30 17:16:04.734838 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:04.733104 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc54c\" (UniqueName: \"kubernetes.io/projected/35026397-efc0-4b45-b548-01ce4d2ac26e-kube-api-access-cc54c\") pod \"payload-pre-processing-e2e-inf-b-e5ac8f-98c6fb647-mlcx8\" (UID: \"35026397-efc0-4b45-b548-01ce4d2ac26e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e5ac8f-98c6fb647-mlcx8" Oct 30 17:16:04.827393 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:04.826391 4522 patch_prober.go:29] interesting pod/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body= Oct 30 17:16:04.827393 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:04.826463 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz" podUID="7a845206-e2dc-4996-8e24-4f59233146b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" Oct 30 17:16:04.857608 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:04.857573 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e5ac8f-98c6fb647-mlcx8" Oct 30 17:16:05.228921 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:16:05.228880 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35026397_efc0_4b45_b548_01ce4d2ac26e.slice/crio-830fc9c467af787afb4c157c4af3754200e101516c9d4d72c2f2826f04fc0466 WatchSource:0}: Error finding container 830fc9c467af787afb4c157c4af3754200e101516c9d4d72c2f2826f04fc0466: Status 404 returned error can't find the container with id 830fc9c467af787afb4c157c4af3754200e101516c9d4d72c2f2826f04fc0466 Oct 30 17:16:05.230676 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:05.230655 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e5ac8f-98c6fb647-mlcx8"] Oct 30 17:16:05.628367 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:05.628235 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e5ac8f-98c6fb647-mlcx8" event={"ID":"35026397-efc0-4b45-b548-01ce4d2ac26e","Type":"ContainerStarted","Data":"08fbfa4e57ab86881fde7563462c59344b02946269addf1e9bf72520ced2f371"} Oct 30 17:16:05.628582 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:05.628449 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e5ac8f-98c6fb647-mlcx8" event={"ID":"35026397-efc0-4b45-b548-01ce4d2ac26e","Type":"ContainerStarted","Data":"830fc9c467af787afb4c157c4af3754200e101516c9d4d72c2f2826f04fc0466"} Oct 30 17:16:05.628582 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:05.628498 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e5ac8f-98c6fb647-mlcx8" Oct 30 17:16:05.673855 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:05.673802 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e5ac8f-98c6fb647-mlcx8" podStartSLOduration=1.673789553 podStartE2EDuration="1.673789553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:16:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:16:05.672678001 +0000 UTC m=+1739.673432901" watchObservedRunningTime="2026-10-30 17:16:05.673789553 +0000 UTC m=+1739.674544454" Oct 30 17:16:05.827064 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:05.824379 4522 patch_prober.go:29] interesting pod/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body= Oct 30 17:16:05.827064 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:05.824453 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz" podUID="7a845206-e2dc-4996-8e24-4f59233146b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" Oct 30 17:16:06.828179 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:06.828149 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz" Oct 30 17:16:06.828695 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:06.828504 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz" Oct 30 17:16:06.829339 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:06.829285 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz" Oct 30 17:16:08.329622 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:08.329591 4522 scope.go:122] "RemoveContainer" containerID="6fd76ad60df611663e7eb95993ac17a802e7373ad61d70f0d3ce959b1d6afc22" Oct 30 17:16:10.524839 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:10.524798 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs"] Oct 30 17:16:10.525665 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:10.525628 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs" podUID="89585321-089e-477a-a6fa-d8b5a8478639" containerName="istio-proxy" containerID="cri-o://c2d48bf0a257b79b6553b3f993854a291adf4d0a65f772ef1503224773dc8cdb" gracePeriod=30 Oct 30 17:16:12.951220 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:16:12.951187 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:16:15.720418 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:15.720380 4522 generic.go:356] "Generic (PLEG): container finished" podID="89585321-089e-477a-a6fa-d8b5a8478639" containerID="c2d48bf0a257b79b6553b3f993854a291adf4d0a65f772ef1503224773dc8cdb" exitCode=0 Oct 30 17:16:15.720897 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:15.720508 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs" event={"ID":"89585321-089e-477a-a6fa-d8b5a8478639","Type":"ContainerDied","Data":"c2d48bf0a257b79b6553b3f993854a291adf4d0a65f772ef1503224773dc8cdb"} Oct 30 17:16:15.821074 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:15.821046 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs" Oct 30 17:16:15.986734 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:15.986699 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89585321-089e-477a-a6fa-d8b5a8478639-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89585321-089e-477a-a6fa-d8b5a8478639-istio-envoy\") pod \"89585321-089e-477a-a6fa-d8b5a8478639\" (UID: \"89585321-089e-477a-a6fa-d8b5a8478639\") " Oct 30 17:16:15.986899 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:15.986754 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89585321-089e-477a-a6fa-d8b5a8478639-kube-api-access-7rwxw\" (UniqueName: \"kubernetes.io/projected/89585321-089e-477a-a6fa-d8b5a8478639-kube-api-access-7rwxw\") pod \"89585321-089e-477a-a6fa-d8b5a8478639\" (UID: \"89585321-089e-477a-a6fa-d8b5a8478639\") " Oct 30 17:16:15.986899 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:15.986881 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89585321-089e-477a-a6fa-d8b5a8478639-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89585321-089e-477a-a6fa-d8b5a8478639-credential-socket\") pod \"89585321-089e-477a-a6fa-d8b5a8478639\" (UID: \"89585321-089e-477a-a6fa-d8b5a8478639\") " Oct 30 17:16:15.987010 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:15.986994 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/89585321-089e-477a-a6fa-d8b5a8478639-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89585321-089e-477a-a6fa-d8b5a8478639-istio-podinfo\") pod \"89585321-089e-477a-a6fa-d8b5a8478639\" (UID: \"89585321-089e-477a-a6fa-d8b5a8478639\") " Oct 30 17:16:15.987075 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:15.987051 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89585321-089e-477a-a6fa-d8b5a8478639-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89585321-089e-477a-a6fa-d8b5a8478639-workload-certs\") pod \"89585321-089e-477a-a6fa-d8b5a8478639\" (UID: \"89585321-089e-477a-a6fa-d8b5a8478639\") " Oct 30 17:16:15.987279 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:15.987173 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89585321-089e-477a-a6fa-d8b5a8478639-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89585321-089e-477a-a6fa-d8b5a8478639-istiod-ca-cert\") pod \"89585321-089e-477a-a6fa-d8b5a8478639\" (UID: \"89585321-089e-477a-a6fa-d8b5a8478639\") " Oct 30 17:16:15.987279 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:15.987225 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89585321-089e-477a-a6fa-d8b5a8478639-credential-socket" pod "89585321-089e-477a-a6fa-d8b5a8478639" (UID: "89585321-089e-477a-a6fa-d8b5a8478639"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:15.987440 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:15.987354 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89585321-089e-477a-a6fa-d8b5a8478639-workload-certs" pod "89585321-089e-477a-a6fa-d8b5a8478639" (UID: "89585321-089e-477a-a6fa-d8b5a8478639"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:15.987595 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:15.987571 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89585321-089e-477a-a6fa-d8b5a8478639-istiod-ca-cert" pod "89585321-089e-477a-a6fa-d8b5a8478639" (UID: "89585321-089e-477a-a6fa-d8b5a8478639"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:16:15.987695 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:15.987676 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89585321-089e-477a-a6fa-d8b5a8478639-istio-token\" (UniqueName: \"kubernetes.io/projected/89585321-089e-477a-a6fa-d8b5a8478639-istio-token\") pod \"89585321-089e-477a-a6fa-d8b5a8478639\" (UID: \"89585321-089e-477a-a6fa-d8b5a8478639\") " Oct 30 17:16:15.987795 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:15.987777 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89585321-089e-477a-a6fa-d8b5a8478639-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89585321-089e-477a-a6fa-d8b5a8478639-workload-socket\") pod \"89585321-089e-477a-a6fa-d8b5a8478639\" (UID: \"89585321-089e-477a-a6fa-d8b5a8478639\") " Oct 30 17:16:15.987863 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:15.987833 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89585321-089e-477a-a6fa-d8b5a8478639-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89585321-089e-477a-a6fa-d8b5a8478639-istio-data\") pod \"89585321-089e-477a-a6fa-d8b5a8478639\" (UID: \"89585321-089e-477a-a6fa-d8b5a8478639\") " Oct 30 17:16:15.988085 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:15.988049 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89585321-089e-477a-a6fa-d8b5a8478639-workload-socket" pod "89585321-089e-477a-a6fa-d8b5a8478639" (UID: "89585321-089e-477a-a6fa-d8b5a8478639"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:15.988199 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:15.988085 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89585321-089e-477a-a6fa-d8b5a8478639-istio-data" pod "89585321-089e-477a-a6fa-d8b5a8478639" (UID: "89585321-089e-477a-a6fa-d8b5a8478639"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:15.988833 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:15.988812 4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89585321-089e-477a-a6fa-d8b5a8478639-credential-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:16:15.988833 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:15.988831 4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89585321-089e-477a-a6fa-d8b5a8478639-workload-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:16:15.988999 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:15.988841 4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89585321-089e-477a-a6fa-d8b5a8478639-istiod-ca-cert\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:16:15.988999 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:15.988848 4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89585321-089e-477a-a6fa-d8b5a8478639-workload-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:16:15.988999 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:15.988857 4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89585321-089e-477a-a6fa-d8b5a8478639-istio-data\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:16:15.989299 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:15.989272 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89585321-089e-477a-a6fa-d8b5a8478639-kube-api-access-7rwxw" pod "89585321-089e-477a-a6fa-d8b5a8478639" (UID: "89585321-089e-477a-a6fa-d8b5a8478639"). InnerVolumeSpecName "kube-api-access-7rwxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:15.989422 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:15.989324 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89585321-089e-477a-a6fa-d8b5a8478639-istio-podinfo" pod "89585321-089e-477a-a6fa-d8b5a8478639" (UID: "89585321-089e-477a-a6fa-d8b5a8478639"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:16:15.989471 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:15.989442 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89585321-089e-477a-a6fa-d8b5a8478639-istio-envoy" pod "89585321-089e-477a-a6fa-d8b5a8478639" (UID: "89585321-089e-477a-a6fa-d8b5a8478639"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:15.989831 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:15.989808 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89585321-089e-477a-a6fa-d8b5a8478639-istio-token" pod "89585321-089e-477a-a6fa-d8b5a8478639" (UID: "89585321-089e-477a-a6fa-d8b5a8478639"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:16.091037 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:16.090953 4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89585321-089e-477a-a6fa-d8b5a8478639-istio-podinfo\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:16:16.091037 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:16.090984 4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89585321-089e-477a-a6fa-d8b5a8478639-istio-token\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:16:16.091037 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:16.090995 4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89585321-089e-477a-a6fa-d8b5a8478639-istio-envoy\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:16:16.091037 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:16.091004 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rwxw\" (UniqueName: \"kubernetes.io/projected/89585321-089e-477a-a6fa-d8b5a8478639-kube-api-access-7rwxw\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:16:16.641668 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:16.641635 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e5ac8f-98c6fb647-mlcx8" Oct 30 17:16:16.733326 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:16.733288 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs" Oct 30 17:16:16.733788 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:16.733345 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs" event={"ID":"89585321-089e-477a-a6fa-d8b5a8478639","Type":"ContainerDied","Data":"79a3d622a78ff539f26306bc0f8665f57bee81b352ba897e301ed66d30ccd49a"} Oct 30 17:16:16.733788 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:16.733403 4522 scope.go:122] "RemoveContainer" containerID="c2d48bf0a257b79b6553b3f993854a291adf4d0a65f772ef1503224773dc8cdb" Oct 30 17:16:16.793370 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:16.793339 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs"] Oct 30 17:16:16.807594 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:16.807565 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-l26gs"] Oct 30 17:16:18.147561 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:18.147528 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89585321-089e-477a-a6fa-d8b5a8478639" path="/var/lib/kubelet/pods/89585321-089e-477a-a6fa-d8b5a8478639/volumes" Oct 30 17:16:21.233788 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:21.233536 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl"] Oct 30 17:16:21.234727 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:21.234703 4522 cpu_manager.go:414] "containerMap: removing container" podUID="89585321-089e-477a-a6fa-d8b5a8478639" containerName="istio-proxy" Oct 30 17:16:21.234727 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:21.234725 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89585321-089e-477a-a6fa-d8b5a8478639" containerName="istio-proxy" Oct 30 17:16:21.234953 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:21.234942 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="89585321-089e-477a-a6fa-d8b5a8478639" containerName="istio-proxy" Oct 30 17:16:21.241593 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:21.241570 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl" Oct 30 17:16:21.261102 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:21.261067 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl"] Oct 30 17:16:21.276953 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:21.276918 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e42c53c-946a-40fc-b126-5cd8a39b532d-istio-token\") pod \"e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl\" (UID: \"0e42c53c-946a-40fc-b126-5cd8a39b532d\") " pod="openshift-ingress/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl" Oct 30 17:16:21.277269 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:21.277235 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e42c53c-946a-40fc-b126-5cd8a39b532d-workload-certs\") pod \"e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl\" (UID: \"0e42c53c-946a-40fc-b126-5cd8a39b532d\") " pod="openshift-ingress/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl" Oct 30 17:16:21.277433 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:21.277416 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvfrv\" (UniqueName: \"kubernetes.io/projected/0e42c53c-946a-40fc-b126-5cd8a39b532d-kube-api-access-lvfrv\") pod \"e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl\" (UID: \"0e42c53c-946a-40fc-b126-5cd8a39b532d\") " pod="openshift-ingress/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl" Oct 30 17:16:21.277594 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:21.277573 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e42c53c-946a-40fc-b126-5cd8a39b532d-istiod-ca-cert\") pod \"e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl\" (UID: \"0e42c53c-946a-40fc-b126-5cd8a39b532d\") " pod="openshift-ingress/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl" Oct 30 17:16:21.277713 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:21.277700 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e42c53c-946a-40fc-b126-5cd8a39b532d-istio-data\") pod \"e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl\" (UID: \"0e42c53c-946a-40fc-b126-5cd8a39b532d\") " pod="openshift-ingress/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl" Oct 30 17:16:21.277928 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:21.277914 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e42c53c-946a-40fc-b126-5cd8a39b532d-credential-socket\") pod \"e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl\" (UID: \"0e42c53c-946a-40fc-b126-5cd8a39b532d\") " pod="openshift-ingress/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl" Oct 30 17:16:21.278057 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:21.278044 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e42c53c-946a-40fc-b126-5cd8a39b532d-workload-socket\") pod \"e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl\" (UID: \"0e42c53c-946a-40fc-b126-5cd8a39b532d\") " pod="openshift-ingress/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl" Oct 30 17:16:21.278228 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:21.278215 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e42c53c-946a-40fc-b126-5cd8a39b532d-istio-podinfo\") pod \"e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl\" (UID: \"0e42c53c-946a-40fc-b126-5cd8a39b532d\") " pod="openshift-ingress/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl" Oct 30 17:16:21.278378 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:21.278358 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e42c53c-946a-40fc-b126-5cd8a39b532d-istio-envoy\") pod \"e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl\" (UID: \"0e42c53c-946a-40fc-b126-5cd8a39b532d\") " pod="openshift-ingress/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl" Oct 30 17:16:21.298499 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:21.298468 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-dd5c8f-openshift-default-dockercfg-rlfmn\"" Oct 30 17:16:21.379707 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:21.379670 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e42c53c-946a-40fc-b126-5cd8a39b532d-istio-token\") pod \"e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl\" (UID: \"0e42c53c-946a-40fc-b126-5cd8a39b532d\") " pod="openshift-ingress/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl" Oct 30 17:16:21.379879 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:21.379794 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e42c53c-946a-40fc-b126-5cd8a39b532d-workload-certs\") pod \"e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl\" (UID: \"0e42c53c-946a-40fc-b126-5cd8a39b532d\") " pod="openshift-ingress/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl" Oct 30 17:16:21.379879 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:21.379835 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvfrv\" (UniqueName: \"kubernetes.io/projected/0e42c53c-946a-40fc-b126-5cd8a39b532d-kube-api-access-lvfrv\") pod \"e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl\" (UID: \"0e42c53c-946a-40fc-b126-5cd8a39b532d\") " pod="openshift-ingress/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl" Oct 30 17:16:21.380014 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:21.379899 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e42c53c-946a-40fc-b126-5cd8a39b532d-istiod-ca-cert\") pod \"e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl\" (UID: \"0e42c53c-946a-40fc-b126-5cd8a39b532d\") " pod="openshift-ingress/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl" Oct 30 17:16:21.380014 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:21.379933 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e42c53c-946a-40fc-b126-5cd8a39b532d-istio-data\") pod \"e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl\" (UID: \"0e42c53c-946a-40fc-b126-5cd8a39b532d\") " pod="openshift-ingress/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl" Oct 30 17:16:21.380138 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:21.380085 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e42c53c-946a-40fc-b126-5cd8a39b532d-credential-socket\") pod \"e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl\" (UID: \"0e42c53c-946a-40fc-b126-5cd8a39b532d\") " pod="openshift-ingress/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl" Oct 30 17:16:21.380201 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:21.380152 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e42c53c-946a-40fc-b126-5cd8a39b532d-workload-socket\") pod \"e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl\" (UID: \"0e42c53c-946a-40fc-b126-5cd8a39b532d\") " pod="openshift-ingress/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl" Oct 30 17:16:21.380264 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:21.380210 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e42c53c-946a-40fc-b126-5cd8a39b532d-workload-certs\") pod \"e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl\" (UID: \"0e42c53c-946a-40fc-b126-5cd8a39b532d\") " pod="openshift-ingress/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl" Oct 30 17:16:21.380264 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:21.380238 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e42c53c-946a-40fc-b126-5cd8a39b532d-istio-podinfo\") pod \"e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl\" (UID: \"0e42c53c-946a-40fc-b126-5cd8a39b532d\") " pod="openshift-ingress/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl" Oct 30 17:16:21.380422 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:21.380291 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e42c53c-946a-40fc-b126-5cd8a39b532d-istio-envoy\") pod \"e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl\" (UID: \"0e42c53c-946a-40fc-b126-5cd8a39b532d\") " pod="openshift-ingress/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl" Oct 30 17:16:21.380422 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:21.380353 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e42c53c-946a-40fc-b126-5cd8a39b532d-istio-data\") pod \"e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl\" (UID: \"0e42c53c-946a-40fc-b126-5cd8a39b532d\") " pod="openshift-ingress/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl" Oct 30 17:16:21.380559 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:21.380440 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e42c53c-946a-40fc-b126-5cd8a39b532d-credential-socket\") pod \"e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl\" (UID: \"0e42c53c-946a-40fc-b126-5cd8a39b532d\") " pod="openshift-ingress/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl" Oct 30 17:16:21.380623 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:21.380584 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e42c53c-946a-40fc-b126-5cd8a39b532d-workload-socket\") pod \"e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl\" (UID: \"0e42c53c-946a-40fc-b126-5cd8a39b532d\") " pod="openshift-ingress/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl" Oct 30 17:16:21.380685 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:21.380651 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e42c53c-946a-40fc-b126-5cd8a39b532d-istiod-ca-cert\") pod \"e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl\" (UID: \"0e42c53c-946a-40fc-b126-5cd8a39b532d\") " pod="openshift-ingress/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl" Oct 30 17:16:21.382631 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:21.382605 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e42c53c-946a-40fc-b126-5cd8a39b532d-istio-envoy\") pod \"e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl\" (UID: \"0e42c53c-946a-40fc-b126-5cd8a39b532d\") " pod="openshift-ingress/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl" Oct 30 17:16:21.382911 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:21.382890 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e42c53c-946a-40fc-b126-5cd8a39b532d-istio-podinfo\") pod \"e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl\" (UID: \"0e42c53c-946a-40fc-b126-5cd8a39b532d\") " pod="openshift-ingress/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl" Oct 30 17:16:21.405093 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:21.405058 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e42c53c-946a-40fc-b126-5cd8a39b532d-istio-token\") pod \"e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl\" (UID: \"0e42c53c-946a-40fc-b126-5cd8a39b532d\") " pod="openshift-ingress/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl" Oct 30 17:16:21.406101 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:21.406075 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvfrv\" (UniqueName: \"kubernetes.io/projected/0e42c53c-946a-40fc-b126-5cd8a39b532d-kube-api-access-lvfrv\") pod \"e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl\" (UID: \"0e42c53c-946a-40fc-b126-5cd8a39b532d\") " pod="openshift-ingress/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl" Oct 30 17:16:21.561659 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:21.561569 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl" Oct 30 17:16:21.901274 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:16:21.901227 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e42c53c_946a_40fc_b126_5cd8a39b532d.slice/crio-414c6c9d2272248abe38aeca19782937499358eecdd920ef1e7532aa59a4e25c WatchSource:0}: Error finding container 414c6c9d2272248abe38aeca19782937499358eecdd920ef1e7532aa59a4e25c: Status 404 returned error can't find the container with id 414c6c9d2272248abe38aeca19782937499358eecdd920ef1e7532aa59a4e25c Oct 30 17:16:21.906836 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:21.905164 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl"] Oct 30 17:16:21.908919 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:21.908883 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:16:21.909013 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:21.908954 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:16:21.909013 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:21.908983 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:16:22.790395 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:22.790359 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl" event={"ID":"0e42c53c-946a-40fc-b126-5cd8a39b532d","Type":"ContainerStarted","Data":"e34ffbc603e71d8b1737bd3ffa4ea74e7266c5f8a4ae4c6d447947ba2255893b"} Oct 30 17:16:22.790922 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:22.790422 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl" event={"ID":"0e42c53c-946a-40fc-b126-5cd8a39b532d","Type":"ContainerStarted","Data":"414c6c9d2272248abe38aeca19782937499358eecdd920ef1e7532aa59a4e25c"} Oct 30 17:16:22.865786 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:22.865733 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl" podStartSLOduration=1.865717407 podStartE2EDuration="1.865717407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:16:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:16:22.851609064 +0000 UTC m=+1756.852363964" watchObservedRunningTime="2026-10-30 17:16:22.865717407 +0000 UTC m=+1756.866472307" Oct 30 17:16:23.562129 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:23.562096 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl" Oct 30 17:16:23.568734 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:23.568699 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl" Oct 30 17:16:23.798560 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:23.798532 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl" Oct 30 17:16:23.799593 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:23.799570 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl" Oct 30 17:16:24.334101 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:24.334068 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b2c3fb-7c87c57c7c-pz9ls"] Oct 30 17:16:24.334363 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:24.334335 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b2c3fb-7c87c57c7c-pz9ls" podUID="60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96" containerName="maas-api" containerID="cri-o://8e00516e6bc728fe10a6c71fe46b6b1f3d35a3d376864451f9522c595f4ef903" gracePeriod=30 Oct 30 17:16:24.675346 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:24.675296 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b2c3fb-7c87c57c7c-pz9ls" Oct 30 17:16:24.767329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:24.755072 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96-metrics-certs\" (UniqueName: \"kubernetes.io/secret/60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96-metrics-certs\") pod \"60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96\" (UID: \"60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96\") " Oct 30 17:16:24.767329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:24.755137 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96-openshift-service-ca\") pod \"60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96\" (UID: \"60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96\") " Oct 30 17:16:24.767329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:24.755212 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96-kube-api-access-9d6dw\" (UniqueName: \"kubernetes.io/projected/60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96-kube-api-access-9d6dw\") pod \"60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96\" (UID: \"60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96\") " Oct 30 17:16:24.767329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:24.755331 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96-maas-api-tls\") pod \"60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96\" (UID: \"60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96\") " Oct 30 17:16:24.781881 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:24.772347 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96-kube-api-access-9d6dw" pod "60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96" (UID: "60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96"). InnerVolumeSpecName "kube-api-access-9d6dw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:24.781881 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:24.774474 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96-metrics-certs" pod "60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96" (UID: "60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:16:24.782675 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:24.782644 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96-openshift-service-ca" pod "60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96" (UID: "60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:16:24.790813 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:24.790767 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96-maas-api-tls" pod "60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96" (UID: "60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:16:24.823270 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:24.823229 4522 generic.go:356] "Generic (PLEG): container finished" podID="60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96" containerID="8e00516e6bc728fe10a6c71fe46b6b1f3d35a3d376864451f9522c595f4ef903" exitCode=0 Oct 30 17:16:24.823795 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:24.823328 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b2c3fb-7c87c57c7c-pz9ls" event={"ID":"60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96","Type":"ContainerDied","Data":"8e00516e6bc728fe10a6c71fe46b6b1f3d35a3d376864451f9522c595f4ef903"} Oct 30 17:16:24.823795 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:24.823429 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b2c3fb-7c87c57c7c-pz9ls" event={"ID":"60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96","Type":"ContainerDied","Data":"adca166344b239a1178badd7c393470a086ab5bba15250ec3580e2dd4564e11d"} Oct 30 17:16:24.823795 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:24.823342 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b2c3fb-7c87c57c7c-pz9ls" Oct 30 17:16:24.823795 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:24.823466 4522 scope.go:122] "RemoveContainer" containerID="8e00516e6bc728fe10a6c71fe46b6b1f3d35a3d376864451f9522c595f4ef903" Oct 30 17:16:24.839350 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:24.839267 4522 scope.go:122] "RemoveContainer" containerID="8e00516e6bc728fe10a6c71fe46b6b1f3d35a3d376864451f9522c595f4ef903" Oct 30 17:16:24.839690 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:16:24.839663 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e00516e6bc728fe10a6c71fe46b6b1f3d35a3d376864451f9522c595f4ef903\": container with ID starting with 8e00516e6bc728fe10a6c71fe46b6b1f3d35a3d376864451f9522c595f4ef903 not found: ID does not exist" containerID="8e00516e6bc728fe10a6c71fe46b6b1f3d35a3d376864451f9522c595f4ef903" Oct 30 17:16:24.839913 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:24.839702 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e00516e6bc728fe10a6c71fe46b6b1f3d35a3d376864451f9522c595f4ef903"} err="failed to get container status \"8e00516e6bc728fe10a6c71fe46b6b1f3d35a3d376864451f9522c595f4ef903\": rpc error: code = NotFound desc = could not find container \"8e00516e6bc728fe10a6c71fe46b6b1f3d35a3d376864451f9522c595f4ef903\": container with ID starting with 8e00516e6bc728fe10a6c71fe46b6b1f3d35a3d376864451f9522c595f4ef903 not found: ID does not exist" Oct 30 17:16:24.845721 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:24.845696 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs"] Oct 30 17:16:24.845886 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:24.845868 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs" podUID="2ecae9c7-ada3-4612-909f-1c1493369daf" containerName="istio-proxy" containerID="cri-o://8be21e369bb85f51030c572ee03a89ee10e1ca6c400efcf5ad5e38d1941d3d51" gracePeriod=30 Oct 30 17:16:24.857739 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:24.857703 4522 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96-metrics-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:16:24.857739 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:24.857729 4522 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96-openshift-service-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:16:24.857739 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:24.857742 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d6dw\" (UniqueName: \"kubernetes.io/projected/60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96-kube-api-access-9d6dw\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:16:24.857952 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:24.857754 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96-maas-api-tls\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:16:24.888901 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:24.888871 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b2c3fb-7c87c57c7c-pz9ls"] Oct 30 17:16:24.895004 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:24.894927 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b2c3fb-7c87c57c7c-pz9ls"] Oct 30 17:16:25.697285 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:25.697247 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b2c3fb-qs5xm"] Oct 30 17:16:25.709092 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:25.709054 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b2c3fb-qs5xm"] Oct 30 17:16:26.147812 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:26.147783 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="486d41a9-c5ac-43c0-bb8e-5e9032acf5d2" path="/var/lib/kubelet/pods/486d41a9-c5ac-43c0-bb8e-5e9032acf5d2/volumes" Oct 30 17:16:26.148294 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:26.148278 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96" path="/var/lib/kubelet/pods/60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96/volumes" Oct 30 17:16:27.794904 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:27.794868 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-dd5c8f-6b45577955-57qp6"] Oct 30 17:16:27.795700 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:27.795683 4522 cpu_manager.go:414] "containerMap: removing container" podUID="60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96" containerName="maas-api" Oct 30 17:16:27.795764 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:27.795702 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96" containerName="maas-api" Oct 30 17:16:27.795832 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:27.795824 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="60a8f0d7-aea8-48ff-beb3-c8c0a7a8cf96" containerName="maas-api" Oct 30 17:16:27.800071 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:27.800051 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd5c8f-6b45577955-57qp6" Oct 30 17:16:27.820219 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:27.820189 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-dd5c8f-6b45577955-57qp6"] Oct 30 17:16:27.831010 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:27.828176 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-dd5c8f\"" Oct 30 17:16:27.831010 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:27.828238 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-dd5c8f-dockercfg-brr4x\"" Oct 30 17:16:27.900364 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:27.890893 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-dd5c8f-796cddf7f4-w8wvf"] Oct 30 17:16:27.900364 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:27.896482 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-dd5c8f-796cddf7f4-w8wvf" Oct 30 17:16:27.949188 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:27.948906 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8226519e-a60e-4b3c-b4b3-83479839c8bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-dd5c8f-6b45577955-57qp6\" (UID: \"8226519e-a60e-4b3c-b4b3-83479839c8bb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd5c8f-6b45577955-57qp6" Oct 30 17:16:27.949733 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:27.949705 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd4ns\" (UniqueName: \"kubernetes.io/projected/8226519e-a60e-4b3c-b4b3-83479839c8bb-kube-api-access-sd4ns\") pod \"payload-pre-processing-e2e-inf-b-dd5c8f-6b45577955-57qp6\" (UID: \"8226519e-a60e-4b3c-b4b3-83479839c8bb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd5c8f-6b45577955-57qp6" Oct 30 17:16:27.967323 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:27.967265 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-dd5c8f-796cddf7f4-w8wvf"] Oct 30 17:16:28.054031 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:28.053602 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd4ns\" (UniqueName: \"kubernetes.io/projected/8226519e-a60e-4b3c-b4b3-83479839c8bb-kube-api-access-sd4ns\") pod \"payload-pre-processing-e2e-inf-b-dd5c8f-6b45577955-57qp6\" (UID: \"8226519e-a60e-4b3c-b4b3-83479839c8bb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd5c8f-6b45577955-57qp6" Oct 30 17:16:28.054031 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:28.054003 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x58xt\" (UniqueName: \"kubernetes.io/projected/f897ea49-4823-457b-b3bc-940202d02d40-kube-api-access-x58xt\") pod \"payload-processing-e2e-inf-b-dd5c8f-796cddf7f4-w8wvf\" (UID: \"f897ea49-4823-457b-b3bc-940202d02d40\") " pod="openshift-ingress/payload-processing-e2e-inf-b-dd5c8f-796cddf7f4-w8wvf" Oct 30 17:16:28.060360 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:28.054432 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8226519e-a60e-4b3c-b4b3-83479839c8bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-dd5c8f-6b45577955-57qp6\" (UID: \"8226519e-a60e-4b3c-b4b3-83479839c8bb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd5c8f-6b45577955-57qp6" Oct 30 17:16:28.060360 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:28.054478 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f897ea49-4823-457b-b3bc-940202d02d40-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-dd5c8f-796cddf7f4-w8wvf\" (UID: \"f897ea49-4823-457b-b3bc-940202d02d40\") " pod="openshift-ingress/payload-processing-e2e-inf-b-dd5c8f-796cddf7f4-w8wvf" Oct 30 17:16:28.060360 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:28.059067 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8226519e-a60e-4b3c-b4b3-83479839c8bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-dd5c8f-6b45577955-57qp6\" (UID: \"8226519e-a60e-4b3c-b4b3-83479839c8bb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd5c8f-6b45577955-57qp6" Oct 30 17:16:28.080914 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:28.080888 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd4ns\" (UniqueName: \"kubernetes.io/projected/8226519e-a60e-4b3c-b4b3-83479839c8bb-kube-api-access-sd4ns\") pod \"payload-pre-processing-e2e-inf-b-dd5c8f-6b45577955-57qp6\" (UID: \"8226519e-a60e-4b3c-b4b3-83479839c8bb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd5c8f-6b45577955-57qp6" Oct 30 17:16:28.115639 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:28.115610 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd5c8f-6b45577955-57qp6" Oct 30 17:16:28.156611 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:28.156568 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x58xt\" (UniqueName: \"kubernetes.io/projected/f897ea49-4823-457b-b3bc-940202d02d40-kube-api-access-x58xt\") pod \"payload-processing-e2e-inf-b-dd5c8f-796cddf7f4-w8wvf\" (UID: \"f897ea49-4823-457b-b3bc-940202d02d40\") " pod="openshift-ingress/payload-processing-e2e-inf-b-dd5c8f-796cddf7f4-w8wvf" Oct 30 17:16:28.156970 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:28.156952 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f897ea49-4823-457b-b3bc-940202d02d40-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-dd5c8f-796cddf7f4-w8wvf\" (UID: \"f897ea49-4823-457b-b3bc-940202d02d40\") " pod="openshift-ingress/payload-processing-e2e-inf-b-dd5c8f-796cddf7f4-w8wvf" Oct 30 17:16:28.157655 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:28.157638 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f897ea49-4823-457b-b3bc-940202d02d40-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-dd5c8f-796cddf7f4-w8wvf\" (UID: \"f897ea49-4823-457b-b3bc-940202d02d40\") " pod="openshift-ingress/payload-processing-e2e-inf-b-dd5c8f-796cddf7f4-w8wvf" Oct 30 17:16:28.190081 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:28.190054 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x58xt\" (UniqueName: \"kubernetes.io/projected/f897ea49-4823-457b-b3bc-940202d02d40-kube-api-access-x58xt\") pod \"payload-processing-e2e-inf-b-dd5c8f-796cddf7f4-w8wvf\" (UID: \"f897ea49-4823-457b-b3bc-940202d02d40\") " pod="openshift-ingress/payload-processing-e2e-inf-b-dd5c8f-796cddf7f4-w8wvf" Oct 30 17:16:28.274909 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:28.274878 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-dd5c8f-796cddf7f4-w8wvf" Oct 30 17:16:28.441728 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:16:28.441688 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8226519e_a60e_4b3c_b4b3_83479839c8bb.slice/crio-3f9f84696bd368284ea1e0d8ca2625589e9b5a811b41bc416f1dc94c9069bf76 WatchSource:0}: Error finding container 3f9f84696bd368284ea1e0d8ca2625589e9b5a811b41bc416f1dc94c9069bf76: Status 404 returned error can't find the container with id 3f9f84696bd368284ea1e0d8ca2625589e9b5a811b41bc416f1dc94c9069bf76 Oct 30 17:16:28.442925 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:28.442898 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-dd5c8f-6b45577955-57qp6"] Oct 30 17:16:28.617599 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:16:28.617569 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf897ea49_4823_457b_b3bc_940202d02d40.slice/crio-e16e79b8d1873465de66b503b664c91612820ebacbf999d55d8676bb343d8341 WatchSource:0}: Error finding container e16e79b8d1873465de66b503b664c91612820ebacbf999d55d8676bb343d8341: Status 404 returned error can't find the container with id e16e79b8d1873465de66b503b664c91612820ebacbf999d55d8676bb343d8341 Oct 30 17:16:28.620887 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:28.620863 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-dd5c8f-796cddf7f4-w8wvf"] Oct 30 17:16:28.876448 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:28.876340 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd5c8f-6b45577955-57qp6" event={"ID":"8226519e-a60e-4b3c-b4b3-83479839c8bb","Type":"ContainerStarted","Data":"05d34dc64a6d870192f7f10e27988f3ecd79d0874121e06516da2a4a9a6e3d9d"} Oct 30 17:16:28.876448 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:28.876404 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd5c8f-6b45577955-57qp6" event={"ID":"8226519e-a60e-4b3c-b4b3-83479839c8bb","Type":"ContainerStarted","Data":"3f9f84696bd368284ea1e0d8ca2625589e9b5a811b41bc416f1dc94c9069bf76"} Oct 30 17:16:28.876448 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:28.876433 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd5c8f-6b45577955-57qp6" Oct 30 17:16:28.878092 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:28.878066 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-dd5c8f-796cddf7f4-w8wvf" event={"ID":"f897ea49-4823-457b-b3bc-940202d02d40","Type":"ContainerStarted","Data":"23df8e36fd7f395e403fb6b80bb2f33a09a5a245ae0ee0695c25798d36f44509"} Oct 30 17:16:28.878214 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:28.878116 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-dd5c8f-796cddf7f4-w8wvf" event={"ID":"f897ea49-4823-457b-b3bc-940202d02d40","Type":"ContainerStarted","Data":"e16e79b8d1873465de66b503b664c91612820ebacbf999d55d8676bb343d8341"} Oct 30 17:16:28.878214 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:28.878146 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-dd5c8f-796cddf7f4-w8wvf" Oct 30 17:16:28.921349 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:28.921283 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd5c8f-6b45577955-57qp6" podStartSLOduration=1.921270456 podStartE2EDuration="1.921270456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:16:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:16:28.920837558 +0000 UTC m=+1762.921592462" watchObservedRunningTime="2026-10-30 17:16:28.921270456 +0000 UTC m=+1762.922025357" Oct 30 17:16:28.967833 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:28.967777 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-dd5c8f-796cddf7f4-w8wvf" podStartSLOduration=1.967763985 podStartE2EDuration="1.967763985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:16:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:16:28.967390891 +0000 UTC m=+1762.968145792" watchObservedRunningTime="2026-10-30 17:16:28.967763985 +0000 UTC m=+1762.968518886" Oct 30 17:16:30.157433 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:30.157407 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs" Oct 30 17:16:30.219857 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:30.219824 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ecae9c7-ada3-4612-909f-1c1493369daf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecae9c7-ada3-4612-909f-1c1493369daf-workload-socket\") pod \"2ecae9c7-ada3-4612-909f-1c1493369daf\" (UID: \"2ecae9c7-ada3-4612-909f-1c1493369daf\") " Oct 30 17:16:30.220177 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:30.220137 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2ecae9c7-ada3-4612-909f-1c1493369daf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ecae9c7-ada3-4612-909f-1c1493369daf-istio-podinfo\") pod \"2ecae9c7-ada3-4612-909f-1c1493369daf\" (UID: \"2ecae9c7-ada3-4612-909f-1c1493369daf\") " Oct 30 17:16:30.220322 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:30.220222 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ecae9c7-ada3-4612-909f-1c1493369daf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ecae9c7-ada3-4612-909f-1c1493369daf-workload-certs\") pod \"2ecae9c7-ada3-4612-909f-1c1493369daf\" (UID: \"2ecae9c7-ada3-4612-909f-1c1493369daf\") " Oct 30 17:16:30.220322 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:30.220258 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ecae9c7-ada3-4612-909f-1c1493369daf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ecae9c7-ada3-4612-909f-1c1493369daf-istiod-ca-cert\") pod \"2ecae9c7-ada3-4612-909f-1c1493369daf\" (UID: \"2ecae9c7-ada3-4612-909f-1c1493369daf\") " Oct 30 17:16:30.220452 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:30.220321 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ecae9c7-ada3-4612-909f-1c1493369daf-kube-api-access-rp7rj\" (UniqueName: \"kubernetes.io/projected/2ecae9c7-ada3-4612-909f-1c1493369daf-kube-api-access-rp7rj\") pod \"2ecae9c7-ada3-4612-909f-1c1493369daf\" (UID: \"2ecae9c7-ada3-4612-909f-1c1493369daf\") " Oct 30 17:16:30.220579 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:30.220543 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ecae9c7-ada3-4612-909f-1c1493369daf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ecae9c7-ada3-4612-909f-1c1493369daf-istio-data\") pod \"2ecae9c7-ada3-4612-909f-1c1493369daf\" (UID: \"2ecae9c7-ada3-4612-909f-1c1493369daf\") " Oct 30 17:16:30.220734 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:30.220713 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ecae9c7-ada3-4612-909f-1c1493369daf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecae9c7-ada3-4612-909f-1c1493369daf-credential-socket\") pod \"2ecae9c7-ada3-4612-909f-1c1493369daf\" (UID: \"2ecae9c7-ada3-4612-909f-1c1493369daf\") " Oct 30 17:16:30.220805 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:30.220765 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ecae9c7-ada3-4612-909f-1c1493369daf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ecae9c7-ada3-4612-909f-1c1493369daf-istio-envoy\") pod \"2ecae9c7-ada3-4612-909f-1c1493369daf\" (UID: \"2ecae9c7-ada3-4612-909f-1c1493369daf\") " Oct 30 17:16:30.220872 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:30.220853 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ecae9c7-ada3-4612-909f-1c1493369daf-istio-token\" (UniqueName: \"kubernetes.io/projected/2ecae9c7-ada3-4612-909f-1c1493369daf-istio-token\") pod \"2ecae9c7-ada3-4612-909f-1c1493369daf\" (UID: \"2ecae9c7-ada3-4612-909f-1c1493369daf\") " Oct 30 17:16:30.221139 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:30.221114 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ecae9c7-ada3-4612-909f-1c1493369daf-credential-socket" pod "2ecae9c7-ada3-4612-909f-1c1493369daf" (UID: "2ecae9c7-ada3-4612-909f-1c1493369daf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:30.221231 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:30.221210 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ecae9c7-ada3-4612-909f-1c1493369daf-istio-data" pod "2ecae9c7-ada3-4612-909f-1c1493369daf" (UID: "2ecae9c7-ada3-4612-909f-1c1493369daf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:30.221291 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:30.221264 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ecae9c7-ada3-4612-909f-1c1493369daf-workload-certs" pod "2ecae9c7-ada3-4612-909f-1c1493369daf" (UID: "2ecae9c7-ada3-4612-909f-1c1493369daf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:30.221407 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:30.221279 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ecae9c7-ada3-4612-909f-1c1493369daf-workload-socket" pod "2ecae9c7-ada3-4612-909f-1c1493369daf" (UID: "2ecae9c7-ada3-4612-909f-1c1493369daf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:30.221474 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:30.221445 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ecae9c7-ada3-4612-909f-1c1493369daf-istiod-ca-cert" pod "2ecae9c7-ada3-4612-909f-1c1493369daf" (UID: "2ecae9c7-ada3-4612-909f-1c1493369daf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:16:30.222228 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:30.222196 4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecae9c7-ada3-4612-909f-1c1493369daf-workload-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:16:30.222373 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:30.222231 4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ecae9c7-ada3-4612-909f-1c1493369daf-workload-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:16:30.222373 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:30.222246 4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ecae9c7-ada3-4612-909f-1c1493369daf-istiod-ca-cert\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:16:30.222373 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:30.222260 4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ecae9c7-ada3-4612-909f-1c1493369daf-istio-data\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:16:30.222373 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:30.222274 4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecae9c7-ada3-4612-909f-1c1493369daf-credential-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:16:30.223262 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:30.223217 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ecae9c7-ada3-4612-909f-1c1493369daf-istio-podinfo" pod "2ecae9c7-ada3-4612-909f-1c1493369daf" (UID: "2ecae9c7-ada3-4612-909f-1c1493369daf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:16:30.223801 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:30.223768 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ecae9c7-ada3-4612-909f-1c1493369daf-istio-token" pod "2ecae9c7-ada3-4612-909f-1c1493369daf" (UID: "2ecae9c7-ada3-4612-909f-1c1493369daf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:30.223912 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:30.223890 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ecae9c7-ada3-4612-909f-1c1493369daf-kube-api-access-rp7rj" pod "2ecae9c7-ada3-4612-909f-1c1493369daf" (UID: "2ecae9c7-ada3-4612-909f-1c1493369daf"). InnerVolumeSpecName "kube-api-access-rp7rj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:30.223975 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:30.223921 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ecae9c7-ada3-4612-909f-1c1493369daf-istio-envoy" pod "2ecae9c7-ada3-4612-909f-1c1493369daf" (UID: "2ecae9c7-ada3-4612-909f-1c1493369daf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:30.323321 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:30.323268 4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ecae9c7-ada3-4612-909f-1c1493369daf-istio-envoy\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:16:30.323321 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:30.323297 4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ecae9c7-ada3-4612-909f-1c1493369daf-istio-token\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:16:30.323321 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:30.323321 4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ecae9c7-ada3-4612-909f-1c1493369daf-istio-podinfo\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:16:30.323564 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:30.323336 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp7rj\" (UniqueName: \"kubernetes.io/projected/2ecae9c7-ada3-4612-909f-1c1493369daf-kube-api-access-rp7rj\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:16:30.896387 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:30.896351 4522 generic.go:356] "Generic (PLEG): container finished" podID="2ecae9c7-ada3-4612-909f-1c1493369daf" containerID="8be21e369bb85f51030c572ee03a89ee10e1ca6c400efcf5ad5e38d1941d3d51" exitCode=0 Oct 30 17:16:30.896552 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:30.896421 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs" Oct 30 17:16:30.896552 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:30.896438 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs" event={"ID":"2ecae9c7-ada3-4612-909f-1c1493369daf","Type":"ContainerDied","Data":"8be21e369bb85f51030c572ee03a89ee10e1ca6c400efcf5ad5e38d1941d3d51"} Oct 30 17:16:30.896552 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:30.896532 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs" event={"ID":"2ecae9c7-ada3-4612-909f-1c1493369daf","Type":"ContainerDied","Data":"5bf13af2d1d8183b5ee4cc9007f301ebabd4a0bfa12b329c0dc1a675201e50d3"} Oct 30 17:16:30.896687 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:30.896560 4522 scope.go:122] "RemoveContainer" containerID="8be21e369bb85f51030c572ee03a89ee10e1ca6c400efcf5ad5e38d1941d3d51" Oct 30 17:16:30.908858 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:30.908838 4522 scope.go:122] "RemoveContainer" containerID="8be21e369bb85f51030c572ee03a89ee10e1ca6c400efcf5ad5e38d1941d3d51" Oct 30 17:16:30.909113 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:16:30.909092 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8be21e369bb85f51030c572ee03a89ee10e1ca6c400efcf5ad5e38d1941d3d51\": container with ID starting with 8be21e369bb85f51030c572ee03a89ee10e1ca6c400efcf5ad5e38d1941d3d51 not found: ID does not exist" containerID="8be21e369bb85f51030c572ee03a89ee10e1ca6c400efcf5ad5e38d1941d3d51" Oct 30 17:16:30.909175 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:30.909121 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8be21e369bb85f51030c572ee03a89ee10e1ca6c400efcf5ad5e38d1941d3d51"} err="failed to get container status \"8be21e369bb85f51030c572ee03a89ee10e1ca6c400efcf5ad5e38d1941d3d51\": rpc error: code = NotFound desc = could not find container \"8be21e369bb85f51030c572ee03a89ee10e1ca6c400efcf5ad5e38d1941d3d51\": container with ID starting with 8be21e369bb85f51030c572ee03a89ee10e1ca6c400efcf5ad5e38d1941d3d51 not found: ID does not exist" Oct 30 17:16:30.971802 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:30.971760 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs"] Oct 30 17:16:30.979696 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:30.979666 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs"] Oct 30 17:16:31.128577 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:31.128539 4522 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:16:31.128759 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:31.128616 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-m8vvs" podUID="2ecae9c7-ada3-4612-909f-1c1493369daf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:16:32.147203 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:32.147158 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ecae9c7-ada3-4612-909f-1c1493369daf" path="/var/lib/kubelet/pods/2ecae9c7-ada3-4612-909f-1c1493369daf/volumes" Oct 30 17:16:36.829344 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:36.829270 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7"] Oct 30 17:16:36.831634 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:36.831608 4522 cpu_manager.go:414] "containerMap: removing container" podUID="2ecae9c7-ada3-4612-909f-1c1493369daf" containerName="istio-proxy" Oct 30 17:16:36.831634 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:36.831635 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ecae9c7-ada3-4612-909f-1c1493369daf" containerName="istio-proxy" Oct 30 17:16:36.831886 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:36.831822 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ecae9c7-ada3-4612-909f-1c1493369daf" containerName="istio-proxy" Oct 30 17:16:36.840770 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:36.840741 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7" Oct 30 17:16:36.856442 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:36.856414 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7"] Oct 30 17:16:36.873848 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:36.873825 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-42d381-openshift-default-dockercfg-gf7bd\"" Oct 30 17:16:36.967041 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:36.967012 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb91a3a6-25a0-4c9b-9564-8221f24efa30-workload-certs\") pod \"e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7\" (UID: \"eb91a3a6-25a0-4c9b-9564-8221f24efa30\") " pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7" Oct 30 17:16:36.967219 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:36.967074 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb91a3a6-25a0-4c9b-9564-8221f24efa30-istio-token\") pod \"e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7\" (UID: \"eb91a3a6-25a0-4c9b-9564-8221f24efa30\") " pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7" Oct 30 17:16:36.967219 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:36.967131 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvkf5\" (UniqueName: \"kubernetes.io/projected/eb91a3a6-25a0-4c9b-9564-8221f24efa30-kube-api-access-fvkf5\") pod \"e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7\" (UID: \"eb91a3a6-25a0-4c9b-9564-8221f24efa30\") " pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7" Oct 30 17:16:36.967351 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:36.967301 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb91a3a6-25a0-4c9b-9564-8221f24efa30-istio-podinfo\") pod \"e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7\" (UID: \"eb91a3a6-25a0-4c9b-9564-8221f24efa30\") " pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7" Oct 30 17:16:36.967429 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:36.967416 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb91a3a6-25a0-4c9b-9564-8221f24efa30-credential-socket\") pod \"e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7\" (UID: \"eb91a3a6-25a0-4c9b-9564-8221f24efa30\") " pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7" Oct 30 17:16:36.967486 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:36.967467 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb91a3a6-25a0-4c9b-9564-8221f24efa30-istiod-ca-cert\") pod \"e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7\" (UID: \"eb91a3a6-25a0-4c9b-9564-8221f24efa30\") " pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7" Oct 30 17:16:36.967574 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:36.967560 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb91a3a6-25a0-4c9b-9564-8221f24efa30-workload-socket\") pod \"e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7\" (UID: \"eb91a3a6-25a0-4c9b-9564-8221f24efa30\") " pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7" Oct 30 17:16:36.967616 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:36.967603 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb91a3a6-25a0-4c9b-9564-8221f24efa30-istio-envoy\") pod \"e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7\" (UID: \"eb91a3a6-25a0-4c9b-9564-8221f24efa30\") " pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7" Oct 30 17:16:36.967681 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:36.967667 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb91a3a6-25a0-4c9b-9564-8221f24efa30-istio-data\") pod \"e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7\" (UID: \"eb91a3a6-25a0-4c9b-9564-8221f24efa30\") " pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7" Oct 30 17:16:37.069677 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:37.069637 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb91a3a6-25a0-4c9b-9564-8221f24efa30-istio-podinfo\") pod \"e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7\" (UID: \"eb91a3a6-25a0-4c9b-9564-8221f24efa30\") " pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7" Oct 30 17:16:37.069895 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:37.069751 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb91a3a6-25a0-4c9b-9564-8221f24efa30-credential-socket\") pod \"e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7\" (UID: \"eb91a3a6-25a0-4c9b-9564-8221f24efa30\") " pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7" Oct 30 17:16:37.069895 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:37.069782 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb91a3a6-25a0-4c9b-9564-8221f24efa30-istiod-ca-cert\") pod \"e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7\" (UID: \"eb91a3a6-25a0-4c9b-9564-8221f24efa30\") " pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7" Oct 30 17:16:37.069895 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:37.069871 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb91a3a6-25a0-4c9b-9564-8221f24efa30-workload-socket\") pod \"e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7\" (UID: \"eb91a3a6-25a0-4c9b-9564-8221f24efa30\") " pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7" Oct 30 17:16:37.070081 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:37.069934 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb91a3a6-25a0-4c9b-9564-8221f24efa30-istio-envoy\") pod \"e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7\" (UID: \"eb91a3a6-25a0-4c9b-9564-8221f24efa30\") " pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7" Oct 30 17:16:37.070081 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:37.069989 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb91a3a6-25a0-4c9b-9564-8221f24efa30-istio-data\") pod \"e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7\" (UID: \"eb91a3a6-25a0-4c9b-9564-8221f24efa30\") " pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7" Oct 30 17:16:37.070081 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:37.070060 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb91a3a6-25a0-4c9b-9564-8221f24efa30-workload-certs\") pod \"e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7\" (UID: \"eb91a3a6-25a0-4c9b-9564-8221f24efa30\") " pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7" Oct 30 17:16:37.070240 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:37.070142 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb91a3a6-25a0-4c9b-9564-8221f24efa30-istio-token\") pod \"e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7\" (UID: \"eb91a3a6-25a0-4c9b-9564-8221f24efa30\") " pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7" Oct 30 17:16:37.070240 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:37.070168 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvkf5\" (UniqueName: \"kubernetes.io/projected/eb91a3a6-25a0-4c9b-9564-8221f24efa30-kube-api-access-fvkf5\") pod \"e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7\" (UID: \"eb91a3a6-25a0-4c9b-9564-8221f24efa30\") " pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7" Oct 30 17:16:37.070373 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:37.070281 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb91a3a6-25a0-4c9b-9564-8221f24efa30-credential-socket\") pod \"e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7\" (UID: \"eb91a3a6-25a0-4c9b-9564-8221f24efa30\") " pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7" Oct 30 17:16:37.070638 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:37.070609 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb91a3a6-25a0-4c9b-9564-8221f24efa30-workload-certs\") pod \"e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7\" (UID: \"eb91a3a6-25a0-4c9b-9564-8221f24efa30\") " pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7" Oct 30 17:16:37.070946 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:37.070918 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb91a3a6-25a0-4c9b-9564-8221f24efa30-istio-data\") pod \"e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7\" (UID: \"eb91a3a6-25a0-4c9b-9564-8221f24efa30\") " pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7" Oct 30 17:16:37.070946 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:37.070935 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb91a3a6-25a0-4c9b-9564-8221f24efa30-istiod-ca-cert\") pod \"e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7\" (UID: \"eb91a3a6-25a0-4c9b-9564-8221f24efa30\") " pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7" Oct 30 17:16:37.071094 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:37.070937 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb91a3a6-25a0-4c9b-9564-8221f24efa30-workload-socket\") pod \"e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7\" (UID: \"eb91a3a6-25a0-4c9b-9564-8221f24efa30\") " pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7" Oct 30 17:16:37.073928 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:37.073878 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb91a3a6-25a0-4c9b-9564-8221f24efa30-istio-podinfo\") pod \"e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7\" (UID: \"eb91a3a6-25a0-4c9b-9564-8221f24efa30\") " pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7" Oct 30 17:16:37.074815 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:37.074793 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb91a3a6-25a0-4c9b-9564-8221f24efa30-istio-envoy\") pod \"e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7\" (UID: \"eb91a3a6-25a0-4c9b-9564-8221f24efa30\") " pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7" Oct 30 17:16:37.143426 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:37.126224 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb91a3a6-25a0-4c9b-9564-8221f24efa30-istio-token\") pod \"e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7\" (UID: \"eb91a3a6-25a0-4c9b-9564-8221f24efa30\") " pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7" Oct 30 17:16:37.163291 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:37.163255 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvkf5\" (UniqueName: \"kubernetes.io/projected/eb91a3a6-25a0-4c9b-9564-8221f24efa30-kube-api-access-fvkf5\") pod \"e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7\" (UID: \"eb91a3a6-25a0-4c9b-9564-8221f24efa30\") " pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7" Oct 30 17:16:37.183780 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:37.183744 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7" Oct 30 17:16:37.685289 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:16:37.685251 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb91a3a6_25a0_4c9b_9564_8221f24efa30.slice/crio-5b7ecc8ff20f732d9d7620bbf154d0fb2eb3ff44d0b2392791953c4aef64562d WatchSource:0}: Error finding container 5b7ecc8ff20f732d9d7620bbf154d0fb2eb3ff44d0b2392791953c4aef64562d: Status 404 returned error can't find the container with id 5b7ecc8ff20f732d9d7620bbf154d0fb2eb3ff44d0b2392791953c4aef64562d Oct 30 17:16:37.687572 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:37.687544 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7"] Oct 30 17:16:37.688948 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:37.688913 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:16:37.689032 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:37.688981 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:16:37.689032 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:37.689011 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:16:37.962829 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:37.962795 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7" event={"ID":"eb91a3a6-25a0-4c9b-9564-8221f24efa30","Type":"ContainerStarted","Data":"67a9433870f1ece7c693445db5d4e4ec6ac14a3d8448ef0a3cfcfc178b256f36"} Oct 30 17:16:37.963349 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:37.962862 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7" event={"ID":"eb91a3a6-25a0-4c9b-9564-8221f24efa30","Type":"ContainerStarted","Data":"5b7ecc8ff20f732d9d7620bbf154d0fb2eb3ff44d0b2392791953c4aef64562d"} Oct 30 17:16:38.020518 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:38.020443 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7" podStartSLOduration=2.020425632 podStartE2EDuration="2.020425632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:16:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:16:38.019591119 +0000 UTC m=+1772.020346020" watchObservedRunningTime="2026-10-30 17:16:38.020425632 +0000 UTC m=+1772.021180532" Oct 30 17:16:38.185058 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:38.185028 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7" Oct 30 17:16:38.186602 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:38.186579 4522 patch_prober.go:29] interesting pod/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Oct 30 17:16:38.186703 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:38.186635 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7" podUID="eb91a3a6-25a0-4c9b-9564-8221f24efa30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Oct 30 17:16:39.190067 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:39.190036 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7" Oct 30 17:16:39.890112 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:39.890083 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-dd5c8f-796cddf7f4-w8wvf" Oct 30 17:16:39.890391 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:39.890213 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd5c8f-6b45577955-57qp6" Oct 30 17:16:39.983479 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:39.983447 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7" Oct 30 17:16:39.984497 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:39.984469 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7" Oct 30 17:16:42.950436 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:16:42.950393 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:16:43.267450 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:43.267412 4522 patch_prober.go:29] interesting pod/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:16:43.267740 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:43.267466 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd" podUID="065164fc-1823-49af-9921-d14d9a94e997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:16:43.990089 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:43.990047 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-42d381-85f9dccd5f-fkl4k"] Oct 30 17:16:43.995020 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:43.994996 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-42d381-85f9dccd5f-fkl4k" Oct 30 17:16:44.015416 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:44.015392 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-42d381-85f9dccd5f-fkl4k"] Oct 30 17:16:44.023582 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:44.023555 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-42d381\"" Oct 30 17:16:44.023740 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:44.023617 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-42d381-dockercfg-8bvjw\"" Oct 30 17:16:44.028045 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:44.028020 4522 generic.go:356] "Generic (PLEG): container finished" podID="065164fc-1823-49af-9921-d14d9a94e997" containerID="290cce3c9c779f0915bd8a6795025cd875f0cd66d2d3588424cfd0536ee49648" exitCode=0 Oct 30 17:16:44.028336 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:44.028052 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd" event={"ID":"065164fc-1823-49af-9921-d14d9a94e997","Type":"ContainerDied","Data":"290cce3c9c779f0915bd8a6795025cd875f0cd66d2d3588424cfd0536ee49648"} Oct 30 17:16:44.028996 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:44.028963 4522 scope.go:122] "RemoveContainer" containerID="290cce3c9c779f0915bd8a6795025cd875f0cd66d2d3588424cfd0536ee49648" Oct 30 17:16:44.031185 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:44.031145 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:16:44.031336 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:44.031236 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:16:44.031336 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:44.031283 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:16:44.049022 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:44.048991 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9710d6d6-5a90-4b5d-b7fe-5835b6323136-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-42d381-85f9dccd5f-fkl4k\" (UID: \"9710d6d6-5a90-4b5d-b7fe-5835b6323136\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-42d381-85f9dccd5f-fkl4k" Oct 30 17:16:44.049192 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:44.049174 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfwvx\" (UniqueName: \"kubernetes.io/projected/9710d6d6-5a90-4b5d-b7fe-5835b6323136-kube-api-access-rfwvx\") pod \"payload-pre-processing-e2e-inf-a-42d381-85f9dccd5f-fkl4k\" (UID: \"9710d6d6-5a90-4b5d-b7fe-5835b6323136\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-42d381-85f9dccd5f-fkl4k" Oct 30 17:16:44.151016 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:44.150976 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9710d6d6-5a90-4b5d-b7fe-5835b6323136-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-42d381-85f9dccd5f-fkl4k\" (UID: \"9710d6d6-5a90-4b5d-b7fe-5835b6323136\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-42d381-85f9dccd5f-fkl4k" Oct 30 17:16:44.151186 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:44.151169 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfwvx\" (UniqueName: \"kubernetes.io/projected/9710d6d6-5a90-4b5d-b7fe-5835b6323136-kube-api-access-rfwvx\") pod \"payload-pre-processing-e2e-inf-a-42d381-85f9dccd5f-fkl4k\" (UID: \"9710d6d6-5a90-4b5d-b7fe-5835b6323136\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-42d381-85f9dccd5f-fkl4k" Oct 30 17:16:44.152712 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:44.152681 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9710d6d6-5a90-4b5d-b7fe-5835b6323136-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-42d381-85f9dccd5f-fkl4k\" (UID: \"9710d6d6-5a90-4b5d-b7fe-5835b6323136\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-42d381-85f9dccd5f-fkl4k" Oct 30 17:16:44.182087 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:44.182057 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfwvx\" (UniqueName: \"kubernetes.io/projected/9710d6d6-5a90-4b5d-b7fe-5835b6323136-kube-api-access-rfwvx\") pod \"payload-pre-processing-e2e-inf-a-42d381-85f9dccd5f-fkl4k\" (UID: \"9710d6d6-5a90-4b5d-b7fe-5835b6323136\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-42d381-85f9dccd5f-fkl4k" Oct 30 17:16:44.331868 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:44.331770 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-42d381-85f9dccd5f-fkl4k" Oct 30 17:16:44.827983 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:16:44.827940 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9710d6d6_5a90_4b5d_b7fe_5835b6323136.slice/crio-f1f078a6fb778018a58bb7a0963ba3c6e9783f264cc912f30036a651b65ebf36 WatchSource:0}: Error finding container f1f078a6fb778018a58bb7a0963ba3c6e9783f264cc912f30036a651b65ebf36: Status 404 returned error can't find the container with id f1f078a6fb778018a58bb7a0963ba3c6e9783f264cc912f30036a651b65ebf36 Oct 30 17:16:44.835407 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:44.835342 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-42d381-85f9dccd5f-fkl4k"] Oct 30 17:16:45.038775 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:45.038741 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-42d381-85f9dccd5f-fkl4k" event={"ID":"9710d6d6-5a90-4b5d-b7fe-5835b6323136","Type":"ContainerStarted","Data":"57380519ba24aab22c522722eae8974a4907ec55450e7d27c9fe9b979e774459"} Oct 30 17:16:45.039227 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:45.038807 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-42d381-85f9dccd5f-fkl4k" Oct 30 17:16:45.039227 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:45.038829 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-42d381-85f9dccd5f-fkl4k" event={"ID":"9710d6d6-5a90-4b5d-b7fe-5835b6323136","Type":"ContainerStarted","Data":"f1f078a6fb778018a58bb7a0963ba3c6e9783f264cc912f30036a651b65ebf36"} Oct 30 17:16:45.040974 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:45.040947 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd" event={"ID":"065164fc-1823-49af-9921-d14d9a94e997","Type":"ContainerStarted","Data":"c821d519a29e83e3094a31fc7b7075ba60e52c3c1105c62637d854ce8de266fb"} Oct 30 17:16:45.086144 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:45.086052 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-42d381-85f9dccd5f-fkl4k" podStartSLOduration=2.086039302 podStartE2EDuration="2.086039302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:16:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:16:45.084357479 +0000 UTC m=+1779.085112382" watchObservedRunningTime="2026-10-30 17:16:45.086039302 +0000 UTC m=+1779.086794197" Oct 30 17:16:45.416396 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:45.415922 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd" Oct 30 17:16:45.416396 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:45.416097 4522 patch_prober.go:29] interesting pod/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 30 17:16:45.416396 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:45.416145 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd" podUID="065164fc-1823-49af-9921-d14d9a94e997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 30 17:16:46.416767 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:46.416723 4522 patch_prober.go:29] interesting pod/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 30 17:16:46.417256 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:46.416795 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd" podUID="065164fc-1823-49af-9921-d14d9a94e997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 30 17:16:47.407329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:47.404679 4522 patch_prober.go:29] interesting pod/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 30 17:16:47.407329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:47.404739 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd" podUID="065164fc-1823-49af-9921-d14d9a94e997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 30 17:16:48.410331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:48.400701 4522 patch_prober.go:29] interesting pod/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 30 17:16:48.410331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:48.400764 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd" podUID="065164fc-1823-49af-9921-d14d9a94e997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 30 17:16:49.414924 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:49.414887 4522 patch_prober.go:29] interesting pod/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 30 17:16:49.415561 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:49.415523 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd" podUID="065164fc-1823-49af-9921-d14d9a94e997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 30 17:16:49.740872 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:49.740835 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39fecf-2gx5j"] Oct 30 17:16:49.744929 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:49.744903 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39fecf-2gx5j"] Oct 30 17:16:50.147320 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:50.147229 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35e117d2-0492-4c63-8af7-2aaa14e7edb3" path="/var/lib/kubelet/pods/35e117d2-0492-4c63-8af7-2aaa14e7edb3/volumes" Oct 30 17:16:50.396737 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:50.396690 4522 patch_prober.go:29] interesting pod/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 30 17:16:50.396951 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:50.396749 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd" podUID="065164fc-1823-49af-9921-d14d9a94e997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 30 17:16:51.400487 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:51.400443 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd" Oct 30 17:16:51.401024 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:51.400890 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd" Oct 30 17:16:51.401687 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:51.401665 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd" Oct 30 17:16:56.053463 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:16:56.053435 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-42d381-85f9dccd5f-fkl4k" Oct 30 17:17:06.120700 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:06.120666 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-987dfc-85f675bb7f-phjx7"] Oct 30 17:17:06.126155 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:06.126131 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-987dfc-85f675bb7f-phjx7" Oct 30 17:17:06.135217 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:06.135188 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-987dfc-85f675bb7f-phjx7"] Oct 30 17:17:06.160826 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:06.160770 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-987dfc\"" Oct 30 17:17:06.163152 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:06.163107 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-987dfc-dockercfg-f45hh\"" Oct 30 17:17:06.305582 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:06.288492 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceac6a2a-ea4a-45ca-a6f0-2beb70ca52d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-987dfc-85f675bb7f-phjx7\" (UID: \"ceac6a2a-ea4a-45ca-a6f0-2beb70ca52d2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-987dfc-85f675bb7f-phjx7" Oct 30 17:17:06.306125 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:06.306095 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jclr2\" (UniqueName: \"kubernetes.io/projected/ceac6a2a-ea4a-45ca-a6f0-2beb70ca52d2-kube-api-access-jclr2\") pod \"payload-pre-processing-e2e-shared-a-w4-987dfc-85f675bb7f-phjx7\" (UID: \"ceac6a2a-ea4a-45ca-a6f0-2beb70ca52d2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-987dfc-85f675bb7f-phjx7" Oct 30 17:17:06.412188 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:06.410983 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceac6a2a-ea4a-45ca-a6f0-2beb70ca52d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-987dfc-85f675bb7f-phjx7\" (UID: \"ceac6a2a-ea4a-45ca-a6f0-2beb70ca52d2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-987dfc-85f675bb7f-phjx7" Oct 30 17:17:06.462349 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:06.458521 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jclr2\" (UniqueName: \"kubernetes.io/projected/ceac6a2a-ea4a-45ca-a6f0-2beb70ca52d2-kube-api-access-jclr2\") pod \"payload-pre-processing-e2e-shared-a-w4-987dfc-85f675bb7f-phjx7\" (UID: \"ceac6a2a-ea4a-45ca-a6f0-2beb70ca52d2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-987dfc-85f675bb7f-phjx7" Oct 30 17:17:06.462349 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:06.413340 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceac6a2a-ea4a-45ca-a6f0-2beb70ca52d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-987dfc-85f675bb7f-phjx7\" (UID: \"ceac6a2a-ea4a-45ca-a6f0-2beb70ca52d2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-987dfc-85f675bb7f-phjx7" Oct 30 17:17:06.489424 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:06.489230 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jclr2\" (UniqueName: \"kubernetes.io/projected/ceac6a2a-ea4a-45ca-a6f0-2beb70ca52d2-kube-api-access-jclr2\") pod \"payload-pre-processing-e2e-shared-a-w4-987dfc-85f675bb7f-phjx7\" (UID: \"ceac6a2a-ea4a-45ca-a6f0-2beb70ca52d2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-987dfc-85f675bb7f-phjx7" Oct 30 17:17:06.770391 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:06.770356 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-987dfc-85f675bb7f-phjx7" Oct 30 17:17:07.122300 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:17:07.122269 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceac6a2a_ea4a_45ca_a6f0_2beb70ca52d2.slice/crio-a1134d581c3f833d33bff30d9ff0c085bf077c710c8b2b56fbbe53035726cf75 WatchSource:0}: Error finding container a1134d581c3f833d33bff30d9ff0c085bf077c710c8b2b56fbbe53035726cf75: Status 404 returned error can't find the container with id a1134d581c3f833d33bff30d9ff0c085bf077c710c8b2b56fbbe53035726cf75 Oct 30 17:17:07.122715 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:07.122689 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-987dfc-85f675bb7f-phjx7"] Oct 30 17:17:07.543913 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:07.543876 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-987dfc-85f675bb7f-phjx7" event={"ID":"ceac6a2a-ea4a-45ca-a6f0-2beb70ca52d2","Type":"ContainerStarted","Data":"917a434c5fca903d113330d152bee12c3180782fa101ba7603c82ba4e1a5e8d7"} Oct 30 17:17:07.544112 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:07.543933 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-987dfc-85f675bb7f-phjx7" event={"ID":"ceac6a2a-ea4a-45ca-a6f0-2beb70ca52d2","Type":"ContainerStarted","Data":"a1134d581c3f833d33bff30d9ff0c085bf077c710c8b2b56fbbe53035726cf75"} Oct 30 17:17:07.544112 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:07.543963 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-987dfc-85f675bb7f-phjx7" Oct 30 17:17:07.619217 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:07.619142 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-987dfc-85f675bb7f-phjx7" podStartSLOduration=1.619120463 podStartE2EDuration="1.619120463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:17:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:17:07.617536879 +0000 UTC m=+1801.618291780" watchObservedRunningTime="2026-10-30 17:17:07.619120463 +0000 UTC m=+1801.619875365" Oct 30 17:17:08.444052 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:08.444012 4522 scope.go:122] "RemoveContainer" containerID="41456b0436842af2976abe5a579dbb7f43c0d600fe5649cc9e0aa17ab6a710f0" Oct 30 17:17:08.456161 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:08.456140 4522 scope.go:122] "RemoveContainer" containerID="c96540258c0cfd6aa00452fd8e8b2577dc5b7f5cccbe23146422a6272051a866" Oct 30 17:17:08.468645 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:08.468623 4522 scope.go:122] "RemoveContainer" containerID="602aeed98b61acf04200090738ece7e608656c04e428e159c582b6075dce6eb1" Oct 30 17:17:12.949451 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:17:12.949412 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:17:14.608489 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:14.608448 4522 generic.go:356] "Generic (PLEG): container finished" podID="40baffd6-35b6-412d-a7bd-f3155ed306db" containerID="674dcc578e0637f40a21dfbc09d9216f80795539a9e7fd2fd1c29bf09d61fc05" exitCode=0 Oct 30 17:17:14.609002 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:14.608523 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm" event={"ID":"40baffd6-35b6-412d-a7bd-f3155ed306db","Type":"ContainerDied","Data":"674dcc578e0637f40a21dfbc09d9216f80795539a9e7fd2fd1c29bf09d61fc05"} Oct 30 17:17:14.609058 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:14.609016 4522 scope.go:122] "RemoveContainer" containerID="674dcc578e0637f40a21dfbc09d9216f80795539a9e7fd2fd1c29bf09d61fc05" Oct 30 17:17:14.611087 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:14.611058 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:17:14.611181 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:14.611132 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:17:14.611181 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:14.611166 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:17:15.625895 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:15.625851 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm" event={"ID":"40baffd6-35b6-412d-a7bd-f3155ed306db","Type":"ContainerStarted","Data":"3f880d54c1afb9e761182f68fcd858d1e2da519556f253e8e462e711925cd876"} Oct 30 17:17:15.739879 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:15.738860 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm" Oct 30 17:17:15.740818 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:15.740781 4522 patch_prober.go:29] interesting pod/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body= Oct 30 17:17:15.740927 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:15.740834 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm" podUID="40baffd6-35b6-412d-a7bd-f3155ed306db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" Oct 30 17:17:16.339199 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:16.339166 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm" Oct 30 17:17:16.724882 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:16.724845 4522 patch_prober.go:29] interesting pod/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body= Oct 30 17:17:16.725284 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:16.724940 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm" podUID="40baffd6-35b6-412d-a7bd-f3155ed306db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" Oct 30 17:17:17.724758 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:17.724726 4522 patch_prober.go:29] interesting pod/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body= Oct 30 17:17:17.725043 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:17.724775 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm" podUID="40baffd6-35b6-412d-a7bd-f3155ed306db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" Oct 30 17:17:18.556568 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:18.556539 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-987dfc-85f675bb7f-phjx7" Oct 30 17:17:18.724211 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:18.724175 4522 patch_prober.go:29] interesting pod/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body= Oct 30 17:17:18.724424 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:18.724242 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm" podUID="40baffd6-35b6-412d-a7bd-f3155ed306db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" Oct 30 17:17:19.725244 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:19.725210 4522 patch_prober.go:29] interesting pod/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body= Oct 30 17:17:19.725708 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:19.725274 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm" podUID="40baffd6-35b6-412d-a7bd-f3155ed306db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" Oct 30 17:17:20.724564 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:20.724525 4522 patch_prober.go:29] interesting pod/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body= Oct 30 17:17:20.724786 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:20.724583 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm" podUID="40baffd6-35b6-412d-a7bd-f3155ed306db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" Oct 30 17:17:21.731530 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:21.731494 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm" Oct 30 17:17:21.734032 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:21.734002 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm" Oct 30 17:17:25.508423 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:25.508389 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m"] Oct 30 17:17:25.508899 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:25.508602 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m" podUID="232a5af4-ecfb-4e38-a937-fc555a9b946b" containerName="istio-proxy" containerID="cri-o://907d3419675b1ee04fb79572c2238986099b43909a692f3d98dcc358efc70ce2" gracePeriod=30 Oct 30 17:17:30.791281 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:30.785897 4522 generic.go:356] "Generic (PLEG): container finished" podID="232a5af4-ecfb-4e38-a937-fc555a9b946b" containerID="907d3419675b1ee04fb79572c2238986099b43909a692f3d98dcc358efc70ce2" exitCode=0 Oct 30 17:17:30.791281 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:30.786156 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m" event={"ID":"232a5af4-ecfb-4e38-a937-fc555a9b946b","Type":"ContainerDied","Data":"907d3419675b1ee04fb79572c2238986099b43909a692f3d98dcc358efc70ce2"} Oct 30 17:17:30.850830 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:30.850799 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m" Oct 30 17:17:30.938204 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:30.938170 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/232a5af4-ecfb-4e38-a937-fc555a9b946b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/232a5af4-ecfb-4e38-a937-fc555a9b946b-istiod-ca-cert\") pod \"232a5af4-ecfb-4e38-a937-fc555a9b946b\" (UID: \"232a5af4-ecfb-4e38-a937-fc555a9b946b\") " Oct 30 17:17:30.938438 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:30.938293 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/232a5af4-ecfb-4e38-a937-fc555a9b946b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/232a5af4-ecfb-4e38-a937-fc555a9b946b-workload-certs\") pod \"232a5af4-ecfb-4e38-a937-fc555a9b946b\" (UID: \"232a5af4-ecfb-4e38-a937-fc555a9b946b\") " Oct 30 17:17:30.938438 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:30.938428 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/232a5af4-ecfb-4e38-a937-fc555a9b946b-istio-token\" (UniqueName: \"kubernetes.io/projected/232a5af4-ecfb-4e38-a937-fc555a9b946b-istio-token\") pod \"232a5af4-ecfb-4e38-a937-fc555a9b946b\" (UID: \"232a5af4-ecfb-4e38-a937-fc555a9b946b\") " Oct 30 17:17:30.938572 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:30.938491 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/232a5af4-ecfb-4e38-a937-fc555a9b946b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/232a5af4-ecfb-4e38-a937-fc555a9b946b-istio-data\") pod \"232a5af4-ecfb-4e38-a937-fc555a9b946b\" (UID: \"232a5af4-ecfb-4e38-a937-fc555a9b946b\") " Oct 30 17:17:30.938572 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:30.938542 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/232a5af4-ecfb-4e38-a937-fc555a9b946b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/232a5af4-ecfb-4e38-a937-fc555a9b946b-istio-podinfo\") pod \"232a5af4-ecfb-4e38-a937-fc555a9b946b\" (UID: \"232a5af4-ecfb-4e38-a937-fc555a9b946b\") " Oct 30 17:17:30.938680 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:30.938602 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/232a5af4-ecfb-4e38-a937-fc555a9b946b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/232a5af4-ecfb-4e38-a937-fc555a9b946b-workload-socket\") pod \"232a5af4-ecfb-4e38-a937-fc555a9b946b\" (UID: \"232a5af4-ecfb-4e38-a937-fc555a9b946b\") " Oct 30 17:17:30.938795 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:30.938675 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/232a5af4-ecfb-4e38-a937-fc555a9b946b-kube-api-access-v6z5q\" (UniqueName: \"kubernetes.io/projected/232a5af4-ecfb-4e38-a937-fc555a9b946b-kube-api-access-v6z5q\") pod \"232a5af4-ecfb-4e38-a937-fc555a9b946b\" (UID: \"232a5af4-ecfb-4e38-a937-fc555a9b946b\") " Oct 30 17:17:30.938795 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:30.938730 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/232a5af4-ecfb-4e38-a937-fc555a9b946b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/232a5af4-ecfb-4e38-a937-fc555a9b946b-credential-socket\") pod \"232a5af4-ecfb-4e38-a937-fc555a9b946b\" (UID: \"232a5af4-ecfb-4e38-a937-fc555a9b946b\") " Oct 30 17:17:30.938795 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:30.938768 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/232a5af4-ecfb-4e38-a937-fc555a9b946b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/232a5af4-ecfb-4e38-a937-fc555a9b946b-istio-envoy\") pod \"232a5af4-ecfb-4e38-a937-fc555a9b946b\" (UID: \"232a5af4-ecfb-4e38-a937-fc555a9b946b\") " Oct 30 17:17:30.939932 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:30.939905 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/232a5af4-ecfb-4e38-a937-fc555a9b946b-workload-certs" pod "232a5af4-ecfb-4e38-a937-fc555a9b946b" (UID: "232a5af4-ecfb-4e38-a937-fc555a9b946b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:30.940278 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:30.940255 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/232a5af4-ecfb-4e38-a937-fc555a9b946b-istiod-ca-cert" pod "232a5af4-ecfb-4e38-a937-fc555a9b946b" (UID: "232a5af4-ecfb-4e38-a937-fc555a9b946b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:17:30.940576 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:30.940550 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/232a5af4-ecfb-4e38-a937-fc555a9b946b-workload-socket" pod "232a5af4-ecfb-4e38-a937-fc555a9b946b" (UID: "232a5af4-ecfb-4e38-a937-fc555a9b946b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:30.941365 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:30.941329 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/232a5af4-ecfb-4e38-a937-fc555a9b946b-credential-socket" pod "232a5af4-ecfb-4e38-a937-fc555a9b946b" (UID: "232a5af4-ecfb-4e38-a937-fc555a9b946b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:30.941675 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:30.941654 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/232a5af4-ecfb-4e38-a937-fc555a9b946b-istio-data" pod "232a5af4-ecfb-4e38-a937-fc555a9b946b" (UID: "232a5af4-ecfb-4e38-a937-fc555a9b946b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:30.943817 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:30.943793 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/232a5af4-ecfb-4e38-a937-fc555a9b946b-istio-envoy" pod "232a5af4-ecfb-4e38-a937-fc555a9b946b" (UID: "232a5af4-ecfb-4e38-a937-fc555a9b946b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:30.945582 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:30.945556 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/232a5af4-ecfb-4e38-a937-fc555a9b946b-istio-podinfo" pod "232a5af4-ecfb-4e38-a937-fc555a9b946b" (UID: "232a5af4-ecfb-4e38-a937-fc555a9b946b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:17:30.945841 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:30.945820 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/232a5af4-ecfb-4e38-a937-fc555a9b946b-kube-api-access-v6z5q" pod "232a5af4-ecfb-4e38-a937-fc555a9b946b" (UID: "232a5af4-ecfb-4e38-a937-fc555a9b946b"). InnerVolumeSpecName "kube-api-access-v6z5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:30.946039 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:30.945999 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/232a5af4-ecfb-4e38-a937-fc555a9b946b-istio-token" pod "232a5af4-ecfb-4e38-a937-fc555a9b946b" (UID: "232a5af4-ecfb-4e38-a937-fc555a9b946b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:31.041108 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:31.041067 4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/232a5af4-ecfb-4e38-a937-fc555a9b946b-istiod-ca-cert\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:17:31.041424 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:31.041371 4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/232a5af4-ecfb-4e38-a937-fc555a9b946b-workload-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:17:31.041653 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:31.041638 4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/232a5af4-ecfb-4e38-a937-fc555a9b946b-istio-token\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:17:31.041760 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:31.041749 4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/232a5af4-ecfb-4e38-a937-fc555a9b946b-istio-data\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:17:31.042129 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:31.042101 4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/232a5af4-ecfb-4e38-a937-fc555a9b946b-istio-podinfo\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:17:31.042129 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:31.042127 4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/232a5af4-ecfb-4e38-a937-fc555a9b946b-workload-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:17:31.042364 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:31.042146 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6z5q\" (UniqueName: \"kubernetes.io/projected/232a5af4-ecfb-4e38-a937-fc555a9b946b-kube-api-access-v6z5q\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:17:31.042364 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:31.042162 4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/232a5af4-ecfb-4e38-a937-fc555a9b946b-credential-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:17:31.042364 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:31.042176 4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/232a5af4-ecfb-4e38-a937-fc555a9b946b-istio-envoy\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:17:31.781876 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:31.781838 4522 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:17:31.782047 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:31.781899 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m" podUID="232a5af4-ecfb-4e38-a937-fc555a9b946b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:17:31.796634 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:31.796602 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m" Oct 30 17:17:31.797071 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:31.796633 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m" event={"ID":"232a5af4-ecfb-4e38-a937-fc555a9b946b","Type":"ContainerDied","Data":"660681750b12d50ea7a28ce5662f28db03ae9464219b5aea90d98e54a6a9d594"} Oct 30 17:17:31.797071 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:31.796706 4522 scope.go:122] "RemoveContainer" containerID="907d3419675b1ee04fb79572c2238986099b43909a692f3d98dcc358efc70ce2" Oct 30 17:17:31.858251 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:31.858218 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m"] Oct 30 17:17:31.862546 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:31.862524 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-54x6m"] Oct 30 17:17:32.147122 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:32.147038 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="232a5af4-ecfb-4e38-a937-fc555a9b946b" path="/var/lib/kubelet/pods/232a5af4-ecfb-4e38-a937-fc555a9b946b/volumes" Oct 30 17:17:35.394346 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:35.394315 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p"] Oct 30 17:17:35.395130 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:35.395116 4522 cpu_manager.go:414] "containerMap: removing container" podUID="232a5af4-ecfb-4e38-a937-fc555a9b946b" containerName="istio-proxy" Oct 30 17:17:35.395178 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:35.395131 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="232a5af4-ecfb-4e38-a937-fc555a9b946b" containerName="istio-proxy" Oct 30 17:17:35.395258 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:35.395247 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="232a5af4-ecfb-4e38-a937-fc555a9b946b" containerName="istio-proxy" Oct 30 17:17:35.400492 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:35.400465 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p" Oct 30 17:17:35.430887 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:35.430861 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-d003a5-openshift-default-dockercfg-8dqnq\"" Oct 30 17:17:35.438388 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:35.438357 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p"] Oct 30 17:17:35.511941 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:35.511912 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smpgj\" (UniqueName: \"kubernetes.io/projected/d30e44c6-cafd-4b54-9784-2f3743de61dc-kube-api-access-smpgj\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p\" (UID: \"d30e44c6-cafd-4b54-9784-2f3743de61dc\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p" Oct 30 17:17:35.512107 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:35.511976 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d30e44c6-cafd-4b54-9784-2f3743de61dc-workload-socket\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p\" (UID: \"d30e44c6-cafd-4b54-9784-2f3743de61dc\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p" Oct 30 17:17:35.512107 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:35.512047 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d30e44c6-cafd-4b54-9784-2f3743de61dc-workload-certs\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p\" (UID: \"d30e44c6-cafd-4b54-9784-2f3743de61dc\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p" Oct 30 17:17:35.512198 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:35.512112 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d30e44c6-cafd-4b54-9784-2f3743de61dc-istio-token\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p\" (UID: \"d30e44c6-cafd-4b54-9784-2f3743de61dc\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p" Oct 30 17:17:35.512198 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:35.512171 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d30e44c6-cafd-4b54-9784-2f3743de61dc-credential-socket\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p\" (UID: \"d30e44c6-cafd-4b54-9784-2f3743de61dc\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p" Oct 30 17:17:35.512298 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:35.512212 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d30e44c6-cafd-4b54-9784-2f3743de61dc-istio-envoy\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p\" (UID: \"d30e44c6-cafd-4b54-9784-2f3743de61dc\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p" Oct 30 17:17:35.512382 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:35.512369 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d30e44c6-cafd-4b54-9784-2f3743de61dc-istiod-ca-cert\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p\" (UID: \"d30e44c6-cafd-4b54-9784-2f3743de61dc\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p" Oct 30 17:17:35.512429 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:35.512417 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d30e44c6-cafd-4b54-9784-2f3743de61dc-istio-data\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p\" (UID: \"d30e44c6-cafd-4b54-9784-2f3743de61dc\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p" Oct 30 17:17:35.512560 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:35.512541 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d30e44c6-cafd-4b54-9784-2f3743de61dc-istio-podinfo\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p\" (UID: \"d30e44c6-cafd-4b54-9784-2f3743de61dc\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p" Oct 30 17:17:35.614149 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:35.614111 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smpgj\" (UniqueName: \"kubernetes.io/projected/d30e44c6-cafd-4b54-9784-2f3743de61dc-kube-api-access-smpgj\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p\" (UID: \"d30e44c6-cafd-4b54-9784-2f3743de61dc\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p" Oct 30 17:17:35.614396 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:35.614202 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d30e44c6-cafd-4b54-9784-2f3743de61dc-workload-socket\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p\" (UID: \"d30e44c6-cafd-4b54-9784-2f3743de61dc\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p" Oct 30 17:17:35.614396 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:35.614240 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d30e44c6-cafd-4b54-9784-2f3743de61dc-workload-certs\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p\" (UID: \"d30e44c6-cafd-4b54-9784-2f3743de61dc\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p" Oct 30 17:17:35.614396 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:35.614277 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d30e44c6-cafd-4b54-9784-2f3743de61dc-istio-token\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p\" (UID: \"d30e44c6-cafd-4b54-9784-2f3743de61dc\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p" Oct 30 17:17:35.614396 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:35.614386 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d30e44c6-cafd-4b54-9784-2f3743de61dc-credential-socket\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p\" (UID: \"d30e44c6-cafd-4b54-9784-2f3743de61dc\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p" Oct 30 17:17:35.614639 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:35.614442 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d30e44c6-cafd-4b54-9784-2f3743de61dc-istio-envoy\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p\" (UID: \"d30e44c6-cafd-4b54-9784-2f3743de61dc\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p" Oct 30 17:17:35.614639 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:35.614553 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d30e44c6-cafd-4b54-9784-2f3743de61dc-istiod-ca-cert\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p\" (UID: \"d30e44c6-cafd-4b54-9784-2f3743de61dc\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p" Oct 30 17:17:35.614639 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:35.614614 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d30e44c6-cafd-4b54-9784-2f3743de61dc-istio-data\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p\" (UID: \"d30e44c6-cafd-4b54-9784-2f3743de61dc\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p" Oct 30 17:17:35.614858 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:35.614700 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d30e44c6-cafd-4b54-9784-2f3743de61dc-istio-podinfo\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p\" (UID: \"d30e44c6-cafd-4b54-9784-2f3743de61dc\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p" Oct 30 17:17:35.614858 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:35.614725 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d30e44c6-cafd-4b54-9784-2f3743de61dc-workload-socket\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p\" (UID: \"d30e44c6-cafd-4b54-9784-2f3743de61dc\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p" Oct 30 17:17:35.614858 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:35.614761 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d30e44c6-cafd-4b54-9784-2f3743de61dc-workload-certs\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p\" (UID: \"d30e44c6-cafd-4b54-9784-2f3743de61dc\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p" Oct 30 17:17:35.614858 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:35.614773 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d30e44c6-cafd-4b54-9784-2f3743de61dc-credential-socket\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p\" (UID: \"d30e44c6-cafd-4b54-9784-2f3743de61dc\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p" Oct 30 17:17:35.615154 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:35.615122 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d30e44c6-cafd-4b54-9784-2f3743de61dc-istio-data\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p\" (UID: \"d30e44c6-cafd-4b54-9784-2f3743de61dc\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p" Oct 30 17:17:35.615465 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:35.615444 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d30e44c6-cafd-4b54-9784-2f3743de61dc-istiod-ca-cert\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p\" (UID: \"d30e44c6-cafd-4b54-9784-2f3743de61dc\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p" Oct 30 17:17:35.616835 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:35.616812 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d30e44c6-cafd-4b54-9784-2f3743de61dc-istio-envoy\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p\" (UID: \"d30e44c6-cafd-4b54-9784-2f3743de61dc\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p" Oct 30 17:17:35.617186 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:35.617166 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d30e44c6-cafd-4b54-9784-2f3743de61dc-istio-podinfo\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p\" (UID: \"d30e44c6-cafd-4b54-9784-2f3743de61dc\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p" Oct 30 17:17:35.639360 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:35.639334 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d30e44c6-cafd-4b54-9784-2f3743de61dc-istio-token\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p\" (UID: \"d30e44c6-cafd-4b54-9784-2f3743de61dc\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p" Oct 30 17:17:35.639456 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:35.639379 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smpgj\" (UniqueName: \"kubernetes.io/projected/d30e44c6-cafd-4b54-9784-2f3743de61dc-kube-api-access-smpgj\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p\" (UID: \"d30e44c6-cafd-4b54-9784-2f3743de61dc\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p" Oct 30 17:17:35.714976 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:35.714943 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p" Oct 30 17:17:36.039390 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:36.039257 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16704da5-9m4g2"] Oct 30 17:17:36.045101 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:36.045069 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16704da5-9m4g2"] Oct 30 17:17:36.049269 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:36.049238 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p"] Oct 30 17:17:36.049546 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:17:36.049522 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd30e44c6_cafd_4b54_9784_2f3743de61dc.slice/crio-a1940e1ec998aa031ab9b824c71cefbac52cb3252aa30b173124fdab08f6f766 WatchSource:0}: Error finding container a1940e1ec998aa031ab9b824c71cefbac52cb3252aa30b173124fdab08f6f766: Status 404 returned error can't find the container with id a1940e1ec998aa031ab9b824c71cefbac52cb3252aa30b173124fdab08f6f766 Oct 30 17:17:36.052438 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:36.052411 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:17:36.052529 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:36.052470 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:17:36.052529 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:36.052509 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:17:36.149040 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:36.149010 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46502a59-2f4b-4dbe-aef2-196aa1be6893" path="/var/lib/kubelet/pods/46502a59-2f4b-4dbe-aef2-196aa1be6893/volumes" Oct 30 17:17:36.842822 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:36.842786 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p" event={"ID":"d30e44c6-cafd-4b54-9784-2f3743de61dc","Type":"ContainerStarted","Data":"9284d9134005736d19ff6360e76307560b38dfa730f4a784569011b9498b2e78"} Oct 30 17:17:36.843247 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:36.842845 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p" event={"ID":"d30e44c6-cafd-4b54-9784-2f3743de61dc","Type":"ContainerStarted","Data":"a1940e1ec998aa031ab9b824c71cefbac52cb3252aa30b173124fdab08f6f766"} Oct 30 17:17:36.912689 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:36.912621 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p" podStartSLOduration=1.912597816 podStartE2EDuration="1.912597816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:17:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:17:36.891229648 +0000 UTC m=+1830.891984570" watchObservedRunningTime="2026-10-30 17:17:36.912597816 +0000 UTC m=+1830.913352718" Oct 30 17:17:37.716647 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:37.716605 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p" Oct 30 17:17:37.718616 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:37.718578 4522 patch_prober.go:29] interesting pod/e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body= Oct 30 17:17:37.718770 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:37.718651 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p" podUID="d30e44c6-cafd-4b54-9784-2f3743de61dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" Oct 30 17:17:38.716334 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:38.716050 4522 patch_prober.go:29] interesting pod/e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body= Oct 30 17:17:38.716334 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:38.716141 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p" podUID="d30e44c6-cafd-4b54-9784-2f3743de61dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" Oct 30 17:17:39.717291 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:39.717252 4522 patch_prober.go:29] interesting pod/e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body= Oct 30 17:17:39.717971 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:39.717936 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p" podUID="d30e44c6-cafd-4b54-9784-2f3743de61dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" Oct 30 17:17:40.717332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:40.716149 4522 patch_prober.go:29] interesting pod/e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body= Oct 30 17:17:40.717841 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:40.717469 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p" podUID="d30e44c6-cafd-4b54-9784-2f3743de61dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" Oct 30 17:17:41.716792 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:41.716759 4522 patch_prober.go:29] interesting pod/e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body= Oct 30 17:17:41.716990 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:41.716829 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p" podUID="d30e44c6-cafd-4b54-9784-2f3743de61dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" Oct 30 17:17:42.941361 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:42.941294 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p" Oct 30 17:17:42.942217 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:42.942188 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p" Oct 30 17:17:42.943445 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:17:42.943420 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p" Oct 30 17:17:42.955117 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:17:42.955086 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:18:02.315131 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:02.315100 4522 generic.go:356] "Generic (PLEG): container finished" podID="eb91a3a6-25a0-4c9b-9564-8221f24efa30" containerID="67a9433870f1ece7c693445db5d4e4ec6ac14a3d8448ef0a3cfcfc178b256f36" exitCode=0 Oct 30 17:18:02.315742 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:02.315142 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7" event={"ID":"eb91a3a6-25a0-4c9b-9564-8221f24efa30","Type":"ContainerDied","Data":"67a9433870f1ece7c693445db5d4e4ec6ac14a3d8448ef0a3cfcfc178b256f36"} Oct 30 17:18:02.315742 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:02.315641 4522 scope.go:122] "RemoveContainer" containerID="67a9433870f1ece7c693445db5d4e4ec6ac14a3d8448ef0a3cfcfc178b256f36" Oct 30 17:18:02.317646 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:02.317617 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:18:02.317735 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:02.317688 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:18:02.317735 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:02.317721 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:18:03.327131 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:03.327093 4522 generic.go:356] "Generic (PLEG): container finished" podID="0e42c53c-946a-40fc-b126-5cd8a39b532d" containerID="e34ffbc603e71d8b1737bd3ffa4ea74e7266c5f8a4ae4c6d447947ba2255893b" exitCode=0 Oct 30 17:18:03.327655 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:03.327169 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl" event={"ID":"0e42c53c-946a-40fc-b126-5cd8a39b532d","Type":"ContainerDied","Data":"e34ffbc603e71d8b1737bd3ffa4ea74e7266c5f8a4ae4c6d447947ba2255893b"} Oct 30 17:18:03.327752 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:03.327734 4522 scope.go:122] "RemoveContainer" containerID="e34ffbc603e71d8b1737bd3ffa4ea74e7266c5f8a4ae4c6d447947ba2255893b" Oct 30 17:18:03.329750 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:03.329730 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7" event={"ID":"eb91a3a6-25a0-4c9b-9564-8221f24efa30","Type":"ContainerStarted","Data":"6d84f39c8ce66309ebc1baa1a57661e955e8f2e7f098d192d1a008b7c0c50443"} Oct 30 17:18:03.330479 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:03.330441 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:18:03.330633 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:03.330613 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:18:03.330740 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:03.330721 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:18:04.184996 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:04.184962 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7" Oct 30 17:18:04.185213 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:04.185184 4522 patch_prober.go:29] interesting pod/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Oct 30 17:18:04.185269 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:04.185251 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7" podUID="eb91a3a6-25a0-4c9b-9564-8221f24efa30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Oct 30 17:18:04.340323 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:04.340268 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl" event={"ID":"0e42c53c-946a-40fc-b126-5cd8a39b532d","Type":"ContainerStarted","Data":"2e763d6c5d0687d027d9c909c38d2c9655886d3b45fb6eb611720bb3278c4423"} Oct 30 17:18:04.562244 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:04.562206 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl" Oct 30 17:18:04.563201 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:04.563174 4522 patch_prober.go:29] interesting pod/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body= Oct 30 17:18:04.563363 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:04.563228 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl" podUID="0e42c53c-946a-40fc-b126-5cd8a39b532d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" Oct 30 17:18:05.200331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:05.195457 4522 patch_prober.go:29] interesting pod/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Oct 30 17:18:05.200331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:05.196331 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7" podUID="eb91a3a6-25a0-4c9b-9564-8221f24efa30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Oct 30 17:18:05.576817 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:05.576780 4522 patch_prober.go:29] interesting pod/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body= Oct 30 17:18:05.577475 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:05.577441 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl" podUID="0e42c53c-946a-40fc-b126-5cd8a39b532d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" Oct 30 17:18:06.184812 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:06.184779 4522 patch_prober.go:29] interesting pod/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Oct 30 17:18:06.185018 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:06.184826 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7" podUID="eb91a3a6-25a0-4c9b-9564-8221f24efa30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Oct 30 17:18:06.562424 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:06.562385 4522 patch_prober.go:29] interesting pod/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body= Oct 30 17:18:06.562581 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:06.562461 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl" podUID="0e42c53c-946a-40fc-b126-5cd8a39b532d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" Oct 30 17:18:07.184673 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:07.184635 4522 patch_prober.go:29] interesting pod/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Oct 30 17:18:07.185160 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:07.184707 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7" podUID="eb91a3a6-25a0-4c9b-9564-8221f24efa30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Oct 30 17:18:07.562936 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:07.562903 4522 patch_prober.go:29] interesting pod/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body= Oct 30 17:18:07.563116 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:07.562952 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl" podUID="0e42c53c-946a-40fc-b126-5cd8a39b532d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" Oct 30 17:18:08.184582 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:08.184543 4522 patch_prober.go:29] interesting pod/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Oct 30 17:18:08.184839 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:08.184817 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7" podUID="eb91a3a6-25a0-4c9b-9564-8221f24efa30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Oct 30 17:18:08.554790 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:08.554454 4522 scope.go:122] "RemoveContainer" containerID="961dc7a7a9dfa0767419f46e4a780f01a8e2547f066471f56db69e4b032dfe71" Oct 30 17:18:08.562591 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:08.562563 4522 patch_prober.go:29] interesting pod/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body= Oct 30 17:18:08.562828 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:08.562798 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl" podUID="0e42c53c-946a-40fc-b126-5cd8a39b532d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" Oct 30 17:18:08.566539 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:08.566518 4522 scope.go:122] "RemoveContainer" containerID="4f6d281faa42f5af7a398425b5ef284424679fd73c868a7aedc368eac32a3a13" Oct 30 17:18:08.800518 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:08.799424 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl" Oct 30 17:18:09.190076 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:09.190043 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7" Oct 30 17:18:09.190593 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:09.190422 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7" Oct 30 17:18:09.191082 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:09.191061 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7" Oct 30 17:18:09.562900 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:09.562869 4522 patch_prober.go:29] interesting pod/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body= Oct 30 17:18:09.563085 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:09.562931 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl" podUID="0e42c53c-946a-40fc-b126-5cd8a39b532d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" Oct 30 17:18:10.568123 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:10.568096 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl" Oct 30 17:18:10.569283 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:10.569263 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl" Oct 30 17:18:12.950113 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:18:12.950083 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:18:17.407790 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:17.407744 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-j47tj"] Oct 30 17:18:17.414370 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:17.414348 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-j47tj" Oct 30 17:18:17.430989 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:17.430959 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-j47tj"] Oct 30 17:18:17.438048 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:17.438022 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-c82666-openshift-default-dockercfg-z7xj6\"" Oct 30 17:18:17.505288 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:17.505255 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587-istio-data\") pod \"e2e-inf-b-c82666-openshift-default-767d987659-j47tj\" (UID: \"3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587\") " pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-j47tj" Oct 30 17:18:17.505478 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:17.505299 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587-istio-envoy\") pod \"e2e-inf-b-c82666-openshift-default-767d987659-j47tj\" (UID: \"3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587\") " pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-j47tj" Oct 30 17:18:17.505478 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:17.505383 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587-workload-certs\") pod \"e2e-inf-b-c82666-openshift-default-767d987659-j47tj\" (UID: \"3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587\") " pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-j47tj" Oct 30 17:18:17.505478 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:17.505436 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587-credential-socket\") pod \"e2e-inf-b-c82666-openshift-default-767d987659-j47tj\" (UID: \"3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587\") " pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-j47tj" Oct 30 17:18:17.505596 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:17.505498 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587-istio-podinfo\") pod \"e2e-inf-b-c82666-openshift-default-767d987659-j47tj\" (UID: \"3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587\") " pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-j47tj" Oct 30 17:18:17.505596 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:17.505551 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bscjk\" (UniqueName: \"kubernetes.io/projected/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587-kube-api-access-bscjk\") pod \"e2e-inf-b-c82666-openshift-default-767d987659-j47tj\" (UID: \"3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587\") " pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-j47tj" Oct 30 17:18:17.505686 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:17.505672 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587-istio-token\") pod \"e2e-inf-b-c82666-openshift-default-767d987659-j47tj\" (UID: \"3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587\") " pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-j47tj" Oct 30 17:18:17.505722 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:17.505709 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587-istiod-ca-cert\") pod \"e2e-inf-b-c82666-openshift-default-767d987659-j47tj\" (UID: \"3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587\") " pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-j47tj" Oct 30 17:18:17.505819 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:17.505805 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587-workload-socket\") pod \"e2e-inf-b-c82666-openshift-default-767d987659-j47tj\" (UID: \"3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587\") " pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-j47tj" Oct 30 17:18:17.607115 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:17.607083 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587-istio-data\") pod \"e2e-inf-b-c82666-openshift-default-767d987659-j47tj\" (UID: \"3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587\") " pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-j47tj" Oct 30 17:18:17.607338 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:17.607137 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587-istio-envoy\") pod \"e2e-inf-b-c82666-openshift-default-767d987659-j47tj\" (UID: \"3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587\") " pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-j47tj" Oct 30 17:18:17.607338 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:17.607156 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587-workload-certs\") pod \"e2e-inf-b-c82666-openshift-default-767d987659-j47tj\" (UID: \"3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587\") " pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-j47tj" Oct 30 17:18:17.607338 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:17.607189 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587-credential-socket\") pod \"e2e-inf-b-c82666-openshift-default-767d987659-j47tj\" (UID: \"3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587\") " pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-j47tj" Oct 30 17:18:17.607338 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:17.607203 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587-istio-podinfo\") pod \"e2e-inf-b-c82666-openshift-default-767d987659-j47tj\" (UID: \"3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587\") " pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-j47tj" Oct 30 17:18:17.607338 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:17.607265 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bscjk\" (UniqueName: \"kubernetes.io/projected/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587-kube-api-access-bscjk\") pod \"e2e-inf-b-c82666-openshift-default-767d987659-j47tj\" (UID: \"3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587\") " pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-j47tj" Oct 30 17:18:17.607639 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:17.607405 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587-istio-token\") pod \"e2e-inf-b-c82666-openshift-default-767d987659-j47tj\" (UID: \"3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587\") " pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-j47tj" Oct 30 17:18:17.607639 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:17.607456 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587-istiod-ca-cert\") pod \"e2e-inf-b-c82666-openshift-default-767d987659-j47tj\" (UID: \"3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587\") " pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-j47tj" Oct 30 17:18:17.607639 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:17.607530 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587-workload-socket\") pod \"e2e-inf-b-c82666-openshift-default-767d987659-j47tj\" (UID: \"3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587\") " pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-j47tj" Oct 30 17:18:17.607639 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:17.607542 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587-istio-data\") pod \"e2e-inf-b-c82666-openshift-default-767d987659-j47tj\" (UID: \"3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587\") " pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-j47tj" Oct 30 17:18:17.607639 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:17.607587 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587-workload-certs\") pod \"e2e-inf-b-c82666-openshift-default-767d987659-j47tj\" (UID: \"3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587\") " pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-j47tj" Oct 30 17:18:17.607916 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:17.607832 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587-credential-socket\") pod \"e2e-inf-b-c82666-openshift-default-767d987659-j47tj\" (UID: \"3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587\") " pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-j47tj" Oct 30 17:18:17.608101 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:17.608073 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587-workload-socket\") pod \"e2e-inf-b-c82666-openshift-default-767d987659-j47tj\" (UID: \"3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587\") " pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-j47tj" Oct 30 17:18:17.608173 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:17.608139 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587-istiod-ca-cert\") pod \"e2e-inf-b-c82666-openshift-default-767d987659-j47tj\" (UID: \"3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587\") " pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-j47tj" Oct 30 17:18:17.609705 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:17.609677 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587-istio-envoy\") pod \"e2e-inf-b-c82666-openshift-default-767d987659-j47tj\" (UID: \"3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587\") " pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-j47tj" Oct 30 17:18:17.609864 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:17.609848 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587-istio-podinfo\") pod \"e2e-inf-b-c82666-openshift-default-767d987659-j47tj\" (UID: \"3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587\") " pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-j47tj" Oct 30 17:18:17.632403 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:17.632377 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587-istio-token\") pod \"e2e-inf-b-c82666-openshift-default-767d987659-j47tj\" (UID: \"3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587\") " pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-j47tj" Oct 30 17:18:17.632514 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:17.632410 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bscjk\" (UniqueName: \"kubernetes.io/projected/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587-kube-api-access-bscjk\") pod \"e2e-inf-b-c82666-openshift-default-767d987659-j47tj\" (UID: \"3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587\") " pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-j47tj" Oct 30 17:18:17.731265 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:17.731233 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-j47tj" Oct 30 17:18:18.064550 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:18:18.064516 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f6ad9f3_0cb9_4e5b_96e1_42d9d0798587.slice/crio-a49a51d774bb6635b2fac508d918a99f34edb0b2a80ac09522592367101bc1f8 WatchSource:0}: Error finding container a49a51d774bb6635b2fac508d918a99f34edb0b2a80ac09522592367101bc1f8: Status 404 returned error can't find the container with id a49a51d774bb6635b2fac508d918a99f34edb0b2a80ac09522592367101bc1f8 Oct 30 17:18:18.065457 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:18.065429 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-j47tj"] Oct 30 17:18:18.067940 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:18.067888 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:18:18.068022 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:18.067981 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:18:18.068058 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:18.068016 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:18:18.474807 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:18.474758 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-j47tj" event={"ID":"3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587","Type":"ContainerStarted","Data":"be87840f212e5810b5dfc76e4934a87cb9d7cb82442e4c126d6e5ba2653377b9"} Oct 30 17:18:18.475396 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:18.474867 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-j47tj" event={"ID":"3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587","Type":"ContainerStarted","Data":"a49a51d774bb6635b2fac508d918a99f34edb0b2a80ac09522592367101bc1f8"} Oct 30 17:18:18.529135 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:18.528729 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-j47tj" podStartSLOduration=1.52871034 podStartE2EDuration="1.52871034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:18:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:18:18.525558331 +0000 UTC m=+1872.526313234" watchObservedRunningTime="2026-10-30 17:18:18.52871034 +0000 UTC m=+1872.529465245" Oct 30 17:18:18.732083 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:18.731978 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-j47tj" Oct 30 17:18:19.733691 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:19.733653 4522 patch_prober.go:29] interesting pod/e2e-inf-b-c82666-openshift-default-767d987659-j47tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Oct 30 17:18:19.734185 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:19.733715 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-j47tj" podUID="3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Oct 30 17:18:20.732089 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:20.732057 4522 patch_prober.go:29] interesting pod/e2e-inf-b-c82666-openshift-default-767d987659-j47tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Oct 30 17:18:20.732276 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:20.732107 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-j47tj" podUID="3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Oct 30 17:18:21.732478 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:21.732434 4522 patch_prober.go:29] interesting pod/e2e-inf-b-c82666-openshift-default-767d987659-j47tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Oct 30 17:18:21.732899 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:21.732500 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-j47tj" podUID="3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Oct 30 17:18:22.731771 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:22.731735 4522 patch_prober.go:29] interesting pod/e2e-inf-b-c82666-openshift-default-767d987659-j47tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Oct 30 17:18:22.731956 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:22.731812 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-j47tj" podUID="3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Oct 30 17:18:23.731976 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:23.731943 4522 patch_prober.go:29] interesting pod/e2e-inf-b-c82666-openshift-default-767d987659-j47tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Oct 30 17:18:23.732410 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:23.731999 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-j47tj" podUID="3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Oct 30 17:18:24.226149 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:24.226119 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh"] Oct 30 17:18:24.231071 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:24.231050 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh" Oct 30 17:18:24.263342 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:24.262850 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh"] Oct 30 17:18:24.268696 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:24.268669 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-c45877-openshift-default-dockercfg-29smt\"" Oct 30 17:18:24.353609 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:24.353560 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40f8b984-d0d8-4938-859e-268ae92cf8f2-istio-token\") pod \"e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh\" (UID: \"40f8b984-d0d8-4938-859e-268ae92cf8f2\") " pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh" Oct 30 17:18:24.353849 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:24.353662 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40f8b984-d0d8-4938-859e-268ae92cf8f2-credential-socket\") pod \"e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh\" (UID: \"40f8b984-d0d8-4938-859e-268ae92cf8f2\") " pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh" Oct 30 17:18:24.353849 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:24.353704 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40f8b984-d0d8-4938-859e-268ae92cf8f2-workload-socket\") pod \"e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh\" (UID: \"40f8b984-d0d8-4938-859e-268ae92cf8f2\") " pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh" Oct 30 17:18:24.353849 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:24.353755 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40f8b984-d0d8-4938-859e-268ae92cf8f2-istio-envoy\") pod \"e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh\" (UID: \"40f8b984-d0d8-4938-859e-268ae92cf8f2\") " pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh" Oct 30 17:18:24.353849 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:24.353783 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40f8b984-d0d8-4938-859e-268ae92cf8f2-workload-certs\") pod \"e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh\" (UID: \"40f8b984-d0d8-4938-859e-268ae92cf8f2\") " pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh" Oct 30 17:18:24.353849 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:24.353828 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40f8b984-d0d8-4938-859e-268ae92cf8f2-istio-data\") pod \"e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh\" (UID: \"40f8b984-d0d8-4938-859e-268ae92cf8f2\") " pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh" Oct 30 17:18:24.354229 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:24.353954 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx92p\" (UniqueName: \"kubernetes.io/projected/40f8b984-d0d8-4938-859e-268ae92cf8f2-kube-api-access-gx92p\") pod \"e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh\" (UID: \"40f8b984-d0d8-4938-859e-268ae92cf8f2\") " pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh" Oct 30 17:18:24.354229 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:24.354188 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40f8b984-d0d8-4938-859e-268ae92cf8f2-istio-podinfo\") pod \"e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh\" (UID: \"40f8b984-d0d8-4938-859e-268ae92cf8f2\") " pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh" Oct 30 17:18:24.354434 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:24.354256 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40f8b984-d0d8-4938-859e-268ae92cf8f2-istiod-ca-cert\") pod \"e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh\" (UID: \"40f8b984-d0d8-4938-859e-268ae92cf8f2\") " pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh" Oct 30 17:18:24.456707 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:24.456663 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40f8b984-d0d8-4938-859e-268ae92cf8f2-credential-socket\") pod \"e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh\" (UID: \"40f8b984-d0d8-4938-859e-268ae92cf8f2\") " pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh" Oct 30 17:18:24.464467 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:24.464433 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40f8b984-d0d8-4938-859e-268ae92cf8f2-workload-socket\") pod \"e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh\" (UID: \"40f8b984-d0d8-4938-859e-268ae92cf8f2\") " pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh" Oct 30 17:18:24.464788 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:24.464763 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40f8b984-d0d8-4938-859e-268ae92cf8f2-istio-envoy\") pod \"e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh\" (UID: \"40f8b984-d0d8-4938-859e-268ae92cf8f2\") " pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh" Oct 30 17:18:24.464959 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:24.464942 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40f8b984-d0d8-4938-859e-268ae92cf8f2-workload-certs\") pod \"e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh\" (UID: \"40f8b984-d0d8-4938-859e-268ae92cf8f2\") " pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh" Oct 30 17:18:24.465130 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:24.465101 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40f8b984-d0d8-4938-859e-268ae92cf8f2-istio-data\") pod \"e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh\" (UID: \"40f8b984-d0d8-4938-859e-268ae92cf8f2\") " pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh" Oct 30 17:18:24.487125 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:24.487043 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40f8b984-d0d8-4938-859e-268ae92cf8f2-workload-socket\") pod \"e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh\" (UID: \"40f8b984-d0d8-4938-859e-268ae92cf8f2\") " pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh" Oct 30 17:18:24.487664 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:24.487637 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40f8b984-d0d8-4938-859e-268ae92cf8f2-credential-socket\") pod \"e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh\" (UID: \"40f8b984-d0d8-4938-859e-268ae92cf8f2\") " pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh" Oct 30 17:18:24.488468 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:24.488447 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40f8b984-d0d8-4938-859e-268ae92cf8f2-workload-certs\") pod \"e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh\" (UID: \"40f8b984-d0d8-4938-859e-268ae92cf8f2\") " pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh" Oct 30 17:18:24.488863 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:24.488844 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40f8b984-d0d8-4938-859e-268ae92cf8f2-istio-data\") pod \"e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh\" (UID: \"40f8b984-d0d8-4938-859e-268ae92cf8f2\") " pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh" Oct 30 17:18:24.489041 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:24.489022 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx92p\" (UniqueName: \"kubernetes.io/projected/40f8b984-d0d8-4938-859e-268ae92cf8f2-kube-api-access-gx92p\") pod \"e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh\" (UID: \"40f8b984-d0d8-4938-859e-268ae92cf8f2\") " pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh" Oct 30 17:18:24.489425 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:24.489404 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40f8b984-d0d8-4938-859e-268ae92cf8f2-istio-podinfo\") pod \"e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh\" (UID: \"40f8b984-d0d8-4938-859e-268ae92cf8f2\") " pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh" Oct 30 17:18:24.489617 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:24.489601 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40f8b984-d0d8-4938-859e-268ae92cf8f2-istiod-ca-cert\") pod \"e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh\" (UID: \"40f8b984-d0d8-4938-859e-268ae92cf8f2\") " pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh" Oct 30 17:18:24.489750 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:24.489733 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40f8b984-d0d8-4938-859e-268ae92cf8f2-istio-token\") pod \"e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh\" (UID: \"40f8b984-d0d8-4938-859e-268ae92cf8f2\") " pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh" Oct 30 17:18:24.508637 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:24.491765 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40f8b984-d0d8-4938-859e-268ae92cf8f2-istiod-ca-cert\") pod \"e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh\" (UID: \"40f8b984-d0d8-4938-859e-268ae92cf8f2\") " pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh" Oct 30 17:18:24.543471 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:24.538661 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40f8b984-d0d8-4938-859e-268ae92cf8f2-istio-envoy\") pod \"e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh\" (UID: \"40f8b984-d0d8-4938-859e-268ae92cf8f2\") " pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh" Oct 30 17:18:24.568374 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:24.561842 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx92p\" (UniqueName: \"kubernetes.io/projected/40f8b984-d0d8-4938-859e-268ae92cf8f2-kube-api-access-gx92p\") pod \"e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh\" (UID: \"40f8b984-d0d8-4938-859e-268ae92cf8f2\") " pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh" Oct 30 17:18:24.585247 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:24.585209 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40f8b984-d0d8-4938-859e-268ae92cf8f2-istio-token\") pod \"e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh\" (UID: \"40f8b984-d0d8-4938-859e-268ae92cf8f2\") " pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh" Oct 30 17:18:24.600701 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:24.597962 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40f8b984-d0d8-4938-859e-268ae92cf8f2-istio-podinfo\") pod \"e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh\" (UID: \"40f8b984-d0d8-4938-859e-268ae92cf8f2\") " pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh" Oct 30 17:18:24.732051 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:24.732018 4522 patch_prober.go:29] interesting pod/e2e-inf-b-c82666-openshift-default-767d987659-j47tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Oct 30 17:18:24.732602 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:24.732078 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-j47tj" podUID="3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Oct 30 17:18:24.851578 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:24.851486 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh" Oct 30 17:18:25.195939 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:18:25.195909 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40f8b984_d0d8_4938_859e_268ae92cf8f2.slice/crio-13cbd6dbcd8167cc9de63f43e3b789ba1ccbc49f92d3bccd8bb3bbc67605a210 WatchSource:0}: Error finding container 13cbd6dbcd8167cc9de63f43e3b789ba1ccbc49f92d3bccd8bb3bbc67605a210: Status 404 returned error can't find the container with id 13cbd6dbcd8167cc9de63f43e3b789ba1ccbc49f92d3bccd8bb3bbc67605a210 Oct 30 17:18:25.198022 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:25.197995 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh"] Oct 30 17:18:25.199087 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:25.199055 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:18:25.199161 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:25.199121 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:18:25.199161 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:25.199151 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:18:25.620876 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:25.620836 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh" event={"ID":"40f8b984-d0d8-4938-859e-268ae92cf8f2","Type":"ContainerStarted","Data":"19340ec98a9c74003f96585f0347e896a9703170056e3ececf339bf3431d812c"} Oct 30 17:18:25.621035 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:25.620906 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh" event={"ID":"40f8b984-d0d8-4938-859e-268ae92cf8f2","Type":"ContainerStarted","Data":"13cbd6dbcd8167cc9de63f43e3b789ba1ccbc49f92d3bccd8bb3bbc67605a210"} Oct 30 17:18:25.695916 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:25.695856 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh" podStartSLOduration=1.6958371140000001 podStartE2EDuration="1.695837114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:18:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:18:25.691649033 +0000 UTC m=+1879.692403940" watchObservedRunningTime="2026-10-30 17:18:25.695837114 +0000 UTC m=+1879.696592015" Oct 30 17:18:25.737432 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:25.737399 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-j47tj" Oct 30 17:18:25.738101 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:25.738075 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-j47tj" Oct 30 17:18:25.738789 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:25.738767 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-j47tj" Oct 30 17:18:25.854337 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:25.853450 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh" Oct 30 17:18:26.858344 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:26.858292 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh" Oct 30 17:18:27.638756 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:27.638726 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh" Oct 30 17:18:27.639734 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:27.639717 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh" Oct 30 17:18:30.293150 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:30.293110 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c45877-787d5595b7-8mhf5"] Oct 30 17:18:30.302143 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:30.301741 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c45877-787d5595b7-8mhf5" Oct 30 17:18:30.309889 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:30.309860 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c45877-787d5595b7-8mhf5"] Oct 30 17:18:30.411700 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:30.409845 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-c45877-dockercfg-v4h46\"" Oct 30 17:18:30.420437 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:30.413568 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-c45877\"" Oct 30 17:18:30.526071 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:30.526028 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab382b04-4fd2-4172-bc26-8f10347da05c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c45877-787d5595b7-8mhf5\" (UID: \"ab382b04-4fd2-4172-bc26-8f10347da05c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c45877-787d5595b7-8mhf5" Oct 30 17:18:30.526333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:30.526291 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2sjj\" (UniqueName: \"kubernetes.io/projected/ab382b04-4fd2-4172-bc26-8f10347da05c-kube-api-access-v2sjj\") pod \"payload-pre-processing-e2e-inf-b-c45877-787d5595b7-8mhf5\" (UID: \"ab382b04-4fd2-4172-bc26-8f10347da05c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c45877-787d5595b7-8mhf5" Oct 30 17:18:30.628143 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:30.628051 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab382b04-4fd2-4172-bc26-8f10347da05c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c45877-787d5595b7-8mhf5\" (UID: \"ab382b04-4fd2-4172-bc26-8f10347da05c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c45877-787d5595b7-8mhf5" Oct 30 17:18:30.628143 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:30.628088 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2sjj\" (UniqueName: \"kubernetes.io/projected/ab382b04-4fd2-4172-bc26-8f10347da05c-kube-api-access-v2sjj\") pod \"payload-pre-processing-e2e-inf-b-c45877-787d5595b7-8mhf5\" (UID: \"ab382b04-4fd2-4172-bc26-8f10347da05c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c45877-787d5595b7-8mhf5" Oct 30 17:18:30.628855 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:30.628828 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab382b04-4fd2-4172-bc26-8f10347da05c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c45877-787d5595b7-8mhf5\" (UID: \"ab382b04-4fd2-4172-bc26-8f10347da05c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c45877-787d5595b7-8mhf5" Oct 30 17:18:30.653827 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:30.653796 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2sjj\" (UniqueName: \"kubernetes.io/projected/ab382b04-4fd2-4172-bc26-8f10347da05c-kube-api-access-v2sjj\") pod \"payload-pre-processing-e2e-inf-b-c45877-787d5595b7-8mhf5\" (UID: \"ab382b04-4fd2-4172-bc26-8f10347da05c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c45877-787d5595b7-8mhf5" Oct 30 17:18:30.922627 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:30.922539 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c45877-787d5595b7-8mhf5" Oct 30 17:18:31.231230 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:18:31.231192 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab382b04_4fd2_4172_bc26_8f10347da05c.slice/crio-2e1d9d327a593c1312c5ace768ae064f59cf2d9dc6be1f9cc03796be06822a43 WatchSource:0}: Error finding container 2e1d9d327a593c1312c5ace768ae064f59cf2d9dc6be1f9cc03796be06822a43: Status 404 returned error can't find the container with id 2e1d9d327a593c1312c5ace768ae064f59cf2d9dc6be1f9cc03796be06822a43 Oct 30 17:18:31.234797 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:31.234728 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c45877-787d5595b7-8mhf5"] Oct 30 17:18:31.684955 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:31.684866 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c45877-787d5595b7-8mhf5" event={"ID":"ab382b04-4fd2-4172-bc26-8f10347da05c","Type":"ContainerStarted","Data":"9570aca6851573754f290ca5602901a159ee6aa98b9281b81dd46851fe49567e"} Oct 30 17:18:31.684955 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:31.684921 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c45877-787d5595b7-8mhf5" event={"ID":"ab382b04-4fd2-4172-bc26-8f10347da05c","Type":"ContainerStarted","Data":"2e1d9d327a593c1312c5ace768ae064f59cf2d9dc6be1f9cc03796be06822a43"} Oct 30 17:18:31.684955 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:31.684954 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c45877-787d5595b7-8mhf5" Oct 30 17:18:31.729529 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:31.729474 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c45877-787d5595b7-8mhf5" podStartSLOduration=1.729460266 podStartE2EDuration="1.729460266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:18:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:18:31.72828371 +0000 UTC m=+1885.729038611" watchObservedRunningTime="2026-10-30 17:18:31.729460266 +0000 UTC m=+1885.730215166" Oct 30 17:18:42.697630 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:42.697601 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c45877-787d5595b7-8mhf5" Oct 30 17:18:42.950473 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:18:42.950389 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:18:54.331467 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:54.331430 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-001a96-54c64574d7-txw2m"] Oct 30 17:18:54.337576 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:54.337557 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-001a96-54c64574d7-txw2m" Oct 30 17:18:54.358355 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:54.358326 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-001a96-54c64574d7-txw2m"] Oct 30 17:18:54.365015 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:54.364991 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-001a96\"" Oct 30 17:18:54.365168 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:54.365083 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-001a96-dockercfg-rzjm8\"" Oct 30 17:18:54.541337 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:54.537864 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05959f4f-5f66-4a9e-8050-851b8fce2c5b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-001a96-54c64574d7-txw2m\" (UID: \"05959f4f-5f66-4a9e-8050-851b8fce2c5b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-001a96-54c64574d7-txw2m" Oct 30 17:18:54.541337 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:54.538168 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njjjw\" (UniqueName: \"kubernetes.io/projected/05959f4f-5f66-4a9e-8050-851b8fce2c5b-kube-api-access-njjjw\") pod \"payload-pre-processing-e2e-shared-b-w4-001a96-54c64574d7-txw2m\" (UID: \"05959f4f-5f66-4a9e-8050-851b8fce2c5b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-001a96-54c64574d7-txw2m" Oct 30 17:18:54.637449 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:54.637360 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-001a96-67fb95b8d4-qkhqq"] Oct 30 17:18:54.641720 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:54.641683 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05959f4f-5f66-4a9e-8050-851b8fce2c5b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-001a96-54c64574d7-txw2m\" (UID: \"05959f4f-5f66-4a9e-8050-851b8fce2c5b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-001a96-54c64574d7-txw2m" Oct 30 17:18:54.642808 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:54.642011 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njjjw\" (UniqueName: \"kubernetes.io/projected/05959f4f-5f66-4a9e-8050-851b8fce2c5b-kube-api-access-njjjw\") pod \"payload-pre-processing-e2e-shared-b-w4-001a96-54c64574d7-txw2m\" (UID: \"05959f4f-5f66-4a9e-8050-851b8fce2c5b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-001a96-54c64574d7-txw2m" Oct 30 17:18:54.643272 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:54.643229 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05959f4f-5f66-4a9e-8050-851b8fce2c5b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-001a96-54c64574d7-txw2m\" (UID: \"05959f4f-5f66-4a9e-8050-851b8fce2c5b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-001a96-54c64574d7-txw2m" Oct 30 17:18:54.653364 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:54.653336 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-001a96-67fb95b8d4-qkhqq"] Oct 30 17:18:54.653705 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:54.653452 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-001a96-67fb95b8d4-qkhqq" Oct 30 17:18:54.670905 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:54.670879 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njjjw\" (UniqueName: \"kubernetes.io/projected/05959f4f-5f66-4a9e-8050-851b8fce2c5b-kube-api-access-njjjw\") pod \"payload-pre-processing-e2e-shared-b-w4-001a96-54c64574d7-txw2m\" (UID: \"05959f4f-5f66-4a9e-8050-851b8fce2c5b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-001a96-54c64574d7-txw2m" Oct 30 17:18:54.745378 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:54.745344 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4c8fa39-e889-46e8-9afc-d1bbefbe57c3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-001a96-67fb95b8d4-qkhqq\" (UID: \"e4c8fa39-e889-46e8-9afc-d1bbefbe57c3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-001a96-67fb95b8d4-qkhqq" Oct 30 17:18:54.745550 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:54.745389 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj6g9\" (UniqueName: \"kubernetes.io/projected/e4c8fa39-e889-46e8-9afc-d1bbefbe57c3-kube-api-access-zj6g9\") pod \"payload-processing-e2e-shared-b-w4-001a96-67fb95b8d4-qkhqq\" (UID: \"e4c8fa39-e889-46e8-9afc-d1bbefbe57c3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-001a96-67fb95b8d4-qkhqq" Oct 30 17:18:54.847071 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:54.847029 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4c8fa39-e889-46e8-9afc-d1bbefbe57c3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-001a96-67fb95b8d4-qkhqq\" (UID: \"e4c8fa39-e889-46e8-9afc-d1bbefbe57c3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-001a96-67fb95b8d4-qkhqq" Oct 30 17:18:54.847273 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:54.847083 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj6g9\" (UniqueName: \"kubernetes.io/projected/e4c8fa39-e889-46e8-9afc-d1bbefbe57c3-kube-api-access-zj6g9\") pod \"payload-processing-e2e-shared-b-w4-001a96-67fb95b8d4-qkhqq\" (UID: \"e4c8fa39-e889-46e8-9afc-d1bbefbe57c3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-001a96-67fb95b8d4-qkhqq" Oct 30 17:18:54.847739 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:54.847719 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4c8fa39-e889-46e8-9afc-d1bbefbe57c3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-001a96-67fb95b8d4-qkhqq\" (UID: \"e4c8fa39-e889-46e8-9afc-d1bbefbe57c3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-001a96-67fb95b8d4-qkhqq" Oct 30 17:18:54.871796 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:54.871770 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj6g9\" (UniqueName: \"kubernetes.io/projected/e4c8fa39-e889-46e8-9afc-d1bbefbe57c3-kube-api-access-zj6g9\") pod \"payload-processing-e2e-shared-b-w4-001a96-67fb95b8d4-qkhqq\" (UID: \"e4c8fa39-e889-46e8-9afc-d1bbefbe57c3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-001a96-67fb95b8d4-qkhqq" Oct 30 17:18:54.952211 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:54.952180 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-001a96-54c64574d7-txw2m" Oct 30 17:18:54.970958 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:54.970932 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-001a96-67fb95b8d4-qkhqq" Oct 30 17:18:55.267203 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:18:55.267174 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05959f4f_5f66_4a9e_8050_851b8fce2c5b.slice/crio-0ef8c9a18ca8eccaa0c39042c8526fce9b17b1e47d7eca9f04c904ad367e2932 WatchSource:0}: Error finding container 0ef8c9a18ca8eccaa0c39042c8526fce9b17b1e47d7eca9f04c904ad367e2932: Status 404 returned error can't find the container with id 0ef8c9a18ca8eccaa0c39042c8526fce9b17b1e47d7eca9f04c904ad367e2932 Oct 30 17:18:55.269250 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:55.269224 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-001a96-54c64574d7-txw2m"] Oct 30 17:18:55.298630 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:55.298592 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-001a96-67fb95b8d4-qkhqq"] Oct 30 17:18:55.300172 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:18:55.300147 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4c8fa39_e889_46e8_9afc_d1bbefbe57c3.slice/crio-782b0a995930b18b98715326ce6e36344eede328c19087fec26ead7542d8b25f WatchSource:0}: Error finding container 782b0a995930b18b98715326ce6e36344eede328c19087fec26ead7542d8b25f: Status 404 returned error can't find the container with id 782b0a995930b18b98715326ce6e36344eede328c19087fec26ead7542d8b25f Oct 30 17:18:55.978495 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:55.978455 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-001a96-67fb95b8d4-qkhqq" event={"ID":"e4c8fa39-e889-46e8-9afc-d1bbefbe57c3","Type":"ContainerStarted","Data":"d99275c569a87658b0e39c88e0376e7de68f50f04068319f3f4d182d149d260d"} Oct 30 17:18:55.979111 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:55.979084 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-001a96-67fb95b8d4-qkhqq" Oct 30 17:18:55.979266 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:55.979125 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-001a96-67fb95b8d4-qkhqq" event={"ID":"e4c8fa39-e889-46e8-9afc-d1bbefbe57c3","Type":"ContainerStarted","Data":"782b0a995930b18b98715326ce6e36344eede328c19087fec26ead7542d8b25f"} Oct 30 17:18:55.980620 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:55.980597 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-001a96-54c64574d7-txw2m" event={"ID":"05959f4f-5f66-4a9e-8050-851b8fce2c5b","Type":"ContainerStarted","Data":"3f964de1e10177334e5dadcc6593cfd3f99f5a83e3e322871b2ffd413b993270"} Oct 30 17:18:55.980726 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:55.980677 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-001a96-54c64574d7-txw2m" event={"ID":"05959f4f-5f66-4a9e-8050-851b8fce2c5b","Type":"ContainerStarted","Data":"0ef8c9a18ca8eccaa0c39042c8526fce9b17b1e47d7eca9f04c904ad367e2932"} Oct 30 17:18:55.980726 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:55.980711 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-001a96-54c64574d7-txw2m" Oct 30 17:18:56.026658 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:56.026612 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-001a96-67fb95b8d4-qkhqq" podStartSLOduration=2.0265992 podStartE2EDuration="2.0265992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:18:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:18:56.024288208 +0000 UTC m=+1910.025043120" watchObservedRunningTime="2026-10-30 17:18:56.0265992 +0000 UTC m=+1910.027354101" Oct 30 17:18:56.073897 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:56.073849 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-001a96-54c64574d7-txw2m" podStartSLOduration=2.073833905 podStartE2EDuration="2.073833905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:18:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:18:56.07308269 +0000 UTC m=+1910.073837592" watchObservedRunningTime="2026-10-30 17:18:56.073833905 +0000 UTC m=+1910.074588808" Oct 30 17:18:58.035609 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:58.035574 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8496cdb9-67f54"] Oct 30 17:18:58.048614 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:58.048578 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8496cdb9-67f54"] Oct 30 17:18:58.148824 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:18:58.148790 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33afccfc-04f5-4b07-86d0-228ab10bedd2" path="/var/lib/kubelet/pods/33afccfc-04f5-4b07-86d0-228ab10bedd2/volumes" Oct 30 17:19:06.994027 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:06.993996 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-001a96-67fb95b8d4-qkhqq" Oct 30 17:19:06.994441 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:06.994076 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-001a96-54c64574d7-txw2m" Oct 30 17:19:07.083626 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:07.083592 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv"] Oct 30 17:19:07.089491 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:07.089465 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv" Oct 30 17:19:07.109565 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:07.109530 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv"] Oct 30 17:19:07.110557 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:07.110535 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-78c9c2-openshift-default-dockercfg-2lskh\"" Oct 30 17:19:07.158098 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:07.158066 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba5dbb57-7fdc-4509-a870-6c95778f650b-istiod-ca-cert\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv\" (UID: \"ba5dbb57-7fdc-4509-a870-6c95778f650b\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv" Oct 30 17:19:07.158323 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:07.158126 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5dbb57-7fdc-4509-a870-6c95778f650b-workload-socket\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv\" (UID: \"ba5dbb57-7fdc-4509-a870-6c95778f650b\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv" Oct 30 17:19:07.158323 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:07.158148 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba5dbb57-7fdc-4509-a870-6c95778f650b-istio-data\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv\" (UID: \"ba5dbb57-7fdc-4509-a870-6c95778f650b\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv" Oct 30 17:19:07.158569 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:07.158552 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5dbb57-7fdc-4509-a870-6c95778f650b-credential-socket\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv\" (UID: \"ba5dbb57-7fdc-4509-a870-6c95778f650b\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv" Oct 30 17:19:07.158708 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:07.158682 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba5dbb57-7fdc-4509-a870-6c95778f650b-workload-certs\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv\" (UID: \"ba5dbb57-7fdc-4509-a870-6c95778f650b\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv" Oct 30 17:19:07.158853 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:07.158830 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba5dbb57-7fdc-4509-a870-6c95778f650b-istio-envoy\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv\" (UID: \"ba5dbb57-7fdc-4509-a870-6c95778f650b\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv" Oct 30 17:19:07.159122 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:07.159091 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kbjq\" (UniqueName: \"kubernetes.io/projected/ba5dbb57-7fdc-4509-a870-6c95778f650b-kube-api-access-7kbjq\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv\" (UID: \"ba5dbb57-7fdc-4509-a870-6c95778f650b\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv" Oct 30 17:19:07.159469 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:07.159443 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba5dbb57-7fdc-4509-a870-6c95778f650b-istio-token\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv\" (UID: \"ba5dbb57-7fdc-4509-a870-6c95778f650b\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv" Oct 30 17:19:07.159597 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:07.159573 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba5dbb57-7fdc-4509-a870-6c95778f650b-istio-podinfo\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv\" (UID: \"ba5dbb57-7fdc-4509-a870-6c95778f650b\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv" Oct 30 17:19:07.261203 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:07.261112 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba5dbb57-7fdc-4509-a870-6c95778f650b-workload-certs\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv\" (UID: \"ba5dbb57-7fdc-4509-a870-6c95778f650b\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv" Oct 30 17:19:07.261203 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:07.261198 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba5dbb57-7fdc-4509-a870-6c95778f650b-istio-envoy\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv\" (UID: \"ba5dbb57-7fdc-4509-a870-6c95778f650b\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv" Oct 30 17:19:07.261478 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:07.261330 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kbjq\" (UniqueName: \"kubernetes.io/projected/ba5dbb57-7fdc-4509-a870-6c95778f650b-kube-api-access-7kbjq\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv\" (UID: \"ba5dbb57-7fdc-4509-a870-6c95778f650b\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv" Oct 30 17:19:07.261478 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:07.261433 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba5dbb57-7fdc-4509-a870-6c95778f650b-istio-token\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv\" (UID: \"ba5dbb57-7fdc-4509-a870-6c95778f650b\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv" Oct 30 17:19:07.261606 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:07.261494 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba5dbb57-7fdc-4509-a870-6c95778f650b-istio-podinfo\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv\" (UID: \"ba5dbb57-7fdc-4509-a870-6c95778f650b\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv" Oct 30 17:19:07.261606 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:07.261587 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba5dbb57-7fdc-4509-a870-6c95778f650b-istiod-ca-cert\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv\" (UID: \"ba5dbb57-7fdc-4509-a870-6c95778f650b\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv" Oct 30 17:19:07.261724 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:07.261597 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba5dbb57-7fdc-4509-a870-6c95778f650b-workload-certs\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv\" (UID: \"ba5dbb57-7fdc-4509-a870-6c95778f650b\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv" Oct 30 17:19:07.261724 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:07.261657 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5dbb57-7fdc-4509-a870-6c95778f650b-workload-socket\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv\" (UID: \"ba5dbb57-7fdc-4509-a870-6c95778f650b\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv" Oct 30 17:19:07.261724 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:07.261684 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba5dbb57-7fdc-4509-a870-6c95778f650b-istio-data\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv\" (UID: \"ba5dbb57-7fdc-4509-a870-6c95778f650b\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv" Oct 30 17:19:07.261724 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:07.261712 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5dbb57-7fdc-4509-a870-6c95778f650b-credential-socket\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv\" (UID: \"ba5dbb57-7fdc-4509-a870-6c95778f650b\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv" Oct 30 17:19:07.262072 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:07.262013 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5dbb57-7fdc-4509-a870-6c95778f650b-workload-socket\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv\" (UID: \"ba5dbb57-7fdc-4509-a870-6c95778f650b\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv" Oct 30 17:19:07.262072 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:07.262038 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5dbb57-7fdc-4509-a870-6c95778f650b-credential-socket\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv\" (UID: \"ba5dbb57-7fdc-4509-a870-6c95778f650b\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv" Oct 30 17:19:07.262202 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:07.262093 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba5dbb57-7fdc-4509-a870-6c95778f650b-istio-data\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv\" (UID: \"ba5dbb57-7fdc-4509-a870-6c95778f650b\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv" Oct 30 17:19:07.262298 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:07.262278 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba5dbb57-7fdc-4509-a870-6c95778f650b-istiod-ca-cert\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv\" (UID: \"ba5dbb57-7fdc-4509-a870-6c95778f650b\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv" Oct 30 17:19:07.263908 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:07.263884 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba5dbb57-7fdc-4509-a870-6c95778f650b-istio-envoy\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv\" (UID: \"ba5dbb57-7fdc-4509-a870-6c95778f650b\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv" Oct 30 17:19:07.264173 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:07.264138 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba5dbb57-7fdc-4509-a870-6c95778f650b-istio-podinfo\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv\" (UID: \"ba5dbb57-7fdc-4509-a870-6c95778f650b\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv" Oct 30 17:19:07.291144 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:07.291119 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba5dbb57-7fdc-4509-a870-6c95778f650b-istio-token\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv\" (UID: \"ba5dbb57-7fdc-4509-a870-6c95778f650b\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv" Oct 30 17:19:07.291875 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:07.291855 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kbjq\" (UniqueName: \"kubernetes.io/projected/ba5dbb57-7fdc-4509-a870-6c95778f650b-kube-api-access-7kbjq\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv\" (UID: \"ba5dbb57-7fdc-4509-a870-6c95778f650b\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv" Oct 30 17:19:07.408821 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:07.408792 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv" Oct 30 17:19:07.744979 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:19:07.744947 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba5dbb57_7fdc_4509_a870_6c95778f650b.slice/crio-eb51f51118247985525b5901c88dfd64c02d8402e5777759bfa3ac1155f71fcf WatchSource:0}: Error finding container eb51f51118247985525b5901c88dfd64c02d8402e5777759bfa3ac1155f71fcf: Status 404 returned error can't find the container with id eb51f51118247985525b5901c88dfd64c02d8402e5777759bfa3ac1155f71fcf Oct 30 17:19:07.747179 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:07.747157 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv"] Oct 30 17:19:07.748148 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:07.748120 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:19:07.748207 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:07.748181 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:19:07.748242 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:07.748211 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:19:08.093197 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:08.093113 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv" event={"ID":"ba5dbb57-7fdc-4509-a870-6c95778f650b","Type":"ContainerStarted","Data":"dccf5ac7282e6ac94a97385dc7ef4b1ce5c239819a40858e0350c069f8e9356b"} Oct 30 17:19:08.093197 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:08.093165 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv" event={"ID":"ba5dbb57-7fdc-4509-a870-6c95778f650b","Type":"ContainerStarted","Data":"eb51f51118247985525b5901c88dfd64c02d8402e5777759bfa3ac1155f71fcf"} Oct 30 17:19:08.144344 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:08.144281 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv" podStartSLOduration=1.144266799 podStartE2EDuration="1.144266799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:19:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:19:08.141657696 +0000 UTC m=+1922.142412615" watchObservedRunningTime="2026-10-30 17:19:08.144266799 +0000 UTC m=+1922.145021699" Oct 30 17:19:08.417603 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:08.417498 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv" Oct 30 17:19:08.419505 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:08.419478 4522 patch_prober.go:29] interesting pod/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body= Oct 30 17:19:08.419638 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:08.419525 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv" podUID="ba5dbb57-7fdc-4509-a870-6c95778f650b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" Oct 30 17:19:08.655922 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:08.655883 4522 scope.go:122] "RemoveContainer" containerID="3a213048b12519db261e4a792b0ba9c8ea3111371369a560000c4068c193e8d4" Oct 30 17:19:08.667862 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:08.667838 4522 scope.go:122] "RemoveContainer" containerID="9fe1a813d428b2732acbf2b35e49c247f2ff770795be0bcb825862e14242c7a4" Oct 30 17:19:08.678770 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:08.678749 4522 scope.go:122] "RemoveContainer" containerID="1efd4e01714cb87d3bed11a295abf1528a19c909ece6233634556584e6118364" Oct 30 17:19:08.689721 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:08.689695 4522 scope.go:122] "RemoveContainer" containerID="8f8b1f55862d413d4b68cd634112b25f4850bd62a62c2e0c6b06d94ee6743796" Oct 30 17:19:09.413332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:09.412458 4522 patch_prober.go:29] interesting pod/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body= Oct 30 17:19:09.413332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:09.412536 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv" podUID="ba5dbb57-7fdc-4509-a870-6c95778f650b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" Oct 30 17:19:10.416761 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:10.416725 4522 patch_prober.go:29] interesting pod/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body= Oct 30 17:19:10.417178 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:10.416821 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv" podUID="ba5dbb57-7fdc-4509-a870-6c95778f650b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" Oct 30 17:19:11.410333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:11.409704 4522 patch_prober.go:29] interesting pod/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body= Oct 30 17:19:11.410333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:11.409786 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv" podUID="ba5dbb57-7fdc-4509-a870-6c95778f650b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" Oct 30 17:19:12.413330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:12.412746 4522 patch_prober.go:29] interesting pod/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body= Oct 30 17:19:12.413330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:12.412811 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv" podUID="ba5dbb57-7fdc-4509-a870-6c95778f650b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" Oct 30 17:19:12.961658 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:19:12.961618 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:19:13.411863 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:13.411766 4522 patch_prober.go:29] interesting pod/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body= Oct 30 17:19:13.412132 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:13.412100 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv" podUID="ba5dbb57-7fdc-4509-a870-6c95778f650b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" Oct 30 17:19:14.482222 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:14.482184 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv" Oct 30 17:19:14.483504 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:14.483482 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv" Oct 30 17:19:14.484279 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:14.484257 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv" Oct 30 17:19:14.789816 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:14.789737 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v"] Oct 30 17:19:14.794659 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:14.794638 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v" Oct 30 17:19:14.823169 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:14.823141 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v"] Oct 30 17:19:14.924107 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:14.924072 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/056b7890-d03e-4f62-8585-b3f7a7777a24-istio-data\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v\" (UID: \"056b7890-d03e-4f62-8585-b3f7a7777a24\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v" Oct 30 17:19:14.924350 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:14.924143 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/056b7890-d03e-4f62-8585-b3f7a7777a24-istio-token\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v\" (UID: \"056b7890-d03e-4f62-8585-b3f7a7777a24\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v" Oct 30 17:19:14.924350 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:14.924243 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/056b7890-d03e-4f62-8585-b3f7a7777a24-istio-podinfo\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v\" (UID: \"056b7890-d03e-4f62-8585-b3f7a7777a24\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v" Oct 30 17:19:14.924350 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:14.924277 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/056b7890-d03e-4f62-8585-b3f7a7777a24-workload-socket\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v\" (UID: \"056b7890-d03e-4f62-8585-b3f7a7777a24\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v" Oct 30 17:19:14.924503 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:14.924467 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fh8k\" (UniqueName: \"kubernetes.io/projected/056b7890-d03e-4f62-8585-b3f7a7777a24-kube-api-access-9fh8k\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v\" (UID: \"056b7890-d03e-4f62-8585-b3f7a7777a24\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v" Oct 30 17:19:14.924553 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:14.924525 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/056b7890-d03e-4f62-8585-b3f7a7777a24-credential-socket\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v\" (UID: \"056b7890-d03e-4f62-8585-b3f7a7777a24\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v" Oct 30 17:19:14.924643 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:14.924626 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/056b7890-d03e-4f62-8585-b3f7a7777a24-istiod-ca-cert\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v\" (UID: \"056b7890-d03e-4f62-8585-b3f7a7777a24\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v" Oct 30 17:19:14.924687 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:14.924675 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/056b7890-d03e-4f62-8585-b3f7a7777a24-istio-envoy\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v\" (UID: \"056b7890-d03e-4f62-8585-b3f7a7777a24\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v" Oct 30 17:19:14.924724 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:14.924713 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/056b7890-d03e-4f62-8585-b3f7a7777a24-workload-certs\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v\" (UID: \"056b7890-d03e-4f62-8585-b3f7a7777a24\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v" Oct 30 17:19:15.026707 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:15.026661 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/056b7890-d03e-4f62-8585-b3f7a7777a24-credential-socket\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v\" (UID: \"056b7890-d03e-4f62-8585-b3f7a7777a24\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v" Oct 30 17:19:15.026904 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:15.026726 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/056b7890-d03e-4f62-8585-b3f7a7777a24-istiod-ca-cert\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v\" (UID: \"056b7890-d03e-4f62-8585-b3f7a7777a24\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v" Oct 30 17:19:15.026904 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:15.026758 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/056b7890-d03e-4f62-8585-b3f7a7777a24-istio-envoy\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v\" (UID: \"056b7890-d03e-4f62-8585-b3f7a7777a24\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v" Oct 30 17:19:15.026904 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:15.026800 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/056b7890-d03e-4f62-8585-b3f7a7777a24-workload-certs\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v\" (UID: \"056b7890-d03e-4f62-8585-b3f7a7777a24\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v" Oct 30 17:19:15.027100 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:15.026936 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/056b7890-d03e-4f62-8585-b3f7a7777a24-istio-data\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v\" (UID: \"056b7890-d03e-4f62-8585-b3f7a7777a24\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v" Oct 30 17:19:15.027100 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:15.026978 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/056b7890-d03e-4f62-8585-b3f7a7777a24-istio-token\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v\" (UID: \"056b7890-d03e-4f62-8585-b3f7a7777a24\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v" Oct 30 17:19:15.027100 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:15.027057 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/056b7890-d03e-4f62-8585-b3f7a7777a24-istio-podinfo\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v\" (UID: \"056b7890-d03e-4f62-8585-b3f7a7777a24\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v" Oct 30 17:19:15.027100 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:15.027090 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/056b7890-d03e-4f62-8585-b3f7a7777a24-workload-socket\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v\" (UID: \"056b7890-d03e-4f62-8585-b3f7a7777a24\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v" Oct 30 17:19:15.027378 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:15.027220 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fh8k\" (UniqueName: \"kubernetes.io/projected/056b7890-d03e-4f62-8585-b3f7a7777a24-kube-api-access-9fh8k\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v\" (UID: \"056b7890-d03e-4f62-8585-b3f7a7777a24\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v" Oct 30 17:19:15.027455 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:15.027383 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/056b7890-d03e-4f62-8585-b3f7a7777a24-credential-socket\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v\" (UID: \"056b7890-d03e-4f62-8585-b3f7a7777a24\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v" Oct 30 17:19:15.027455 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:15.027442 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/056b7890-d03e-4f62-8585-b3f7a7777a24-workload-certs\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v\" (UID: \"056b7890-d03e-4f62-8585-b3f7a7777a24\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v" Oct 30 17:19:15.027539 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:15.027484 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/056b7890-d03e-4f62-8585-b3f7a7777a24-istio-data\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v\" (UID: \"056b7890-d03e-4f62-8585-b3f7a7777a24\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v" Oct 30 17:19:15.027650 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:15.027627 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/056b7890-d03e-4f62-8585-b3f7a7777a24-workload-socket\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v\" (UID: \"056b7890-d03e-4f62-8585-b3f7a7777a24\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v" Oct 30 17:19:15.027914 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:15.027898 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/056b7890-d03e-4f62-8585-b3f7a7777a24-istiod-ca-cert\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v\" (UID: \"056b7890-d03e-4f62-8585-b3f7a7777a24\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v" Oct 30 17:19:15.029148 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:15.029129 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/056b7890-d03e-4f62-8585-b3f7a7777a24-istio-envoy\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v\" (UID: \"056b7890-d03e-4f62-8585-b3f7a7777a24\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v" Oct 30 17:19:15.031588 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:15.031568 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/056b7890-d03e-4f62-8585-b3f7a7777a24-istio-podinfo\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v\" (UID: \"056b7890-d03e-4f62-8585-b3f7a7777a24\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v" Oct 30 17:19:15.053298 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:15.053235 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/056b7890-d03e-4f62-8585-b3f7a7777a24-istio-token\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v\" (UID: \"056b7890-d03e-4f62-8585-b3f7a7777a24\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v" Oct 30 17:19:15.053442 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:15.053422 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fh8k\" (UniqueName: \"kubernetes.io/projected/056b7890-d03e-4f62-8585-b3f7a7777a24-kube-api-access-9fh8k\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v\" (UID: \"056b7890-d03e-4f62-8585-b3f7a7777a24\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v" Oct 30 17:19:15.111280 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:15.111238 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v" Oct 30 17:19:15.436386 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:19:15.436355 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod056b7890_d03e_4f62_8585_b3f7a7777a24.slice/crio-ac8d7f147344bfe4e84a6ae2f988dc6b28be7860f0bb09bf85b3943056af114a WatchSource:0}: Error finding container ac8d7f147344bfe4e84a6ae2f988dc6b28be7860f0bb09bf85b3943056af114a: Status 404 returned error can't find the container with id ac8d7f147344bfe4e84a6ae2f988dc6b28be7860f0bb09bf85b3943056af114a Oct 30 17:19:15.438758 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:15.438735 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v"] Oct 30 17:19:15.439534 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:15.439506 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:19:15.439588 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:15.439576 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:19:15.439633 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:15.439622 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:19:16.227024 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:16.226981 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v" event={"ID":"056b7890-d03e-4f62-8585-b3f7a7777a24","Type":"ContainerStarted","Data":"6979ce90ee24f03a859277190837f2d9137302028de5e8d48011f33ef367a6c4"} Oct 30 17:19:16.227463 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:16.227052 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v" event={"ID":"056b7890-d03e-4f62-8585-b3f7a7777a24","Type":"ContainerStarted","Data":"ac8d7f147344bfe4e84a6ae2f988dc6b28be7860f0bb09bf85b3943056af114a"} Oct 30 17:19:16.318693 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:16.318637 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v" podStartSLOduration=2.318621246 podStartE2EDuration="2.318621246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:19:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:19:16.289390069 +0000 UTC m=+1930.290144969" watchObservedRunningTime="2026-10-30 17:19:16.318621246 +0000 UTC m=+1930.319376146" Oct 30 17:19:17.117329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:17.116611 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v" Oct 30 17:19:17.118987 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:17.118904 4522 patch_prober.go:29] interesting pod/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Oct 30 17:19:17.118987 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:17.118974 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v" podUID="056b7890-d03e-4f62-8585-b3f7a7777a24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Oct 30 17:19:18.116753 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:18.116716 4522 patch_prober.go:29] interesting pod/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Oct 30 17:19:18.117165 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:18.116771 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v" podUID="056b7890-d03e-4f62-8585-b3f7a7777a24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Oct 30 17:19:19.113903 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:19.113809 4522 patch_prober.go:29] interesting pod/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Oct 30 17:19:19.114180 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:19.114151 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v" podUID="056b7890-d03e-4f62-8585-b3f7a7777a24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Oct 30 17:19:20.111794 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:20.111750 4522 patch_prober.go:29] interesting pod/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Oct 30 17:19:20.112511 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:20.112467 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v" podUID="056b7890-d03e-4f62-8585-b3f7a7777a24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Oct 30 17:19:21.113784 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:21.113743 4522 patch_prober.go:29] interesting pod/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Oct 30 17:19:21.114403 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:21.114368 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v" podUID="056b7890-d03e-4f62-8585-b3f7a7777a24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Oct 30 17:19:22.221474 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:22.221437 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v" Oct 30 17:19:22.221947 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:22.221913 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v" Oct 30 17:19:22.223605 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:22.223583 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v" Oct 30 17:19:41.044863 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:41.044792 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33b8828f-w7xtc"] Oct 30 17:19:41.056388 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:41.056354 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33b8828f-w7xtc"] Oct 30 17:19:42.166360 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:42.165730 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70ccb80a-0b1e-4b73-851c-8c5294468c2c" path="/var/lib/kubelet/pods/70ccb80a-0b1e-4b73-851c-8c5294468c2c/volumes" Oct 30 17:19:42.951893 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:19:42.951836 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:19:53.042299 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:53.042265 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ebacbb2-27rdm"] Oct 30 17:19:53.057673 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:53.057644 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ebacbb2-27rdm"] Oct 30 17:19:53.616256 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:53.616221 4522 generic.go:356] "Generic (PLEG): container finished" podID="40f8b984-d0d8-4938-859e-268ae92cf8f2" containerID="19340ec98a9c74003f96585f0347e896a9703170056e3ececf339bf3431d812c" exitCode=0 Oct 30 17:19:53.616518 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:53.616299 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh" event={"ID":"40f8b984-d0d8-4938-859e-268ae92cf8f2","Type":"ContainerDied","Data":"19340ec98a9c74003f96585f0347e896a9703170056e3ececf339bf3431d812c"} Oct 30 17:19:53.616874 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:53.616855 4522 scope.go:122] "RemoveContainer" containerID="19340ec98a9c74003f96585f0347e896a9703170056e3ececf339bf3431d812c" Oct 30 17:19:53.618758 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:53.618732 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:19:53.618821 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:53.618799 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:19:53.618859 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:53.618833 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:19:54.147731 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:54.147690 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1555702-8acd-4384-acb9-94e3f51495de" path="/var/lib/kubelet/pods/a1555702-8acd-4384-acb9-94e3f51495de/volumes" Oct 30 17:19:54.632373 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:54.632331 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh" event={"ID":"40f8b984-d0d8-4938-859e-268ae92cf8f2","Type":"ContainerStarted","Data":"e77d008790cb6f74989e7fc89e4510f0b2b0dd107c2e75415d2fec9e5e4d0b66"} Oct 30 17:19:54.852520 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:54.852480 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh" Oct 30 17:19:54.855412 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:54.855382 4522 patch_prober.go:29] interesting pod/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 30 17:19:54.855577 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:54.855432 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh" podUID="40f8b984-d0d8-4938-859e-268ae92cf8f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 30 17:19:55.852385 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:55.852352 4522 patch_prober.go:29] interesting pod/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 30 17:19:55.852803 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:55.852403 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh" podUID="40f8b984-d0d8-4938-859e-268ae92cf8f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 30 17:19:56.852257 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:56.852219 4522 patch_prober.go:29] interesting pod/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 30 17:19:56.852477 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:56.852283 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh" podUID="40f8b984-d0d8-4938-859e-268ae92cf8f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 30 17:19:57.638557 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:57.638528 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh" Oct 30 17:19:57.852539 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:57.852507 4522 patch_prober.go:29] interesting pod/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 30 17:19:57.852940 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:57.852560 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh" podUID="40f8b984-d0d8-4938-859e-268ae92cf8f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 30 17:19:58.851874 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:58.851835 4522 patch_prober.go:29] interesting pod/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 30 17:19:58.852175 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:58.852144 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh" podUID="40f8b984-d0d8-4938-859e-268ae92cf8f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 30 17:19:59.852191 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:59.852159 4522 patch_prober.go:29] interesting pod/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 30 17:19:59.852639 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:19:59.852221 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh" podUID="40f8b984-d0d8-4938-859e-268ae92cf8f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 30 17:20:00.857168 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:00.857138 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh" Oct 30 17:20:00.858582 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:00.858558 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh" Oct 30 17:20:06.531560 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:06.531524 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0feca821-pqs6z"] Oct 30 17:20:06.535498 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:06.535477 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0feca821-pqs6z" Oct 30 17:20:06.558043 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:06.558009 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0feca821-pqs6z"] Oct 30 17:20:06.564220 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:06.564178 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:20:06.564397 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:06.564217 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:20:06.617374 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:06.617282 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9d0a693-f2b8-4036-8f23-a3e2c304eb12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0feca821-pqs6z\" (UID: \"e9d0a693-f2b8-4036-8f23-a3e2c304eb12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0feca821-pqs6z" Oct 30 17:20:06.617374 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:06.617351 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9d0a693-f2b8-4036-8f23-a3e2c304eb12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0feca821-pqs6z\" (UID: \"e9d0a693-f2b8-4036-8f23-a3e2c304eb12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0feca821-pqs6z" Oct 30 17:20:06.617594 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:06.617512 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfp8q\" (UniqueName: \"kubernetes.io/projected/e9d0a693-f2b8-4036-8f23-a3e2c304eb12-kube-api-access-zfp8q\") pod \"maas-api-revoke-sub-0feca821-pqs6z\" (UID: \"e9d0a693-f2b8-4036-8f23-a3e2c304eb12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0feca821-pqs6z" Oct 30 17:20:06.719546 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:06.719511 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9d0a693-f2b8-4036-8f23-a3e2c304eb12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0feca821-pqs6z\" (UID: \"e9d0a693-f2b8-4036-8f23-a3e2c304eb12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0feca821-pqs6z" Oct 30 17:20:06.719546 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:06.719552 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9d0a693-f2b8-4036-8f23-a3e2c304eb12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0feca821-pqs6z\" (UID: \"e9d0a693-f2b8-4036-8f23-a3e2c304eb12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0feca821-pqs6z" Oct 30 17:20:06.719814 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:06.719722 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfp8q\" (UniqueName: \"kubernetes.io/projected/e9d0a693-f2b8-4036-8f23-a3e2c304eb12-kube-api-access-zfp8q\") pod \"maas-api-revoke-sub-0feca821-pqs6z\" (UID: \"e9d0a693-f2b8-4036-8f23-a3e2c304eb12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0feca821-pqs6z" Oct 30 17:20:06.720258 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:06.720237 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9d0a693-f2b8-4036-8f23-a3e2c304eb12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0feca821-pqs6z\" (UID: \"e9d0a693-f2b8-4036-8f23-a3e2c304eb12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0feca821-pqs6z" Oct 30 17:20:06.722096 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:06.722064 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9d0a693-f2b8-4036-8f23-a3e2c304eb12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0feca821-pqs6z\" (UID: \"e9d0a693-f2b8-4036-8f23-a3e2c304eb12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0feca821-pqs6z" Oct 30 17:20:06.757408 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:06.757370 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfp8q\" (UniqueName: \"kubernetes.io/projected/e9d0a693-f2b8-4036-8f23-a3e2c304eb12-kube-api-access-zfp8q\") pod \"maas-api-revoke-sub-0feca821-pqs6z\" (UID: \"e9d0a693-f2b8-4036-8f23-a3e2c304eb12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0feca821-pqs6z" Oct 30 17:20:06.852362 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:06.852255 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0feca821-pqs6z" Oct 30 17:20:07.200842 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:07.200807 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0feca821-pqs6z"] Oct 30 17:20:07.201103 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:20:07.201076 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9d0a693_f2b8_4036_8f23_a3e2c304eb12.slice/crio-e930ed79bac8fcd8d396fbfde521016a2a3a5f2aa29d8e0440bfbd9a15b9e82f WatchSource:0}: Error finding container e930ed79bac8fcd8d396fbfde521016a2a3a5f2aa29d8e0440bfbd9a15b9e82f: Status 404 returned error can't find the container with id e930ed79bac8fcd8d396fbfde521016a2a3a5f2aa29d8e0440bfbd9a15b9e82f Oct 30 17:20:07.463612 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:07.463581 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef03b6bc-rzdkj"] Oct 30 17:20:07.468118 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:07.468100 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef03b6bc-rzdkj" Oct 30 17:20:07.488044 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:07.488010 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef03b6bc-rzdkj"] Oct 30 17:20:07.635124 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:07.635086 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13d8213f-ae96-45a6-99c3-26d15d56313e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef03b6bc-rzdkj\" (UID: \"13d8213f-ae96-45a6-99c3-26d15d56313e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef03b6bc-rzdkj" Oct 30 17:20:07.635124 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:07.635126 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13d8213f-ae96-45a6-99c3-26d15d56313e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef03b6bc-rzdkj\" (UID: \"13d8213f-ae96-45a6-99c3-26d15d56313e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef03b6bc-rzdkj" Oct 30 17:20:07.635570 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:07.635395 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dkfr\" (UniqueName: \"kubernetes.io/projected/13d8213f-ae96-45a6-99c3-26d15d56313e-kube-api-access-8dkfr\") pod \"maas-api-revoke-sub-ef03b6bc-rzdkj\" (UID: \"13d8213f-ae96-45a6-99c3-26d15d56313e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef03b6bc-rzdkj" Oct 30 17:20:07.737120 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:07.737048 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dkfr\" (UniqueName: \"kubernetes.io/projected/13d8213f-ae96-45a6-99c3-26d15d56313e-kube-api-access-8dkfr\") pod \"maas-api-revoke-sub-ef03b6bc-rzdkj\" (UID: \"13d8213f-ae96-45a6-99c3-26d15d56313e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef03b6bc-rzdkj" Oct 30 17:20:07.737316 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:07.737284 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13d8213f-ae96-45a6-99c3-26d15d56313e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef03b6bc-rzdkj\" (UID: \"13d8213f-ae96-45a6-99c3-26d15d56313e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef03b6bc-rzdkj" Oct 30 17:20:07.737394 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:07.737335 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13d8213f-ae96-45a6-99c3-26d15d56313e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef03b6bc-rzdkj\" (UID: \"13d8213f-ae96-45a6-99c3-26d15d56313e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef03b6bc-rzdkj" Oct 30 17:20:07.737941 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:07.737920 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13d8213f-ae96-45a6-99c3-26d15d56313e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef03b6bc-rzdkj\" (UID: \"13d8213f-ae96-45a6-99c3-26d15d56313e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef03b6bc-rzdkj" Oct 30 17:20:07.739637 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:07.739619 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13d8213f-ae96-45a6-99c3-26d15d56313e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef03b6bc-rzdkj\" (UID: \"13d8213f-ae96-45a6-99c3-26d15d56313e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef03b6bc-rzdkj" Oct 30 17:20:07.761891 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:07.761855 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0feca821-pqs6z" event={"ID":"e9d0a693-f2b8-4036-8f23-a3e2c304eb12","Type":"ContainerStarted","Data":"18792109e2ce3e9b48b0bb3564c835bfe6b384f2fb0209b1a911de94311485b7"} Oct 30 17:20:07.762027 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:07.761907 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0feca821-pqs6z" event={"ID":"e9d0a693-f2b8-4036-8f23-a3e2c304eb12","Type":"ContainerStarted","Data":"e930ed79bac8fcd8d396fbfde521016a2a3a5f2aa29d8e0440bfbd9a15b9e82f"} Oct 30 17:20:07.762377 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:07.762356 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dkfr\" (UniqueName: \"kubernetes.io/projected/13d8213f-ae96-45a6-99c3-26d15d56313e-kube-api-access-8dkfr\") pod \"maas-api-revoke-sub-ef03b6bc-rzdkj\" (UID: \"13d8213f-ae96-45a6-99c3-26d15d56313e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef03b6bc-rzdkj" Oct 30 17:20:07.782431 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:07.782408 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef03b6bc-rzdkj" Oct 30 17:20:07.806678 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:07.806632 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0feca821-pqs6z" podStartSLOduration=1.806617754 podStartE2EDuration="1.806617754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:20:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:20:07.805263546 +0000 UTC m=+1981.806018443" watchObservedRunningTime="2026-10-30 17:20:07.806617754 +0000 UTC m=+1981.807372654" Oct 30 17:20:08.225341 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:08.225289 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef03b6bc-rzdkj"] Oct 30 17:20:08.762731 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:08.762702 4522 scope.go:122] "RemoveContainer" containerID="91671f414fcdef316b94568815a631fa9abf542b37a15f6552d7a4556e7be1f0" Oct 30 17:20:08.772516 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:08.772453 4522 generic.go:356] "Generic (PLEG): container finished" podID="e9d0a693-f2b8-4036-8f23-a3e2c304eb12" containerID="18792109e2ce3e9b48b0bb3564c835bfe6b384f2fb0209b1a911de94311485b7" exitCode=0 Oct 30 17:20:08.772643 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:08.772539 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0feca821-pqs6z" event={"ID":"e9d0a693-f2b8-4036-8f23-a3e2c304eb12","Type":"ContainerDied","Data":"18792109e2ce3e9b48b0bb3564c835bfe6b384f2fb0209b1a911de94311485b7"} Oct 30 17:20:08.774319 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:08.774287 4522 scope.go:122] "RemoveContainer" containerID="e6257cf785e7c6eb273e6c8c1cd921ca9a9973b8a084bdb970d8bb683d301e6d" Oct 30 17:20:08.774414 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:08.774348 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef03b6bc-rzdkj" event={"ID":"13d8213f-ae96-45a6-99c3-26d15d56313e","Type":"ContainerStarted","Data":"a96de25733c9705950c19e3cf107472e4de3b4c502ecbab04d85c017126e5918"} Oct 30 17:20:08.774414 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:08.774386 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef03b6bc-rzdkj" event={"ID":"13d8213f-ae96-45a6-99c3-26d15d56313e","Type":"ContainerStarted","Data":"d483a74232a8e2fdc43b2356f81f799c01624afdc0504caf659d39553976023f"} Oct 30 17:20:08.862769 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:08.862717 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef03b6bc-rzdkj" podStartSLOduration=1.862704072 podStartE2EDuration="1.862704072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:20:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:20:08.859578044 +0000 UTC m=+1982.860332947" watchObservedRunningTime="2026-10-30 17:20:08.862704072 +0000 UTC m=+1982.863458973" Oct 30 17:20:09.784293 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:09.784263 4522 generic.go:356] "Generic (PLEG): container finished" podID="13d8213f-ae96-45a6-99c3-26d15d56313e" containerID="a96de25733c9705950c19e3cf107472e4de3b4c502ecbab04d85c017126e5918" exitCode=0 Oct 30 17:20:09.784737 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:09.784348 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef03b6bc-rzdkj" event={"ID":"13d8213f-ae96-45a6-99c3-26d15d56313e","Type":"ContainerDied","Data":"a96de25733c9705950c19e3cf107472e4de3b4c502ecbab04d85c017126e5918"} Oct 30 17:20:09.980612 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:09.980588 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0feca821-pqs6z" Oct 30 17:20:10.093509 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:10.093411 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9d0a693-f2b8-4036-8f23-a3e2c304eb12-kube-api-access-zfp8q\" (UniqueName: \"kubernetes.io/projected/e9d0a693-f2b8-4036-8f23-a3e2c304eb12-kube-api-access-zfp8q\") pod \"e9d0a693-f2b8-4036-8f23-a3e2c304eb12\" (UID: \"e9d0a693-f2b8-4036-8f23-a3e2c304eb12\") " Oct 30 17:20:10.093509 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:10.093485 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9d0a693-f2b8-4036-8f23-a3e2c304eb12-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9d0a693-f2b8-4036-8f23-a3e2c304eb12-maas-api-xks-ca\") pod \"e9d0a693-f2b8-4036-8f23-a3e2c304eb12\" (UID: \"e9d0a693-f2b8-4036-8f23-a3e2c304eb12\") " Oct 30 17:20:10.093718 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:10.093658 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9d0a693-f2b8-4036-8f23-a3e2c304eb12-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9d0a693-f2b8-4036-8f23-a3e2c304eb12-maas-api-openshift-ca\") pod \"e9d0a693-f2b8-4036-8f23-a3e2c304eb12\" (UID: \"e9d0a693-f2b8-4036-8f23-a3e2c304eb12\") " Oct 30 17:20:10.097102 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:10.096982 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9d0a693-f2b8-4036-8f23-a3e2c304eb12-maas-api-openshift-ca" pod "e9d0a693-f2b8-4036-8f23-a3e2c304eb12" (UID: "e9d0a693-f2b8-4036-8f23-a3e2c304eb12"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:20:10.100495 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:10.100452 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9d0a693-f2b8-4036-8f23-a3e2c304eb12-maas-api-xks-ca" pod "e9d0a693-f2b8-4036-8f23-a3e2c304eb12" (UID: "e9d0a693-f2b8-4036-8f23-a3e2c304eb12"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:20:10.103138 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:10.103110 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9d0a693-f2b8-4036-8f23-a3e2c304eb12-kube-api-access-zfp8q" pod "e9d0a693-f2b8-4036-8f23-a3e2c304eb12" (UID: "e9d0a693-f2b8-4036-8f23-a3e2c304eb12"). InnerVolumeSpecName "kube-api-access-zfp8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:20:10.198044 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:10.198006 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfp8q\" (UniqueName: \"kubernetes.io/projected/e9d0a693-f2b8-4036-8f23-a3e2c304eb12-kube-api-access-zfp8q\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:20:10.198044 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:10.198040 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9d0a693-f2b8-4036-8f23-a3e2c304eb12-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:20:10.198257 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:10.198054 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9d0a693-f2b8-4036-8f23-a3e2c304eb12-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:20:10.794963 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:10.794932 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0feca821-pqs6z" Oct 30 17:20:10.794963 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:10.794953 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0feca821-pqs6z" event={"ID":"e9d0a693-f2b8-4036-8f23-a3e2c304eb12","Type":"ContainerDied","Data":"e930ed79bac8fcd8d396fbfde521016a2a3a5f2aa29d8e0440bfbd9a15b9e82f"} Oct 30 17:20:10.795464 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:10.795027 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e930ed79bac8fcd8d396fbfde521016a2a3a5f2aa29d8e0440bfbd9a15b9e82f" Oct 30 17:20:10.993165 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:10.993136 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef03b6bc-rzdkj" Oct 30 17:20:11.011540 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:11.011513 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13d8213f-ae96-45a6-99c3-26d15d56313e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13d8213f-ae96-45a6-99c3-26d15d56313e-maas-api-xks-ca\") pod \"13d8213f-ae96-45a6-99c3-26d15d56313e\" (UID: \"13d8213f-ae96-45a6-99c3-26d15d56313e\") " Oct 30 17:20:11.011763 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:11.011742 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13d8213f-ae96-45a6-99c3-26d15d56313e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13d8213f-ae96-45a6-99c3-26d15d56313e-maas-api-openshift-ca\") pod \"13d8213f-ae96-45a6-99c3-26d15d56313e\" (UID: \"13d8213f-ae96-45a6-99c3-26d15d56313e\") " Oct 30 17:20:11.011807 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:11.011790 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13d8213f-ae96-45a6-99c3-26d15d56313e-kube-api-access-8dkfr\" (UniqueName: \"kubernetes.io/projected/13d8213f-ae96-45a6-99c3-26d15d56313e-kube-api-access-8dkfr\") pod \"13d8213f-ae96-45a6-99c3-26d15d56313e\" (UID: \"13d8213f-ae96-45a6-99c3-26d15d56313e\") " Oct 30 17:20:11.012073 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:11.012045 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13d8213f-ae96-45a6-99c3-26d15d56313e-maas-api-openshift-ca" pod "13d8213f-ae96-45a6-99c3-26d15d56313e" (UID: "13d8213f-ae96-45a6-99c3-26d15d56313e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:20:11.012617 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:11.012595 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13d8213f-ae96-45a6-99c3-26d15d56313e-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:20:11.013703 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:11.013675 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13d8213f-ae96-45a6-99c3-26d15d56313e-maas-api-xks-ca" pod "13d8213f-ae96-45a6-99c3-26d15d56313e" (UID: "13d8213f-ae96-45a6-99c3-26d15d56313e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:20:11.013956 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:11.013932 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13d8213f-ae96-45a6-99c3-26d15d56313e-kube-api-access-8dkfr" pod "13d8213f-ae96-45a6-99c3-26d15d56313e" (UID: "13d8213f-ae96-45a6-99c3-26d15d56313e"). InnerVolumeSpecName "kube-api-access-8dkfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:20:11.114526 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:11.114437 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dkfr\" (UniqueName: \"kubernetes.io/projected/13d8213f-ae96-45a6-99c3-26d15d56313e-kube-api-access-8dkfr\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:20:11.114526 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:11.114467 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13d8213f-ae96-45a6-99c3-26d15d56313e-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:20:11.428872 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:11.428793 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-v527k"] Oct 30 17:20:11.429025 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:11.428991 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-v527k" podUID="8b6f1a27-3c5a-447a-9043-72c9f70fbe70" containerName="istio-proxy" containerID="cri-o://a5862a44e6344ff74cf1e8f204bff6a2ee606520ce08168979bf651011e21cde" gracePeriod=30 Oct 30 17:20:11.805048 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:11.805021 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef03b6bc-rzdkj" Oct 30 17:20:11.805514 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:11.805043 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef03b6bc-rzdkj" event={"ID":"13d8213f-ae96-45a6-99c3-26d15d56313e","Type":"ContainerDied","Data":"d483a74232a8e2fdc43b2356f81f799c01624afdc0504caf659d39553976023f"} Oct 30 17:20:11.805514 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:11.805098 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d483a74232a8e2fdc43b2356f81f799c01624afdc0504caf659d39553976023f" Oct 30 17:20:12.978185 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:20:12.978141 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:20:16.732496 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:16.732471 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-v527k" Oct 30 17:20:16.823942 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:16.823903 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b6f1a27-3c5a-447a-9043-72c9f70fbe70-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b6f1a27-3c5a-447a-9043-72c9f70fbe70-istio-podinfo\") pod \"8b6f1a27-3c5a-447a-9043-72c9f70fbe70\" (UID: \"8b6f1a27-3c5a-447a-9043-72c9f70fbe70\") " Oct 30 17:20:16.824174 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:16.824081 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b6f1a27-3c5a-447a-9043-72c9f70fbe70-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6f1a27-3c5a-447a-9043-72c9f70fbe70-credential-socket\") pod \"8b6f1a27-3c5a-447a-9043-72c9f70fbe70\" (UID: \"8b6f1a27-3c5a-447a-9043-72c9f70fbe70\") " Oct 30 17:20:16.824341 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:16.824232 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b6f1a27-3c5a-447a-9043-72c9f70fbe70-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6f1a27-3c5a-447a-9043-72c9f70fbe70-workload-socket\") pod \"8b6f1a27-3c5a-447a-9043-72c9f70fbe70\" (UID: \"8b6f1a27-3c5a-447a-9043-72c9f70fbe70\") " Oct 30 17:20:16.824341 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:16.824275 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b6f1a27-3c5a-447a-9043-72c9f70fbe70-kube-api-access-nvsq7\" (UniqueName: \"kubernetes.io/projected/8b6f1a27-3c5a-447a-9043-72c9f70fbe70-kube-api-access-nvsq7\") pod \"8b6f1a27-3c5a-447a-9043-72c9f70fbe70\" (UID: \"8b6f1a27-3c5a-447a-9043-72c9f70fbe70\") " Oct 30 17:20:16.824540 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:16.824435 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b6f1a27-3c5a-447a-9043-72c9f70fbe70-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b6f1a27-3c5a-447a-9043-72c9f70fbe70-istio-data\") pod \"8b6f1a27-3c5a-447a-9043-72c9f70fbe70\" (UID: \"8b6f1a27-3c5a-447a-9043-72c9f70fbe70\") " Oct 30 17:20:16.824540 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:16.824485 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b6f1a27-3c5a-447a-9043-72c9f70fbe70-istio-token\" (UniqueName: \"kubernetes.io/projected/8b6f1a27-3c5a-447a-9043-72c9f70fbe70-istio-token\") pod \"8b6f1a27-3c5a-447a-9043-72c9f70fbe70\" (UID: \"8b6f1a27-3c5a-447a-9043-72c9f70fbe70\") " Oct 30 17:20:16.824540 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:16.824513 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b6f1a27-3c5a-447a-9043-72c9f70fbe70-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b6f1a27-3c5a-447a-9043-72c9f70fbe70-istiod-ca-cert\") pod \"8b6f1a27-3c5a-447a-9043-72c9f70fbe70\" (UID: \"8b6f1a27-3c5a-447a-9043-72c9f70fbe70\") " Oct 30 17:20:16.824780 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:16.824635 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b6f1a27-3c5a-447a-9043-72c9f70fbe70-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b6f1a27-3c5a-447a-9043-72c9f70fbe70-workload-certs\") pod \"8b6f1a27-3c5a-447a-9043-72c9f70fbe70\" (UID: \"8b6f1a27-3c5a-447a-9043-72c9f70fbe70\") " Oct 30 17:20:16.824780 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:16.824688 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b6f1a27-3c5a-447a-9043-72c9f70fbe70-credential-socket" pod "8b6f1a27-3c5a-447a-9043-72c9f70fbe70" (UID: "8b6f1a27-3c5a-447a-9043-72c9f70fbe70"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:20:16.824780 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:16.824714 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b6f1a27-3c5a-447a-9043-72c9f70fbe70-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b6f1a27-3c5a-447a-9043-72c9f70fbe70-istio-envoy\") pod \"8b6f1a27-3c5a-447a-9043-72c9f70fbe70\" (UID: \"8b6f1a27-3c5a-447a-9043-72c9f70fbe70\") " Oct 30 17:20:16.825012 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:16.824894 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b6f1a27-3c5a-447a-9043-72c9f70fbe70-istio-data" pod "8b6f1a27-3c5a-447a-9043-72c9f70fbe70" (UID: "8b6f1a27-3c5a-447a-9043-72c9f70fbe70"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:20:16.825092 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:16.825074 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b6f1a27-3c5a-447a-9043-72c9f70fbe70-workload-socket" pod "8b6f1a27-3c5a-447a-9043-72c9f70fbe70" (UID: "8b6f1a27-3c5a-447a-9043-72c9f70fbe70"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:20:16.825695 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:16.825667 4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6f1a27-3c5a-447a-9043-72c9f70fbe70-credential-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:20:16.825835 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:16.825699 4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6f1a27-3c5a-447a-9043-72c9f70fbe70-workload-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:20:16.825835 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:16.825714 4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b6f1a27-3c5a-447a-9043-72c9f70fbe70-istio-data\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:20:16.827178 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:16.827147 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b6f1a27-3c5a-447a-9043-72c9f70fbe70-istiod-ca-cert" pod "8b6f1a27-3c5a-447a-9043-72c9f70fbe70" (UID: "8b6f1a27-3c5a-447a-9043-72c9f70fbe70"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:20:16.827292 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:16.827219 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b6f1a27-3c5a-447a-9043-72c9f70fbe70-workload-certs" pod "8b6f1a27-3c5a-447a-9043-72c9f70fbe70" (UID: "8b6f1a27-3c5a-447a-9043-72c9f70fbe70"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:20:16.828183 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:16.828156 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b6f1a27-3c5a-447a-9043-72c9f70fbe70-kube-api-access-nvsq7" pod "8b6f1a27-3c5a-447a-9043-72c9f70fbe70" (UID: "8b6f1a27-3c5a-447a-9043-72c9f70fbe70"). InnerVolumeSpecName "kube-api-access-nvsq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:20:16.830693 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:16.830666 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b6f1a27-3c5a-447a-9043-72c9f70fbe70-istio-podinfo" pod "8b6f1a27-3c5a-447a-9043-72c9f70fbe70" (UID: "8b6f1a27-3c5a-447a-9043-72c9f70fbe70"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:20:16.831272 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:16.831233 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b6f1a27-3c5a-447a-9043-72c9f70fbe70-istio-token" pod "8b6f1a27-3c5a-447a-9043-72c9f70fbe70" (UID: "8b6f1a27-3c5a-447a-9043-72c9f70fbe70"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:20:16.832709 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:16.832689 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b6f1a27-3c5a-447a-9043-72c9f70fbe70-istio-envoy" pod "8b6f1a27-3c5a-447a-9043-72c9f70fbe70" (UID: "8b6f1a27-3c5a-447a-9043-72c9f70fbe70"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:20:16.851527 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:16.851494 4522 generic.go:356] "Generic (PLEG): container finished" podID="8b6f1a27-3c5a-447a-9043-72c9f70fbe70" containerID="a5862a44e6344ff74cf1e8f204bff6a2ee606520ce08168979bf651011e21cde" exitCode=0 Oct 30 17:20:16.851691 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:16.851539 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-v527k" event={"ID":"8b6f1a27-3c5a-447a-9043-72c9f70fbe70","Type":"ContainerDied","Data":"a5862a44e6344ff74cf1e8f204bff6a2ee606520ce08168979bf651011e21cde"} Oct 30 17:20:16.851691 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:16.851565 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-v527k" Oct 30 17:20:16.851691 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:16.851610 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-v527k" event={"ID":"8b6f1a27-3c5a-447a-9043-72c9f70fbe70","Type":"ContainerDied","Data":"eaaa1e3e0323c282727186ef650ae428c3909c2b50c50cb49a464d38b969e57b"} Oct 30 17:20:16.851691 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:16.851642 4522 scope.go:122] "RemoveContainer" containerID="a5862a44e6344ff74cf1e8f204bff6a2ee606520ce08168979bf651011e21cde" Oct 30 17:20:16.874205 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:16.874132 4522 scope.go:122] "RemoveContainer" containerID="a5862a44e6344ff74cf1e8f204bff6a2ee606520ce08168979bf651011e21cde" Oct 30 17:20:16.874493 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:20:16.874470 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5862a44e6344ff74cf1e8f204bff6a2ee606520ce08168979bf651011e21cde\": container with ID starting with a5862a44e6344ff74cf1e8f204bff6a2ee606520ce08168979bf651011e21cde not found: ID does not exist" containerID="a5862a44e6344ff74cf1e8f204bff6a2ee606520ce08168979bf651011e21cde" Oct 30 17:20:16.874566 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:16.874502 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5862a44e6344ff74cf1e8f204bff6a2ee606520ce08168979bf651011e21cde"} err="failed to get container status \"a5862a44e6344ff74cf1e8f204bff6a2ee606520ce08168979bf651011e21cde\": rpc error: code = NotFound desc = could not find container \"a5862a44e6344ff74cf1e8f204bff6a2ee606520ce08168979bf651011e21cde\": container with ID starting with a5862a44e6344ff74cf1e8f204bff6a2ee606520ce08168979bf651011e21cde not found: ID does not exist" Oct 30 17:20:16.921302 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:16.921270 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-v527k"] Oct 30 17:20:16.927724 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:16.927689 4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b6f1a27-3c5a-447a-9043-72c9f70fbe70-istio-token\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:20:16.927724 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:16.927723 4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b6f1a27-3c5a-447a-9043-72c9f70fbe70-istiod-ca-cert\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:20:16.927967 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:16.927737 4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b6f1a27-3c5a-447a-9043-72c9f70fbe70-workload-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:20:16.927967 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:16.927750 4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b6f1a27-3c5a-447a-9043-72c9f70fbe70-istio-envoy\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:20:16.927967 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:16.927763 4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b6f1a27-3c5a-447a-9043-72c9f70fbe70-istio-podinfo\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:20:16.927967 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:16.927777 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvsq7\" (UniqueName: \"kubernetes.io/projected/8b6f1a27-3c5a-447a-9043-72c9f70fbe70-kube-api-access-nvsq7\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:20:16.929186 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:16.929155 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-v527k"] Oct 30 17:20:18.147817 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:18.147782 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b6f1a27-3c5a-447a-9043-72c9f70fbe70" path="/var/lib/kubelet/pods/8b6f1a27-3c5a-447a-9043-72c9f70fbe70/volumes" Oct 30 17:20:42.399331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:42.396356 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbd06511-gw24m"] Oct 30 17:20:42.399331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:42.397892 4522 cpu_manager.go:414] "containerMap: removing container" podUID="e9d0a693-f2b8-4036-8f23-a3e2c304eb12" containerName="revoke-keys" Oct 30 17:20:42.399331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:42.397915 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9d0a693-f2b8-4036-8f23-a3e2c304eb12" containerName="revoke-keys" Oct 30 17:20:42.399331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:42.397982 4522 cpu_manager.go:414] "containerMap: removing container" podUID="8b6f1a27-3c5a-447a-9043-72c9f70fbe70" containerName="istio-proxy" Oct 30 17:20:42.399331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:42.397990 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b6f1a27-3c5a-447a-9043-72c9f70fbe70" containerName="istio-proxy" Oct 30 17:20:42.399331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:42.398007 4522 cpu_manager.go:414] "containerMap: removing container" podUID="13d8213f-ae96-45a6-99c3-26d15d56313e" containerName="revoke-keys" Oct 30 17:20:42.399331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:42.398016 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13d8213f-ae96-45a6-99c3-26d15d56313e" containerName="revoke-keys" Oct 30 17:20:42.399331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:42.398232 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b6f1a27-3c5a-447a-9043-72c9f70fbe70" containerName="istio-proxy" Oct 30 17:20:42.399331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:42.398242 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="13d8213f-ae96-45a6-99c3-26d15d56313e" containerName="revoke-keys" Oct 30 17:20:42.399331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:42.398261 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9d0a693-f2b8-4036-8f23-a3e2c304eb12" containerName="revoke-keys" Oct 30 17:20:42.640325 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:42.637541 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbd06511-gw24m" Oct 30 17:20:42.654906 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:42.651556 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5whjq\" (UniqueName: \"kubernetes.io/projected/7ec65b68-1b8a-49f0-aca2-3c9fd0fcaa56-kube-api-access-5whjq\") pod \"maas-api-revoke-sub-fbd06511-gw24m\" (UID: \"7ec65b68-1b8a-49f0-aca2-3c9fd0fcaa56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbd06511-gw24m" Oct 30 17:20:42.654906 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:42.651690 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ec65b68-1b8a-49f0-aca2-3c9fd0fcaa56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbd06511-gw24m\" (UID: \"7ec65b68-1b8a-49f0-aca2-3c9fd0fcaa56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbd06511-gw24m" Oct 30 17:20:42.654906 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:42.651824 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ec65b68-1b8a-49f0-aca2-3c9fd0fcaa56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbd06511-gw24m\" (UID: \"7ec65b68-1b8a-49f0-aca2-3c9fd0fcaa56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbd06511-gw24m" Oct 30 17:20:42.659733 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:42.659693 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbd06511-gw24m"] Oct 30 17:20:42.661342 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:42.661299 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:20:42.694961 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:42.694919 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:20:42.760436 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:42.760399 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ec65b68-1b8a-49f0-aca2-3c9fd0fcaa56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbd06511-gw24m\" (UID: \"7ec65b68-1b8a-49f0-aca2-3c9fd0fcaa56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbd06511-gw24m" Oct 30 17:20:42.796729 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:42.796153 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ec65b68-1b8a-49f0-aca2-3c9fd0fcaa56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbd06511-gw24m\" (UID: \"7ec65b68-1b8a-49f0-aca2-3c9fd0fcaa56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbd06511-gw24m" Oct 30 17:20:42.805711 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:42.797110 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ec65b68-1b8a-49f0-aca2-3c9fd0fcaa56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbd06511-gw24m\" (UID: \"7ec65b68-1b8a-49f0-aca2-3c9fd0fcaa56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbd06511-gw24m" Oct 30 17:20:42.805711 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:42.803569 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5whjq\" (UniqueName: \"kubernetes.io/projected/7ec65b68-1b8a-49f0-aca2-3c9fd0fcaa56-kube-api-access-5whjq\") pod \"maas-api-revoke-sub-fbd06511-gw24m\" (UID: \"7ec65b68-1b8a-49f0-aca2-3c9fd0fcaa56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbd06511-gw24m" Oct 30 17:20:42.805711 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:42.799144 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ec65b68-1b8a-49f0-aca2-3c9fd0fcaa56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbd06511-gw24m\" (UID: \"7ec65b68-1b8a-49f0-aca2-3c9fd0fcaa56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbd06511-gw24m" Oct 30 17:20:42.894338 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:42.877092 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5whjq\" (UniqueName: \"kubernetes.io/projected/7ec65b68-1b8a-49f0-aca2-3c9fd0fcaa56-kube-api-access-5whjq\") pod \"maas-api-revoke-sub-fbd06511-gw24m\" (UID: \"7ec65b68-1b8a-49f0-aca2-3c9fd0fcaa56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbd06511-gw24m" Oct 30 17:20:42.898071 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:42.898037 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bf44438-kpfmd"] Oct 30 17:20:43.014280 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:43.014235 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bf44438-kpfmd"] Oct 30 17:20:43.014507 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:43.014451 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf44438-kpfmd" Oct 30 17:20:43.191350 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:20:43.179933 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:20:43.354480 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:43.207286 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbd06511-gw24m" Oct 30 17:20:43.366336 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:43.222227 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6196f49-61cf-45eb-bf47-21621604e662-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bf44438-kpfmd\" (UID: \"e6196f49-61cf-45eb-bf47-21621604e662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf44438-kpfmd" Oct 30 17:20:43.366336 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:43.362627 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6196f49-61cf-45eb-bf47-21621604e662-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bf44438-kpfmd\" (UID: \"e6196f49-61cf-45eb-bf47-21621604e662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf44438-kpfmd" Oct 30 17:20:43.467282 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:43.465726 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6196f49-61cf-45eb-bf47-21621604e662-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bf44438-kpfmd\" (UID: \"e6196f49-61cf-45eb-bf47-21621604e662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf44438-kpfmd" Oct 30 17:20:43.467282 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:43.465877 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ctmr\" (UniqueName: \"kubernetes.io/projected/e6196f49-61cf-45eb-bf47-21621604e662-kube-api-access-6ctmr\") pod \"maas-api-revoke-sub-0bf44438-kpfmd\" (UID: \"e6196f49-61cf-45eb-bf47-21621604e662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf44438-kpfmd" Oct 30 17:20:43.467282 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:43.466070 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6196f49-61cf-45eb-bf47-21621604e662-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bf44438-kpfmd\" (UID: \"e6196f49-61cf-45eb-bf47-21621604e662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf44438-kpfmd" Oct 30 17:20:43.482649 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:43.482039 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6196f49-61cf-45eb-bf47-21621604e662-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bf44438-kpfmd\" (UID: \"e6196f49-61cf-45eb-bf47-21621604e662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf44438-kpfmd" Oct 30 17:20:43.515335 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:43.505784 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6196f49-61cf-45eb-bf47-21621604e662-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bf44438-kpfmd\" (UID: \"e6196f49-61cf-45eb-bf47-21621604e662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf44438-kpfmd" Oct 30 17:20:43.573326 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:43.569650 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ctmr\" (UniqueName: \"kubernetes.io/projected/e6196f49-61cf-45eb-bf47-21621604e662-kube-api-access-6ctmr\") pod \"maas-api-revoke-sub-0bf44438-kpfmd\" (UID: \"e6196f49-61cf-45eb-bf47-21621604e662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf44438-kpfmd" Oct 30 17:20:43.646137 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:43.646020 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ctmr\" (UniqueName: \"kubernetes.io/projected/e6196f49-61cf-45eb-bf47-21621604e662-kube-api-access-6ctmr\") pod \"maas-api-revoke-sub-0bf44438-kpfmd\" (UID: \"e6196f49-61cf-45eb-bf47-21621604e662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf44438-kpfmd" Oct 30 17:20:43.757332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:43.745680 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf44438-kpfmd" Oct 30 17:20:44.321349 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:44.305677 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" podUID="75321c5d-8bf8-45c6-8811-81981fe3cad4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 17:20:44.381068 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:44.362462 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq" podUID="f18303fb-6d27-4ff1-9cc9-ed032e43d098" containerName="main" probeResult="failure" output="Get \"https://10.134.0.120:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:20:44.381068 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:44.363229 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" podUID="0b832781-3d41-4496-aa86-e4d92792e590" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:20:44.391576 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:44.391531 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" podUID="2f1e3bbb-ab1c-455d-a05d-d62f72c5d009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:20:44.724074 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:44.724032 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbd06511-gw24m"] Oct 30 17:20:44.827437 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:20:44.827362 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6196f49_61cf_45eb_bf47_21621604e662.slice/crio-1c797af022c8fea95f8bada76fe20c4785277a50ef06d2f94ef6b4a7a93a5b9f WatchSource:0}: Error finding container 1c797af022c8fea95f8bada76fe20c4785277a50ef06d2f94ef6b4a7a93a5b9f: Status 404 returned error can't find the container with id 1c797af022c8fea95f8bada76fe20c4785277a50ef06d2f94ef6b4a7a93a5b9f Oct 30 17:20:44.829255 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:44.829210 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bf44438-kpfmd"] Oct 30 17:20:45.561534 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:45.561492 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbd06511-gw24m" event={"ID":"7ec65b68-1b8a-49f0-aca2-3c9fd0fcaa56","Type":"ContainerStarted","Data":"e33fa64c57006dc50543fc442c4d19f5e7c3e3124b65c0b28eccc70ce507355d"} Oct 30 17:20:45.561753 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:45.561564 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbd06511-gw24m" event={"ID":"7ec65b68-1b8a-49f0-aca2-3c9fd0fcaa56","Type":"ContainerStarted","Data":"eb7955b85834fb0d75152f8fbcc8abafdbc3255f4e37d4ac1d7b36f751b25e85"} Oct 30 17:20:45.563511 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:45.563478 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf44438-kpfmd" event={"ID":"e6196f49-61cf-45eb-bf47-21621604e662","Type":"ContainerStarted","Data":"efa661720c8b048df143196128e076e3d0f4b94c92ff966bfd6932ad5cb94d63"} Oct 30 17:20:45.563621 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:45.563539 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf44438-kpfmd" event={"ID":"e6196f49-61cf-45eb-bf47-21621604e662","Type":"ContainerStarted","Data":"1c797af022c8fea95f8bada76fe20c4785277a50ef06d2f94ef6b4a7a93a5b9f"} Oct 30 17:20:45.607466 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:45.607412 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbd06511-gw24m" podStartSLOduration=3.6073971 podStartE2EDuration="3.6073971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:20:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:20:45.606657615 +0000 UTC m=+2019.607412516" watchObservedRunningTime="2026-10-30 17:20:45.6073971 +0000 UTC m=+2019.608152000" Oct 30 17:20:45.653554 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:45.653506 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf44438-kpfmd" podStartSLOduration=3.6534917399999998 podStartE2EDuration="3.65349174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:20:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:20:45.650188283 +0000 UTC m=+2019.650943195" watchObservedRunningTime="2026-10-30 17:20:45.65349174 +0000 UTC m=+2019.654246640" Oct 30 17:20:46.573938 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:46.573902 4522 generic.go:356] "Generic (PLEG): container finished" podID="7ec65b68-1b8a-49f0-aca2-3c9fd0fcaa56" containerID="e33fa64c57006dc50543fc442c4d19f5e7c3e3124b65c0b28eccc70ce507355d" exitCode=0 Oct 30 17:20:46.574495 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:46.573995 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbd06511-gw24m" event={"ID":"7ec65b68-1b8a-49f0-aca2-3c9fd0fcaa56","Type":"ContainerDied","Data":"e33fa64c57006dc50543fc442c4d19f5e7c3e3124b65c0b28eccc70ce507355d"} Oct 30 17:20:46.576149 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:46.576122 4522 generic.go:356] "Generic (PLEG): container finished" podID="e6196f49-61cf-45eb-bf47-21621604e662" containerID="efa661720c8b048df143196128e076e3d0f4b94c92ff966bfd6932ad5cb94d63" exitCode=0 Oct 30 17:20:46.576278 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:46.576189 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf44438-kpfmd" event={"ID":"e6196f49-61cf-45eb-bf47-21621604e662","Type":"ContainerDied","Data":"efa661720c8b048df143196128e076e3d0f4b94c92ff966bfd6932ad5cb94d63"} Oct 30 17:20:47.807037 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:47.807007 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbd06511-gw24m" Oct 30 17:20:47.810859 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:47.810836 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf44438-kpfmd" Oct 30 17:20:47.874476 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:47.874403 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6196f49-61cf-45eb-bf47-21621604e662-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6196f49-61cf-45eb-bf47-21621604e662-maas-api-xks-ca\") pod \"e6196f49-61cf-45eb-bf47-21621604e662\" (UID: \"e6196f49-61cf-45eb-bf47-21621604e662\") " Oct 30 17:20:47.874624 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:47.874515 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6196f49-61cf-45eb-bf47-21621604e662-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6196f49-61cf-45eb-bf47-21621604e662-maas-api-openshift-ca\") pod \"e6196f49-61cf-45eb-bf47-21621604e662\" (UID: \"e6196f49-61cf-45eb-bf47-21621604e662\") " Oct 30 17:20:47.874624 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:47.874590 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ec65b68-1b8a-49f0-aca2-3c9fd0fcaa56-kube-api-access-5whjq\" (UniqueName: \"kubernetes.io/projected/7ec65b68-1b8a-49f0-aca2-3c9fd0fcaa56-kube-api-access-5whjq\") pod \"7ec65b68-1b8a-49f0-aca2-3c9fd0fcaa56\" (UID: \"7ec65b68-1b8a-49f0-aca2-3c9fd0fcaa56\") " Oct 30 17:20:47.874712 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:47.874643 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ec65b68-1b8a-49f0-aca2-3c9fd0fcaa56-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ec65b68-1b8a-49f0-aca2-3c9fd0fcaa56-maas-api-xks-ca\") pod \"7ec65b68-1b8a-49f0-aca2-3c9fd0fcaa56\" (UID: \"7ec65b68-1b8a-49f0-aca2-3c9fd0fcaa56\") " Oct 30 17:20:47.874763 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:47.874744 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ec65b68-1b8a-49f0-aca2-3c9fd0fcaa56-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ec65b68-1b8a-49f0-aca2-3c9fd0fcaa56-maas-api-openshift-ca\") pod \"7ec65b68-1b8a-49f0-aca2-3c9fd0fcaa56\" (UID: \"7ec65b68-1b8a-49f0-aca2-3c9fd0fcaa56\") " Oct 30 17:20:47.874846 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:47.874828 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6196f49-61cf-45eb-bf47-21621604e662-kube-api-access-6ctmr\" (UniqueName: \"kubernetes.io/projected/e6196f49-61cf-45eb-bf47-21621604e662-kube-api-access-6ctmr\") pod \"e6196f49-61cf-45eb-bf47-21621604e662\" (UID: \"e6196f49-61cf-45eb-bf47-21621604e662\") " Oct 30 17:20:47.874920 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:47.874860 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6196f49-61cf-45eb-bf47-21621604e662-maas-api-openshift-ca" pod "e6196f49-61cf-45eb-bf47-21621604e662" (UID: "e6196f49-61cf-45eb-bf47-21621604e662"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:20:47.875569 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:47.875542 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ec65b68-1b8a-49f0-aca2-3c9fd0fcaa56-maas-api-openshift-ca" pod "7ec65b68-1b8a-49f0-aca2-3c9fd0fcaa56" (UID: "7ec65b68-1b8a-49f0-aca2-3c9fd0fcaa56"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:20:47.875708 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:47.875622 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6196f49-61cf-45eb-bf47-21621604e662-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:20:47.877008 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:47.876981 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ec65b68-1b8a-49f0-aca2-3c9fd0fcaa56-maas-api-xks-ca" pod "7ec65b68-1b8a-49f0-aca2-3c9fd0fcaa56" (UID: "7ec65b68-1b8a-49f0-aca2-3c9fd0fcaa56"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:20:47.877526 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:47.877496 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ec65b68-1b8a-49f0-aca2-3c9fd0fcaa56-kube-api-access-5whjq" pod "7ec65b68-1b8a-49f0-aca2-3c9fd0fcaa56" (UID: "7ec65b68-1b8a-49f0-aca2-3c9fd0fcaa56"). InnerVolumeSpecName "kube-api-access-5whjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:20:47.877641 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:47.877538 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6196f49-61cf-45eb-bf47-21621604e662-kube-api-access-6ctmr" pod "e6196f49-61cf-45eb-bf47-21621604e662" (UID: "e6196f49-61cf-45eb-bf47-21621604e662"). InnerVolumeSpecName "kube-api-access-6ctmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:20:47.877641 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:47.877619 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6196f49-61cf-45eb-bf47-21621604e662-maas-api-xks-ca" pod "e6196f49-61cf-45eb-bf47-21621604e662" (UID: "e6196f49-61cf-45eb-bf47-21621604e662"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:20:47.987964 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:47.984578 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6196f49-61cf-45eb-bf47-21621604e662-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:20:47.987964 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:47.984622 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5whjq\" (UniqueName: \"kubernetes.io/projected/7ec65b68-1b8a-49f0-aca2-3c9fd0fcaa56-kube-api-access-5whjq\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:20:47.987964 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:47.984649 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ec65b68-1b8a-49f0-aca2-3c9fd0fcaa56-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:20:47.987964 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:47.984662 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ec65b68-1b8a-49f0-aca2-3c9fd0fcaa56-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:20:47.987964 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:47.984675 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ctmr\" (UniqueName: \"kubernetes.io/projected/e6196f49-61cf-45eb-bf47-21621604e662-kube-api-access-6ctmr\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:20:48.595018 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:48.594984 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbd06511-gw24m" Oct 30 17:20:48.595018 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:48.595012 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbd06511-gw24m" event={"ID":"7ec65b68-1b8a-49f0-aca2-3c9fd0fcaa56","Type":"ContainerDied","Data":"eb7955b85834fb0d75152f8fbcc8abafdbc3255f4e37d4ac1d7b36f751b25e85"} Oct 30 17:20:48.595269 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:48.595102 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb7955b85834fb0d75152f8fbcc8abafdbc3255f4e37d4ac1d7b36f751b25e85" Oct 30 17:20:48.596537 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:48.596518 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf44438-kpfmd" Oct 30 17:20:48.596681 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:48.596568 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf44438-kpfmd" event={"ID":"e6196f49-61cf-45eb-bf47-21621604e662","Type":"ContainerDied","Data":"1c797af022c8fea95f8bada76fe20c4785277a50ef06d2f94ef6b4a7a93a5b9f"} Oct 30 17:20:48.596681 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:48.596605 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c797af022c8fea95f8bada76fe20c4785277a50ef06d2f94ef6b4a7a93a5b9f" Oct 30 17:20:58.031869 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:58.031832 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f42baf-8w7lj"] Oct 30 17:20:58.032743 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:58.032727 4522 cpu_manager.go:414] "containerMap: removing container" podUID="7ec65b68-1b8a-49f0-aca2-3c9fd0fcaa56" containerName="revoke-keys" Oct 30 17:20:58.032803 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:58.032745 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ec65b68-1b8a-49f0-aca2-3c9fd0fcaa56" containerName="revoke-keys" Oct 30 17:20:58.032803 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:58.032778 4522 cpu_manager.go:414] "containerMap: removing container" podUID="e6196f49-61cf-45eb-bf47-21621604e662" containerName="revoke-keys" Oct 30 17:20:58.032803 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:58.032783 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6196f49-61cf-45eb-bf47-21621604e662" containerName="revoke-keys" Oct 30 17:20:58.032944 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:58.032934 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ec65b68-1b8a-49f0-aca2-3c9fd0fcaa56" containerName="revoke-keys" Oct 30 17:20:58.032986 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:58.032952 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6196f49-61cf-45eb-bf47-21621604e662" containerName="revoke-keys" Oct 30 17:20:58.038346 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:58.038327 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f42baf-8w7lj" Oct 30 17:20:58.056820 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:58.056793 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f42baf-8w7lj"] Oct 30 17:20:58.064186 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:58.064159 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:20:58.064394 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:58.064283 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:20:58.094031 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:58.093998 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31840135-fd6a-4f90-a610-c4ff17f0601a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f42baf-8w7lj\" (UID: \"31840135-fd6a-4f90-a610-c4ff17f0601a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f42baf-8w7lj" Oct 30 17:20:58.094217 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:58.094175 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnnql\" (UniqueName: \"kubernetes.io/projected/31840135-fd6a-4f90-a610-c4ff17f0601a-kube-api-access-lnnql\") pod \"maas-api-revoke-keys-e2e-inf-a-f42baf-8w7lj\" (UID: \"31840135-fd6a-4f90-a610-c4ff17f0601a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f42baf-8w7lj" Oct 30 17:20:58.094430 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:58.094413 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31840135-fd6a-4f90-a610-c4ff17f0601a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f42baf-8w7lj\" (UID: \"31840135-fd6a-4f90-a610-c4ff17f0601a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f42baf-8w7lj" Oct 30 17:20:58.196010 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:58.195965 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31840135-fd6a-4f90-a610-c4ff17f0601a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f42baf-8w7lj\" (UID: \"31840135-fd6a-4f90-a610-c4ff17f0601a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f42baf-8w7lj" Oct 30 17:20:58.196430 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:58.196396 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31840135-fd6a-4f90-a610-c4ff17f0601a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f42baf-8w7lj\" (UID: \"31840135-fd6a-4f90-a610-c4ff17f0601a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f42baf-8w7lj" Oct 30 17:20:58.196637 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:58.196615 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnnql\" (UniqueName: \"kubernetes.io/projected/31840135-fd6a-4f90-a610-c4ff17f0601a-kube-api-access-lnnql\") pod \"maas-api-revoke-keys-e2e-inf-a-f42baf-8w7lj\" (UID: \"31840135-fd6a-4f90-a610-c4ff17f0601a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f42baf-8w7lj" Oct 30 17:20:58.196970 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:58.196947 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31840135-fd6a-4f90-a610-c4ff17f0601a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f42baf-8w7lj\" (UID: \"31840135-fd6a-4f90-a610-c4ff17f0601a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f42baf-8w7lj" Oct 30 17:20:58.198585 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:58.198565 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31840135-fd6a-4f90-a610-c4ff17f0601a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f42baf-8w7lj\" (UID: \"31840135-fd6a-4f90-a610-c4ff17f0601a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f42baf-8w7lj" Oct 30 17:20:58.223986 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:58.223955 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnnql\" (UniqueName: \"kubernetes.io/projected/31840135-fd6a-4f90-a610-c4ff17f0601a-kube-api-access-lnnql\") pod \"maas-api-revoke-keys-e2e-inf-a-f42baf-8w7lj\" (UID: \"31840135-fd6a-4f90-a610-c4ff17f0601a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f42baf-8w7lj" Oct 30 17:20:58.354000 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:58.353916 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f42baf-8w7lj" Oct 30 17:20:58.672425 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:20:58.672393 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31840135_fd6a_4f90_a610_c4ff17f0601a.slice/crio-6a0e2d3e59e1b0627e81aa4d9c3db0623cbdef7fdaab5e9a5b986d3044768c61 WatchSource:0}: Error finding container 6a0e2d3e59e1b0627e81aa4d9c3db0623cbdef7fdaab5e9a5b986d3044768c61: Status 404 returned error can't find the container with id 6a0e2d3e59e1b0627e81aa4d9c3db0623cbdef7fdaab5e9a5b986d3044768c61 Oct 30 17:20:58.674494 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:58.674435 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f42baf-8w7lj"] Oct 30 17:20:58.704713 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:58.704681 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f42baf-8w7lj" event={"ID":"31840135-fd6a-4f90-a610-c4ff17f0601a","Type":"ContainerStarted","Data":"6a0e2d3e59e1b0627e81aa4d9c3db0623cbdef7fdaab5e9a5b986d3044768c61"} Oct 30 17:20:59.724372 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:59.724225 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f42baf-8w7lj" event={"ID":"31840135-fd6a-4f90-a610-c4ff17f0601a","Type":"ContainerStarted","Data":"5199b0bb903fd1ced149feb230811eaaa4d92c725f7c96c5899f710ebc50408e"} Oct 30 17:20:59.777609 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:20:59.777557 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f42baf-8w7lj" podStartSLOduration=1.7775424690000001 podStartE2EDuration="1.777542469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:20:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:20:59.773131275 +0000 UTC m=+2033.773886196" watchObservedRunningTime="2026-10-30 17:20:59.777542469 +0000 UTC m=+2033.778297371" Oct 30 17:21:00.734892 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:00.734862 4522 generic.go:356] "Generic (PLEG): container finished" podID="31840135-fd6a-4f90-a610-c4ff17f0601a" containerID="5199b0bb903fd1ced149feb230811eaaa4d92c725f7c96c5899f710ebc50408e" exitCode=0 Oct 30 17:21:00.735336 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:00.734948 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f42baf-8w7lj" event={"ID":"31840135-fd6a-4f90-a610-c4ff17f0601a","Type":"ContainerDied","Data":"5199b0bb903fd1ced149feb230811eaaa4d92c725f7c96c5899f710ebc50408e"} Oct 30 17:21:01.932679 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:01.932649 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f42baf-8w7lj" Oct 30 17:21:02.082808 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:02.082678 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31840135-fd6a-4f90-a610-c4ff17f0601a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31840135-fd6a-4f90-a610-c4ff17f0601a-maas-api-openshift-ca\") pod \"31840135-fd6a-4f90-a610-c4ff17f0601a\" (UID: \"31840135-fd6a-4f90-a610-c4ff17f0601a\") " Oct 30 17:21:02.082808 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:02.082752 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31840135-fd6a-4f90-a610-c4ff17f0601a-kube-api-access-lnnql\" (UniqueName: \"kubernetes.io/projected/31840135-fd6a-4f90-a610-c4ff17f0601a-kube-api-access-lnnql\") pod \"31840135-fd6a-4f90-a610-c4ff17f0601a\" (UID: \"31840135-fd6a-4f90-a610-c4ff17f0601a\") " Oct 30 17:21:02.083074 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:02.082972 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31840135-fd6a-4f90-a610-c4ff17f0601a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31840135-fd6a-4f90-a610-c4ff17f0601a-maas-api-xks-ca\") pod \"31840135-fd6a-4f90-a610-c4ff17f0601a\" (UID: \"31840135-fd6a-4f90-a610-c4ff17f0601a\") " Oct 30 17:21:02.083074 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:02.083058 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31840135-fd6a-4f90-a610-c4ff17f0601a-maas-api-openshift-ca" pod "31840135-fd6a-4f90-a610-c4ff17f0601a" (UID: "31840135-fd6a-4f90-a610-c4ff17f0601a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:21:02.083702 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:02.083680 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31840135-fd6a-4f90-a610-c4ff17f0601a-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:21:02.085041 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:02.085012 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31840135-fd6a-4f90-a610-c4ff17f0601a-kube-api-access-lnnql" pod "31840135-fd6a-4f90-a610-c4ff17f0601a" (UID: "31840135-fd6a-4f90-a610-c4ff17f0601a"). InnerVolumeSpecName "kube-api-access-lnnql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:21:02.085145 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:02.085069 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31840135-fd6a-4f90-a610-c4ff17f0601a-maas-api-xks-ca" pod "31840135-fd6a-4f90-a610-c4ff17f0601a" (UID: "31840135-fd6a-4f90-a610-c4ff17f0601a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:21:02.185899 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:02.185860 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnnql\" (UniqueName: \"kubernetes.io/projected/31840135-fd6a-4f90-a610-c4ff17f0601a-kube-api-access-lnnql\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:21:02.185899 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:02.185892 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31840135-fd6a-4f90-a610-c4ff17f0601a-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:21:02.754134 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:02.754093 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f42baf-8w7lj" event={"ID":"31840135-fd6a-4f90-a610-c4ff17f0601a","Type":"ContainerDied","Data":"6a0e2d3e59e1b0627e81aa4d9c3db0623cbdef7fdaab5e9a5b986d3044768c61"} Oct 30 17:21:02.754384 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:02.754155 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a0e2d3e59e1b0627e81aa4d9c3db0623cbdef7fdaab5e9a5b986d3044768c61" Oct 30 17:21:02.754384 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:02.754111 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f42baf-8w7lj" Oct 30 17:21:08.158740 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:08.158696 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f42baf-58fcd7df8-kshhs"] Oct 30 17:21:08.194099 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:08.194062 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f42baf-6c555f88f9-k8szt"] Oct 30 17:21:08.809505 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:08.809472 4522 scope.go:122] "RemoveContainer" containerID="4cb06a02a69d533e98a56f4837a9754e20888bc0b185861ffa135936a0d6cffe" Oct 30 17:21:08.821196 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:08.821173 4522 scope.go:122] "RemoveContainer" containerID="66e3281cce76250b6de5c4fc4dd447e13e29d387dbda945fc8a43ee9df4209c5" Oct 30 17:21:08.839800 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:08.839764 4522 scope.go:122] "RemoveContainer" containerID="60437ae41bb5026b169ea16b93008db4ac72a1ce6e6dbc9db561917e40c5b72b" Oct 30 17:21:09.010573 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:09.003512 4522 scope.go:122] "RemoveContainer" containerID="fb8d4d3acf269d2b84411b262a8ec1655ae4632c28030b5cdc397f4bc75cff6a" Oct 30 17:21:10.710647 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:10.710607 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq"] Oct 30 17:21:10.711178 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:10.710927 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq" podUID="77c327fe-cc04-492f-bc0e-efb01a43a67f" containerName="istio-proxy" containerID="cri-o://5e7604ac9a3c4cbddd2348117d5f83dcef0bdf184cd82f06067a46d4d08d0fa2" gracePeriod=30 Oct 30 17:21:11.922633 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:11.922319 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d003a5-n64mh"] Oct 30 17:21:11.926937 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:11.926896 4522 cpu_manager.go:414] "containerMap: removing container" podUID="31840135-fd6a-4f90-a610-c4ff17f0601a" containerName="revoke-keys" Oct 30 17:21:11.927256 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:11.927225 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31840135-fd6a-4f90-a610-c4ff17f0601a" containerName="revoke-keys" Oct 30 17:21:11.927799 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:11.927766 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="31840135-fd6a-4f90-a610-c4ff17f0601a" containerName="revoke-keys" Oct 30 17:21:11.934921 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:11.934849 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d003a5-n64mh"] Oct 30 17:21:11.935090 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:11.934966 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d003a5-n64mh" Oct 30 17:21:11.953647 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:11.953619 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:21:11.965250 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:11.965217 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:21:12.002648 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:12.002610 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/949ded2a-41f3-4046-b83f-e2d7b3d13334-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d003a5-n64mh\" (UID: \"949ded2a-41f3-4046-b83f-e2d7b3d13334\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d003a5-n64mh" Oct 30 17:21:12.002841 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:12.002668 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/949ded2a-41f3-4046-b83f-e2d7b3d13334-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d003a5-n64mh\" (UID: \"949ded2a-41f3-4046-b83f-e2d7b3d13334\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d003a5-n64mh" Oct 30 17:21:12.002841 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:12.002813 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzs74\" (UniqueName: \"kubernetes.io/projected/949ded2a-41f3-4046-b83f-e2d7b3d13334-kube-api-access-bzs74\") pod \"maas-api-revoke-keys-e2e-inf-a-d003a5-n64mh\" (UID: \"949ded2a-41f3-4046-b83f-e2d7b3d13334\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d003a5-n64mh" Oct 30 17:21:12.104222 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:12.104184 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/949ded2a-41f3-4046-b83f-e2d7b3d13334-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d003a5-n64mh\" (UID: \"949ded2a-41f3-4046-b83f-e2d7b3d13334\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d003a5-n64mh" Oct 30 17:21:12.104460 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:12.104237 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/949ded2a-41f3-4046-b83f-e2d7b3d13334-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d003a5-n64mh\" (UID: \"949ded2a-41f3-4046-b83f-e2d7b3d13334\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d003a5-n64mh" Oct 30 17:21:12.104460 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:12.104363 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzs74\" (UniqueName: \"kubernetes.io/projected/949ded2a-41f3-4046-b83f-e2d7b3d13334-kube-api-access-bzs74\") pod \"maas-api-revoke-keys-e2e-inf-a-d003a5-n64mh\" (UID: \"949ded2a-41f3-4046-b83f-e2d7b3d13334\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d003a5-n64mh" Oct 30 17:21:12.105006 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:12.104987 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/949ded2a-41f3-4046-b83f-e2d7b3d13334-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d003a5-n64mh\" (UID: \"949ded2a-41f3-4046-b83f-e2d7b3d13334\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d003a5-n64mh" Oct 30 17:21:12.106749 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:12.106728 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/949ded2a-41f3-4046-b83f-e2d7b3d13334-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d003a5-n64mh\" (UID: \"949ded2a-41f3-4046-b83f-e2d7b3d13334\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d003a5-n64mh" Oct 30 17:21:12.129131 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:12.129105 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzs74\" (UniqueName: \"kubernetes.io/projected/949ded2a-41f3-4046-b83f-e2d7b3d13334-kube-api-access-bzs74\") pod \"maas-api-revoke-keys-e2e-inf-a-d003a5-n64mh\" (UID: \"949ded2a-41f3-4046-b83f-e2d7b3d13334\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d003a5-n64mh" Oct 30 17:21:12.256103 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:12.256070 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d003a5-n64mh" Oct 30 17:21:12.573999 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:21:12.573972 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod949ded2a_41f3_4046_b83f_e2d7b3d13334.slice/crio-e9feb5992766e249478e86dc8118a91b1e78283a94efd834cef30fa889901c79 WatchSource:0}: Error finding container e9feb5992766e249478e86dc8118a91b1e78283a94efd834cef30fa889901c79: Status 404 returned error can't find the container with id e9feb5992766e249478e86dc8118a91b1e78283a94efd834cef30fa889901c79 Oct 30 17:21:12.576099 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:12.576073 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d003a5-n64mh"] Oct 30 17:21:12.903608 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:12.903521 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d003a5-n64mh" event={"ID":"949ded2a-41f3-4046-b83f-e2d7b3d13334","Type":"ContainerStarted","Data":"e9feb5992766e249478e86dc8118a91b1e78283a94efd834cef30fa889901c79"} Oct 30 17:21:12.964459 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:21:12.964420 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:21:13.159032 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:13.158933 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-f42baf-58fcd7df8-kshhs" podUID="e9629e12-7b8a-4e59-acb9-8e7476994826" containerName="payload-processing" containerID="cri-o://aebbd3d5e02197db7c1da22c7e704f05a5d794038600d9d84a395bcbb5b1e83f" gracePeriod=25 Oct 30 17:21:13.194449 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:13.194403 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f42baf-6c555f88f9-k8szt" podUID="46c462ae-1044-4037-939b-43ad0906630d" containerName="payload-pre-processing" containerID="cri-o://708c7b66d9f4c6246fcc1c976856835d6200f2ab9683119c0813ff60c3b2348a" gracePeriod=25 Oct 30 17:21:13.916287 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:13.916248 4522 generic.go:356] "Generic (PLEG): container finished" podID="949ded2a-41f3-4046-b83f-e2d7b3d13334" containerID="f54fb7d618e31f7718806d99270edc505f1aabe25e80009f1ae698eaef73f815" exitCode=0 Oct 30 17:21:13.916627 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:13.916286 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d003a5-n64mh" event={"ID":"949ded2a-41f3-4046-b83f-e2d7b3d13334","Type":"ContainerDied","Data":"f54fb7d618e31f7718806d99270edc505f1aabe25e80009f1ae698eaef73f815"} Oct 30 17:21:15.099204 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:15.099176 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d003a5-n64mh" Oct 30 17:21:15.149630 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:15.149593 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/949ded2a-41f3-4046-b83f-e2d7b3d13334-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/949ded2a-41f3-4046-b83f-e2d7b3d13334-maas-api-openshift-ca\") pod \"949ded2a-41f3-4046-b83f-e2d7b3d13334\" (UID: \"949ded2a-41f3-4046-b83f-e2d7b3d13334\") " Oct 30 17:21:15.149819 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:15.149783 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/949ded2a-41f3-4046-b83f-e2d7b3d13334-kube-api-access-bzs74\" (UniqueName: \"kubernetes.io/projected/949ded2a-41f3-4046-b83f-e2d7b3d13334-kube-api-access-bzs74\") pod \"949ded2a-41f3-4046-b83f-e2d7b3d13334\" (UID: \"949ded2a-41f3-4046-b83f-e2d7b3d13334\") " Oct 30 17:21:15.149907 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:15.149868 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/949ded2a-41f3-4046-b83f-e2d7b3d13334-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/949ded2a-41f3-4046-b83f-e2d7b3d13334-maas-api-xks-ca\") pod \"949ded2a-41f3-4046-b83f-e2d7b3d13334\" (UID: \"949ded2a-41f3-4046-b83f-e2d7b3d13334\") " Oct 30 17:21:15.150002 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:15.149975 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/949ded2a-41f3-4046-b83f-e2d7b3d13334-maas-api-openshift-ca" pod "949ded2a-41f3-4046-b83f-e2d7b3d13334" (UID: "949ded2a-41f3-4046-b83f-e2d7b3d13334"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:21:15.150846 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:15.150818 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/949ded2a-41f3-4046-b83f-e2d7b3d13334-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:21:15.152469 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:15.152437 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/949ded2a-41f3-4046-b83f-e2d7b3d13334-maas-api-xks-ca" pod "949ded2a-41f3-4046-b83f-e2d7b3d13334" (UID: "949ded2a-41f3-4046-b83f-e2d7b3d13334"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:21:15.152772 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:15.152739 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/949ded2a-41f3-4046-b83f-e2d7b3d13334-kube-api-access-bzs74" pod "949ded2a-41f3-4046-b83f-e2d7b3d13334" (UID: "949ded2a-41f3-4046-b83f-e2d7b3d13334"). InnerVolumeSpecName "kube-api-access-bzs74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:21:15.252925 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:15.252889 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/949ded2a-41f3-4046-b83f-e2d7b3d13334-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:21:15.252925 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:15.252919 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzs74\" (UniqueName: \"kubernetes.io/projected/949ded2a-41f3-4046-b83f-e2d7b3d13334-kube-api-access-bzs74\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:21:15.584939 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:15.584858 4522 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:21:15.585087 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:15.584935 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq" podUID="77c327fe-cc04-492f-bc0e-efb01a43a67f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:21:15.940815 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:15.940780 4522 generic.go:356] "Generic (PLEG): container finished" podID="77c327fe-cc04-492f-bc0e-efb01a43a67f" containerID="5e7604ac9a3c4cbddd2348117d5f83dcef0bdf184cd82f06067a46d4d08d0fa2" exitCode=0 Oct 30 17:21:15.940998 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:15.940865 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq" event={"ID":"77c327fe-cc04-492f-bc0e-efb01a43a67f","Type":"ContainerDied","Data":"5e7604ac9a3c4cbddd2348117d5f83dcef0bdf184cd82f06067a46d4d08d0fa2"} Oct 30 17:21:15.942764 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:15.942737 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d003a5-n64mh" Oct 30 17:21:15.942902 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:15.942745 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d003a5-n64mh" event={"ID":"949ded2a-41f3-4046-b83f-e2d7b3d13334","Type":"ContainerDied","Data":"e9feb5992766e249478e86dc8118a91b1e78283a94efd834cef30fa889901c79"} Oct 30 17:21:15.942902 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:15.942884 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9feb5992766e249478e86dc8118a91b1e78283a94efd834cef30fa889901c79" Oct 30 17:21:15.990959 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:15.990930 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq" Oct 30 17:21:16.074648 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:16.074618 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77c327fe-cc04-492f-bc0e-efb01a43a67f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77c327fe-cc04-492f-bc0e-efb01a43a67f-credential-socket\") pod \"77c327fe-cc04-492f-bc0e-efb01a43a67f\" (UID: \"77c327fe-cc04-492f-bc0e-efb01a43a67f\") " Oct 30 17:21:16.075214 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:16.075190 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77c327fe-cc04-492f-bc0e-efb01a43a67f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77c327fe-cc04-492f-bc0e-efb01a43a67f-istio-data\") pod \"77c327fe-cc04-492f-bc0e-efb01a43a67f\" (UID: \"77c327fe-cc04-492f-bc0e-efb01a43a67f\") " Oct 30 17:21:16.075358 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:16.075330 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77c327fe-cc04-492f-bc0e-efb01a43a67f-credential-socket" pod "77c327fe-cc04-492f-bc0e-efb01a43a67f" (UID: "77c327fe-cc04-492f-bc0e-efb01a43a67f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:16.075539 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:16.075496 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77c327fe-cc04-492f-bc0e-efb01a43a67f-istio-data" pod "77c327fe-cc04-492f-bc0e-efb01a43a67f" (UID: "77c327fe-cc04-492f-bc0e-efb01a43a67f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:16.075707 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:16.075534 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77c327fe-cc04-492f-bc0e-efb01a43a67f-kube-api-access-fzvx7\" (UniqueName: \"kubernetes.io/projected/77c327fe-cc04-492f-bc0e-efb01a43a67f-kube-api-access-fzvx7\") pod \"77c327fe-cc04-492f-bc0e-efb01a43a67f\" (UID: \"77c327fe-cc04-492f-bc0e-efb01a43a67f\") " Oct 30 17:21:16.075707 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:16.075571 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77c327fe-cc04-492f-bc0e-efb01a43a67f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77c327fe-cc04-492f-bc0e-efb01a43a67f-istio-envoy\") pod \"77c327fe-cc04-492f-bc0e-efb01a43a67f\" (UID: \"77c327fe-cc04-492f-bc0e-efb01a43a67f\") " Oct 30 17:21:16.075707 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:16.075647 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77c327fe-cc04-492f-bc0e-efb01a43a67f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77c327fe-cc04-492f-bc0e-efb01a43a67f-workload-certs\") pod \"77c327fe-cc04-492f-bc0e-efb01a43a67f\" (UID: \"77c327fe-cc04-492f-bc0e-efb01a43a67f\") " Oct 30 17:21:16.075873 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:16.075724 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77c327fe-cc04-492f-bc0e-efb01a43a67f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77c327fe-cc04-492f-bc0e-efb01a43a67f-istiod-ca-cert\") pod \"77c327fe-cc04-492f-bc0e-efb01a43a67f\" (UID: \"77c327fe-cc04-492f-bc0e-efb01a43a67f\") " Oct 30 17:21:16.075873 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:16.075752 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/77c327fe-cc04-492f-bc0e-efb01a43a67f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77c327fe-cc04-492f-bc0e-efb01a43a67f-istio-podinfo\") pod \"77c327fe-cc04-492f-bc0e-efb01a43a67f\" (UID: \"77c327fe-cc04-492f-bc0e-efb01a43a67f\") " Oct 30 17:21:16.075873 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:16.075813 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77c327fe-cc04-492f-bc0e-efb01a43a67f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77c327fe-cc04-492f-bc0e-efb01a43a67f-workload-socket\") pod \"77c327fe-cc04-492f-bc0e-efb01a43a67f\" (UID: \"77c327fe-cc04-492f-bc0e-efb01a43a67f\") " Oct 30 17:21:16.076037 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:16.075882 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77c327fe-cc04-492f-bc0e-efb01a43a67f-istio-token\" (UniqueName: \"kubernetes.io/projected/77c327fe-cc04-492f-bc0e-efb01a43a67f-istio-token\") pod \"77c327fe-cc04-492f-bc0e-efb01a43a67f\" (UID: \"77c327fe-cc04-492f-bc0e-efb01a43a67f\") " Oct 30 17:21:16.076696 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:16.076372 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77c327fe-cc04-492f-bc0e-efb01a43a67f-workload-certs" pod "77c327fe-cc04-492f-bc0e-efb01a43a67f" (UID: "77c327fe-cc04-492f-bc0e-efb01a43a67f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:16.077179 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:16.076973 4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77c327fe-cc04-492f-bc0e-efb01a43a67f-credential-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:21:16.077179 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:16.076998 4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77c327fe-cc04-492f-bc0e-efb01a43a67f-istio-data\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:21:16.077179 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:16.077020 4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77c327fe-cc04-492f-bc0e-efb01a43a67f-workload-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:21:16.078030 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:16.078008 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77c327fe-cc04-492f-bc0e-efb01a43a67f-workload-socket" pod "77c327fe-cc04-492f-bc0e-efb01a43a67f" (UID: "77c327fe-cc04-492f-bc0e-efb01a43a67f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:16.078549 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:16.078529 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77c327fe-cc04-492f-bc0e-efb01a43a67f-istiod-ca-cert" pod "77c327fe-cc04-492f-bc0e-efb01a43a67f" (UID: "77c327fe-cc04-492f-bc0e-efb01a43a67f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:21:16.079562 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:16.079473 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77c327fe-cc04-492f-bc0e-efb01a43a67f-istio-token" pod "77c327fe-cc04-492f-bc0e-efb01a43a67f" (UID: "77c327fe-cc04-492f-bc0e-efb01a43a67f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:21:16.079562 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:16.079530 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77c327fe-cc04-492f-bc0e-efb01a43a67f-kube-api-access-fzvx7" pod "77c327fe-cc04-492f-bc0e-efb01a43a67f" (UID: "77c327fe-cc04-492f-bc0e-efb01a43a67f"). InnerVolumeSpecName "kube-api-access-fzvx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:21:16.079782 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:16.079689 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77c327fe-cc04-492f-bc0e-efb01a43a67f-istio-podinfo" pod "77c327fe-cc04-492f-bc0e-efb01a43a67f" (UID: "77c327fe-cc04-492f-bc0e-efb01a43a67f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:21:16.081450 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:16.081424 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77c327fe-cc04-492f-bc0e-efb01a43a67f-istio-envoy" pod "77c327fe-cc04-492f-bc0e-efb01a43a67f" (UID: "77c327fe-cc04-492f-bc0e-efb01a43a67f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:16.179252 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:16.179218 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzvx7\" (UniqueName: \"kubernetes.io/projected/77c327fe-cc04-492f-bc0e-efb01a43a67f-kube-api-access-fzvx7\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:21:16.179252 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:16.179253 4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77c327fe-cc04-492f-bc0e-efb01a43a67f-istio-envoy\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:21:16.179821 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:16.179276 4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77c327fe-cc04-492f-bc0e-efb01a43a67f-istiod-ca-cert\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:21:16.179821 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:16.179290 4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77c327fe-cc04-492f-bc0e-efb01a43a67f-istio-podinfo\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:21:16.180477 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:16.180445 4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77c327fe-cc04-492f-bc0e-efb01a43a67f-workload-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:21:16.180608 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:16.180485 4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77c327fe-cc04-492f-bc0e-efb01a43a67f-istio-token\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:21:16.727803 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:16.727763 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f42baf-6c555f88f9-k8szt" podUID="46c462ae-1044-4037-939b-43ad0906630d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:21:16.727977 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:16.727763 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-f42baf-58fcd7df8-kshhs" podUID="e9629e12-7b8a-4e59-acb9-8e7476994826" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:21:16.953945 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:16.953904 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq" event={"ID":"77c327fe-cc04-492f-bc0e-efb01a43a67f","Type":"ContainerDied","Data":"99f143979e62bfac740520e8fa141397e6311d90a9a61b1da3ad699d48a3378c"} Oct 30 17:21:16.953945 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:16.953928 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq" Oct 30 17:21:16.954202 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:16.953997 4522 scope.go:122] "RemoveContainer" containerID="5e7604ac9a3c4cbddd2348117d5f83dcef0bdf184cd82f06067a46d4d08d0fa2" Oct 30 17:21:17.013790 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:17.013701 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq"] Oct 30 17:21:17.027039 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:17.027004 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-h5bxq"] Oct 30 17:21:18.147960 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:18.147926 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77c327fe-cc04-492f-bc0e-efb01a43a67f" path="/var/lib/kubelet/pods/77c327fe-cc04-492f-bc0e-efb01a43a67f/volumes" Oct 30 17:21:22.109761 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:22.109728 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d003a5-5cc689c7b6-z4b8q"] Oct 30 17:21:26.077526 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:26.077471 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f42baf-8w7lj"] Oct 30 17:21:26.090243 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:26.090210 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f42baf-8w7lj"] Oct 30 17:21:26.158917 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:26.158885 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31840135-fd6a-4f90-a610-c4ff17f0601a" path="/var/lib/kubelet/pods/31840135-fd6a-4f90-a610-c4ff17f0601a/volumes" Oct 30 17:21:26.728448 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:26.728406 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-f42baf-58fcd7df8-kshhs" podUID="e9629e12-7b8a-4e59-acb9-8e7476994826" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:21:26.728448 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:26.728442 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f42baf-6c555f88f9-k8szt" podUID="46c462ae-1044-4037-939b-43ad0906630d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:21:27.110461 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:27.110370 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-d003a5-5cc689c7b6-z4b8q" podUID="8f8c9da3-072b-4c71-8cd9-be0dd174e99d" containerName="payload-processing" containerID="cri-o://a8d72af0c8590b93f2e8150276dced3eeefa36c637b9fe8a35bac273dec88400" gracePeriod=25 Oct 30 17:21:29.628767 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:29.628728 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-d003a5-5cc689c7b6-z4b8q" podUID="8f8c9da3-072b-4c71-8cd9-be0dd174e99d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:21:33.537354 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:33.534291 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f42baf-58fcd7df8-kshhs" Oct 30 17:21:33.582857 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:33.582817 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9629e12-7b8a-4e59-acb9-8e7476994826-kube-api-access-hrbzd\" (UniqueName: \"kubernetes.io/projected/e9629e12-7b8a-4e59-acb9-8e7476994826-kube-api-access-hrbzd\") pod \"e9629e12-7b8a-4e59-acb9-8e7476994826\" (UID: \"e9629e12-7b8a-4e59-acb9-8e7476994826\") " Oct 30 17:21:33.583045 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:33.583015 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9629e12-7b8a-4e59-acb9-8e7476994826-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9629e12-7b8a-4e59-acb9-8e7476994826-plugins-config-volume\") pod \"e9629e12-7b8a-4e59-acb9-8e7476994826\" (UID: \"e9629e12-7b8a-4e59-acb9-8e7476994826\") " Oct 30 17:21:33.588399 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:33.588369 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f42baf-6c555f88f9-k8szt" Oct 30 17:21:33.589285 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:33.589260 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9629e12-7b8a-4e59-acb9-8e7476994826-plugins-config-volume" pod "e9629e12-7b8a-4e59-acb9-8e7476994826" (UID: "e9629e12-7b8a-4e59-acb9-8e7476994826"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:21:33.600224 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:33.600191 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9629e12-7b8a-4e59-acb9-8e7476994826-kube-api-access-hrbzd" pod "e9629e12-7b8a-4e59-acb9-8e7476994826" (UID: "e9629e12-7b8a-4e59-acb9-8e7476994826"). InnerVolumeSpecName "kube-api-access-hrbzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:21:33.684851 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:33.684748 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46c462ae-1044-4037-939b-43ad0906630d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46c462ae-1044-4037-939b-43ad0906630d-plugins-config-volume\") pod \"46c462ae-1044-4037-939b-43ad0906630d\" (UID: \"46c462ae-1044-4037-939b-43ad0906630d\") " Oct 30 17:21:33.685022 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:33.684957 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46c462ae-1044-4037-939b-43ad0906630d-kube-api-access-fdct4\" (UniqueName: \"kubernetes.io/projected/46c462ae-1044-4037-939b-43ad0906630d-kube-api-access-fdct4\") pod \"46c462ae-1044-4037-939b-43ad0906630d\" (UID: \"46c462ae-1044-4037-939b-43ad0906630d\") " Oct 30 17:21:33.685180 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:33.685144 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46c462ae-1044-4037-939b-43ad0906630d-plugins-config-volume" pod "46c462ae-1044-4037-939b-43ad0906630d" (UID: "46c462ae-1044-4037-939b-43ad0906630d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:21:33.685946 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:33.685920 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9629e12-7b8a-4e59-acb9-8e7476994826-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:21:33.686071 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:33.685948 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46c462ae-1044-4037-939b-43ad0906630d-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:21:33.686071 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:33.685964 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrbzd\" (UniqueName: \"kubernetes.io/projected/e9629e12-7b8a-4e59-acb9-8e7476994826-kube-api-access-hrbzd\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:21:33.687500 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:33.687472 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46c462ae-1044-4037-939b-43ad0906630d-kube-api-access-fdct4" pod "46c462ae-1044-4037-939b-43ad0906630d" (UID: "46c462ae-1044-4037-939b-43ad0906630d"). InnerVolumeSpecName "kube-api-access-fdct4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:21:33.788022 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:33.787980 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdct4\" (UniqueName: \"kubernetes.io/projected/46c462ae-1044-4037-939b-43ad0906630d-kube-api-access-fdct4\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:21:34.087625 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:34.087587 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv"] Oct 30 17:21:34.087945 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:34.087879 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv" podUID="5d6e738a-984b-463c-acbd-e2b38e345f6d" containerName="istio-proxy" containerID="cri-o://5e5da11a1b7cd82130b6dbe321235e4b8a130a3d4c2bce8d77ab30f878040097" gracePeriod=30 Oct 30 17:21:34.229720 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:34.218057 4522 generic.go:356] "Generic (PLEG): container finished" podID="e9629e12-7b8a-4e59-acb9-8e7476994826" containerID="aebbd3d5e02197db7c1da22c7e704f05a5d794038600d9d84a395bcbb5b1e83f" exitCode=0 Oct 30 17:21:34.229720 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:34.218447 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f42baf-58fcd7df8-kshhs" event={"ID":"e9629e12-7b8a-4e59-acb9-8e7476994826","Type":"ContainerDied","Data":"aebbd3d5e02197db7c1da22c7e704f05a5d794038600d9d84a395bcbb5b1e83f"} Oct 30 17:21:34.229720 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:34.218521 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f42baf-58fcd7df8-kshhs" event={"ID":"e9629e12-7b8a-4e59-acb9-8e7476994826","Type":"ContainerDied","Data":"5be3a18557a5f04f323b0e7e7e1fa8af0937d0aa333b14f07c0127ed1a88a607"} Oct 30 17:21:34.229720 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:34.218562 4522 scope.go:122] "RemoveContainer" containerID="aebbd3d5e02197db7c1da22c7e704f05a5d794038600d9d84a395bcbb5b1e83f" Oct 30 17:21:34.229720 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:34.219378 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f42baf-58fcd7df8-kshhs" Oct 30 17:21:34.356598 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:34.327353 4522 scope.go:122] "RemoveContainer" containerID="aebbd3d5e02197db7c1da22c7e704f05a5d794038600d9d84a395bcbb5b1e83f" Oct 30 17:21:34.356598 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:34.348056 4522 patch_prober.go:29] interesting pod/e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:21:34.356598 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:34.348119 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv" podUID="5d6e738a-984b-463c-acbd-e2b38e345f6d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:21:34.356598 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:21:34.349543 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aebbd3d5e02197db7c1da22c7e704f05a5d794038600d9d84a395bcbb5b1e83f\": container with ID starting with aebbd3d5e02197db7c1da22c7e704f05a5d794038600d9d84a395bcbb5b1e83f not found: ID does not exist" containerID="aebbd3d5e02197db7c1da22c7e704f05a5d794038600d9d84a395bcbb5b1e83f" Oct 30 17:21:34.356598 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:34.349585 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aebbd3d5e02197db7c1da22c7e704f05a5d794038600d9d84a395bcbb5b1e83f"} err="failed to get container status \"aebbd3d5e02197db7c1da22c7e704f05a5d794038600d9d84a395bcbb5b1e83f\": rpc error: code = NotFound desc = could not find container \"aebbd3d5e02197db7c1da22c7e704f05a5d794038600d9d84a395bcbb5b1e83f\": container with ID starting with aebbd3d5e02197db7c1da22c7e704f05a5d794038600d9d84a395bcbb5b1e83f not found: ID does not exist" Oct 30 17:21:34.356598 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:34.349798 4522 generic.go:356] "Generic (PLEG): container finished" podID="46c462ae-1044-4037-939b-43ad0906630d" containerID="708c7b66d9f4c6246fcc1c976856835d6200f2ab9683119c0813ff60c3b2348a" exitCode=0 Oct 30 17:21:34.356598 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:34.349944 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f42baf-6c555f88f9-k8szt" event={"ID":"46c462ae-1044-4037-939b-43ad0906630d","Type":"ContainerDied","Data":"708c7b66d9f4c6246fcc1c976856835d6200f2ab9683119c0813ff60c3b2348a"} Oct 30 17:21:34.356598 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:34.349992 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f42baf-6c555f88f9-k8szt" event={"ID":"46c462ae-1044-4037-939b-43ad0906630d","Type":"ContainerDied","Data":"a80ffd6513c902d82a591996c25b9d6755f6e8d84eada3c04cf9d085b846c596"} Oct 30 17:21:34.356598 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:34.350021 4522 scope.go:122] "RemoveContainer" containerID="708c7b66d9f4c6246fcc1c976856835d6200f2ab9683119c0813ff60c3b2348a" Oct 30 17:21:34.356598 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:34.350169 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f42baf-6c555f88f9-k8szt" Oct 30 17:21:34.402138 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:34.402030 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f42baf-58fcd7df8-kshhs"] Oct 30 17:21:34.408091 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:34.407918 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f42baf-58fcd7df8-kshhs"] Oct 30 17:21:34.408196 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:34.408173 4522 scope.go:122] "RemoveContainer" containerID="708c7b66d9f4c6246fcc1c976856835d6200f2ab9683119c0813ff60c3b2348a" Oct 30 17:21:34.408664 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:21:34.408632 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"708c7b66d9f4c6246fcc1c976856835d6200f2ab9683119c0813ff60c3b2348a\": container with ID starting with 708c7b66d9f4c6246fcc1c976856835d6200f2ab9683119c0813ff60c3b2348a not found: ID does not exist" containerID="708c7b66d9f4c6246fcc1c976856835d6200f2ab9683119c0813ff60c3b2348a" Oct 30 17:21:34.408784 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:34.408674 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"708c7b66d9f4c6246fcc1c976856835d6200f2ab9683119c0813ff60c3b2348a"} err="failed to get container status \"708c7b66d9f4c6246fcc1c976856835d6200f2ab9683119c0813ff60c3b2348a\": rpc error: code = NotFound desc = could not find container \"708c7b66d9f4c6246fcc1c976856835d6200f2ab9683119c0813ff60c3b2348a\": container with ID starting with 708c7b66d9f4c6246fcc1c976856835d6200f2ab9683119c0813ff60c3b2348a not found: ID does not exist" Oct 30 17:21:34.482401 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:34.482363 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f42baf-6c555f88f9-k8szt"] Oct 30 17:21:34.490614 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:34.490582 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f42baf-6c555f88f9-k8szt"] Oct 30 17:21:34.583324 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:34.583268 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5d67ca-sfwzm"] Oct 30 17:21:34.584616 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:34.584593 4522 cpu_manager.go:414] "containerMap: removing container" podUID="46c462ae-1044-4037-939b-43ad0906630d" containerName="payload-pre-processing" Oct 30 17:21:34.584681 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:34.584619 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46c462ae-1044-4037-939b-43ad0906630d" containerName="payload-pre-processing" Oct 30 17:21:34.584681 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:34.584667 4522 cpu_manager.go:414] "containerMap: removing container" podUID="949ded2a-41f3-4046-b83f-e2d7b3d13334" containerName="revoke-keys" Oct 30 17:21:34.584681 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:34.584675 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="949ded2a-41f3-4046-b83f-e2d7b3d13334" containerName="revoke-keys" Oct 30 17:21:34.584792 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:34.584687 4522 cpu_manager.go:414] "containerMap: removing container" podUID="77c327fe-cc04-492f-bc0e-efb01a43a67f" containerName="istio-proxy" Oct 30 17:21:34.584792 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:34.584694 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77c327fe-cc04-492f-bc0e-efb01a43a67f" containerName="istio-proxy" Oct 30 17:21:34.584792 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:34.584733 4522 cpu_manager.go:414] "containerMap: removing container" podUID="e9629e12-7b8a-4e59-acb9-8e7476994826" containerName="payload-processing" Oct 30 17:21:34.584792 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:34.584741 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9629e12-7b8a-4e59-acb9-8e7476994826" containerName="payload-processing" Oct 30 17:21:34.584963 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:34.584951 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9629e12-7b8a-4e59-acb9-8e7476994826" containerName="payload-processing" Oct 30 17:21:34.584999 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:34.584971 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="46c462ae-1044-4037-939b-43ad0906630d" containerName="payload-pre-processing" Oct 30 17:21:34.584999 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:34.584986 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="949ded2a-41f3-4046-b83f-e2d7b3d13334" containerName="revoke-keys" Oct 30 17:21:34.585062 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:34.584997 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="77c327fe-cc04-492f-bc0e-efb01a43a67f" containerName="istio-proxy" Oct 30 17:21:34.590728 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:34.590703 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5d67ca-sfwzm" Oct 30 17:21:34.606341 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:34.606301 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5d67ca-sfwzm"] Oct 30 17:21:34.621408 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:34.621230 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:21:34.621739 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:34.621701 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:21:34.656560 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:34.655492 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eeeb5f83-d6f8-4fd4-9c78-c5cb3640b538-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5d67ca-sfwzm\" (UID: \"eeeb5f83-d6f8-4fd4-9c78-c5cb3640b538\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5d67ca-sfwzm" Oct 30 17:21:34.656560 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:34.655563 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlx68\" (UniqueName: \"kubernetes.io/projected/eeeb5f83-d6f8-4fd4-9c78-c5cb3640b538-kube-api-access-qlx68\") pod \"maas-api-revoke-keys-e2e-inf-b-5d67ca-sfwzm\" (UID: \"eeeb5f83-d6f8-4fd4-9c78-c5cb3640b538\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5d67ca-sfwzm" Oct 30 17:21:34.656560 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:34.655707 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eeeb5f83-d6f8-4fd4-9c78-c5cb3640b538-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5d67ca-sfwzm\" (UID: \"eeeb5f83-d6f8-4fd4-9c78-c5cb3640b538\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5d67ca-sfwzm" Oct 30 17:21:34.757794 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:34.757755 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlx68\" (UniqueName: \"kubernetes.io/projected/eeeb5f83-d6f8-4fd4-9c78-c5cb3640b538-kube-api-access-qlx68\") pod \"maas-api-revoke-keys-e2e-inf-b-5d67ca-sfwzm\" (UID: \"eeeb5f83-d6f8-4fd4-9c78-c5cb3640b538\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5d67ca-sfwzm" Oct 30 17:21:34.757947 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:34.757847 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eeeb5f83-d6f8-4fd4-9c78-c5cb3640b538-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5d67ca-sfwzm\" (UID: \"eeeb5f83-d6f8-4fd4-9c78-c5cb3640b538\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5d67ca-sfwzm" Oct 30 17:21:34.758198 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:34.758178 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eeeb5f83-d6f8-4fd4-9c78-c5cb3640b538-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5d67ca-sfwzm\" (UID: \"eeeb5f83-d6f8-4fd4-9c78-c5cb3640b538\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5d67ca-sfwzm" Oct 30 17:21:34.758937 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:34.758915 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eeeb5f83-d6f8-4fd4-9c78-c5cb3640b538-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5d67ca-sfwzm\" (UID: \"eeeb5f83-d6f8-4fd4-9c78-c5cb3640b538\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5d67ca-sfwzm" Oct 30 17:21:34.760336 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:34.760272 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eeeb5f83-d6f8-4fd4-9c78-c5cb3640b538-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5d67ca-sfwzm\" (UID: \"eeeb5f83-d6f8-4fd4-9c78-c5cb3640b538\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5d67ca-sfwzm" Oct 30 17:21:34.783169 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:34.783141 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlx68\" (UniqueName: \"kubernetes.io/projected/eeeb5f83-d6f8-4fd4-9c78-c5cb3640b538-kube-api-access-qlx68\") pod \"maas-api-revoke-keys-e2e-inf-b-5d67ca-sfwzm\" (UID: \"eeeb5f83-d6f8-4fd4-9c78-c5cb3640b538\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5d67ca-sfwzm" Oct 30 17:21:34.908610 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:34.908524 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5d67ca-sfwzm" Oct 30 17:21:35.242703 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:21:35.242673 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeeb5f83_d6f8_4fd4_9c78_c5cb3640b538.slice/crio-207365678ed7fa74382bd8ac6e59573be2e2275aab85d71ef3f7507da30e11da WatchSource:0}: Error finding container 207365678ed7fa74382bd8ac6e59573be2e2275aab85d71ef3f7507da30e11da: Status 404 returned error can't find the container with id 207365678ed7fa74382bd8ac6e59573be2e2275aab85d71ef3f7507da30e11da Oct 30 17:21:35.244803 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:35.244742 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5d67ca-sfwzm"] Oct 30 17:21:35.361377 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:35.361340 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5d67ca-sfwzm" event={"ID":"eeeb5f83-d6f8-4fd4-9c78-c5cb3640b538","Type":"ContainerStarted","Data":"207365678ed7fa74382bd8ac6e59573be2e2275aab85d71ef3f7507da30e11da"} Oct 30 17:21:36.147837 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:36.147804 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46c462ae-1044-4037-939b-43ad0906630d" path="/var/lib/kubelet/pods/46c462ae-1044-4037-939b-43ad0906630d/volumes" Oct 30 17:21:36.148406 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:36.148389 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9629e12-7b8a-4e59-acb9-8e7476994826" path="/var/lib/kubelet/pods/e9629e12-7b8a-4e59-acb9-8e7476994826/volumes" Oct 30 17:21:36.373684 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:36.373089 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5d67ca-sfwzm" event={"ID":"eeeb5f83-d6f8-4fd4-9c78-c5cb3640b538","Type":"ContainerStarted","Data":"deee53d28ea23c79e2d1215ebc7faefca10a653dc8ceebb4854a5a7687b71009"} Oct 30 17:21:36.424379 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:36.424299 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5d67ca-sfwzm" podStartSLOduration=2.424279951 podStartE2EDuration="2.424279951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:21:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:21:36.416610133 +0000 UTC m=+2070.417365034" watchObservedRunningTime="2026-10-30 17:21:36.424279951 +0000 UTC m=+2070.425034853" Oct 30 17:21:37.385671 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:37.385637 4522 generic.go:356] "Generic (PLEG): container finished" podID="eeeb5f83-d6f8-4fd4-9c78-c5cb3640b538" containerID="deee53d28ea23c79e2d1215ebc7faefca10a653dc8ceebb4854a5a7687b71009" exitCode=0 Oct 30 17:21:37.386092 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:37.385718 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5d67ca-sfwzm" event={"ID":"eeeb5f83-d6f8-4fd4-9c78-c5cb3640b538","Type":"ContainerDied","Data":"deee53d28ea23c79e2d1215ebc7faefca10a653dc8ceebb4854a5a7687b71009"} Oct 30 17:21:39.224470 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:39.224436 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5d67ca-sfwzm" Oct 30 17:21:39.374378 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:39.366277 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eeeb5f83-d6f8-4fd4-9c78-c5cb3640b538-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eeeb5f83-d6f8-4fd4-9c78-c5cb3640b538-maas-api-xks-ca\") pod \"eeeb5f83-d6f8-4fd4-9c78-c5cb3640b538\" (UID: \"eeeb5f83-d6f8-4fd4-9c78-c5cb3640b538\") " Oct 30 17:21:39.374378 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:39.366378 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eeeb5f83-d6f8-4fd4-9c78-c5cb3640b538-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eeeb5f83-d6f8-4fd4-9c78-c5cb3640b538-maas-api-openshift-ca\") pod \"eeeb5f83-d6f8-4fd4-9c78-c5cb3640b538\" (UID: \"eeeb5f83-d6f8-4fd4-9c78-c5cb3640b538\") " Oct 30 17:21:39.374378 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:39.366427 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eeeb5f83-d6f8-4fd4-9c78-c5cb3640b538-kube-api-access-qlx68\" (UniqueName: \"kubernetes.io/projected/eeeb5f83-d6f8-4fd4-9c78-c5cb3640b538-kube-api-access-qlx68\") pod \"eeeb5f83-d6f8-4fd4-9c78-c5cb3640b538\" (UID: \"eeeb5f83-d6f8-4fd4-9c78-c5cb3640b538\") " Oct 30 17:21:39.374378 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:39.371100 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eeeb5f83-d6f8-4fd4-9c78-c5cb3640b538-maas-api-openshift-ca" pod "eeeb5f83-d6f8-4fd4-9c78-c5cb3640b538" (UID: "eeeb5f83-d6f8-4fd4-9c78-c5cb3640b538"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:21:39.393391 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:39.386600 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eeeb5f83-d6f8-4fd4-9c78-c5cb3640b538-maas-api-xks-ca" pod "eeeb5f83-d6f8-4fd4-9c78-c5cb3640b538" (UID: "eeeb5f83-d6f8-4fd4-9c78-c5cb3640b538"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:21:39.394110 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:39.393953 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeeb5f83-d6f8-4fd4-9c78-c5cb3640b538-kube-api-access-qlx68" pod "eeeb5f83-d6f8-4fd4-9c78-c5cb3640b538" (UID: "eeeb5f83-d6f8-4fd4-9c78-c5cb3640b538"). InnerVolumeSpecName "kube-api-access-qlx68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:21:39.415372 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:39.413805 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5d67ca-sfwzm" Oct 30 17:21:39.415372 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:39.413836 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5d67ca-sfwzm" event={"ID":"eeeb5f83-d6f8-4fd4-9c78-c5cb3640b538","Type":"ContainerDied","Data":"207365678ed7fa74382bd8ac6e59573be2e2275aab85d71ef3f7507da30e11da"} Oct 30 17:21:39.415372 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:39.413899 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="207365678ed7fa74382bd8ac6e59573be2e2275aab85d71ef3f7507da30e11da" Oct 30 17:21:39.419366 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:39.419285 4522 generic.go:356] "Generic (PLEG): container finished" podID="5d6e738a-984b-463c-acbd-e2b38e345f6d" containerID="5e5da11a1b7cd82130b6dbe321235e4b8a130a3d4c2bce8d77ab30f878040097" exitCode=0 Oct 30 17:21:39.419492 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:39.419369 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv" event={"ID":"5d6e738a-984b-463c-acbd-e2b38e345f6d","Type":"ContainerDied","Data":"5e5da11a1b7cd82130b6dbe321235e4b8a130a3d4c2bce8d77ab30f878040097"} Oct 30 17:21:39.468549 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:39.468524 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eeeb5f83-d6f8-4fd4-9c78-c5cb3640b538-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:21:39.468549 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:39.468548 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eeeb5f83-d6f8-4fd4-9c78-c5cb3640b538-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:21:39.468756 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:39.468558 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlx68\" (UniqueName: \"kubernetes.io/projected/eeeb5f83-d6f8-4fd4-9c78-c5cb3640b538-kube-api-access-qlx68\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:21:39.474570 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:39.474551 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv" Oct 30 17:21:39.628866 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:39.628765 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-d003a5-5cc689c7b6-z4b8q" podUID="8f8c9da3-072b-4c71-8cd9-be0dd174e99d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:21:39.671746 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:39.671691 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d6e738a-984b-463c-acbd-e2b38e345f6d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d6e738a-984b-463c-acbd-e2b38e345f6d-istio-data\") pod \"5d6e738a-984b-463c-acbd-e2b38e345f6d\" (UID: \"5d6e738a-984b-463c-acbd-e2b38e345f6d\") " Oct 30 17:21:39.671984 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:39.671773 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d6e738a-984b-463c-acbd-e2b38e345f6d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6e738a-984b-463c-acbd-e2b38e345f6d-workload-socket\") pod \"5d6e738a-984b-463c-acbd-e2b38e345f6d\" (UID: \"5d6e738a-984b-463c-acbd-e2b38e345f6d\") " Oct 30 17:21:39.671984 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:39.671878 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d6e738a-984b-463c-acbd-e2b38e345f6d-kube-api-access-phcz8\" (UniqueName: \"kubernetes.io/projected/5d6e738a-984b-463c-acbd-e2b38e345f6d-kube-api-access-phcz8\") pod \"5d6e738a-984b-463c-acbd-e2b38e345f6d\" (UID: \"5d6e738a-984b-463c-acbd-e2b38e345f6d\") " Oct 30 17:21:39.671984 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:39.671930 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d6e738a-984b-463c-acbd-e2b38e345f6d-istio-token\" (UniqueName: \"kubernetes.io/projected/5d6e738a-984b-463c-acbd-e2b38e345f6d-istio-token\") pod \"5d6e738a-984b-463c-acbd-e2b38e345f6d\" (UID: \"5d6e738a-984b-463c-acbd-e2b38e345f6d\") " Oct 30 17:21:39.671984 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:39.671957 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d6e738a-984b-463c-acbd-e2b38e345f6d-istio-data" pod "5d6e738a-984b-463c-acbd-e2b38e345f6d" (UID: "5d6e738a-984b-463c-acbd-e2b38e345f6d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:39.672800 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:39.672063 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d6e738a-984b-463c-acbd-e2b38e345f6d-workload-socket" pod "5d6e738a-984b-463c-acbd-e2b38e345f6d" (UID: "5d6e738a-984b-463c-acbd-e2b38e345f6d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:39.672800 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:39.672150 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d6e738a-984b-463c-acbd-e2b38e345f6d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d6e738a-984b-463c-acbd-e2b38e345f6d-istio-podinfo\") pod \"5d6e738a-984b-463c-acbd-e2b38e345f6d\" (UID: \"5d6e738a-984b-463c-acbd-e2b38e345f6d\") " Oct 30 17:21:39.672800 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:39.672217 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d6e738a-984b-463c-acbd-e2b38e345f6d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6e738a-984b-463c-acbd-e2b38e345f6d-credential-socket\") pod \"5d6e738a-984b-463c-acbd-e2b38e345f6d\" (UID: \"5d6e738a-984b-463c-acbd-e2b38e345f6d\") " Oct 30 17:21:39.672800 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:39.672275 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d6e738a-984b-463c-acbd-e2b38e345f6d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d6e738a-984b-463c-acbd-e2b38e345f6d-workload-certs\") pod \"5d6e738a-984b-463c-acbd-e2b38e345f6d\" (UID: \"5d6e738a-984b-463c-acbd-e2b38e345f6d\") " Oct 30 17:21:39.672800 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:39.672510 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d6e738a-984b-463c-acbd-e2b38e345f6d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d6e738a-984b-463c-acbd-e2b38e345f6d-istiod-ca-cert\") pod \"5d6e738a-984b-463c-acbd-e2b38e345f6d\" (UID: \"5d6e738a-984b-463c-acbd-e2b38e345f6d\") " Oct 30 17:21:39.672800 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:39.672571 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d6e738a-984b-463c-acbd-e2b38e345f6d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d6e738a-984b-463c-acbd-e2b38e345f6d-istio-envoy\") pod \"5d6e738a-984b-463c-acbd-e2b38e345f6d\" (UID: \"5d6e738a-984b-463c-acbd-e2b38e345f6d\") " Oct 30 17:21:39.673192 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:39.673166 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d6e738a-984b-463c-acbd-e2b38e345f6d-credential-socket" pod "5d6e738a-984b-463c-acbd-e2b38e345f6d" (UID: "5d6e738a-984b-463c-acbd-e2b38e345f6d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:39.673350 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:39.673297 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d6e738a-984b-463c-acbd-e2b38e345f6d-istiod-ca-cert" pod "5d6e738a-984b-463c-acbd-e2b38e345f6d" (UID: "5d6e738a-984b-463c-acbd-e2b38e345f6d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:21:39.673497 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:39.673470 4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d6e738a-984b-463c-acbd-e2b38e345f6d-istiod-ca-cert\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:21:39.673497 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:39.673494 4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d6e738a-984b-463c-acbd-e2b38e345f6d-istio-data\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:21:39.673649 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:39.673508 4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6e738a-984b-463c-acbd-e2b38e345f6d-workload-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:21:39.673649 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:39.673522 4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6e738a-984b-463c-acbd-e2b38e345f6d-credential-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:21:39.673649 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:39.673548 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d6e738a-984b-463c-acbd-e2b38e345f6d-workload-certs" pod "5d6e738a-984b-463c-acbd-e2b38e345f6d" (UID: "5d6e738a-984b-463c-acbd-e2b38e345f6d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:39.674526 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:39.674500 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d6e738a-984b-463c-acbd-e2b38e345f6d-kube-api-access-phcz8" pod "5d6e738a-984b-463c-acbd-e2b38e345f6d" (UID: "5d6e738a-984b-463c-acbd-e2b38e345f6d"). InnerVolumeSpecName "kube-api-access-phcz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:21:39.674625 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:39.674524 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d6e738a-984b-463c-acbd-e2b38e345f6d-istio-token" pod "5d6e738a-984b-463c-acbd-e2b38e345f6d" (UID: "5d6e738a-984b-463c-acbd-e2b38e345f6d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:21:39.674678 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:39.674664 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d6e738a-984b-463c-acbd-e2b38e345f6d-istio-podinfo" pod "5d6e738a-984b-463c-acbd-e2b38e345f6d" (UID: "5d6e738a-984b-463c-acbd-e2b38e345f6d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:21:39.675093 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:39.675077 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d6e738a-984b-463c-acbd-e2b38e345f6d-istio-envoy" pod "5d6e738a-984b-463c-acbd-e2b38e345f6d" (UID: "5d6e738a-984b-463c-acbd-e2b38e345f6d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:39.775472 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:39.775440 4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d6e738a-984b-463c-acbd-e2b38e345f6d-workload-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:21:39.775472 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:39.775468 4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d6e738a-984b-463c-acbd-e2b38e345f6d-istio-envoy\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:21:39.775472 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:39.775477 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phcz8\" (UniqueName: \"kubernetes.io/projected/5d6e738a-984b-463c-acbd-e2b38e345f6d-kube-api-access-phcz8\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:21:39.775715 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:39.775490 4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d6e738a-984b-463c-acbd-e2b38e345f6d-istio-token\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:21:39.775715 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:39.775502 4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d6e738a-984b-463c-acbd-e2b38e345f6d-istio-podinfo\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:21:40.428707 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:40.428620 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv" Oct 30 17:21:40.428707 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:40.428647 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv" event={"ID":"5d6e738a-984b-463c-acbd-e2b38e345f6d","Type":"ContainerDied","Data":"692bf5f4ecaf6df6db79c010a14ee70c3beef567d53c8ab50b6576c1e5385ea9"} Oct 30 17:21:40.428707 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:40.428704 4522 scope.go:122] "RemoveContainer" containerID="5e5da11a1b7cd82130b6dbe321235e4b8a130a3d4c2bce8d77ab30f878040097" Oct 30 17:21:40.503166 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:40.503134 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv"] Oct 30 17:21:40.516188 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:40.516155 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-n9bkv"] Oct 30 17:21:41.488004 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:41.487971 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16339e0f-bvkvm"] Oct 30 17:21:41.488874 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:41.488857 4522 cpu_manager.go:414] "containerMap: removing container" podUID="eeeb5f83-d6f8-4fd4-9c78-c5cb3640b538" containerName="revoke-keys" Oct 30 17:21:41.488924 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:41.488876 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eeeb5f83-d6f8-4fd4-9c78-c5cb3640b538" containerName="revoke-keys" Oct 30 17:21:41.488924 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:41.488892 4522 cpu_manager.go:414] "containerMap: removing container" podUID="5d6e738a-984b-463c-acbd-e2b38e345f6d" containerName="istio-proxy" Oct 30 17:21:41.488924 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:41.488897 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d6e738a-984b-463c-acbd-e2b38e345f6d" containerName="istio-proxy" Oct 30 17:21:41.489047 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:41.489037 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="eeeb5f83-d6f8-4fd4-9c78-c5cb3640b538" containerName="revoke-keys" Oct 30 17:21:41.489081 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:41.489050 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d6e738a-984b-463c-acbd-e2b38e345f6d" containerName="istio-proxy" Oct 30 17:21:41.493736 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:41.493720 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16339e0f-bvkvm" Oct 30 17:21:41.513949 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:41.513911 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16339e0f-bvkvm"] Oct 30 17:21:41.516928 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:41.516897 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fc7054e-4f5c-4fdf-8b7a-d6561d562ae5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16339e0f-bvkvm\" (UID: \"1fc7054e-4f5c-4fdf-8b7a-d6561d562ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16339e0f-bvkvm" Oct 30 17:21:41.517094 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:41.517069 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fc7054e-4f5c-4fdf-8b7a-d6561d562ae5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16339e0f-bvkvm\" (UID: \"1fc7054e-4f5c-4fdf-8b7a-d6561d562ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16339e0f-bvkvm" Oct 30 17:21:41.517336 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:41.517292 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpwhv\" (UniqueName: \"kubernetes.io/projected/1fc7054e-4f5c-4fdf-8b7a-d6561d562ae5-kube-api-access-gpwhv\") pod \"maas-api-revoke-sub-16339e0f-bvkvm\" (UID: \"1fc7054e-4f5c-4fdf-8b7a-d6561d562ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16339e0f-bvkvm" Oct 30 17:21:41.521159 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:41.521132 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:21:41.521159 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:41.521169 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:21:41.619253 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:41.619216 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpwhv\" (UniqueName: \"kubernetes.io/projected/1fc7054e-4f5c-4fdf-8b7a-d6561d562ae5-kube-api-access-gpwhv\") pod \"maas-api-revoke-sub-16339e0f-bvkvm\" (UID: \"1fc7054e-4f5c-4fdf-8b7a-d6561d562ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16339e0f-bvkvm" Oct 30 17:21:41.619451 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:41.619293 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fc7054e-4f5c-4fdf-8b7a-d6561d562ae5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16339e0f-bvkvm\" (UID: \"1fc7054e-4f5c-4fdf-8b7a-d6561d562ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16339e0f-bvkvm" Oct 30 17:21:41.619524 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:41.619505 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fc7054e-4f5c-4fdf-8b7a-d6561d562ae5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16339e0f-bvkvm\" (UID: \"1fc7054e-4f5c-4fdf-8b7a-d6561d562ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16339e0f-bvkvm" Oct 30 17:21:41.619904 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:41.619886 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fc7054e-4f5c-4fdf-8b7a-d6561d562ae5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16339e0f-bvkvm\" (UID: \"1fc7054e-4f5c-4fdf-8b7a-d6561d562ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16339e0f-bvkvm" Oct 30 17:21:41.621934 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:41.621914 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fc7054e-4f5c-4fdf-8b7a-d6561d562ae5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16339e0f-bvkvm\" (UID: \"1fc7054e-4f5c-4fdf-8b7a-d6561d562ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16339e0f-bvkvm" Oct 30 17:21:41.644244 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:41.644218 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpwhv\" (UniqueName: \"kubernetes.io/projected/1fc7054e-4f5c-4fdf-8b7a-d6561d562ae5-kube-api-access-gpwhv\") pod \"maas-api-revoke-sub-16339e0f-bvkvm\" (UID: \"1fc7054e-4f5c-4fdf-8b7a-d6561d562ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16339e0f-bvkvm" Oct 30 17:21:41.815586 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:41.815503 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16339e0f-bvkvm" Oct 30 17:21:42.125572 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:21:42.125539 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fc7054e_4f5c_4fdf_8b7a_d6561d562ae5.slice/crio-c1ce49b0b4e6f57388144ffcc41010eec4a266c4add317e70a1670958ffa5f0c WatchSource:0}: Error finding container c1ce49b0b4e6f57388144ffcc41010eec4a266c4add317e70a1670958ffa5f0c: Status 404 returned error can't find the container with id c1ce49b0b4e6f57388144ffcc41010eec4a266c4add317e70a1670958ffa5f0c Oct 30 17:21:42.127778 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:42.127752 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16339e0f-bvkvm"] Oct 30 17:21:42.150182 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:42.150155 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d6e738a-984b-463c-acbd-e2b38e345f6d" path="/var/lib/kubelet/pods/5d6e738a-984b-463c-acbd-e2b38e345f6d/volumes" Oct 30 17:21:42.450071 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:42.449988 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16339e0f-bvkvm" event={"ID":"1fc7054e-4f5c-4fdf-8b7a-d6561d562ae5","Type":"ContainerStarted","Data":"c1ce49b0b4e6f57388144ffcc41010eec4a266c4add317e70a1670958ffa5f0c"} Oct 30 17:21:42.949690 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:21:42.949660 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:21:43.461139 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:43.461102 4522 generic.go:356] "Generic (PLEG): container finished" podID="1fc7054e-4f5c-4fdf-8b7a-d6561d562ae5" containerID="93568b5b3ab2a334c1e0a062fa4ff50394d0d0a00a8493afe23d54a57f71263e" exitCode=0 Oct 30 17:21:43.461300 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:43.461145 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16339e0f-bvkvm" event={"ID":"1fc7054e-4f5c-4fdf-8b7a-d6561d562ae5","Type":"ContainerDied","Data":"93568b5b3ab2a334c1e0a062fa4ff50394d0d0a00a8493afe23d54a57f71263e"} Oct 30 17:21:44.684382 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:44.684350 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5d67ca-6c5bcd5578-n5lgb"] Oct 30 17:21:44.688804 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:44.688782 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16339e0f-bvkvm" Oct 30 17:21:44.783511 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:44.782203 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fc7054e-4f5c-4fdf-8b7a-d6561d562ae5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fc7054e-4f5c-4fdf-8b7a-d6561d562ae5-maas-api-openshift-ca\") pod \"1fc7054e-4f5c-4fdf-8b7a-d6561d562ae5\" (UID: \"1fc7054e-4f5c-4fdf-8b7a-d6561d562ae5\") " Oct 30 17:21:44.783511 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:44.782258 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fc7054e-4f5c-4fdf-8b7a-d6561d562ae5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fc7054e-4f5c-4fdf-8b7a-d6561d562ae5-maas-api-xks-ca\") pod \"1fc7054e-4f5c-4fdf-8b7a-d6561d562ae5\" (UID: \"1fc7054e-4f5c-4fdf-8b7a-d6561d562ae5\") " Oct 30 17:21:44.783511 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:44.782509 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fc7054e-4f5c-4fdf-8b7a-d6561d562ae5-kube-api-access-gpwhv\" (UniqueName: \"kubernetes.io/projected/1fc7054e-4f5c-4fdf-8b7a-d6561d562ae5-kube-api-access-gpwhv\") pod \"1fc7054e-4f5c-4fdf-8b7a-d6561d562ae5\" (UID: \"1fc7054e-4f5c-4fdf-8b7a-d6561d562ae5\") " Oct 30 17:21:44.784160 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:44.784134 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fc7054e-4f5c-4fdf-8b7a-d6561d562ae5-maas-api-openshift-ca" pod "1fc7054e-4f5c-4fdf-8b7a-d6561d562ae5" (UID: "1fc7054e-4f5c-4fdf-8b7a-d6561d562ae5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:21:44.787467 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:44.787440 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fc7054e-4f5c-4fdf-8b7a-d6561d562ae5-kube-api-access-gpwhv" pod "1fc7054e-4f5c-4fdf-8b7a-d6561d562ae5" (UID: "1fc7054e-4f5c-4fdf-8b7a-d6561d562ae5"). InnerVolumeSpecName "kube-api-access-gpwhv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:21:44.792489 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:44.789468 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fc7054e-4f5c-4fdf-8b7a-d6561d562ae5-maas-api-xks-ca" pod "1fc7054e-4f5c-4fdf-8b7a-d6561d562ae5" (UID: "1fc7054e-4f5c-4fdf-8b7a-d6561d562ae5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:21:44.885100 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:44.885062 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fc7054e-4f5c-4fdf-8b7a-d6561d562ae5-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:21:44.885100 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:44.885101 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fc7054e-4f5c-4fdf-8b7a-d6561d562ae5-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:21:44.885428 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:44.885119 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpwhv\" (UniqueName: \"kubernetes.io/projected/1fc7054e-4f5c-4fdf-8b7a-d6561d562ae5-kube-api-access-gpwhv\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:21:45.526331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:45.522567 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16339e0f-bvkvm" event={"ID":"1fc7054e-4f5c-4fdf-8b7a-d6561d562ae5","Type":"ContainerDied","Data":"c1ce49b0b4e6f57388144ffcc41010eec4a266c4add317e70a1670958ffa5f0c"} Oct 30 17:21:45.526331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:45.522638 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1ce49b0b4e6f57388144ffcc41010eec4a266c4add317e70a1670958ffa5f0c" Oct 30 17:21:45.526331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:45.522709 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16339e0f-bvkvm" Oct 30 17:21:47.405224 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:47.405192 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d003a5-5cc689c7b6-z4b8q" Oct 30 17:21:47.537445 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:47.537413 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f8c9da3-072b-4c71-8cd9-be0dd174e99d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f8c9da3-072b-4c71-8cd9-be0dd174e99d-plugins-config-volume\") pod \"8f8c9da3-072b-4c71-8cd9-be0dd174e99d\" (UID: \"8f8c9da3-072b-4c71-8cd9-be0dd174e99d\") " Oct 30 17:21:47.537664 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:47.537562 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f8c9da3-072b-4c71-8cd9-be0dd174e99d-kube-api-access-sc96q\" (UniqueName: \"kubernetes.io/projected/8f8c9da3-072b-4c71-8cd9-be0dd174e99d-kube-api-access-sc96q\") pod \"8f8c9da3-072b-4c71-8cd9-be0dd174e99d\" (UID: \"8f8c9da3-072b-4c71-8cd9-be0dd174e99d\") " Oct 30 17:21:47.537987 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:47.537958 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f8c9da3-072b-4c71-8cd9-be0dd174e99d-plugins-config-volume" pod "8f8c9da3-072b-4c71-8cd9-be0dd174e99d" (UID: "8f8c9da3-072b-4c71-8cd9-be0dd174e99d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:21:47.538252 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:47.538230 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f8c9da3-072b-4c71-8cd9-be0dd174e99d-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:21:47.540633 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:47.540605 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f8c9da3-072b-4c71-8cd9-be0dd174e99d-kube-api-access-sc96q" pod "8f8c9da3-072b-4c71-8cd9-be0dd174e99d" (UID: "8f8c9da3-072b-4c71-8cd9-be0dd174e99d"). InnerVolumeSpecName "kube-api-access-sc96q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:21:47.548787 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:47.548760 4522 generic.go:356] "Generic (PLEG): container finished" podID="8f8c9da3-072b-4c71-8cd9-be0dd174e99d" containerID="a8d72af0c8590b93f2e8150276dced3eeefa36c637b9fe8a35bac273dec88400" exitCode=0 Oct 30 17:21:47.549113 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:47.549080 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d003a5-5cc689c7b6-z4b8q" event={"ID":"8f8c9da3-072b-4c71-8cd9-be0dd174e99d","Type":"ContainerDied","Data":"a8d72af0c8590b93f2e8150276dced3eeefa36c637b9fe8a35bac273dec88400"} Oct 30 17:21:47.549223 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:47.549136 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d003a5-5cc689c7b6-z4b8q" event={"ID":"8f8c9da3-072b-4c71-8cd9-be0dd174e99d","Type":"ContainerDied","Data":"359c4a92f67f080e8144a397ff536807f8ad98c28965e8aff3f7a5ed541478a6"} Oct 30 17:21:47.549223 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:47.549165 4522 scope.go:122] "RemoveContainer" containerID="a8d72af0c8590b93f2e8150276dced3eeefa36c637b9fe8a35bac273dec88400" Oct 30 17:21:47.549369 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:47.549351 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d003a5-5cc689c7b6-z4b8q" Oct 30 17:21:47.566720 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:47.566696 4522 scope.go:122] "RemoveContainer" containerID="a8d72af0c8590b93f2e8150276dced3eeefa36c637b9fe8a35bac273dec88400" Oct 30 17:21:47.567034 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:21:47.567015 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8d72af0c8590b93f2e8150276dced3eeefa36c637b9fe8a35bac273dec88400\": container with ID starting with a8d72af0c8590b93f2e8150276dced3eeefa36c637b9fe8a35bac273dec88400 not found: ID does not exist" containerID="a8d72af0c8590b93f2e8150276dced3eeefa36c637b9fe8a35bac273dec88400" Oct 30 17:21:47.567091 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:47.567043 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8d72af0c8590b93f2e8150276dced3eeefa36c637b9fe8a35bac273dec88400"} err="failed to get container status \"a8d72af0c8590b93f2e8150276dced3eeefa36c637b9fe8a35bac273dec88400\": rpc error: code = NotFound desc = could not find container \"a8d72af0c8590b93f2e8150276dced3eeefa36c637b9fe8a35bac273dec88400\": container with ID starting with a8d72af0c8590b93f2e8150276dced3eeefa36c637b9fe8a35bac273dec88400 not found: ID does not exist" Oct 30 17:21:47.614380 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:47.614345 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d003a5-5cc689c7b6-z4b8q"] Oct 30 17:21:47.618407 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:47.618350 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d003a5-5cc689c7b6-z4b8q"] Oct 30 17:21:47.639518 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:47.639491 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sc96q\" (UniqueName: \"kubernetes.io/projected/8f8c9da3-072b-4c71-8cd9-be0dd174e99d-kube-api-access-sc96q\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:21:48.147791 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:48.147760 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f8c9da3-072b-4c71-8cd9-be0dd174e99d" path="/var/lib/kubelet/pods/8f8c9da3-072b-4c71-8cd9-be0dd174e99d/volumes" Oct 30 17:21:49.684961 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:49.684917 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-5d67ca-6c5bcd5578-n5lgb" podUID="75475e2a-de0d-429f-afc2-34714ef410a1" containerName="payload-processing" containerID="cri-o://104af81e4d837c87be3d47899e54a5a4cb672cb1a368f644e7ddf2c66027feed" gracePeriod=25 Oct 30 17:21:51.825322 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:51.825278 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f89e1d2-dcf7c"] Oct 30 17:21:51.826533 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:51.826514 4522 cpu_manager.go:414] "containerMap: removing container" podUID="1fc7054e-4f5c-4fdf-8b7a-d6561d562ae5" containerName="revoke-keys" Oct 30 17:21:51.826595 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:51.826535 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fc7054e-4f5c-4fdf-8b7a-d6561d562ae5" containerName="revoke-keys" Oct 30 17:21:51.826633 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:51.826606 4522 cpu_manager.go:414] "containerMap: removing container" podUID="8f8c9da3-072b-4c71-8cd9-be0dd174e99d" containerName="payload-processing" Oct 30 17:21:51.826633 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:51.826615 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f8c9da3-072b-4c71-8cd9-be0dd174e99d" containerName="payload-processing" Oct 30 17:21:51.826785 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:51.826773 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f8c9da3-072b-4c71-8cd9-be0dd174e99d" containerName="payload-processing" Oct 30 17:21:51.826825 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:51.826798 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fc7054e-4f5c-4fdf-8b7a-d6561d562ae5" containerName="revoke-keys" Oct 30 17:21:51.832549 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:51.832527 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f89e1d2-dcf7c" Oct 30 17:21:51.855090 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:51.855059 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f89e1d2-dcf7c"] Oct 30 17:21:51.862986 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:51.862918 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:21:51.863152 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:51.863032 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:21:52.014604 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:52.014562 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfc2s\" (UniqueName: \"kubernetes.io/projected/cc35c2ee-ed6b-4cf3-8bd9-0e13ed0ede13-kube-api-access-jfc2s\") pod \"maas-api-revoke-sub-8f89e1d2-dcf7c\" (UID: \"cc35c2ee-ed6b-4cf3-8bd9-0e13ed0ede13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f89e1d2-dcf7c" Oct 30 17:21:52.014785 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:52.014704 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc35c2ee-ed6b-4cf3-8bd9-0e13ed0ede13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f89e1d2-dcf7c\" (UID: \"cc35c2ee-ed6b-4cf3-8bd9-0e13ed0ede13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f89e1d2-dcf7c" Oct 30 17:21:52.014879 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:52.014863 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc35c2ee-ed6b-4cf3-8bd9-0e13ed0ede13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f89e1d2-dcf7c\" (UID: \"cc35c2ee-ed6b-4cf3-8bd9-0e13ed0ede13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f89e1d2-dcf7c" Oct 30 17:21:52.116756 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:52.116661 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfc2s\" (UniqueName: \"kubernetes.io/projected/cc35c2ee-ed6b-4cf3-8bd9-0e13ed0ede13-kube-api-access-jfc2s\") pod \"maas-api-revoke-sub-8f89e1d2-dcf7c\" (UID: \"cc35c2ee-ed6b-4cf3-8bd9-0e13ed0ede13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f89e1d2-dcf7c" Oct 30 17:21:52.116973 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:52.116765 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc35c2ee-ed6b-4cf3-8bd9-0e13ed0ede13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f89e1d2-dcf7c\" (UID: \"cc35c2ee-ed6b-4cf3-8bd9-0e13ed0ede13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f89e1d2-dcf7c" Oct 30 17:21:52.116973 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:52.116902 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc35c2ee-ed6b-4cf3-8bd9-0e13ed0ede13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f89e1d2-dcf7c\" (UID: \"cc35c2ee-ed6b-4cf3-8bd9-0e13ed0ede13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f89e1d2-dcf7c" Oct 30 17:21:52.117438 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:52.117416 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc35c2ee-ed6b-4cf3-8bd9-0e13ed0ede13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f89e1d2-dcf7c\" (UID: \"cc35c2ee-ed6b-4cf3-8bd9-0e13ed0ede13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f89e1d2-dcf7c" Oct 30 17:21:52.119286 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:52.119262 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc35c2ee-ed6b-4cf3-8bd9-0e13ed0ede13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f89e1d2-dcf7c\" (UID: \"cc35c2ee-ed6b-4cf3-8bd9-0e13ed0ede13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f89e1d2-dcf7c" Oct 30 17:21:52.142982 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:52.142947 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfc2s\" (UniqueName: \"kubernetes.io/projected/cc35c2ee-ed6b-4cf3-8bd9-0e13ed0ede13-kube-api-access-jfc2s\") pod \"maas-api-revoke-sub-8f89e1d2-dcf7c\" (UID: \"cc35c2ee-ed6b-4cf3-8bd9-0e13ed0ede13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f89e1d2-dcf7c" Oct 30 17:21:52.147674 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:52.147648 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f89e1d2-dcf7c" Oct 30 17:21:52.460237 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:21:52.460207 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc35c2ee_ed6b_4cf3_8bd9_0e13ed0ede13.slice/crio-04742e5323781fee1835466c69b8a970b6eb8f43860045b02ff25b60d3375a8e WatchSource:0}: Error finding container 04742e5323781fee1835466c69b8a970b6eb8f43860045b02ff25b60d3375a8e: Status 404 returned error can't find the container with id 04742e5323781fee1835466c69b8a970b6eb8f43860045b02ff25b60d3375a8e Oct 30 17:21:52.464905 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:52.464876 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f89e1d2-dcf7c"] Oct 30 17:21:52.595172 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:52.595139 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f89e1d2-dcf7c" event={"ID":"cc35c2ee-ed6b-4cf3-8bd9-0e13ed0ede13","Type":"ContainerStarted","Data":"04742e5323781fee1835466c69b8a970b6eb8f43860045b02ff25b60d3375a8e"} Oct 30 17:21:53.606997 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:53.606962 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f89e1d2-dcf7c" event={"ID":"cc35c2ee-ed6b-4cf3-8bd9-0e13ed0ede13","Type":"ContainerStarted","Data":"c76350e808b0571974a268287f88334bc42c818495753b7ff746a66aa92b8a36"} Oct 30 17:21:54.160570 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:54.160533 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9v2l"] Oct 30 17:21:54.160779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:54.160755 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9v2l" podUID="6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a" containerName="discovery" containerID="cri-o://efd44db602cacb6307d931fa4263aae0a699d6c56b6691862282811d4c651389" gracePeriod=30 Oct 30 17:21:54.468066 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:54.468040 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9v2l" Oct 30 17:21:54.472470 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:54.472437 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a-istio-csr-dns-cert\") pod \"6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a\" (UID: \"6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a\") " Oct 30 17:21:54.472622 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:54.472521 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a-istio-kubeconfig\") pod \"6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a\" (UID: \"6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a\") " Oct 30 17:21:54.472622 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:54.472564 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a-kube-api-access-d7bfx\" (UniqueName: \"kubernetes.io/projected/6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a-kube-api-access-d7bfx\") pod \"6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a\" (UID: \"6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a\") " Oct 30 17:21:54.472748 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:54.472646 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a-istio-token\" (UniqueName: \"kubernetes.io/projected/6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a-istio-token\") pod \"6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a\" (UID: \"6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a\") " Oct 30 17:21:54.472857 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:54.472840 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a-istio-csr-ca-configmap\") pod \"6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a\" (UID: \"6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a\") " Oct 30 17:21:54.472971 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:54.472957 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a-local-certs\") pod \"6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a\" (UID: \"6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a\") " Oct 30 17:21:54.473059 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:54.472988 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a-cacerts\" (UniqueName: \"kubernetes.io/secret/6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a-cacerts\") pod \"6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a\" (UID: \"6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a\") " Oct 30 17:21:54.473834 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:54.473324 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a-istio-csr-ca-configmap" pod "6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a" (UID: "6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:21:54.473834 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:54.473778 4522 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a-istio-csr-ca-configmap\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:21:54.476110 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:54.475333 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a-istio-token" pod "6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a" (UID: "6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:21:54.476110 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:54.475526 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a-istio-csr-dns-cert" pod "6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a" (UID: "6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:21:54.476110 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:54.475580 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a-local-certs" pod "6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a" (UID: "6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:54.476110 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:54.475846 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a-kube-api-access-d7bfx" pod "6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a" (UID: "6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a"). InnerVolumeSpecName "kube-api-access-d7bfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:21:54.476110 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:54.476032 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a-cacerts" pod "6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a" (UID: "6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:21:54.476799 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:54.476778 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a-istio-kubeconfig" pod "6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a" (UID: "6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:21:54.574842 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:54.574793 4522 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a-istio-kubeconfig\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:21:54.574842 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:54.574838 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7bfx\" (UniqueName: \"kubernetes.io/projected/6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a-kube-api-access-d7bfx\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:21:54.574842 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:54.574850 4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a-istio-token\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:21:54.574842 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:54.574859 4522 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a-local-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:21:54.574842 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:54.574867 4522 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a-cacerts\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:21:54.575289 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:54.574874 4522 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a-istio-csr-dns-cert\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:21:54.617703 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:54.617668 4522 generic.go:356] "Generic (PLEG): container finished" podID="cc35c2ee-ed6b-4cf3-8bd9-0e13ed0ede13" containerID="c76350e808b0571974a268287f88334bc42c818495753b7ff746a66aa92b8a36" exitCode=0 Oct 30 17:21:54.618118 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:54.617757 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f89e1d2-dcf7c" event={"ID":"cc35c2ee-ed6b-4cf3-8bd9-0e13ed0ede13","Type":"ContainerDied","Data":"c76350e808b0571974a268287f88334bc42c818495753b7ff746a66aa92b8a36"} Oct 30 17:21:54.619631 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:54.619605 4522 generic.go:356] "Generic (PLEG): container finished" podID="6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a" containerID="efd44db602cacb6307d931fa4263aae0a699d6c56b6691862282811d4c651389" exitCode=0 Oct 30 17:21:54.619754 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:54.619657 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9v2l" Oct 30 17:21:54.619754 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:54.619685 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9v2l" event={"ID":"6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a","Type":"ContainerDied","Data":"efd44db602cacb6307d931fa4263aae0a699d6c56b6691862282811d4c651389"} Oct 30 17:21:54.619754 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:54.619730 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9v2l" event={"ID":"6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a","Type":"ContainerDied","Data":"a5a20873ab3dc1319765403e58ab55e07da380043a91a92386b564c71230aca3"} Oct 30 17:21:54.619754 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:54.619754 4522 scope.go:122] "RemoveContainer" containerID="efd44db602cacb6307d931fa4263aae0a699d6c56b6691862282811d4c651389" Oct 30 17:21:54.643676 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:54.643655 4522 scope.go:122] "RemoveContainer" containerID="efd44db602cacb6307d931fa4263aae0a699d6c56b6691862282811d4c651389" Oct 30 17:21:54.644233 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:21:54.644138 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efd44db602cacb6307d931fa4263aae0a699d6c56b6691862282811d4c651389\": container with ID starting with efd44db602cacb6307d931fa4263aae0a699d6c56b6691862282811d4c651389 not found: ID does not exist" containerID="efd44db602cacb6307d931fa4263aae0a699d6c56b6691862282811d4c651389" Oct 30 17:21:54.644233 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:54.644194 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efd44db602cacb6307d931fa4263aae0a699d6c56b6691862282811d4c651389"} err="failed to get container status \"efd44db602cacb6307d931fa4263aae0a699d6c56b6691862282811d4c651389\": rpc error: code = NotFound desc = could not find container \"efd44db602cacb6307d931fa4263aae0a699d6c56b6691862282811d4c651389\": container with ID starting with efd44db602cacb6307d931fa4263aae0a699d6c56b6691862282811d4c651389 not found: ID does not exist" Oct 30 17:21:54.682079 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:54.681982 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9v2l"] Oct 30 17:21:54.686676 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:54.686650 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9v2l"] Oct 30 17:21:54.803907 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:54.803880 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f89e1d2-dcf7c" Oct 30 17:21:54.878916 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:54.878873 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc35c2ee-ed6b-4cf3-8bd9-0e13ed0ede13-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc35c2ee-ed6b-4cf3-8bd9-0e13ed0ede13-maas-api-xks-ca\") pod \"cc35c2ee-ed6b-4cf3-8bd9-0e13ed0ede13\" (UID: \"cc35c2ee-ed6b-4cf3-8bd9-0e13ed0ede13\") " Oct 30 17:21:54.879114 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:54.878988 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc35c2ee-ed6b-4cf3-8bd9-0e13ed0ede13-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc35c2ee-ed6b-4cf3-8bd9-0e13ed0ede13-maas-api-openshift-ca\") pod \"cc35c2ee-ed6b-4cf3-8bd9-0e13ed0ede13\" (UID: \"cc35c2ee-ed6b-4cf3-8bd9-0e13ed0ede13\") " Oct 30 17:21:54.879114 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:54.879087 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc35c2ee-ed6b-4cf3-8bd9-0e13ed0ede13-kube-api-access-jfc2s\" (UniqueName: \"kubernetes.io/projected/cc35c2ee-ed6b-4cf3-8bd9-0e13ed0ede13-kube-api-access-jfc2s\") pod \"cc35c2ee-ed6b-4cf3-8bd9-0e13ed0ede13\" (UID: \"cc35c2ee-ed6b-4cf3-8bd9-0e13ed0ede13\") " Oct 30 17:21:54.879463 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:54.879435 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc35c2ee-ed6b-4cf3-8bd9-0e13ed0ede13-maas-api-openshift-ca" pod "cc35c2ee-ed6b-4cf3-8bd9-0e13ed0ede13" (UID: "cc35c2ee-ed6b-4cf3-8bd9-0e13ed0ede13"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:21:54.879842 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:54.879824 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc35c2ee-ed6b-4cf3-8bd9-0e13ed0ede13-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:21:54.881126 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:54.881100 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc35c2ee-ed6b-4cf3-8bd9-0e13ed0ede13-maas-api-xks-ca" pod "cc35c2ee-ed6b-4cf3-8bd9-0e13ed0ede13" (UID: "cc35c2ee-ed6b-4cf3-8bd9-0e13ed0ede13"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:21:54.881226 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:54.881168 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc35c2ee-ed6b-4cf3-8bd9-0e13ed0ede13-kube-api-access-jfc2s" pod "cc35c2ee-ed6b-4cf3-8bd9-0e13ed0ede13" (UID: "cc35c2ee-ed6b-4cf3-8bd9-0e13ed0ede13"). InnerVolumeSpecName "kube-api-access-jfc2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:21:54.982225 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:54.982176 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfc2s\" (UniqueName: \"kubernetes.io/projected/cc35c2ee-ed6b-4cf3-8bd9-0e13ed0ede13-kube-api-access-jfc2s\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:21:54.982225 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:54.982226 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc35c2ee-ed6b-4cf3-8bd9-0e13ed0ede13-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:21:55.629664 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:55.629632 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f89e1d2-dcf7c" Oct 30 17:21:55.630121 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:55.629661 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f89e1d2-dcf7c" event={"ID":"cc35c2ee-ed6b-4cf3-8bd9-0e13ed0ede13","Type":"ContainerDied","Data":"04742e5323781fee1835466c69b8a970b6eb8f43860045b02ff25b60d3375a8e"} Oct 30 17:21:55.630121 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:55.629710 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04742e5323781fee1835466c69b8a970b6eb8f43860045b02ff25b60d3375a8e" Oct 30 17:21:56.150703 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:56.150673 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a" path="/var/lib/kubelet/pods/6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a/volumes" Oct 30 17:21:57.721530 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:21:57.721494 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-5d67ca-6c5bcd5578-n5lgb" podUID="75475e2a-de0d-429f-afc2-34714ef410a1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:22:00.628707 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:00.628674 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz"] Oct 30 17:22:00.629540 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:00.629523 4522 cpu_manager.go:414] "containerMap: removing container" podUID="cc35c2ee-ed6b-4cf3-8bd9-0e13ed0ede13" containerName="revoke-keys" Oct 30 17:22:00.629606 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:00.629540 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc35c2ee-ed6b-4cf3-8bd9-0e13ed0ede13" containerName="revoke-keys" Oct 30 17:22:00.629606 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:00.629575 4522 cpu_manager.go:414] "containerMap: removing container" podUID="6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a" containerName="discovery" Oct 30 17:22:00.629606 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:00.629580 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a" containerName="discovery" Oct 30 17:22:00.629731 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:00.629722 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ae5f037-50a6-4b57-b59e-b5bcbfb99f3a" containerName="discovery" Oct 30 17:22:00.629767 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:00.629740 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc35c2ee-ed6b-4cf3-8bd9-0e13ed0ede13" containerName="revoke-keys" Oct 30 17:22:00.634763 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:00.634744 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz" Oct 30 17:22:00.666880 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:00.666742 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-987dfc-openshift-default-dockercfg-ggt87\"" Oct 30 17:22:00.674110 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:00.674084 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz"] Oct 30 17:22:00.776759 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:00.776724 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm8q4\" (UniqueName: \"kubernetes.io/projected/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc-kube-api-access-sm8q4\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz\" (UID: \"b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz" Oct 30 17:22:00.776943 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:00.776771 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc-istio-data\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz\" (UID: \"b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz" Oct 30 17:22:00.776993 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:00.776945 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc-istiod-ca-cert\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz\" (UID: \"b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz" Oct 30 17:22:00.776993 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:00.776976 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc-workload-socket\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz\" (UID: \"b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz" Oct 30 17:22:00.777065 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:00.776996 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc-workload-certs\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz\" (UID: \"b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz" Oct 30 17:22:00.777123 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:00.777104 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc-credential-socket\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz\" (UID: \"b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz" Oct 30 17:22:00.777170 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:00.777143 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc-istio-token\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz\" (UID: \"b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz" Oct 30 17:22:00.777170 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:00.777165 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc-istio-envoy\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz\" (UID: \"b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz" Oct 30 17:22:00.777243 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:00.777210 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc-istio-podinfo\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz\" (UID: \"b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz" Oct 30 17:22:00.879169 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:00.879083 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm8q4\" (UniqueName: \"kubernetes.io/projected/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc-kube-api-access-sm8q4\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz\" (UID: \"b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz" Oct 30 17:22:00.879169 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:00.879128 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc-istio-data\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz\" (UID: \"b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz" Oct 30 17:22:00.879442 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:00.879211 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc-istiod-ca-cert\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz\" (UID: \"b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz" Oct 30 17:22:00.879442 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:00.879229 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc-workload-socket\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz\" (UID: \"b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz" Oct 30 17:22:00.879442 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:00.879248 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc-workload-certs\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz\" (UID: \"b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz" Oct 30 17:22:00.879442 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:00.879275 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc-credential-socket\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz\" (UID: \"b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz" Oct 30 17:22:00.879667 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:00.879441 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc-istio-token\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz\" (UID: \"b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz" Oct 30 17:22:00.879667 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:00.879487 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc-istio-envoy\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz\" (UID: \"b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz" Oct 30 17:22:00.879667 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:00.879512 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc-istio-podinfo\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz\" (UID: \"b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz" Oct 30 17:22:00.879667 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:00.879623 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc-istio-data\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz\" (UID: \"b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz" Oct 30 17:22:00.879667 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:00.879631 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc-credential-socket\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz\" (UID: \"b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz" Oct 30 17:22:00.879941 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:00.879687 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc-workload-socket\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz\" (UID: \"b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz" Oct 30 17:22:00.880014 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:00.879994 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc-workload-certs\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz\" (UID: \"b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz" Oct 30 17:22:00.880061 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:00.880046 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc-istiod-ca-cert\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz\" (UID: \"b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz" Oct 30 17:22:00.881798 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:00.881780 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc-istio-envoy\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz\" (UID: \"b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz" Oct 30 17:22:00.882005 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:00.881989 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc-istio-podinfo\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz\" (UID: \"b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz" Oct 30 17:22:00.905000 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:00.904969 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc-istio-token\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz\" (UID: \"b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz" Oct 30 17:22:00.905111 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:00.905095 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm8q4\" (UniqueName: \"kubernetes.io/projected/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc-kube-api-access-sm8q4\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz\" (UID: \"b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz" Oct 30 17:22:00.949571 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:00.949541 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz" Oct 30 17:22:01.290601 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:01.281848 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz"] Oct 30 17:22:01.290601 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:01.284129 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:22:01.290601 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:01.284208 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:22:01.290601 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:01.284247 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:22:01.701081 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:01.701043 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz" event={"ID":"b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc","Type":"ContainerStarted","Data":"7f2c75141c115bd014075858751b48d33608e1a5877f0ba2ede04d93a3c899ad"} Oct 30 17:22:01.701927 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:01.701111 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz" event={"ID":"b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc","Type":"ContainerStarted","Data":"4fff6a76f7727a7f279c29bc13adc1d9c007cec815984b225686a4af716f5e15"} Oct 30 17:22:01.753093 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:01.753037 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz" podStartSLOduration=1.7530214549999998 podStartE2EDuration="1.753021455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:22:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:22:01.748963492 +0000 UTC m=+2095.749718394" watchObservedRunningTime="2026-10-30 17:22:01.753021455 +0000 UTC m=+2095.753776355" Oct 30 17:22:01.951417 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:01.950580 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz" Oct 30 17:22:02.619389 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:02.619341 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d44f8491-5k49b"] Oct 30 17:22:02.624423 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:02.624398 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d44f8491-5k49b" Oct 30 17:22:02.634500 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:02.634468 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d44f8491-5k49b"] Oct 30 17:22:02.642594 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:02.641342 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:22:02.642594 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:02.641429 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:22:02.724737 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:02.724703 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df7wc\" (UniqueName: \"kubernetes.io/projected/fada1708-60a3-43d6-b7cc-8884c8e11341-kube-api-access-df7wc\") pod \"maas-api-revoke-sub-d44f8491-5k49b\" (UID: \"fada1708-60a3-43d6-b7cc-8884c8e11341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d44f8491-5k49b" Oct 30 17:22:02.725157 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:02.724753 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fada1708-60a3-43d6-b7cc-8884c8e11341-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d44f8491-5k49b\" (UID: \"fada1708-60a3-43d6-b7cc-8884c8e11341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d44f8491-5k49b" Oct 30 17:22:02.725157 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:02.724831 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fada1708-60a3-43d6-b7cc-8884c8e11341-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d44f8491-5k49b\" (UID: \"fada1708-60a3-43d6-b7cc-8884c8e11341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d44f8491-5k49b" Oct 30 17:22:02.827189 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:02.827149 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df7wc\" (UniqueName: \"kubernetes.io/projected/fada1708-60a3-43d6-b7cc-8884c8e11341-kube-api-access-df7wc\") pod \"maas-api-revoke-sub-d44f8491-5k49b\" (UID: \"fada1708-60a3-43d6-b7cc-8884c8e11341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d44f8491-5k49b" Oct 30 17:22:02.827423 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:02.827212 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fada1708-60a3-43d6-b7cc-8884c8e11341-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d44f8491-5k49b\" (UID: \"fada1708-60a3-43d6-b7cc-8884c8e11341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d44f8491-5k49b" Oct 30 17:22:02.827423 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:02.827298 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fada1708-60a3-43d6-b7cc-8884c8e11341-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d44f8491-5k49b\" (UID: \"fada1708-60a3-43d6-b7cc-8884c8e11341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d44f8491-5k49b" Oct 30 17:22:02.827950 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:02.827928 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fada1708-60a3-43d6-b7cc-8884c8e11341-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d44f8491-5k49b\" (UID: \"fada1708-60a3-43d6-b7cc-8884c8e11341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d44f8491-5k49b" Oct 30 17:22:02.830061 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:02.830039 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fada1708-60a3-43d6-b7cc-8884c8e11341-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d44f8491-5k49b\" (UID: \"fada1708-60a3-43d6-b7cc-8884c8e11341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d44f8491-5k49b" Oct 30 17:22:02.852244 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:02.852221 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-df7wc\" (UniqueName: \"kubernetes.io/projected/fada1708-60a3-43d6-b7cc-8884c8e11341-kube-api-access-df7wc\") pod \"maas-api-revoke-sub-d44f8491-5k49b\" (UID: \"fada1708-60a3-43d6-b7cc-8884c8e11341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d44f8491-5k49b" Oct 30 17:22:02.951637 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:02.951602 4522 patch_prober.go:29] interesting pod/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body= Oct 30 17:22:02.951810 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:02.951655 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz" podUID="b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" Oct 30 17:22:02.952839 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:02.952818 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d44f8491-5k49b" Oct 30 17:22:03.290044 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:03.290014 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d44f8491-5k49b"] Oct 30 17:22:03.290171 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:22:03.290055 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfada1708_60a3_43d6_b7cc_8884c8e11341.slice/crio-0cf4f4230134587ddc3692a7b69bf2e26c1a07e578e20fe7467ad14fc8c24193 WatchSource:0}: Error finding container 0cf4f4230134587ddc3692a7b69bf2e26c1a07e578e20fe7467ad14fc8c24193: Status 404 returned error can't find the container with id 0cf4f4230134587ddc3692a7b69bf2e26c1a07e578e20fe7467ad14fc8c24193 Oct 30 17:22:03.737200 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:03.737164 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d44f8491-5k49b" event={"ID":"fada1708-60a3-43d6-b7cc-8884c8e11341","Type":"ContainerStarted","Data":"0cf4f4230134587ddc3692a7b69bf2e26c1a07e578e20fe7467ad14fc8c24193"} Oct 30 17:22:03.951382 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:03.951348 4522 patch_prober.go:29] interesting pod/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body= Oct 30 17:22:03.951560 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:03.951404 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz" podUID="b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" Oct 30 17:22:04.747656 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:04.747620 4522 generic.go:356] "Generic (PLEG): container finished" podID="fada1708-60a3-43d6-b7cc-8884c8e11341" containerID="55e15fe15e61e17f1d74b22dd655c95b7253f1d198b15a4e84b472670e05a342" exitCode=0 Oct 30 17:22:04.748118 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:04.747723 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d44f8491-5k49b" event={"ID":"fada1708-60a3-43d6-b7cc-8884c8e11341","Type":"ContainerDied","Data":"55e15fe15e61e17f1d74b22dd655c95b7253f1d198b15a4e84b472670e05a342"} Oct 30 17:22:04.951930 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:04.951869 4522 patch_prober.go:29] interesting pod/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body= Oct 30 17:22:04.952157 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:04.951982 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz" podUID="b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" Oct 30 17:22:05.950735 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:05.950664 4522 patch_prober.go:29] interesting pod/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body= Oct 30 17:22:05.951869 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:05.951394 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz" podUID="b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" Oct 30 17:22:06.071611 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:06.071586 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d44f8491-5k49b" Oct 30 17:22:06.135136 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:06.135102 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fada1708-60a3-43d6-b7cc-8884c8e11341-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fada1708-60a3-43d6-b7cc-8884c8e11341-maas-api-xks-ca\") pod \"fada1708-60a3-43d6-b7cc-8884c8e11341\" (UID: \"fada1708-60a3-43d6-b7cc-8884c8e11341\") " Oct 30 17:22:06.135465 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:06.135173 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fada1708-60a3-43d6-b7cc-8884c8e11341-kube-api-access-df7wc\" (UniqueName: \"kubernetes.io/projected/fada1708-60a3-43d6-b7cc-8884c8e11341-kube-api-access-df7wc\") pod \"fada1708-60a3-43d6-b7cc-8884c8e11341\" (UID: \"fada1708-60a3-43d6-b7cc-8884c8e11341\") " Oct 30 17:22:06.135465 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:06.135263 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fada1708-60a3-43d6-b7cc-8884c8e11341-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fada1708-60a3-43d6-b7cc-8884c8e11341-maas-api-openshift-ca\") pod \"fada1708-60a3-43d6-b7cc-8884c8e11341\" (UID: \"fada1708-60a3-43d6-b7cc-8884c8e11341\") " Oct 30 17:22:06.136107 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:06.136083 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fada1708-60a3-43d6-b7cc-8884c8e11341-maas-api-openshift-ca" pod "fada1708-60a3-43d6-b7cc-8884c8e11341" (UID: "fada1708-60a3-43d6-b7cc-8884c8e11341"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:22:06.142450 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:06.141844 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fada1708-60a3-43d6-b7cc-8884c8e11341-maas-api-xks-ca" pod "fada1708-60a3-43d6-b7cc-8884c8e11341" (UID: "fada1708-60a3-43d6-b7cc-8884c8e11341"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:22:06.142450 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:06.142371 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fada1708-60a3-43d6-b7cc-8884c8e11341-kube-api-access-df7wc" pod "fada1708-60a3-43d6-b7cc-8884c8e11341" (UID: "fada1708-60a3-43d6-b7cc-8884c8e11341"). InnerVolumeSpecName "kube-api-access-df7wc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:22:06.177440 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:06.177409 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d003a5-n64mh"] Oct 30 17:22:06.183867 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:06.183841 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d003a5-n64mh"] Oct 30 17:22:06.237996 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:06.237961 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-df7wc\" (UniqueName: \"kubernetes.io/projected/fada1708-60a3-43d6-b7cc-8884c8e11341-kube-api-access-df7wc\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:22:06.238244 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:06.238221 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fada1708-60a3-43d6-b7cc-8884c8e11341-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:22:06.238374 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:06.238360 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fada1708-60a3-43d6-b7cc-8884c8e11341-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:22:06.774996 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:06.774965 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d44f8491-5k49b" event={"ID":"fada1708-60a3-43d6-b7cc-8884c8e11341","Type":"ContainerDied","Data":"0cf4f4230134587ddc3692a7b69bf2e26c1a07e578e20fe7467ad14fc8c24193"} Oct 30 17:22:06.774996 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:06.774997 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d44f8491-5k49b" Oct 30 17:22:06.775218 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:06.775011 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0cf4f4230134587ddc3692a7b69bf2e26c1a07e578e20fe7467ad14fc8c24193" Oct 30 17:22:06.949968 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:06.949929 4522 patch_prober.go:29] interesting pod/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body= Oct 30 17:22:06.950153 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:06.949984 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz" podUID="b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" Oct 30 17:22:07.721871 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:07.721832 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-5d67ca-6c5bcd5578-n5lgb" podUID="75475e2a-de0d-429f-afc2-34714ef410a1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:22:08.003042 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:08.002953 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz" Oct 30 17:22:08.003369 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:08.003343 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz" Oct 30 17:22:08.004219 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:08.004198 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz" Oct 30 17:22:08.180767 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:08.180690 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="949ded2a-41f3-4046-b83f-e2d7b3d13334" path="/var/lib/kubelet/pods/949ded2a-41f3-4046-b83f-e2d7b3d13334/volumes" Oct 30 17:22:09.087158 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:09.087126 4522 scope.go:122] "RemoveContainer" containerID="438806e0d021b193c6e01b30ef5233fa7cfb200b307f4f2e11a61590594e2faa" Oct 30 17:22:09.808923 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:09.808895 4522 generic.go:356] "Generic (PLEG): container finished" podID="75475e2a-de0d-429f-afc2-34714ef410a1" containerID="104af81e4d837c87be3d47899e54a5a4cb672cb1a368f644e7ddf2c66027feed" exitCode=0 Oct 30 17:22:09.809051 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:09.808952 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5d67ca-6c5bcd5578-n5lgb" event={"ID":"75475e2a-de0d-429f-afc2-34714ef410a1","Type":"ContainerDied","Data":"104af81e4d837c87be3d47899e54a5a4cb672cb1a368f644e7ddf2c66027feed"} Oct 30 17:22:09.999957 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:09.999932 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5d67ca-6c5bcd5578-n5lgb" Oct 30 17:22:10.104800 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:10.104767 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75475e2a-de0d-429f-afc2-34714ef410a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75475e2a-de0d-429f-afc2-34714ef410a1-plugins-config-volume\") pod \"75475e2a-de0d-429f-afc2-34714ef410a1\" (UID: \"75475e2a-de0d-429f-afc2-34714ef410a1\") " Oct 30 17:22:10.105192 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:10.105081 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75475e2a-de0d-429f-afc2-34714ef410a1-kube-api-access-jjgkr\" (UniqueName: \"kubernetes.io/projected/75475e2a-de0d-429f-afc2-34714ef410a1-kube-api-access-jjgkr\") pod \"75475e2a-de0d-429f-afc2-34714ef410a1\" (UID: \"75475e2a-de0d-429f-afc2-34714ef410a1\") " Oct 30 17:22:10.105192 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:10.105157 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75475e2a-de0d-429f-afc2-34714ef410a1-plugins-config-volume" pod "75475e2a-de0d-429f-afc2-34714ef410a1" (UID: "75475e2a-de0d-429f-afc2-34714ef410a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:22:10.105542 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:10.105519 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75475e2a-de0d-429f-afc2-34714ef410a1-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:22:10.107211 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:10.107184 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75475e2a-de0d-429f-afc2-34714ef410a1-kube-api-access-jjgkr" pod "75475e2a-de0d-429f-afc2-34714ef410a1" (UID: "75475e2a-de0d-429f-afc2-34714ef410a1"). InnerVolumeSpecName "kube-api-access-jjgkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:22:10.206838 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:10.206803 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjgkr\" (UniqueName: \"kubernetes.io/projected/75475e2a-de0d-429f-afc2-34714ef410a1-kube-api-access-jjgkr\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:22:10.818862 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:10.818831 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5d67ca-6c5bcd5578-n5lgb" Oct 30 17:22:10.818862 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:10.818859 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5d67ca-6c5bcd5578-n5lgb" event={"ID":"75475e2a-de0d-429f-afc2-34714ef410a1","Type":"ContainerDied","Data":"57e908bf7bb4bd3bcac9165520307a1a5a88893c9afbe54a0b63442af4bce08f"} Oct 30 17:22:10.819131 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:10.818927 4522 scope.go:122] "RemoveContainer" containerID="104af81e4d837c87be3d47899e54a5a4cb672cb1a368f644e7ddf2c66027feed" Oct 30 17:22:10.877595 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:10.877561 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5d67ca-6c5bcd5578-n5lgb"] Oct 30 17:22:10.882072 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:10.882042 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5d67ca-6c5bcd5578-n5lgb"] Oct 30 17:22:11.420950 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:11.420915 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb"] Oct 30 17:22:11.421475 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:11.421184 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb" podUID="534aeebe-4edf-425e-a16c-712cc498154b" containerName="istio-proxy" containerID="cri-o://87822abdd74a876394d196ef8960d7bb1d259403aac446551fb7fc8fe3b35e9e" gracePeriod=30 Oct 30 17:22:12.147560 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:12.147526 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75475e2a-de0d-429f-afc2-34714ef410a1" path="/var/lib/kubelet/pods/75475e2a-de0d-429f-afc2-34714ef410a1/volumes" Oct 30 17:22:12.951235 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:22:12.951199 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:22:13.904354 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:13.904320 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p"] Oct 30 17:22:13.904685 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:13.904649 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p" podUID="d30e44c6-cafd-4b54-9784-2f3743de61dc" containerName="istio-proxy" containerID="cri-o://9284d9134005736d19ff6360e76307560b38dfa730f4a784569011b9498b2e78" gracePeriod=30 Oct 30 17:22:14.445742 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:14.445697 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c82666-t9m8f"] Oct 30 17:22:14.446561 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:14.446543 4522 cpu_manager.go:414] "containerMap: removing container" podUID="fada1708-60a3-43d6-b7cc-8884c8e11341" containerName="revoke-keys" Oct 30 17:22:14.446561 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:14.446560 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fada1708-60a3-43d6-b7cc-8884c8e11341" containerName="revoke-keys" Oct 30 17:22:14.446675 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:14.446597 4522 cpu_manager.go:414] "containerMap: removing container" podUID="75475e2a-de0d-429f-afc2-34714ef410a1" containerName="payload-processing" Oct 30 17:22:14.446675 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:14.446603 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75475e2a-de0d-429f-afc2-34714ef410a1" containerName="payload-processing" Oct 30 17:22:14.446745 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:14.446719 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="75475e2a-de0d-429f-afc2-34714ef410a1" containerName="payload-processing" Oct 30 17:22:14.446745 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:14.446734 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="fada1708-60a3-43d6-b7cc-8884c8e11341" containerName="revoke-keys" Oct 30 17:22:14.452212 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:14.452193 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c82666-t9m8f" Oct 30 17:22:14.470483 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:14.470453 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c82666-t9m8f"] Oct 30 17:22:14.533712 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:14.533674 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:22:14.533888 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:14.533727 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:22:14.585284 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:14.585242 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkz72\" (UniqueName: \"kubernetes.io/projected/6fa345e3-dbb2-4e58-bb92-6dcc9adcd63c-kube-api-access-dkz72\") pod \"maas-api-revoke-keys-e2e-inf-b-c82666-t9m8f\" (UID: \"6fa345e3-dbb2-4e58-bb92-6dcc9adcd63c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c82666-t9m8f" Oct 30 17:22:14.585594 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:14.585573 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fa345e3-dbb2-4e58-bb92-6dcc9adcd63c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c82666-t9m8f\" (UID: \"6fa345e3-dbb2-4e58-bb92-6dcc9adcd63c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c82666-t9m8f" Oct 30 17:22:14.585675 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:14.585640 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fa345e3-dbb2-4e58-bb92-6dcc9adcd63c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c82666-t9m8f\" (UID: \"6fa345e3-dbb2-4e58-bb92-6dcc9adcd63c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c82666-t9m8f" Oct 30 17:22:14.686764 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:14.686723 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fa345e3-dbb2-4e58-bb92-6dcc9adcd63c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c82666-t9m8f\" (UID: \"6fa345e3-dbb2-4e58-bb92-6dcc9adcd63c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c82666-t9m8f" Oct 30 17:22:14.686940 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:14.686830 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkz72\" (UniqueName: \"kubernetes.io/projected/6fa345e3-dbb2-4e58-bb92-6dcc9adcd63c-kube-api-access-dkz72\") pod \"maas-api-revoke-keys-e2e-inf-b-c82666-t9m8f\" (UID: \"6fa345e3-dbb2-4e58-bb92-6dcc9adcd63c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c82666-t9m8f" Oct 30 17:22:14.687136 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:14.687116 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fa345e3-dbb2-4e58-bb92-6dcc9adcd63c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c82666-t9m8f\" (UID: \"6fa345e3-dbb2-4e58-bb92-6dcc9adcd63c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c82666-t9m8f" Oct 30 17:22:14.687976 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:14.687956 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fa345e3-dbb2-4e58-bb92-6dcc9adcd63c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c82666-t9m8f\" (UID: \"6fa345e3-dbb2-4e58-bb92-6dcc9adcd63c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c82666-t9m8f" Oct 30 17:22:14.689238 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:14.689216 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fa345e3-dbb2-4e58-bb92-6dcc9adcd63c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c82666-t9m8f\" (UID: \"6fa345e3-dbb2-4e58-bb92-6dcc9adcd63c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c82666-t9m8f" Oct 30 17:22:14.711835 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:14.711806 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkz72\" (UniqueName: \"kubernetes.io/projected/6fa345e3-dbb2-4e58-bb92-6dcc9adcd63c-kube-api-access-dkz72\") pod \"maas-api-revoke-keys-e2e-inf-b-c82666-t9m8f\" (UID: \"6fa345e3-dbb2-4e58-bb92-6dcc9adcd63c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c82666-t9m8f" Oct 30 17:22:14.767105 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:14.767075 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c82666-t9m8f" Oct 30 17:22:15.086771 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:15.086163 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c82666-t9m8f"] Oct 30 17:22:15.089564 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:22:15.089534 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fa345e3_dbb2_4e58_bb92_6dcc9adcd63c.slice/crio-ac79052b9e986eca5d315abf94289233302add79b380474f6d37115ffea4a3f8 WatchSource:0}: Error finding container ac79052b9e986eca5d315abf94289233302add79b380474f6d37115ffea4a3f8: Status 404 returned error can't find the container with id ac79052b9e986eca5d315abf94289233302add79b380474f6d37115ffea4a3f8 Oct 30 17:22:15.833858 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:15.833815 4522 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:22:15.834258 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:15.833882 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb" podUID="534aeebe-4edf-425e-a16c-712cc498154b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:22:15.877964 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:15.877928 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c82666-t9m8f" event={"ID":"6fa345e3-dbb2-4e58-bb92-6dcc9adcd63c","Type":"ContainerStarted","Data":"eeb6730f9a6bb710f6c16e9cd118989127194b6b0be57a2883ef2ada20333088"} Oct 30 17:22:15.878161 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:15.878013 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c82666-t9m8f" event={"ID":"6fa345e3-dbb2-4e58-bb92-6dcc9adcd63c","Type":"ContainerStarted","Data":"ac79052b9e986eca5d315abf94289233302add79b380474f6d37115ffea4a3f8"} Oct 30 17:22:15.924163 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:15.924112 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c82666-t9m8f" podStartSLOduration=1.924098018 podStartE2EDuration="1.924098018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:22:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:22:15.92053539 +0000 UTC m=+2109.921290309" watchObservedRunningTime="2026-10-30 17:22:15.924098018 +0000 UTC m=+2109.924852918" Oct 30 17:22:16.706777 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:16.706753 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb" Oct 30 17:22:16.824607 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:16.824566 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/534aeebe-4edf-425e-a16c-712cc498154b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/534aeebe-4edf-425e-a16c-712cc498154b-istio-envoy\") pod \"534aeebe-4edf-425e-a16c-712cc498154b\" (UID: \"534aeebe-4edf-425e-a16c-712cc498154b\") " Oct 30 17:22:16.824818 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:16.824622 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/534aeebe-4edf-425e-a16c-712cc498154b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/534aeebe-4edf-425e-a16c-712cc498154b-istio-podinfo\") pod \"534aeebe-4edf-425e-a16c-712cc498154b\" (UID: \"534aeebe-4edf-425e-a16c-712cc498154b\") " Oct 30 17:22:16.824818 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:16.824666 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/534aeebe-4edf-425e-a16c-712cc498154b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/534aeebe-4edf-425e-a16c-712cc498154b-workload-certs\") pod \"534aeebe-4edf-425e-a16c-712cc498154b\" (UID: \"534aeebe-4edf-425e-a16c-712cc498154b\") " Oct 30 17:22:16.824818 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:16.824712 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/534aeebe-4edf-425e-a16c-712cc498154b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/534aeebe-4edf-425e-a16c-712cc498154b-credential-socket\") pod \"534aeebe-4edf-425e-a16c-712cc498154b\" (UID: \"534aeebe-4edf-425e-a16c-712cc498154b\") " Oct 30 17:22:16.825010 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:16.824857 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/534aeebe-4edf-425e-a16c-712cc498154b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/534aeebe-4edf-425e-a16c-712cc498154b-workload-socket\") pod \"534aeebe-4edf-425e-a16c-712cc498154b\" (UID: \"534aeebe-4edf-425e-a16c-712cc498154b\") " Oct 30 17:22:16.825010 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:16.824892 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/534aeebe-4edf-425e-a16c-712cc498154b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/534aeebe-4edf-425e-a16c-712cc498154b-istio-data\") pod \"534aeebe-4edf-425e-a16c-712cc498154b\" (UID: \"534aeebe-4edf-425e-a16c-712cc498154b\") " Oct 30 17:22:16.825141 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:16.825088 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/534aeebe-4edf-425e-a16c-712cc498154b-istio-token\" (UniqueName: \"kubernetes.io/projected/534aeebe-4edf-425e-a16c-712cc498154b-istio-token\") pod \"534aeebe-4edf-425e-a16c-712cc498154b\" (UID: \"534aeebe-4edf-425e-a16c-712cc498154b\") " Oct 30 17:22:16.825141 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:16.825129 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/534aeebe-4edf-425e-a16c-712cc498154b-kube-api-access-29tcv\" (UniqueName: \"kubernetes.io/projected/534aeebe-4edf-425e-a16c-712cc498154b-kube-api-access-29tcv\") pod \"534aeebe-4edf-425e-a16c-712cc498154b\" (UID: \"534aeebe-4edf-425e-a16c-712cc498154b\") " Oct 30 17:22:16.825246 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:16.825164 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/534aeebe-4edf-425e-a16c-712cc498154b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/534aeebe-4edf-425e-a16c-712cc498154b-istiod-ca-cert\") pod \"534aeebe-4edf-425e-a16c-712cc498154b\" (UID: \"534aeebe-4edf-425e-a16c-712cc498154b\") " Oct 30 17:22:16.825777 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:16.825743 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/534aeebe-4edf-425e-a16c-712cc498154b-credential-socket" pod "534aeebe-4edf-425e-a16c-712cc498154b" (UID: "534aeebe-4edf-425e-a16c-712cc498154b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:22:16.826116 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:16.826089 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/534aeebe-4edf-425e-a16c-712cc498154b-istiod-ca-cert" pod "534aeebe-4edf-425e-a16c-712cc498154b" (UID: "534aeebe-4edf-425e-a16c-712cc498154b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:22:16.826246 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:16.826211 4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/534aeebe-4edf-425e-a16c-712cc498154b-credential-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:22:16.827440 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:16.827404 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/534aeebe-4edf-425e-a16c-712cc498154b-istio-data" pod "534aeebe-4edf-425e-a16c-712cc498154b" (UID: "534aeebe-4edf-425e-a16c-712cc498154b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:22:16.827639 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:16.827612 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/534aeebe-4edf-425e-a16c-712cc498154b-istio-envoy" pod "534aeebe-4edf-425e-a16c-712cc498154b" (UID: "534aeebe-4edf-425e-a16c-712cc498154b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:22:16.828530 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:16.828505 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/534aeebe-4edf-425e-a16c-712cc498154b-workload-certs" pod "534aeebe-4edf-425e-a16c-712cc498154b" (UID: "534aeebe-4edf-425e-a16c-712cc498154b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:22:16.828713 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:16.828690 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/534aeebe-4edf-425e-a16c-712cc498154b-workload-socket" pod "534aeebe-4edf-425e-a16c-712cc498154b" (UID: "534aeebe-4edf-425e-a16c-712cc498154b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:22:16.829104 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:16.829080 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/534aeebe-4edf-425e-a16c-712cc498154b-istio-podinfo" pod "534aeebe-4edf-425e-a16c-712cc498154b" (UID: "534aeebe-4edf-425e-a16c-712cc498154b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:22:16.831017 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:16.830994 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/534aeebe-4edf-425e-a16c-712cc498154b-kube-api-access-29tcv" pod "534aeebe-4edf-425e-a16c-712cc498154b" (UID: "534aeebe-4edf-425e-a16c-712cc498154b"). InnerVolumeSpecName "kube-api-access-29tcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:22:16.832520 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:16.832499 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/534aeebe-4edf-425e-a16c-712cc498154b-istio-token" pod "534aeebe-4edf-425e-a16c-712cc498154b" (UID: "534aeebe-4edf-425e-a16c-712cc498154b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:22:16.891982 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:16.891949 4522 generic.go:356] "Generic (PLEG): container finished" podID="534aeebe-4edf-425e-a16c-712cc498154b" containerID="87822abdd74a876394d196ef8960d7bb1d259403aac446551fb7fc8fe3b35e9e" exitCode=0 Oct 30 17:22:16.892514 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:16.891999 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb" Oct 30 17:22:16.892514 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:16.892031 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb" event={"ID":"534aeebe-4edf-425e-a16c-712cc498154b","Type":"ContainerDied","Data":"87822abdd74a876394d196ef8960d7bb1d259403aac446551fb7fc8fe3b35e9e"} Oct 30 17:22:16.892514 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:16.892087 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb" event={"ID":"534aeebe-4edf-425e-a16c-712cc498154b","Type":"ContainerDied","Data":"25677683e9a4cd4b17e3ad740bc10af5d6c5e4482c96a58783f04cb5438479d9"} Oct 30 17:22:16.892514 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:16.892109 4522 scope.go:122] "RemoveContainer" containerID="87822abdd74a876394d196ef8960d7bb1d259403aac446551fb7fc8fe3b35e9e" Oct 30 17:22:16.893939 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:16.893915 4522 generic.go:356] "Generic (PLEG): container finished" podID="6fa345e3-dbb2-4e58-bb92-6dcc9adcd63c" containerID="eeb6730f9a6bb710f6c16e9cd118989127194b6b0be57a2883ef2ada20333088" exitCode=0 Oct 30 17:22:16.894073 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:16.893950 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c82666-t9m8f" event={"ID":"6fa345e3-dbb2-4e58-bb92-6dcc9adcd63c","Type":"ContainerDied","Data":"eeb6730f9a6bb710f6c16e9cd118989127194b6b0be57a2883ef2ada20333088"} Oct 30 17:22:16.904278 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:16.904261 4522 scope.go:122] "RemoveContainer" containerID="87822abdd74a876394d196ef8960d7bb1d259403aac446551fb7fc8fe3b35e9e" Oct 30 17:22:16.904606 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:22:16.904584 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87822abdd74a876394d196ef8960d7bb1d259403aac446551fb7fc8fe3b35e9e\": container with ID starting with 87822abdd74a876394d196ef8960d7bb1d259403aac446551fb7fc8fe3b35e9e not found: ID does not exist" containerID="87822abdd74a876394d196ef8960d7bb1d259403aac446551fb7fc8fe3b35e9e" Oct 30 17:22:16.904685 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:16.904614 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87822abdd74a876394d196ef8960d7bb1d259403aac446551fb7fc8fe3b35e9e"} err="failed to get container status \"87822abdd74a876394d196ef8960d7bb1d259403aac446551fb7fc8fe3b35e9e\": rpc error: code = NotFound desc = could not find container \"87822abdd74a876394d196ef8960d7bb1d259403aac446551fb7fc8fe3b35e9e\": container with ID starting with 87822abdd74a876394d196ef8960d7bb1d259403aac446551fb7fc8fe3b35e9e not found: ID does not exist" Oct 30 17:22:16.927601 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:16.927575 4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/534aeebe-4edf-425e-a16c-712cc498154b-istio-envoy\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:22:16.927601 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:16.927597 4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/534aeebe-4edf-425e-a16c-712cc498154b-istio-podinfo\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:22:16.927771 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:16.927606 4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/534aeebe-4edf-425e-a16c-712cc498154b-workload-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:22:16.927771 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:16.927629 4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/534aeebe-4edf-425e-a16c-712cc498154b-workload-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:22:16.927771 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:16.927643 4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/534aeebe-4edf-425e-a16c-712cc498154b-istio-data\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:22:16.927771 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:16.927651 4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/534aeebe-4edf-425e-a16c-712cc498154b-istio-token\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:22:16.927771 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:16.927660 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29tcv\" (UniqueName: \"kubernetes.io/projected/534aeebe-4edf-425e-a16c-712cc498154b-kube-api-access-29tcv\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:22:16.927771 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:16.927669 4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/534aeebe-4edf-425e-a16c-712cc498154b-istiod-ca-cert\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:22:16.996441 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:16.996361 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb"] Oct 30 17:22:17.000830 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:17.000801 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-9h6sb"] Oct 30 17:22:18.075692 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:18.075664 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c82666-t9m8f" Oct 30 17:22:18.149515 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:18.149470 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="534aeebe-4edf-425e-a16c-712cc498154b" path="/var/lib/kubelet/pods/534aeebe-4edf-425e-a16c-712cc498154b/volumes" Oct 30 17:22:18.249077 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:18.249039 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fa345e3-dbb2-4e58-bb92-6dcc9adcd63c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fa345e3-dbb2-4e58-bb92-6dcc9adcd63c-maas-api-openshift-ca\") pod \"6fa345e3-dbb2-4e58-bb92-6dcc9adcd63c\" (UID: \"6fa345e3-dbb2-4e58-bb92-6dcc9adcd63c\") " Oct 30 17:22:18.249285 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:18.249236 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fa345e3-dbb2-4e58-bb92-6dcc9adcd63c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fa345e3-dbb2-4e58-bb92-6dcc9adcd63c-maas-api-xks-ca\") pod \"6fa345e3-dbb2-4e58-bb92-6dcc9adcd63c\" (UID: \"6fa345e3-dbb2-4e58-bb92-6dcc9adcd63c\") " Oct 30 17:22:18.249377 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:18.249348 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fa345e3-dbb2-4e58-bb92-6dcc9adcd63c-kube-api-access-dkz72\" (UniqueName: \"kubernetes.io/projected/6fa345e3-dbb2-4e58-bb92-6dcc9adcd63c-kube-api-access-dkz72\") pod \"6fa345e3-dbb2-4e58-bb92-6dcc9adcd63c\" (UID: \"6fa345e3-dbb2-4e58-bb92-6dcc9adcd63c\") " Oct 30 17:22:18.249439 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:18.249417 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fa345e3-dbb2-4e58-bb92-6dcc9adcd63c-maas-api-openshift-ca" pod "6fa345e3-dbb2-4e58-bb92-6dcc9adcd63c" (UID: "6fa345e3-dbb2-4e58-bb92-6dcc9adcd63c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:22:18.250074 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:18.250053 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fa345e3-dbb2-4e58-bb92-6dcc9adcd63c-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:22:18.251654 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:18.251621 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fa345e3-dbb2-4e58-bb92-6dcc9adcd63c-kube-api-access-dkz72" pod "6fa345e3-dbb2-4e58-bb92-6dcc9adcd63c" (UID: "6fa345e3-dbb2-4e58-bb92-6dcc9adcd63c"). InnerVolumeSpecName "kube-api-access-dkz72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:22:18.251784 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:18.251736 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fa345e3-dbb2-4e58-bb92-6dcc9adcd63c-maas-api-xks-ca" pod "6fa345e3-dbb2-4e58-bb92-6dcc9adcd63c" (UID: "6fa345e3-dbb2-4e58-bb92-6dcc9adcd63c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:22:18.352094 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:18.352062 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fa345e3-dbb2-4e58-bb92-6dcc9adcd63c-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:22:18.352094 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:18.352090 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkz72\" (UniqueName: \"kubernetes.io/projected/6fa345e3-dbb2-4e58-bb92-6dcc9adcd63c-kube-api-access-dkz72\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:22:18.916152 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:18.916073 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c82666-t9m8f" Oct 30 17:22:18.916152 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:18.916101 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c82666-t9m8f" event={"ID":"6fa345e3-dbb2-4e58-bb92-6dcc9adcd63c","Type":"ContainerDied","Data":"ac79052b9e986eca5d315abf94289233302add79b380474f6d37115ffea4a3f8"} Oct 30 17:22:18.916408 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:18.916195 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac79052b9e986eca5d315abf94289233302add79b380474f6d37115ffea4a3f8" Oct 30 17:22:19.207058 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:19.207031 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p" Oct 30 17:22:19.268280 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:19.268247 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d30e44c6-cafd-4b54-9784-2f3743de61dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d30e44c6-cafd-4b54-9784-2f3743de61dc-istio-envoy\") pod \"d30e44c6-cafd-4b54-9784-2f3743de61dc\" (UID: \"d30e44c6-cafd-4b54-9784-2f3743de61dc\") " Oct 30 17:22:19.268509 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:19.268400 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d30e44c6-cafd-4b54-9784-2f3743de61dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d30e44c6-cafd-4b54-9784-2f3743de61dc-istio-podinfo\") pod \"d30e44c6-cafd-4b54-9784-2f3743de61dc\" (UID: \"d30e44c6-cafd-4b54-9784-2f3743de61dc\") " Oct 30 17:22:19.268509 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:19.268454 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d30e44c6-cafd-4b54-9784-2f3743de61dc-kube-api-access-smpgj\" (UniqueName: \"kubernetes.io/projected/d30e44c6-cafd-4b54-9784-2f3743de61dc-kube-api-access-smpgj\") pod \"d30e44c6-cafd-4b54-9784-2f3743de61dc\" (UID: \"d30e44c6-cafd-4b54-9784-2f3743de61dc\") " Oct 30 17:22:19.268654 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:19.268518 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d30e44c6-cafd-4b54-9784-2f3743de61dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d30e44c6-cafd-4b54-9784-2f3743de61dc-istio-data\") pod \"d30e44c6-cafd-4b54-9784-2f3743de61dc\" (UID: \"d30e44c6-cafd-4b54-9784-2f3743de61dc\") " Oct 30 17:22:19.268654 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:19.268567 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d30e44c6-cafd-4b54-9784-2f3743de61dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d30e44c6-cafd-4b54-9784-2f3743de61dc-istiod-ca-cert\") pod \"d30e44c6-cafd-4b54-9784-2f3743de61dc\" (UID: \"d30e44c6-cafd-4b54-9784-2f3743de61dc\") " Oct 30 17:22:19.268780 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:19.268758 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d30e44c6-cafd-4b54-9784-2f3743de61dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d30e44c6-cafd-4b54-9784-2f3743de61dc-credential-socket\") pod \"d30e44c6-cafd-4b54-9784-2f3743de61dc\" (UID: \"d30e44c6-cafd-4b54-9784-2f3743de61dc\") " Oct 30 17:22:19.268888 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:19.268874 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d30e44c6-cafd-4b54-9784-2f3743de61dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d30e44c6-cafd-4b54-9784-2f3743de61dc-workload-certs\") pod \"d30e44c6-cafd-4b54-9784-2f3743de61dc\" (UID: \"d30e44c6-cafd-4b54-9784-2f3743de61dc\") " Oct 30 17:22:19.268961 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:19.268949 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d30e44c6-cafd-4b54-9784-2f3743de61dc-istio-token\" (UniqueName: \"kubernetes.io/projected/d30e44c6-cafd-4b54-9784-2f3743de61dc-istio-token\") pod \"d30e44c6-cafd-4b54-9784-2f3743de61dc\" (UID: \"d30e44c6-cafd-4b54-9784-2f3743de61dc\") " Oct 30 17:22:19.269027 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:19.268987 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d30e44c6-cafd-4b54-9784-2f3743de61dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d30e44c6-cafd-4b54-9784-2f3743de61dc-workload-socket\") pod \"d30e44c6-cafd-4b54-9784-2f3743de61dc\" (UID: \"d30e44c6-cafd-4b54-9784-2f3743de61dc\") " Oct 30 17:22:19.269339 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:19.269291 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d30e44c6-cafd-4b54-9784-2f3743de61dc-credential-socket" pod "d30e44c6-cafd-4b54-9784-2f3743de61dc" (UID: "d30e44c6-cafd-4b54-9784-2f3743de61dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:22:19.269463 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:19.269355 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d30e44c6-cafd-4b54-9784-2f3743de61dc-istio-data" pod "d30e44c6-cafd-4b54-9784-2f3743de61dc" (UID: "d30e44c6-cafd-4b54-9784-2f3743de61dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:22:19.269596 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:19.269577 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d30e44c6-cafd-4b54-9784-2f3743de61dc-workload-certs" pod "d30e44c6-cafd-4b54-9784-2f3743de61dc" (UID: "d30e44c6-cafd-4b54-9784-2f3743de61dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:22:19.269795 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:19.269770 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d30e44c6-cafd-4b54-9784-2f3743de61dc-workload-socket" pod "d30e44c6-cafd-4b54-9784-2f3743de61dc" (UID: "d30e44c6-cafd-4b54-9784-2f3743de61dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:22:19.270119 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:19.270100 4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d30e44c6-cafd-4b54-9784-2f3743de61dc-workload-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:22:19.270301 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:19.270287 4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d30e44c6-cafd-4b54-9784-2f3743de61dc-workload-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:22:19.270422 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:19.270407 4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d30e44c6-cafd-4b54-9784-2f3743de61dc-istio-data\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:22:19.270551 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:19.270534 4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d30e44c6-cafd-4b54-9784-2f3743de61dc-credential-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:22:19.270644 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:19.270171 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d30e44c6-cafd-4b54-9784-2f3743de61dc-istiod-ca-cert" pod "d30e44c6-cafd-4b54-9784-2f3743de61dc" (UID: "d30e44c6-cafd-4b54-9784-2f3743de61dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:22:19.274191 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:19.274159 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d30e44c6-cafd-4b54-9784-2f3743de61dc-istio-token" pod "d30e44c6-cafd-4b54-9784-2f3743de61dc" (UID: "d30e44c6-cafd-4b54-9784-2f3743de61dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:22:19.282451 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:19.282417 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d30e44c6-cafd-4b54-9784-2f3743de61dc-kube-api-access-smpgj" pod "d30e44c6-cafd-4b54-9784-2f3743de61dc" (UID: "d30e44c6-cafd-4b54-9784-2f3743de61dc"). InnerVolumeSpecName "kube-api-access-smpgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:22:19.287787 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:19.287759 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d30e44c6-cafd-4b54-9784-2f3743de61dc-istio-envoy" pod "d30e44c6-cafd-4b54-9784-2f3743de61dc" (UID: "d30e44c6-cafd-4b54-9784-2f3743de61dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:22:19.302730 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:19.302703 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d30e44c6-cafd-4b54-9784-2f3743de61dc-istio-podinfo" pod "d30e44c6-cafd-4b54-9784-2f3743de61dc" (UID: "d30e44c6-cafd-4b54-9784-2f3743de61dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:22:19.372668 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:19.372633 4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d30e44c6-cafd-4b54-9784-2f3743de61dc-istio-token\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:22:19.372668 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:19.372663 4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d30e44c6-cafd-4b54-9784-2f3743de61dc-istio-envoy\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:22:19.372668 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:19.372672 4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d30e44c6-cafd-4b54-9784-2f3743de61dc-istio-podinfo\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:22:19.372927 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:19.372680 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smpgj\" (UniqueName: \"kubernetes.io/projected/d30e44c6-cafd-4b54-9784-2f3743de61dc-kube-api-access-smpgj\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:22:19.372927 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:19.372692 4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d30e44c6-cafd-4b54-9784-2f3743de61dc-istiod-ca-cert\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:22:19.931094 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:19.931063 4522 generic.go:356] "Generic (PLEG): container finished" podID="d30e44c6-cafd-4b54-9784-2f3743de61dc" containerID="9284d9134005736d19ff6360e76307560b38dfa730f4a784569011b9498b2e78" exitCode=0 Oct 30 17:22:19.931275 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:19.931250 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p" event={"ID":"d30e44c6-cafd-4b54-9784-2f3743de61dc","Type":"ContainerDied","Data":"9284d9134005736d19ff6360e76307560b38dfa730f4a784569011b9498b2e78"} Oct 30 17:22:19.931369 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:19.931291 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p" event={"ID":"d30e44c6-cafd-4b54-9784-2f3743de61dc","Type":"ContainerDied","Data":"a1940e1ec998aa031ab9b824c71cefbac52cb3252aa30b173124fdab08f6f766"} Oct 30 17:22:19.931369 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:19.931334 4522 scope.go:122] "RemoveContainer" containerID="9284d9134005736d19ff6360e76307560b38dfa730f4a784569011b9498b2e78" Oct 30 17:22:19.931499 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:19.931486 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p" Oct 30 17:22:19.947525 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:19.947503 4522 scope.go:122] "RemoveContainer" containerID="9284d9134005736d19ff6360e76307560b38dfa730f4a784569011b9498b2e78" Oct 30 17:22:19.947920 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:22:19.947897 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9284d9134005736d19ff6360e76307560b38dfa730f4a784569011b9498b2e78\": container with ID starting with 9284d9134005736d19ff6360e76307560b38dfa730f4a784569011b9498b2e78 not found: ID does not exist" containerID="9284d9134005736d19ff6360e76307560b38dfa730f4a784569011b9498b2e78" Oct 30 17:22:19.948004 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:19.947930 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9284d9134005736d19ff6360e76307560b38dfa730f4a784569011b9498b2e78"} err="failed to get container status \"9284d9134005736d19ff6360e76307560b38dfa730f4a784569011b9498b2e78\": rpc error: code = NotFound desc = could not find container \"9284d9134005736d19ff6360e76307560b38dfa730f4a784569011b9498b2e78\": container with ID starting with 9284d9134005736d19ff6360e76307560b38dfa730f4a784569011b9498b2e78 not found: ID does not exist" Oct 30 17:22:20.009854 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:20.009815 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p"] Oct 30 17:22:20.022924 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:20.022883 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-kwd9p"] Oct 30 17:22:20.146951 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:20.146921 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d30e44c6-cafd-4b54-9784-2f3743de61dc" path="/var/lib/kubelet/pods/d30e44c6-cafd-4b54-9784-2f3743de61dc/volumes" Oct 30 17:22:20.865024 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:20.864987 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5d67ca-sfwzm"] Oct 30 17:22:20.878169 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:20.878138 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5d67ca-sfwzm"] Oct 30 17:22:22.101368 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:22.101325 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca1e9ef0-rk7j8"] Oct 30 17:22:22.102165 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:22.102148 4522 cpu_manager.go:414] "containerMap: removing container" podUID="534aeebe-4edf-425e-a16c-712cc498154b" containerName="istio-proxy" Oct 30 17:22:22.102216 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:22.102166 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="534aeebe-4edf-425e-a16c-712cc498154b" containerName="istio-proxy" Oct 30 17:22:22.102216 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:22.102186 4522 cpu_manager.go:414] "containerMap: removing container" podUID="6fa345e3-dbb2-4e58-bb92-6dcc9adcd63c" containerName="revoke-keys" Oct 30 17:22:22.102216 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:22.102191 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fa345e3-dbb2-4e58-bb92-6dcc9adcd63c" containerName="revoke-keys" Oct 30 17:22:22.102216 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:22.102202 4522 cpu_manager.go:414] "containerMap: removing container" podUID="d30e44c6-cafd-4b54-9784-2f3743de61dc" containerName="istio-proxy" Oct 30 17:22:22.102216 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:22.102207 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d30e44c6-cafd-4b54-9784-2f3743de61dc" containerName="istio-proxy" Oct 30 17:22:22.102423 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:22.102341 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="534aeebe-4edf-425e-a16c-712cc498154b" containerName="istio-proxy" Oct 30 17:22:22.102423 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:22.102351 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fa345e3-dbb2-4e58-bb92-6dcc9adcd63c" containerName="revoke-keys" Oct 30 17:22:22.102423 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:22.102359 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="d30e44c6-cafd-4b54-9784-2f3743de61dc" containerName="istio-proxy" Oct 30 17:22:22.107915 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:22.107900 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca1e9ef0-rk7j8" Oct 30 17:22:22.128440 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:22.128412 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca1e9ef0-rk7j8"] Oct 30 17:22:22.134833 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:22.134808 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:22:22.134965 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:22.134900 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:22:22.149418 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:22.149384 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eeeb5f83-d6f8-4fd4-9c78-c5cb3640b538" path="/var/lib/kubelet/pods/eeeb5f83-d6f8-4fd4-9c78-c5cb3640b538/volumes" Oct 30 17:22:22.227500 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:22.227465 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8113928a-b589-43fe-ab47-c6e4b4c4fd87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca1e9ef0-rk7j8\" (UID: \"8113928a-b589-43fe-ab47-c6e4b4c4fd87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca1e9ef0-rk7j8" Oct 30 17:22:22.227683 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:22.227650 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqgqb\" (UniqueName: \"kubernetes.io/projected/8113928a-b589-43fe-ab47-c6e4b4c4fd87-kube-api-access-hqgqb\") pod \"maas-api-revoke-sub-ca1e9ef0-rk7j8\" (UID: \"8113928a-b589-43fe-ab47-c6e4b4c4fd87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca1e9ef0-rk7j8" Oct 30 17:22:22.227729 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:22.227690 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8113928a-b589-43fe-ab47-c6e4b4c4fd87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca1e9ef0-rk7j8\" (UID: \"8113928a-b589-43fe-ab47-c6e4b4c4fd87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca1e9ef0-rk7j8" Oct 30 17:22:22.329529 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:22.329493 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8113928a-b589-43fe-ab47-c6e4b4c4fd87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca1e9ef0-rk7j8\" (UID: \"8113928a-b589-43fe-ab47-c6e4b4c4fd87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca1e9ef0-rk7j8" Oct 30 17:22:22.329763 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:22.329743 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8113928a-b589-43fe-ab47-c6e4b4c4fd87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca1e9ef0-rk7j8\" (UID: \"8113928a-b589-43fe-ab47-c6e4b4c4fd87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca1e9ef0-rk7j8" Oct 30 17:22:22.329921 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:22.329899 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqgqb\" (UniqueName: \"kubernetes.io/projected/8113928a-b589-43fe-ab47-c6e4b4c4fd87-kube-api-access-hqgqb\") pod \"maas-api-revoke-sub-ca1e9ef0-rk7j8\" (UID: \"8113928a-b589-43fe-ab47-c6e4b4c4fd87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca1e9ef0-rk7j8" Oct 30 17:22:22.330431 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:22.330411 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8113928a-b589-43fe-ab47-c6e4b4c4fd87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca1e9ef0-rk7j8\" (UID: \"8113928a-b589-43fe-ab47-c6e4b4c4fd87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca1e9ef0-rk7j8" Oct 30 17:22:22.331933 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:22.331911 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8113928a-b589-43fe-ab47-c6e4b4c4fd87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca1e9ef0-rk7j8\" (UID: \"8113928a-b589-43fe-ab47-c6e4b4c4fd87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca1e9ef0-rk7j8" Oct 30 17:22:22.354524 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:22.354452 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqgqb\" (UniqueName: \"kubernetes.io/projected/8113928a-b589-43fe-ab47-c6e4b4c4fd87-kube-api-access-hqgqb\") pod \"maas-api-revoke-sub-ca1e9ef0-rk7j8\" (UID: \"8113928a-b589-43fe-ab47-c6e4b4c4fd87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca1e9ef0-rk7j8" Oct 30 17:22:22.422883 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:22.422850 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca1e9ef0-rk7j8" Oct 30 17:22:22.735543 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:22:22.735497 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8113928a_b589_43fe_ab47_c6e4b4c4fd87.slice/crio-ed3ee3c09c8b77174e4dcb54c5f995dcfce4f03f11cc9f2de038bca9372a2160 WatchSource:0}: Error finding container ed3ee3c09c8b77174e4dcb54c5f995dcfce4f03f11cc9f2de038bca9372a2160: Status 404 returned error can't find the container with id ed3ee3c09c8b77174e4dcb54c5f995dcfce4f03f11cc9f2de038bca9372a2160 Oct 30 17:22:22.737339 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:22.737285 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca1e9ef0-rk7j8"] Oct 30 17:22:22.964026 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:22.963990 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca1e9ef0-rk7j8" event={"ID":"8113928a-b589-43fe-ab47-c6e4b4c4fd87","Type":"ContainerStarted","Data":"ed3ee3c09c8b77174e4dcb54c5f995dcfce4f03f11cc9f2de038bca9372a2160"} Oct 30 17:22:23.974623 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:23.974584 4522 generic.go:356] "Generic (PLEG): container finished" podID="8113928a-b589-43fe-ab47-c6e4b4c4fd87" containerID="b2794999c1c383a7c35424f4744f6eed996cbb2c07e29ce961af4b5c2a72b10d" exitCode=0 Oct 30 17:22:23.975008 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:23.974667 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca1e9ef0-rk7j8" event={"ID":"8113928a-b589-43fe-ab47-c6e4b4c4fd87","Type":"ContainerDied","Data":"b2794999c1c383a7c35424f4744f6eed996cbb2c07e29ce961af4b5c2a72b10d"} Oct 30 17:22:24.541616 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:24.541582 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c82666-6ccbf4bbb4-tw2zk"] Oct 30 17:22:24.578441 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:24.578406 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c82666-7867d55d87-89bwv"] Oct 30 17:22:25.150499 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:25.150471 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca1e9ef0-rk7j8" Oct 30 17:22:25.280115 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:25.279994 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8113928a-b589-43fe-ab47-c6e4b4c4fd87-kube-api-access-hqgqb\" (UniqueName: \"kubernetes.io/projected/8113928a-b589-43fe-ab47-c6e4b4c4fd87-kube-api-access-hqgqb\") pod \"8113928a-b589-43fe-ab47-c6e4b4c4fd87\" (UID: \"8113928a-b589-43fe-ab47-c6e4b4c4fd87\") " Oct 30 17:22:25.283632 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:25.283597 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8113928a-b589-43fe-ab47-c6e4b4c4fd87-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8113928a-b589-43fe-ab47-c6e4b4c4fd87-maas-api-xks-ca\") pod \"8113928a-b589-43fe-ab47-c6e4b4c4fd87\" (UID: \"8113928a-b589-43fe-ab47-c6e4b4c4fd87\") " Oct 30 17:22:25.284209 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:25.284190 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8113928a-b589-43fe-ab47-c6e4b4c4fd87-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8113928a-b589-43fe-ab47-c6e4b4c4fd87-maas-api-openshift-ca\") pod \"8113928a-b589-43fe-ab47-c6e4b4c4fd87\" (UID: \"8113928a-b589-43fe-ab47-c6e4b4c4fd87\") " Oct 30 17:22:25.286183 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:25.284694 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8113928a-b589-43fe-ab47-c6e4b4c4fd87-maas-api-openshift-ca" pod "8113928a-b589-43fe-ab47-c6e4b4c4fd87" (UID: "8113928a-b589-43fe-ab47-c6e4b4c4fd87"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:22:25.313634 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:25.313594 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8113928a-b589-43fe-ab47-c6e4b4c4fd87-kube-api-access-hqgqb" pod "8113928a-b589-43fe-ab47-c6e4b4c4fd87" (UID: "8113928a-b589-43fe-ab47-c6e4b4c4fd87"). InnerVolumeSpecName "kube-api-access-hqgqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:22:25.381084 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:25.381047 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8113928a-b589-43fe-ab47-c6e4b4c4fd87-maas-api-xks-ca" pod "8113928a-b589-43fe-ab47-c6e4b4c4fd87" (UID: "8113928a-b589-43fe-ab47-c6e4b4c4fd87"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:22:25.388071 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:25.388043 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8113928a-b589-43fe-ab47-c6e4b4c4fd87-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:22:25.388261 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:25.388247 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8113928a-b589-43fe-ab47-c6e4b4c4fd87-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:22:25.388379 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:25.388368 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqgqb\" (UniqueName: \"kubernetes.io/projected/8113928a-b589-43fe-ab47-c6e4b4c4fd87-kube-api-access-hqgqb\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:22:25.993384 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:25.993358 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca1e9ef0-rk7j8" Oct 30 17:22:25.993536 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:25.993354 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca1e9ef0-rk7j8" event={"ID":"8113928a-b589-43fe-ab47-c6e4b4c4fd87","Type":"ContainerDied","Data":"ed3ee3c09c8b77174e4dcb54c5f995dcfce4f03f11cc9f2de038bca9372a2160"} Oct 30 17:22:25.993536 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:25.993484 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed3ee3c09c8b77174e4dcb54c5f995dcfce4f03f11cc9f2de038bca9372a2160" Oct 30 17:22:28.719776 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:28.719740 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd"] Oct 30 17:22:28.720260 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:28.720023 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd" podUID="065164fc-1823-49af-9921-d14d9a94e997" containerName="istio-proxy" containerID="cri-o://c821d519a29e83e3094a31fc7b7075ba60e52c3c1105c62637d854ce8de266fb" gracePeriod=30 Oct 30 17:22:29.543002 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:29.542933 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-c82666-6ccbf4bbb4-tw2zk" podUID="e3f6c9b4-919b-4ea3-9909-1c9824339e55" containerName="payload-processing" containerID="cri-o://e73ca24f781d3f592f04c4fddf86fda9afb6a621ae62d504b1cab1bae60e76b4" gracePeriod=25 Oct 30 17:22:29.579148 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:29.579105 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c82666-7867d55d87-89bwv" podUID="a1c26ee9-a385-4923-84f6-d1413e4a8302" containerName="payload-pre-processing" containerID="cri-o://168a7ce10e5e16b41f4bd3a435a416cdcd91764c29b85043b0b472a6f79d5838" gracePeriod=25 Oct 30 17:22:34.075620 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:34.075593 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd" Oct 30 17:22:34.094841 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:34.094800 4522 generic.go:356] "Generic (PLEG): container finished" podID="065164fc-1823-49af-9921-d14d9a94e997" containerID="c821d519a29e83e3094a31fc7b7075ba60e52c3c1105c62637d854ce8de266fb" exitCode=0 Oct 30 17:22:34.095022 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:34.094855 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd" Oct 30 17:22:34.095022 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:34.094869 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd" event={"ID":"065164fc-1823-49af-9921-d14d9a94e997","Type":"ContainerDied","Data":"c821d519a29e83e3094a31fc7b7075ba60e52c3c1105c62637d854ce8de266fb"} Oct 30 17:22:34.095022 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:34.094945 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd" event={"ID":"065164fc-1823-49af-9921-d14d9a94e997","Type":"ContainerDied","Data":"b40e3ac0fe5bb7b9672ec6d437fda14a018a35f3b911b9b8cc4ed5e4116d4216"} Oct 30 17:22:34.095022 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:34.094967 4522 scope.go:122] "RemoveContainer" containerID="c821d519a29e83e3094a31fc7b7075ba60e52c3c1105c62637d854ce8de266fb" Oct 30 17:22:34.106446 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:34.106419 4522 scope.go:122] "RemoveContainer" containerID="290cce3c9c779f0915bd8a6795025cd875f0cd66d2d3588424cfd0536ee49648" Oct 30 17:22:34.117716 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:34.117695 4522 scope.go:122] "RemoveContainer" containerID="c821d519a29e83e3094a31fc7b7075ba60e52c3c1105c62637d854ce8de266fb" Oct 30 17:22:34.118007 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:22:34.117985 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c821d519a29e83e3094a31fc7b7075ba60e52c3c1105c62637d854ce8de266fb\": container with ID starting with c821d519a29e83e3094a31fc7b7075ba60e52c3c1105c62637d854ce8de266fb not found: ID does not exist" containerID="c821d519a29e83e3094a31fc7b7075ba60e52c3c1105c62637d854ce8de266fb" Oct 30 17:22:34.118080 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:34.118015 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c821d519a29e83e3094a31fc7b7075ba60e52c3c1105c62637d854ce8de266fb"} err="failed to get container status \"c821d519a29e83e3094a31fc7b7075ba60e52c3c1105c62637d854ce8de266fb\": rpc error: code = NotFound desc = could not find container \"c821d519a29e83e3094a31fc7b7075ba60e52c3c1105c62637d854ce8de266fb\": container with ID starting with c821d519a29e83e3094a31fc7b7075ba60e52c3c1105c62637d854ce8de266fb not found: ID does not exist" Oct 30 17:22:34.118080 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:34.118034 4522 scope.go:122] "RemoveContainer" containerID="290cce3c9c779f0915bd8a6795025cd875f0cd66d2d3588424cfd0536ee49648" Oct 30 17:22:34.118292 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:22:34.118276 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"290cce3c9c779f0915bd8a6795025cd875f0cd66d2d3588424cfd0536ee49648\": container with ID starting with 290cce3c9c779f0915bd8a6795025cd875f0cd66d2d3588424cfd0536ee49648 not found: ID does not exist" containerID="290cce3c9c779f0915bd8a6795025cd875f0cd66d2d3588424cfd0536ee49648" Oct 30 17:22:34.118356 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:34.118296 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"290cce3c9c779f0915bd8a6795025cd875f0cd66d2d3588424cfd0536ee49648"} err="failed to get container status \"290cce3c9c779f0915bd8a6795025cd875f0cd66d2d3588424cfd0536ee49648\": rpc error: code = NotFound desc = could not find container \"290cce3c9c779f0915bd8a6795025cd875f0cd66d2d3588424cfd0536ee49648\": container with ID starting with 290cce3c9c779f0915bd8a6795025cd875f0cd66d2d3588424cfd0536ee49648 not found: ID does not exist" Oct 30 17:22:34.172063 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:34.172025 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/065164fc-1823-49af-9921-d14d9a94e997-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/065164fc-1823-49af-9921-d14d9a94e997-credential-socket\") pod \"065164fc-1823-49af-9921-d14d9a94e997\" (UID: \"065164fc-1823-49af-9921-d14d9a94e997\") " Oct 30 17:22:34.172231 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:34.172163 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/065164fc-1823-49af-9921-d14d9a94e997-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/065164fc-1823-49af-9921-d14d9a94e997-istio-podinfo\") pod \"065164fc-1823-49af-9921-d14d9a94e997\" (UID: \"065164fc-1823-49af-9921-d14d9a94e997\") " Oct 30 17:22:34.172231 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:34.172218 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/065164fc-1823-49af-9921-d14d9a94e997-istio-token\" (UniqueName: \"kubernetes.io/projected/065164fc-1823-49af-9921-d14d9a94e997-istio-token\") pod \"065164fc-1823-49af-9921-d14d9a94e997\" (UID: \"065164fc-1823-49af-9921-d14d9a94e997\") " Oct 30 17:22:34.172414 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:34.172248 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/065164fc-1823-49af-9921-d14d9a94e997-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/065164fc-1823-49af-9921-d14d9a94e997-istiod-ca-cert\") pod \"065164fc-1823-49af-9921-d14d9a94e997\" (UID: \"065164fc-1823-49af-9921-d14d9a94e997\") " Oct 30 17:22:34.172414 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:34.172291 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/065164fc-1823-49af-9921-d14d9a94e997-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/065164fc-1823-49af-9921-d14d9a94e997-workload-certs\") pod \"065164fc-1823-49af-9921-d14d9a94e997\" (UID: \"065164fc-1823-49af-9921-d14d9a94e997\") " Oct 30 17:22:34.172414 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:34.172365 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/065164fc-1823-49af-9921-d14d9a94e997-credential-socket" pod "065164fc-1823-49af-9921-d14d9a94e997" (UID: "065164fc-1823-49af-9921-d14d9a94e997"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:22:34.172414 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:34.172400 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/065164fc-1823-49af-9921-d14d9a94e997-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/065164fc-1823-49af-9921-d14d9a94e997-workload-socket\") pod \"065164fc-1823-49af-9921-d14d9a94e997\" (UID: \"065164fc-1823-49af-9921-d14d9a94e997\") " Oct 30 17:22:34.172659 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:34.172511 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/065164fc-1823-49af-9921-d14d9a94e997-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/065164fc-1823-49af-9921-d14d9a94e997-istio-data\") pod \"065164fc-1823-49af-9921-d14d9a94e997\" (UID: \"065164fc-1823-49af-9921-d14d9a94e997\") " Oct 30 17:22:34.172659 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:34.172635 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/065164fc-1823-49af-9921-d14d9a94e997-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/065164fc-1823-49af-9921-d14d9a94e997-istio-envoy\") pod \"065164fc-1823-49af-9921-d14d9a94e997\" (UID: \"065164fc-1823-49af-9921-d14d9a94e997\") " Oct 30 17:22:34.172770 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:34.172688 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/065164fc-1823-49af-9921-d14d9a94e997-kube-api-access-qfb27\" (UniqueName: \"kubernetes.io/projected/065164fc-1823-49af-9921-d14d9a94e997-kube-api-access-qfb27\") pod \"065164fc-1823-49af-9921-d14d9a94e997\" (UID: \"065164fc-1823-49af-9921-d14d9a94e997\") " Oct 30 17:22:34.172770 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:34.172698 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/065164fc-1823-49af-9921-d14d9a94e997-istiod-ca-cert" pod "065164fc-1823-49af-9921-d14d9a94e997" (UID: "065164fc-1823-49af-9921-d14d9a94e997"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:22:34.173636 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:34.173604 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/065164fc-1823-49af-9921-d14d9a94e997-workload-socket" pod "065164fc-1823-49af-9921-d14d9a94e997" (UID: "065164fc-1823-49af-9921-d14d9a94e997"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:22:34.173815 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:34.173795 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/065164fc-1823-49af-9921-d14d9a94e997-workload-certs" pod "065164fc-1823-49af-9921-d14d9a94e997" (UID: "065164fc-1823-49af-9921-d14d9a94e997"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:22:34.174122 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:34.174050 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/065164fc-1823-49af-9921-d14d9a94e997-istio-data" pod "065164fc-1823-49af-9921-d14d9a94e997" (UID: "065164fc-1823-49af-9921-d14d9a94e997"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:22:34.174224 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:34.174187 4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/065164fc-1823-49af-9921-d14d9a94e997-credential-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:22:34.174224 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:34.174211 4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/065164fc-1823-49af-9921-d14d9a94e997-istiod-ca-cert\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:22:34.174356 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:34.174227 4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/065164fc-1823-49af-9921-d14d9a94e997-workload-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:22:34.174356 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:34.174242 4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/065164fc-1823-49af-9921-d14d9a94e997-workload-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:22:34.174356 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:34.174257 4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/065164fc-1823-49af-9921-d14d9a94e997-istio-data\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:22:34.175068 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:34.175041 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/065164fc-1823-49af-9921-d14d9a94e997-istio-podinfo" pod "065164fc-1823-49af-9921-d14d9a94e997" (UID: "065164fc-1823-49af-9921-d14d9a94e997"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:22:34.177202 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:34.177171 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/065164fc-1823-49af-9921-d14d9a94e997-kube-api-access-qfb27" pod "065164fc-1823-49af-9921-d14d9a94e997" (UID: "065164fc-1823-49af-9921-d14d9a94e997"). InnerVolumeSpecName "kube-api-access-qfb27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:22:34.178050 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:34.177935 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/065164fc-1823-49af-9921-d14d9a94e997-istio-envoy" pod "065164fc-1823-49af-9921-d14d9a94e997" (UID: "065164fc-1823-49af-9921-d14d9a94e997"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:22:34.178555 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:34.178534 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/065164fc-1823-49af-9921-d14d9a94e997-istio-token" pod "065164fc-1823-49af-9921-d14d9a94e997" (UID: "065164fc-1823-49af-9921-d14d9a94e997"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:22:34.275642 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:34.275606 4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/065164fc-1823-49af-9921-d14d9a94e997-istio-podinfo\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:22:34.275642 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:34.275641 4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/065164fc-1823-49af-9921-d14d9a94e997-istio-token\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:22:34.275642 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:34.275657 4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/065164fc-1823-49af-9921-d14d9a94e997-istio-envoy\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:22:34.275956 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:34.275672 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfb27\" (UniqueName: \"kubernetes.io/projected/065164fc-1823-49af-9921-d14d9a94e997-kube-api-access-qfb27\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:22:34.461635 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:34.461600 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd"] Oct 30 17:22:34.484695 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:34.484661 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-nn2hd"] Oct 30 17:22:35.189122 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:35.189082 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-c82666-6ccbf4bbb4-tw2zk" podUID="e3f6c9b4-919b-4ea3-9909-1c9824339e55" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:22:35.191022 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:35.190995 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c82666-7867d55d87-89bwv" podUID="a1c26ee9-a385-4923-84f6-d1413e4a8302" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:22:36.147580 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:36.147549 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="065164fc-1823-49af-9921-d14d9a94e997" path="/var/lib/kubelet/pods/065164fc-1823-49af-9921-d14d9a94e997/volumes" Oct 30 17:22:42.962692 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:22:42.962649 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:22:45.189048 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:45.189010 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-c82666-6ccbf4bbb4-tw2zk" podUID="e3f6c9b4-919b-4ea3-9909-1c9824339e55" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:22:45.189798 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:45.189765 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c82666-7867d55d87-89bwv" podUID="a1c26ee9-a385-4923-84f6-d1413e4a8302" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:22:49.881020 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:49.880991 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c82666-6ccbf4bbb4-tw2zk" Oct 30 17:22:49.914788 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:49.914763 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c82666-7867d55d87-89bwv" Oct 30 17:22:49.921084 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:49.921054 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3f6c9b4-919b-4ea3-9909-1c9824339e55-kube-api-access-6zh6n\" (UniqueName: \"kubernetes.io/projected/e3f6c9b4-919b-4ea3-9909-1c9824339e55-kube-api-access-6zh6n\") pod \"e3f6c9b4-919b-4ea3-9909-1c9824339e55\" (UID: \"e3f6c9b4-919b-4ea3-9909-1c9824339e55\") " Oct 30 17:22:49.921218 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:49.921126 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3f6c9b4-919b-4ea3-9909-1c9824339e55-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3f6c9b4-919b-4ea3-9909-1c9824339e55-plugins-config-volume\") pod \"e3f6c9b4-919b-4ea3-9909-1c9824339e55\" (UID: \"e3f6c9b4-919b-4ea3-9909-1c9824339e55\") " Oct 30 17:22:49.921495 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:49.921472 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3f6c9b4-919b-4ea3-9909-1c9824339e55-plugins-config-volume" pod "e3f6c9b4-919b-4ea3-9909-1c9824339e55" (UID: "e3f6c9b4-919b-4ea3-9909-1c9824339e55"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:22:49.921853 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:49.921833 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3f6c9b4-919b-4ea3-9909-1c9824339e55-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:22:49.923162 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:49.923143 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3f6c9b4-919b-4ea3-9909-1c9824339e55-kube-api-access-6zh6n" pod "e3f6c9b4-919b-4ea3-9909-1c9824339e55" (UID: "e3f6c9b4-919b-4ea3-9909-1c9824339e55"). InnerVolumeSpecName "kube-api-access-6zh6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:22:50.023196 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:50.023156 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1c26ee9-a385-4923-84f6-d1413e4a8302-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1c26ee9-a385-4923-84f6-d1413e4a8302-plugins-config-volume\") pod \"a1c26ee9-a385-4923-84f6-d1413e4a8302\" (UID: \"a1c26ee9-a385-4923-84f6-d1413e4a8302\") " Oct 30 17:22:50.023433 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:50.023214 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1c26ee9-a385-4923-84f6-d1413e4a8302-kube-api-access-m5fg8\" (UniqueName: \"kubernetes.io/projected/a1c26ee9-a385-4923-84f6-d1413e4a8302-kube-api-access-m5fg8\") pod \"a1c26ee9-a385-4923-84f6-d1413e4a8302\" (UID: \"a1c26ee9-a385-4923-84f6-d1413e4a8302\") " Oct 30 17:22:50.023676 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:50.023651 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1c26ee9-a385-4923-84f6-d1413e4a8302-plugins-config-volume" pod "a1c26ee9-a385-4923-84f6-d1413e4a8302" (UID: "a1c26ee9-a385-4923-84f6-d1413e4a8302"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:22:50.024032 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:50.024006 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zh6n\" (UniqueName: \"kubernetes.io/projected/e3f6c9b4-919b-4ea3-9909-1c9824339e55-kube-api-access-6zh6n\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:22:50.024032 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:50.024033 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1c26ee9-a385-4923-84f6-d1413e4a8302-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:22:50.025547 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:50.025523 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1c26ee9-a385-4923-84f6-d1413e4a8302-kube-api-access-m5fg8" pod "a1c26ee9-a385-4923-84f6-d1413e4a8302" (UID: "a1c26ee9-a385-4923-84f6-d1413e4a8302"). InnerVolumeSpecName "kube-api-access-m5fg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:22:50.125868 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:50.125781 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5fg8\" (UniqueName: \"kubernetes.io/projected/a1c26ee9-a385-4923-84f6-d1413e4a8302-kube-api-access-m5fg8\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:22:50.239782 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:50.239751 4522 generic.go:356] "Generic (PLEG): container finished" podID="a1c26ee9-a385-4923-84f6-d1413e4a8302" containerID="168a7ce10e5e16b41f4bd3a435a416cdcd91764c29b85043b0b472a6f79d5838" exitCode=0 Oct 30 17:22:50.239985 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:50.239794 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c82666-7867d55d87-89bwv" Oct 30 17:22:50.239985 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:50.239837 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c82666-7867d55d87-89bwv" event={"ID":"a1c26ee9-a385-4923-84f6-d1413e4a8302","Type":"ContainerDied","Data":"168a7ce10e5e16b41f4bd3a435a416cdcd91764c29b85043b0b472a6f79d5838"} Oct 30 17:22:50.239985 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:50.239901 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c82666-7867d55d87-89bwv" event={"ID":"a1c26ee9-a385-4923-84f6-d1413e4a8302","Type":"ContainerDied","Data":"95dc8c3277bbd1243ef37fba4cfb6ab6f27fe6e88f3ebdd918acd6551ef70127"} Oct 30 17:22:50.239985 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:50.239934 4522 scope.go:122] "RemoveContainer" containerID="168a7ce10e5e16b41f4bd3a435a416cdcd91764c29b85043b0b472a6f79d5838" Oct 30 17:22:50.241496 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:50.241467 4522 generic.go:356] "Generic (PLEG): container finished" podID="e3f6c9b4-919b-4ea3-9909-1c9824339e55" containerID="e73ca24f781d3f592f04c4fddf86fda9afb6a621ae62d504b1cab1bae60e76b4" exitCode=0 Oct 30 17:22:50.241631 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:50.241493 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c82666-6ccbf4bbb4-tw2zk" event={"ID":"e3f6c9b4-919b-4ea3-9909-1c9824339e55","Type":"ContainerDied","Data":"e73ca24f781d3f592f04c4fddf86fda9afb6a621ae62d504b1cab1bae60e76b4"} Oct 30 17:22:50.241631 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:50.241524 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c82666-6ccbf4bbb4-tw2zk" Oct 30 17:22:50.241631 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:50.241545 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c82666-6ccbf4bbb4-tw2zk" event={"ID":"e3f6c9b4-919b-4ea3-9909-1c9824339e55","Type":"ContainerDied","Data":"568917d413441288e464c8252646d6a25362de4a42e91b61cc96c46d2e9b9e5e"} Oct 30 17:22:50.251239 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:50.251221 4522 scope.go:122] "RemoveContainer" containerID="168a7ce10e5e16b41f4bd3a435a416cdcd91764c29b85043b0b472a6f79d5838" Oct 30 17:22:50.251549 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:22:50.251531 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"168a7ce10e5e16b41f4bd3a435a416cdcd91764c29b85043b0b472a6f79d5838\": container with ID starting with 168a7ce10e5e16b41f4bd3a435a416cdcd91764c29b85043b0b472a6f79d5838 not found: ID does not exist" containerID="168a7ce10e5e16b41f4bd3a435a416cdcd91764c29b85043b0b472a6f79d5838" Oct 30 17:22:50.251598 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:50.251560 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"168a7ce10e5e16b41f4bd3a435a416cdcd91764c29b85043b0b472a6f79d5838"} err="failed to get container status \"168a7ce10e5e16b41f4bd3a435a416cdcd91764c29b85043b0b472a6f79d5838\": rpc error: code = NotFound desc = could not find container \"168a7ce10e5e16b41f4bd3a435a416cdcd91764c29b85043b0b472a6f79d5838\": container with ID starting with 168a7ce10e5e16b41f4bd3a435a416cdcd91764c29b85043b0b472a6f79d5838 not found: ID does not exist" Oct 30 17:22:50.251598 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:50.251576 4522 scope.go:122] "RemoveContainer" containerID="e73ca24f781d3f592f04c4fddf86fda9afb6a621ae62d504b1cab1bae60e76b4" Oct 30 17:22:50.264076 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:50.264054 4522 scope.go:122] "RemoveContainer" containerID="e73ca24f781d3f592f04c4fddf86fda9afb6a621ae62d504b1cab1bae60e76b4" Oct 30 17:22:50.264425 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:22:50.264407 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e73ca24f781d3f592f04c4fddf86fda9afb6a621ae62d504b1cab1bae60e76b4\": container with ID starting with e73ca24f781d3f592f04c4fddf86fda9afb6a621ae62d504b1cab1bae60e76b4 not found: ID does not exist" containerID="e73ca24f781d3f592f04c4fddf86fda9afb6a621ae62d504b1cab1bae60e76b4" Oct 30 17:22:50.264515 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:50.264431 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e73ca24f781d3f592f04c4fddf86fda9afb6a621ae62d504b1cab1bae60e76b4"} err="failed to get container status \"e73ca24f781d3f592f04c4fddf86fda9afb6a621ae62d504b1cab1bae60e76b4\": rpc error: code = NotFound desc = could not find container \"e73ca24f781d3f592f04c4fddf86fda9afb6a621ae62d504b1cab1bae60e76b4\": container with ID starting with e73ca24f781d3f592f04c4fddf86fda9afb6a621ae62d504b1cab1bae60e76b4 not found: ID does not exist" Oct 30 17:22:50.299457 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:50.299429 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c82666-7867d55d87-89bwv"] Oct 30 17:22:50.312382 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:50.312352 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c82666-7867d55d87-89bwv"] Oct 30 17:22:50.349860 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:50.349833 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c82666-6ccbf4bbb4-tw2zk"] Oct 30 17:22:50.354163 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:50.354143 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c82666-6ccbf4bbb4-tw2zk"] Oct 30 17:22:52.148002 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:52.147967 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1c26ee9-a385-4923-84f6-d1413e4a8302" path="/var/lib/kubelet/pods/a1c26ee9-a385-4923-84f6-d1413e4a8302/volumes" Oct 30 17:22:52.148619 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:52.148599 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3f6c9b4-919b-4ea3-9909-1c9824339e55" path="/var/lib/kubelet/pods/e3f6c9b4-919b-4ea3-9909-1c9824339e55/volumes" Oct 30 17:22:53.783625 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:53.783594 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7c52f8-8hhsc"] Oct 30 17:22:53.784357 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:53.784341 4522 cpu_manager.go:414] "containerMap: removing container" podUID="a1c26ee9-a385-4923-84f6-d1413e4a8302" containerName="payload-pre-processing" Oct 30 17:22:53.784421 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:53.784357 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1c26ee9-a385-4923-84f6-d1413e4a8302" containerName="payload-pre-processing" Oct 30 17:22:53.784421 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:53.784385 4522 cpu_manager.go:414] "containerMap: removing container" podUID="065164fc-1823-49af-9921-d14d9a94e997" containerName="istio-proxy" Oct 30 17:22:53.784421 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:53.784390 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="065164fc-1823-49af-9921-d14d9a94e997" containerName="istio-proxy" Oct 30 17:22:53.784421 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:53.784402 4522 cpu_manager.go:414] "containerMap: removing container" podUID="e3f6c9b4-919b-4ea3-9909-1c9824339e55" containerName="payload-processing" Oct 30 17:22:53.784421 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:53.784407 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3f6c9b4-919b-4ea3-9909-1c9824339e55" containerName="payload-processing" Oct 30 17:22:53.784601 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:53.784443 4522 cpu_manager.go:414] "containerMap: removing container" podUID="8113928a-b589-43fe-ab47-c6e4b4c4fd87" containerName="revoke-keys" Oct 30 17:22:53.784601 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:53.784448 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8113928a-b589-43fe-ab47-c6e4b4c4fd87" containerName="revoke-keys" Oct 30 17:22:53.784601 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:53.784566 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="8113928a-b589-43fe-ab47-c6e4b4c4fd87" containerName="revoke-keys" Oct 30 17:22:53.784601 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:53.784577 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="065164fc-1823-49af-9921-d14d9a94e997" containerName="istio-proxy" Oct 30 17:22:53.784601 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:53.784584 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1c26ee9-a385-4923-84f6-d1413e4a8302" containerName="payload-pre-processing" Oct 30 17:22:53.784601 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:53.784595 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="065164fc-1823-49af-9921-d14d9a94e997" containerName="istio-proxy" Oct 30 17:22:53.784601 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:53.784603 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3f6c9b4-919b-4ea3-9909-1c9824339e55" containerName="payload-processing" Oct 30 17:22:53.790288 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:53.790254 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7c52f8-8hhsc" Oct 30 17:22:53.808876 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:53.808845 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7c52f8-8hhsc"] Oct 30 17:22:53.817522 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:53.817496 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:22:53.817732 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:53.817588 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:22:53.884700 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:53.884665 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a423a8b4-3931-4958-8961-358be318996f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7c52f8-8hhsc\" (UID: \"a423a8b4-3931-4958-8961-358be318996f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7c52f8-8hhsc" Oct 30 17:22:53.884883 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:53.884726 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kglgl\" (UniqueName: \"kubernetes.io/projected/a423a8b4-3931-4958-8961-358be318996f-kube-api-access-kglgl\") pod \"maas-api-revoke-keys-e2e-inf-a-7c52f8-8hhsc\" (UID: \"a423a8b4-3931-4958-8961-358be318996f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7c52f8-8hhsc" Oct 30 17:22:53.884928 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:53.884910 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a423a8b4-3931-4958-8961-358be318996f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7c52f8-8hhsc\" (UID: \"a423a8b4-3931-4958-8961-358be318996f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7c52f8-8hhsc" Oct 30 17:22:53.986845 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:53.986801 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a423a8b4-3931-4958-8961-358be318996f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7c52f8-8hhsc\" (UID: \"a423a8b4-3931-4958-8961-358be318996f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7c52f8-8hhsc" Oct 30 17:22:53.987017 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:53.986954 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kglgl\" (UniqueName: \"kubernetes.io/projected/a423a8b4-3931-4958-8961-358be318996f-kube-api-access-kglgl\") pod \"maas-api-revoke-keys-e2e-inf-a-7c52f8-8hhsc\" (UID: \"a423a8b4-3931-4958-8961-358be318996f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7c52f8-8hhsc" Oct 30 17:22:53.987117 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:53.987097 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a423a8b4-3931-4958-8961-358be318996f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7c52f8-8hhsc\" (UID: \"a423a8b4-3931-4958-8961-358be318996f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7c52f8-8hhsc" Oct 30 17:22:53.987612 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:53.987591 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a423a8b4-3931-4958-8961-358be318996f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7c52f8-8hhsc\" (UID: \"a423a8b4-3931-4958-8961-358be318996f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7c52f8-8hhsc" Oct 30 17:22:53.989499 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:53.989478 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a423a8b4-3931-4958-8961-358be318996f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7c52f8-8hhsc\" (UID: \"a423a8b4-3931-4958-8961-358be318996f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7c52f8-8hhsc" Oct 30 17:22:54.012016 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:54.011983 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kglgl\" (UniqueName: \"kubernetes.io/projected/a423a8b4-3931-4958-8961-358be318996f-kube-api-access-kglgl\") pod \"maas-api-revoke-keys-e2e-inf-a-7c52f8-8hhsc\" (UID: \"a423a8b4-3931-4958-8961-358be318996f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7c52f8-8hhsc" Oct 30 17:22:54.106635 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:54.106558 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7c52f8-8hhsc" Oct 30 17:22:54.421054 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:22:54.421018 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda423a8b4_3931_4958_8961_358be318996f.slice/crio-8d879b4c301f2d4792bbd90b87cd1c1856a77b0a7f46254d8a903cc56c8a85c4 WatchSource:0}: Error finding container 8d879b4c301f2d4792bbd90b87cd1c1856a77b0a7f46254d8a903cc56c8a85c4: Status 404 returned error can't find the container with id 8d879b4c301f2d4792bbd90b87cd1c1856a77b0a7f46254d8a903cc56c8a85c4 Oct 30 17:22:54.422722 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:54.422609 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7c52f8-8hhsc"] Oct 30 17:22:54.747923 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:54.747887 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71437729-w8n4k"] Oct 30 17:22:54.748716 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:54.748701 4522 cpu_manager.go:414] "containerMap: removing container" podUID="065164fc-1823-49af-9921-d14d9a94e997" containerName="istio-proxy" Oct 30 17:22:54.748716 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:54.748716 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="065164fc-1823-49af-9921-d14d9a94e997" containerName="istio-proxy" Oct 30 17:22:54.754066 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:54.754046 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71437729-w8n4k" Oct 30 17:22:54.772423 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:54.772396 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71437729-w8n4k"] Oct 30 17:22:54.804530 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:54.804502 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qdl9\" (UniqueName: \"kubernetes.io/projected/ad6583a4-0e86-4e83-8d27-5ec3f17060bd-kube-api-access-9qdl9\") pod \"maas-api-revoke-sub-71437729-w8n4k\" (UID: \"ad6583a4-0e86-4e83-8d27-5ec3f17060bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71437729-w8n4k" Oct 30 17:22:54.804958 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:54.804551 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad6583a4-0e86-4e83-8d27-5ec3f17060bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71437729-w8n4k\" (UID: \"ad6583a4-0e86-4e83-8d27-5ec3f17060bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71437729-w8n4k" Oct 30 17:22:54.804958 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:54.804636 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad6583a4-0e86-4e83-8d27-5ec3f17060bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71437729-w8n4k\" (UID: \"ad6583a4-0e86-4e83-8d27-5ec3f17060bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71437729-w8n4k" Oct 30 17:22:54.906528 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:54.906495 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qdl9\" (UniqueName: \"kubernetes.io/projected/ad6583a4-0e86-4e83-8d27-5ec3f17060bd-kube-api-access-9qdl9\") pod \"maas-api-revoke-sub-71437729-w8n4k\" (UID: \"ad6583a4-0e86-4e83-8d27-5ec3f17060bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71437729-w8n4k" Oct 30 17:22:54.906713 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:54.906560 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad6583a4-0e86-4e83-8d27-5ec3f17060bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71437729-w8n4k\" (UID: \"ad6583a4-0e86-4e83-8d27-5ec3f17060bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71437729-w8n4k" Oct 30 17:22:54.906713 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:54.906619 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad6583a4-0e86-4e83-8d27-5ec3f17060bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71437729-w8n4k\" (UID: \"ad6583a4-0e86-4e83-8d27-5ec3f17060bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71437729-w8n4k" Oct 30 17:22:54.907297 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:54.907277 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad6583a4-0e86-4e83-8d27-5ec3f17060bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71437729-w8n4k\" (UID: \"ad6583a4-0e86-4e83-8d27-5ec3f17060bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71437729-w8n4k" Oct 30 17:22:54.908911 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:54.908891 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad6583a4-0e86-4e83-8d27-5ec3f17060bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71437729-w8n4k\" (UID: \"ad6583a4-0e86-4e83-8d27-5ec3f17060bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71437729-w8n4k" Oct 30 17:22:54.931694 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:54.931668 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qdl9\" (UniqueName: \"kubernetes.io/projected/ad6583a4-0e86-4e83-8d27-5ec3f17060bd-kube-api-access-9qdl9\") pod \"maas-api-revoke-sub-71437729-w8n4k\" (UID: \"ad6583a4-0e86-4e83-8d27-5ec3f17060bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71437729-w8n4k" Oct 30 17:22:55.068380 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:55.068285 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71437729-w8n4k" Oct 30 17:22:55.295085 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:55.295046 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7c52f8-8hhsc" event={"ID":"a423a8b4-3931-4958-8961-358be318996f","Type":"ContainerStarted","Data":"e878ddfaa5b78346ad5a6f3177c3eb364d5b025d9c0edc5b5d80c13291443fc9"} Oct 30 17:22:55.295275 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:55.295103 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7c52f8-8hhsc" event={"ID":"a423a8b4-3931-4958-8961-358be318996f","Type":"ContainerStarted","Data":"8d879b4c301f2d4792bbd90b87cd1c1856a77b0a7f46254d8a903cc56c8a85c4"} Oct 30 17:22:55.342797 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:55.342698 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7c52f8-8hhsc" podStartSLOduration=2.342682446 podStartE2EDuration="2.342682446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:22:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:22:55.338575047 +0000 UTC m=+2149.339329970" watchObservedRunningTime="2026-10-30 17:22:55.342682446 +0000 UTC m=+2149.343437388" Oct 30 17:22:55.378558 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:22:55.378528 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad6583a4_0e86_4e83_8d27_5ec3f17060bd.slice/crio-fa3f7d882a5d1200a9d9adf11200b1a38cad143621d1771b6b8d3c997a95c00e WatchSource:0}: Error finding container fa3f7d882a5d1200a9d9adf11200b1a38cad143621d1771b6b8d3c997a95c00e: Status 404 returned error can't find the container with id fa3f7d882a5d1200a9d9adf11200b1a38cad143621d1771b6b8d3c997a95c00e Oct 30 17:22:55.381148 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:55.381117 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71437729-w8n4k"] Oct 30 17:22:56.007213 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:56.007180 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w5-78c9c2/rate-test-model-78c9c2-kserve-6fbd556d4f-6bbfp"] Oct 30 17:22:56.012838 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:56.012817 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-78c9c2/rate-test-model-78c9c2-kserve-6fbd556d4f-6bbfp" Oct 30 17:22:56.023385 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:56.023360 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-78c9c2/rate-test-model-78c9c2-kserve-6fbd556d4f-6bbfp"] Oct 30 17:22:56.031360 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:56.031333 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-78c9c2\"/\"kube-root-ca.crt\"" Oct 30 17:22:56.031360 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:56.031354 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-78c9c2\"/\"default-dockercfg-4ll8h\"" Oct 30 17:22:56.044239 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:56.044214 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-78c9c2\"/\"openshift-service-ca.crt\"" Oct 30 17:22:56.044239 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:56.044230 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-78c9c2\"/\"rate-test-model-78c9c2-kserve-self-signed-certs\"" Oct 30 17:22:56.127538 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:56.127500 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47d89bea-97aa-48c6-ab76-760d42f18344-model-cache\") pod \"rate-test-model-78c9c2-kserve-6fbd556d4f-6bbfp\" (UID: \"47d89bea-97aa-48c6-ab76-760d42f18344\") " pod="ai-tenant-e2e-shared-a-w5-78c9c2/rate-test-model-78c9c2-kserve-6fbd556d4f-6bbfp" Oct 30 17:22:56.127708 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:56.127558 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47d89bea-97aa-48c6-ab76-760d42f18344-tls-certs\") pod \"rate-test-model-78c9c2-kserve-6fbd556d4f-6bbfp\" (UID: \"47d89bea-97aa-48c6-ab76-760d42f18344\") " pod="ai-tenant-e2e-shared-a-w5-78c9c2/rate-test-model-78c9c2-kserve-6fbd556d4f-6bbfp" Oct 30 17:22:56.127821 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:56.127800 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47d89bea-97aa-48c6-ab76-760d42f18344-tmp-dir\") pod \"rate-test-model-78c9c2-kserve-6fbd556d4f-6bbfp\" (UID: \"47d89bea-97aa-48c6-ab76-760d42f18344\") " pod="ai-tenant-e2e-shared-a-w5-78c9c2/rate-test-model-78c9c2-kserve-6fbd556d4f-6bbfp" Oct 30 17:22:56.127954 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:56.127933 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47d89bea-97aa-48c6-ab76-760d42f18344-dshm\") pod \"rate-test-model-78c9c2-kserve-6fbd556d4f-6bbfp\" (UID: \"47d89bea-97aa-48c6-ab76-760d42f18344\") " pod="ai-tenant-e2e-shared-a-w5-78c9c2/rate-test-model-78c9c2-kserve-6fbd556d4f-6bbfp" Oct 30 17:22:56.128031 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:56.127980 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47d89bea-97aa-48c6-ab76-760d42f18344-home\") pod \"rate-test-model-78c9c2-kserve-6fbd556d4f-6bbfp\" (UID: \"47d89bea-97aa-48c6-ab76-760d42f18344\") " pod="ai-tenant-e2e-shared-a-w5-78c9c2/rate-test-model-78c9c2-kserve-6fbd556d4f-6bbfp" Oct 30 17:22:56.128031 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:56.128011 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59n7b\" (UniqueName: \"kubernetes.io/projected/47d89bea-97aa-48c6-ab76-760d42f18344-kube-api-access-59n7b\") pod \"rate-test-model-78c9c2-kserve-6fbd556d4f-6bbfp\" (UID: \"47d89bea-97aa-48c6-ab76-760d42f18344\") " pod="ai-tenant-e2e-shared-a-w5-78c9c2/rate-test-model-78c9c2-kserve-6fbd556d4f-6bbfp" Oct 30 17:22:56.240329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:56.230099 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47d89bea-97aa-48c6-ab76-760d42f18344-tls-certs\") pod \"rate-test-model-78c9c2-kserve-6fbd556d4f-6bbfp\" (UID: \"47d89bea-97aa-48c6-ab76-760d42f18344\") " pod="ai-tenant-e2e-shared-a-w5-78c9c2/rate-test-model-78c9c2-kserve-6fbd556d4f-6bbfp" Oct 30 17:22:56.240329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:56.230372 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47d89bea-97aa-48c6-ab76-760d42f18344-tmp-dir\") pod \"rate-test-model-78c9c2-kserve-6fbd556d4f-6bbfp\" (UID: \"47d89bea-97aa-48c6-ab76-760d42f18344\") " pod="ai-tenant-e2e-shared-a-w5-78c9c2/rate-test-model-78c9c2-kserve-6fbd556d4f-6bbfp" Oct 30 17:22:56.240329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:56.231345 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47d89bea-97aa-48c6-ab76-760d42f18344-dshm\") pod \"rate-test-model-78c9c2-kserve-6fbd556d4f-6bbfp\" (UID: \"47d89bea-97aa-48c6-ab76-760d42f18344\") " pod="ai-tenant-e2e-shared-a-w5-78c9c2/rate-test-model-78c9c2-kserve-6fbd556d4f-6bbfp" Oct 30 17:22:56.240329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:56.237488 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47d89bea-97aa-48c6-ab76-760d42f18344-home\") pod \"rate-test-model-78c9c2-kserve-6fbd556d4f-6bbfp\" (UID: \"47d89bea-97aa-48c6-ab76-760d42f18344\") " pod="ai-tenant-e2e-shared-a-w5-78c9c2/rate-test-model-78c9c2-kserve-6fbd556d4f-6bbfp" Oct 30 17:22:56.240329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:56.237533 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59n7b\" (UniqueName: \"kubernetes.io/projected/47d89bea-97aa-48c6-ab76-760d42f18344-kube-api-access-59n7b\") pod \"rate-test-model-78c9c2-kserve-6fbd556d4f-6bbfp\" (UID: \"47d89bea-97aa-48c6-ab76-760d42f18344\") " pod="ai-tenant-e2e-shared-a-w5-78c9c2/rate-test-model-78c9c2-kserve-6fbd556d4f-6bbfp" Oct 30 17:22:56.240329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:56.237617 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47d89bea-97aa-48c6-ab76-760d42f18344-model-cache\") pod \"rate-test-model-78c9c2-kserve-6fbd556d4f-6bbfp\" (UID: \"47d89bea-97aa-48c6-ab76-760d42f18344\") " pod="ai-tenant-e2e-shared-a-w5-78c9c2/rate-test-model-78c9c2-kserve-6fbd556d4f-6bbfp" Oct 30 17:22:56.240329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:56.238079 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47d89bea-97aa-48c6-ab76-760d42f18344-tmp-dir\") pod \"rate-test-model-78c9c2-kserve-6fbd556d4f-6bbfp\" (UID: \"47d89bea-97aa-48c6-ab76-760d42f18344\") " pod="ai-tenant-e2e-shared-a-w5-78c9c2/rate-test-model-78c9c2-kserve-6fbd556d4f-6bbfp" Oct 30 17:22:56.240329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:56.238971 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47d89bea-97aa-48c6-ab76-760d42f18344-home\") pod \"rate-test-model-78c9c2-kserve-6fbd556d4f-6bbfp\" (UID: \"47d89bea-97aa-48c6-ab76-760d42f18344\") " pod="ai-tenant-e2e-shared-a-w5-78c9c2/rate-test-model-78c9c2-kserve-6fbd556d4f-6bbfp" Oct 30 17:22:56.240329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:56.239890 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47d89bea-97aa-48c6-ab76-760d42f18344-model-cache\") pod \"rate-test-model-78c9c2-kserve-6fbd556d4f-6bbfp\" (UID: \"47d89bea-97aa-48c6-ab76-760d42f18344\") " pod="ai-tenant-e2e-shared-a-w5-78c9c2/rate-test-model-78c9c2-kserve-6fbd556d4f-6bbfp" Oct 30 17:22:56.297548 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:56.286010 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47d89bea-97aa-48c6-ab76-760d42f18344-tls-certs\") pod \"rate-test-model-78c9c2-kserve-6fbd556d4f-6bbfp\" (UID: \"47d89bea-97aa-48c6-ab76-760d42f18344\") " pod="ai-tenant-e2e-shared-a-w5-78c9c2/rate-test-model-78c9c2-kserve-6fbd556d4f-6bbfp" Oct 30 17:22:56.297548 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:56.288729 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47d89bea-97aa-48c6-ab76-760d42f18344-dshm\") pod \"rate-test-model-78c9c2-kserve-6fbd556d4f-6bbfp\" (UID: \"47d89bea-97aa-48c6-ab76-760d42f18344\") " pod="ai-tenant-e2e-shared-a-w5-78c9c2/rate-test-model-78c9c2-kserve-6fbd556d4f-6bbfp" Oct 30 17:22:56.311379 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:56.310391 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59n7b\" (UniqueName: \"kubernetes.io/projected/47d89bea-97aa-48c6-ab76-760d42f18344-kube-api-access-59n7b\") pod \"rate-test-model-78c9c2-kserve-6fbd556d4f-6bbfp\" (UID: \"47d89bea-97aa-48c6-ab76-760d42f18344\") " pod="ai-tenant-e2e-shared-a-w5-78c9c2/rate-test-model-78c9c2-kserve-6fbd556d4f-6bbfp" Oct 30 17:22:56.327745 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:56.327502 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-78c9c2/rate-test-model-78c9c2-kserve-6fbd556d4f-6bbfp" Oct 30 17:22:56.340862 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:56.340819 4522 generic.go:356] "Generic (PLEG): container finished" podID="a423a8b4-3931-4958-8961-358be318996f" containerID="e878ddfaa5b78346ad5a6f3177c3eb364d5b025d9c0edc5b5d80c13291443fc9" exitCode=0 Oct 30 17:22:56.341142 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:56.341114 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7c52f8-8hhsc" event={"ID":"a423a8b4-3931-4958-8961-358be318996f","Type":"ContainerDied","Data":"e878ddfaa5b78346ad5a6f3177c3eb364d5b025d9c0edc5b5d80c13291443fc9"} Oct 30 17:22:56.343426 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:56.343402 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71437729-w8n4k" event={"ID":"ad6583a4-0e86-4e83-8d27-5ec3f17060bd","Type":"ContainerStarted","Data":"f243f76b3e4047e0e08014a2b2cb0d3d9536f0ea24036a6cfa61cd19858a972e"} Oct 30 17:22:56.343598 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:56.343584 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71437729-w8n4k" event={"ID":"ad6583a4-0e86-4e83-8d27-5ec3f17060bd","Type":"ContainerStarted","Data":"fa3f7d882a5d1200a9d9adf11200b1a38cad143621d1771b6b8d3c997a95c00e"} Oct 30 17:22:56.447835 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:56.447784 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71437729-w8n4k" podStartSLOduration=2.44777068 podStartE2EDuration="2.44777068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:22:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:22:56.446602406 +0000 UTC m=+2150.447357308" watchObservedRunningTime="2026-10-30 17:22:56.44777068 +0000 UTC m=+2150.448525581" Oct 30 17:22:56.655644 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:22:56.655592 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod47d89bea_97aa_48c6_ab76_760d42f18344.slice/crio-3622b41d655d9479bd461b27201c0019fce29c6ea7ee123eefa6c7bb277166e7 WatchSource:0}: Error finding container 3622b41d655d9479bd461b27201c0019fce29c6ea7ee123eefa6c7bb277166e7: Status 404 returned error can't find the container with id 3622b41d655d9479bd461b27201c0019fce29c6ea7ee123eefa6c7bb277166e7 Oct 30 17:22:56.660297 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:56.660262 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-78c9c2/rate-test-model-78c9c2-kserve-6fbd556d4f-6bbfp"] Oct 30 17:22:57.279688 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:57.279595 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-11f480/sub-test-model-11f480-kserve-6946856dd5-j4hmg"] Oct 30 17:22:57.290689 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:57.286299 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-11f480/sub-test-model-11f480-kserve-6946856dd5-j4hmg" Oct 30 17:22:57.304295 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:57.304259 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-11f480\"/\"openshift-service-ca.crt\"" Oct 30 17:22:57.307182 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:57.307150 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-11f480/sub-test-model-11f480-kserve-6946856dd5-j4hmg"] Oct 30 17:22:57.313232 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:57.313202 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-11f480\"/\"sub-test-model-11f480-kserve-self-signed-certs\"" Oct 30 17:22:57.313410 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:57.313247 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-11f480\"/\"default-dockercfg-hnjkj\"" Oct 30 17:22:57.313539 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:57.313400 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-11f480\"/\"kube-root-ca.crt\"" Oct 30 17:22:57.353453 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:57.353407 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-78c9c2/rate-test-model-78c9c2-kserve-6fbd556d4f-6bbfp" event={"ID":"47d89bea-97aa-48c6-ab76-760d42f18344","Type":"ContainerStarted","Data":"bb74e67eae7a465f1b54227cfc19084827ba34acbdbf796c73e4f834219bae71"} Oct 30 17:22:57.353656 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:57.353457 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-78c9c2/rate-test-model-78c9c2-kserve-6fbd556d4f-6bbfp" event={"ID":"47d89bea-97aa-48c6-ab76-760d42f18344","Type":"ContainerStarted","Data":"3622b41d655d9479bd461b27201c0019fce29c6ea7ee123eefa6c7bb277166e7"} Oct 30 17:22:57.354787 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:57.354758 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d98b9965-90fe-44d6-b5de-9f861148d07a-tmp-dir\") pod \"sub-test-model-11f480-kserve-6946856dd5-j4hmg\" (UID: \"d98b9965-90fe-44d6-b5de-9f861148d07a\") " pod="ai-tenant-e2e-shared-a-w2-11f480/sub-test-model-11f480-kserve-6946856dd5-j4hmg" Oct 30 17:22:57.354985 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:57.354964 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d98b9965-90fe-44d6-b5de-9f861148d07a-dshm\") pod \"sub-test-model-11f480-kserve-6946856dd5-j4hmg\" (UID: \"d98b9965-90fe-44d6-b5de-9f861148d07a\") " pod="ai-tenant-e2e-shared-a-w2-11f480/sub-test-model-11f480-kserve-6946856dd5-j4hmg" Oct 30 17:22:57.355075 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:57.355023 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d98b9965-90fe-44d6-b5de-9f861148d07a-home\") pod \"sub-test-model-11f480-kserve-6946856dd5-j4hmg\" (UID: \"d98b9965-90fe-44d6-b5de-9f861148d07a\") " pod="ai-tenant-e2e-shared-a-w2-11f480/sub-test-model-11f480-kserve-6946856dd5-j4hmg" Oct 30 17:22:57.355141 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:57.355117 4522 generic.go:356] "Generic (PLEG): container finished" podID="ad6583a4-0e86-4e83-8d27-5ec3f17060bd" containerID="f243f76b3e4047e0e08014a2b2cb0d3d9536f0ea24036a6cfa61cd19858a972e" exitCode=0 Oct 30 17:22:57.355181 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:57.355164 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmdms\" (UniqueName: \"kubernetes.io/projected/d98b9965-90fe-44d6-b5de-9f861148d07a-kube-api-access-bmdms\") pod \"sub-test-model-11f480-kserve-6946856dd5-j4hmg\" (UID: \"d98b9965-90fe-44d6-b5de-9f861148d07a\") " pod="ai-tenant-e2e-shared-a-w2-11f480/sub-test-model-11f480-kserve-6946856dd5-j4hmg" Oct 30 17:22:57.355237 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:57.355204 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d98b9965-90fe-44d6-b5de-9f861148d07a-model-cache\") pod \"sub-test-model-11f480-kserve-6946856dd5-j4hmg\" (UID: \"d98b9965-90fe-44d6-b5de-9f861148d07a\") " pod="ai-tenant-e2e-shared-a-w2-11f480/sub-test-model-11f480-kserve-6946856dd5-j4hmg" Oct 30 17:22:57.355237 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:57.355203 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71437729-w8n4k" event={"ID":"ad6583a4-0e86-4e83-8d27-5ec3f17060bd","Type":"ContainerDied","Data":"f243f76b3e4047e0e08014a2b2cb0d3d9536f0ea24036a6cfa61cd19858a972e"} Oct 30 17:22:57.355319 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:57.355271 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d98b9965-90fe-44d6-b5de-9f861148d07a-tls-certs\") pod \"sub-test-model-11f480-kserve-6946856dd5-j4hmg\" (UID: \"d98b9965-90fe-44d6-b5de-9f861148d07a\") " pod="ai-tenant-e2e-shared-a-w2-11f480/sub-test-model-11f480-kserve-6946856dd5-j4hmg" Oct 30 17:22:57.448124 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:57.447704 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w5-78c9c2/rate-test-model-78c9c2-kserve-6fbd556d4f-6bbfp" podStartSLOduration=2.447683699 podStartE2EDuration="2.447683699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:22:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:22:57.441422523 +0000 UTC m=+2151.442177456" watchObservedRunningTime="2026-10-30 17:22:57.447683699 +0000 UTC m=+2151.448438601" Oct 30 17:22:57.462354 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:57.457549 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d98b9965-90fe-44d6-b5de-9f861148d07a-tmp-dir\") pod \"sub-test-model-11f480-kserve-6946856dd5-j4hmg\" (UID: \"d98b9965-90fe-44d6-b5de-9f861148d07a\") " pod="ai-tenant-e2e-shared-a-w2-11f480/sub-test-model-11f480-kserve-6946856dd5-j4hmg" Oct 30 17:22:57.462354 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:57.457749 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d98b9965-90fe-44d6-b5de-9f861148d07a-dshm\") pod \"sub-test-model-11f480-kserve-6946856dd5-j4hmg\" (UID: \"d98b9965-90fe-44d6-b5de-9f861148d07a\") " pod="ai-tenant-e2e-shared-a-w2-11f480/sub-test-model-11f480-kserve-6946856dd5-j4hmg" Oct 30 17:22:57.462354 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:57.457804 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d98b9965-90fe-44d6-b5de-9f861148d07a-home\") pod \"sub-test-model-11f480-kserve-6946856dd5-j4hmg\" (UID: \"d98b9965-90fe-44d6-b5de-9f861148d07a\") " pod="ai-tenant-e2e-shared-a-w2-11f480/sub-test-model-11f480-kserve-6946856dd5-j4hmg" Oct 30 17:22:57.462354 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:57.457913 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmdms\" (UniqueName: \"kubernetes.io/projected/d98b9965-90fe-44d6-b5de-9f861148d07a-kube-api-access-bmdms\") pod \"sub-test-model-11f480-kserve-6946856dd5-j4hmg\" (UID: \"d98b9965-90fe-44d6-b5de-9f861148d07a\") " pod="ai-tenant-e2e-shared-a-w2-11f480/sub-test-model-11f480-kserve-6946856dd5-j4hmg" Oct 30 17:22:57.462354 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:57.457953 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d98b9965-90fe-44d6-b5de-9f861148d07a-model-cache\") pod \"sub-test-model-11f480-kserve-6946856dd5-j4hmg\" (UID: \"d98b9965-90fe-44d6-b5de-9f861148d07a\") " pod="ai-tenant-e2e-shared-a-w2-11f480/sub-test-model-11f480-kserve-6946856dd5-j4hmg" Oct 30 17:22:57.462354 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:57.457983 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d98b9965-90fe-44d6-b5de-9f861148d07a-tls-certs\") pod \"sub-test-model-11f480-kserve-6946856dd5-j4hmg\" (UID: \"d98b9965-90fe-44d6-b5de-9f861148d07a\") " pod="ai-tenant-e2e-shared-a-w2-11f480/sub-test-model-11f480-kserve-6946856dd5-j4hmg" Oct 30 17:22:57.462354 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:57.459062 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d98b9965-90fe-44d6-b5de-9f861148d07a-tmp-dir\") pod \"sub-test-model-11f480-kserve-6946856dd5-j4hmg\" (UID: \"d98b9965-90fe-44d6-b5de-9f861148d07a\") " pod="ai-tenant-e2e-shared-a-w2-11f480/sub-test-model-11f480-kserve-6946856dd5-j4hmg" Oct 30 17:22:57.462354 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:57.461784 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d98b9965-90fe-44d6-b5de-9f861148d07a-home\") pod \"sub-test-model-11f480-kserve-6946856dd5-j4hmg\" (UID: \"d98b9965-90fe-44d6-b5de-9f861148d07a\") " pod="ai-tenant-e2e-shared-a-w2-11f480/sub-test-model-11f480-kserve-6946856dd5-j4hmg" Oct 30 17:22:57.462354 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:57.462116 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d98b9965-90fe-44d6-b5de-9f861148d07a-model-cache\") pod \"sub-test-model-11f480-kserve-6946856dd5-j4hmg\" (UID: \"d98b9965-90fe-44d6-b5de-9f861148d07a\") " pod="ai-tenant-e2e-shared-a-w2-11f480/sub-test-model-11f480-kserve-6946856dd5-j4hmg" Oct 30 17:22:57.534267 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:57.534173 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmdms\" (UniqueName: \"kubernetes.io/projected/d98b9965-90fe-44d6-b5de-9f861148d07a-kube-api-access-bmdms\") pod \"sub-test-model-11f480-kserve-6946856dd5-j4hmg\" (UID: \"d98b9965-90fe-44d6-b5de-9f861148d07a\") " pod="ai-tenant-e2e-shared-a-w2-11f480/sub-test-model-11f480-kserve-6946856dd5-j4hmg" Oct 30 17:22:57.569134 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:57.569099 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d98b9965-90fe-44d6-b5de-9f861148d07a-tls-certs\") pod \"sub-test-model-11f480-kserve-6946856dd5-j4hmg\" (UID: \"d98b9965-90fe-44d6-b5de-9f861148d07a\") " pod="ai-tenant-e2e-shared-a-w2-11f480/sub-test-model-11f480-kserve-6946856dd5-j4hmg" Oct 30 17:22:57.583149 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:57.578938 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d98b9965-90fe-44d6-b5de-9f861148d07a-dshm\") pod \"sub-test-model-11f480-kserve-6946856dd5-j4hmg\" (UID: \"d98b9965-90fe-44d6-b5de-9f861148d07a\") " pod="ai-tenant-e2e-shared-a-w2-11f480/sub-test-model-11f480-kserve-6946856dd5-j4hmg" Oct 30 17:22:57.613079 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:57.613052 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7c52f8-8hhsc" Oct 30 17:22:57.616841 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:57.616820 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-11f480/sub-test-model-11f480-kserve-6946856dd5-j4hmg" Oct 30 17:22:57.680326 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:57.677021 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a423a8b4-3931-4958-8961-358be318996f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a423a8b4-3931-4958-8961-358be318996f-maas-api-openshift-ca\") pod \"a423a8b4-3931-4958-8961-358be318996f\" (UID: \"a423a8b4-3931-4958-8961-358be318996f\") " Oct 30 17:22:57.680326 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:57.677196 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a423a8b4-3931-4958-8961-358be318996f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a423a8b4-3931-4958-8961-358be318996f-maas-api-xks-ca\") pod \"a423a8b4-3931-4958-8961-358be318996f\" (UID: \"a423a8b4-3931-4958-8961-358be318996f\") " Oct 30 17:22:57.680326 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:57.677253 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a423a8b4-3931-4958-8961-358be318996f-kube-api-access-kglgl\" (UniqueName: \"kubernetes.io/projected/a423a8b4-3931-4958-8961-358be318996f-kube-api-access-kglgl\") pod \"a423a8b4-3931-4958-8961-358be318996f\" (UID: \"a423a8b4-3931-4958-8961-358be318996f\") " Oct 30 17:22:57.692795 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:57.683492 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a423a8b4-3931-4958-8961-358be318996f-kube-api-access-kglgl" pod "a423a8b4-3931-4958-8961-358be318996f" (UID: "a423a8b4-3931-4958-8961-358be318996f"). InnerVolumeSpecName "kube-api-access-kglgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:22:57.692795 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:57.684056 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a423a8b4-3931-4958-8961-358be318996f-maas-api-openshift-ca" pod "a423a8b4-3931-4958-8961-358be318996f" (UID: "a423a8b4-3931-4958-8961-358be318996f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:22:57.692795 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:57.686597 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a423a8b4-3931-4958-8961-358be318996f-maas-api-xks-ca" pod "a423a8b4-3931-4958-8961-358be318996f" (UID: "a423a8b4-3931-4958-8961-358be318996f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:22:57.779708 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:57.779671 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a423a8b4-3931-4958-8961-358be318996f-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:22:57.779708 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:57.779707 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a423a8b4-3931-4958-8961-358be318996f-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:22:57.779708 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:57.779717 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kglgl\" (UniqueName: \"kubernetes.io/projected/a423a8b4-3931-4958-8961-358be318996f-kube-api-access-kglgl\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:22:57.979003 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:22:57.978975 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd98b9965_90fe_44d6_b5de_9f861148d07a.slice/crio-478dc94e718fc83460e5c32f60df443606a213c73df6a159bbab574d5a7faf8e WatchSource:0}: Error finding container 478dc94e718fc83460e5c32f60df443606a213c73df6a159bbab574d5a7faf8e: Status 404 returned error can't find the container with id 478dc94e718fc83460e5c32f60df443606a213c73df6a159bbab574d5a7faf8e Oct 30 17:22:57.980698 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:57.980673 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-11f480/sub-test-model-11f480-kserve-6946856dd5-j4hmg"] Oct 30 17:22:57.999439 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:57.999411 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c82666-56fc4d955c-29j8s"] Oct 30 17:22:57.999844 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:57.999793 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c82666-56fc4d955c-29j8s" podUID="08e720cb-49b1-4520-bd37-3f791fb9c4b2" containerName="maas-api" containerID="cri-o://1cac4d23fa81e3dbbc96f5abceb3e0960d9629f83d6cdacfa6aba24be1592aa8" gracePeriod=30 Oct 30 17:22:58.164512 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:58.164474 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c82666-56fc4d955c-29j8s" podUID="08e720cb-49b1-4520-bd37-3f791fb9c4b2" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.205:8443/health\": dial tcp 10.134.0.205:8443: connect: connection refused" Oct 30 17:22:58.328547 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:58.328418 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w5-78c9c2/rate-test-model-78c9c2-kserve-6fbd556d4f-6bbfp" Oct 30 17:22:58.372099 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:58.372063 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-11f480/sub-test-model-11f480-kserve-6946856dd5-j4hmg" event={"ID":"d98b9965-90fe-44d6-b5de-9f861148d07a","Type":"ContainerStarted","Data":"ea9b773c9a84fb0fcbc56b6f6884693e98bb1b3c75f1ae3d83b70cd7f663e3bb"} Oct 30 17:22:58.372257 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:58.372114 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-11f480/sub-test-model-11f480-kserve-6946856dd5-j4hmg" event={"ID":"d98b9965-90fe-44d6-b5de-9f861148d07a","Type":"ContainerStarted","Data":"478dc94e718fc83460e5c32f60df443606a213c73df6a159bbab574d5a7faf8e"} Oct 30 17:22:58.373850 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:58.373824 4522 generic.go:356] "Generic (PLEG): container finished" podID="08e720cb-49b1-4520-bd37-3f791fb9c4b2" containerID="1cac4d23fa81e3dbbc96f5abceb3e0960d9629f83d6cdacfa6aba24be1592aa8" exitCode=0 Oct 30 17:22:58.373980 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:58.373893 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c82666-56fc4d955c-29j8s" event={"ID":"08e720cb-49b1-4520-bd37-3f791fb9c4b2","Type":"ContainerDied","Data":"1cac4d23fa81e3dbbc96f5abceb3e0960d9629f83d6cdacfa6aba24be1592aa8"} Oct 30 17:22:58.375452 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:58.375422 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7c52f8-8hhsc" event={"ID":"a423a8b4-3931-4958-8961-358be318996f","Type":"ContainerDied","Data":"8d879b4c301f2d4792bbd90b87cd1c1856a77b0a7f46254d8a903cc56c8a85c4"} Oct 30 17:22:58.375563 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:58.375474 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d879b4c301f2d4792bbd90b87cd1c1856a77b0a7f46254d8a903cc56c8a85c4" Oct 30 17:22:58.375628 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:58.375613 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7c52f8-8hhsc" Oct 30 17:22:58.428221 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:58.428162 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-11f480/sub-test-model-11f480-kserve-6946856dd5-j4hmg" podStartSLOduration=1.42814551 podStartE2EDuration="1.42814551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:22:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:22:58.423328796 +0000 UTC m=+2152.424083697" watchObservedRunningTime="2026-10-30 17:22:58.42814551 +0000 UTC m=+2152.428900411" Oct 30 17:22:58.434858 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:58.434473 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c82666-56fc4d955c-29j8s" Oct 30 17:22:58.509467 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:58.509435 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08e720cb-49b1-4520-bd37-3f791fb9c4b2-kube-api-access-mr7fg\" (UniqueName: \"kubernetes.io/projected/08e720cb-49b1-4520-bd37-3f791fb9c4b2-kube-api-access-mr7fg\") pod \"08e720cb-49b1-4520-bd37-3f791fb9c4b2\" (UID: \"08e720cb-49b1-4520-bd37-3f791fb9c4b2\") " Oct 30 17:22:58.509799 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:58.509777 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08e720cb-49b1-4520-bd37-3f791fb9c4b2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08e720cb-49b1-4520-bd37-3f791fb9c4b2-maas-api-tls\") pod \"08e720cb-49b1-4520-bd37-3f791fb9c4b2\" (UID: \"08e720cb-49b1-4520-bd37-3f791fb9c4b2\") " Oct 30 17:22:58.509982 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:58.509963 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08e720cb-49b1-4520-bd37-3f791fb9c4b2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08e720cb-49b1-4520-bd37-3f791fb9c4b2-openshift-service-ca\") pod \"08e720cb-49b1-4520-bd37-3f791fb9c4b2\" (UID: \"08e720cb-49b1-4520-bd37-3f791fb9c4b2\") " Oct 30 17:22:58.510208 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:58.510195 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08e720cb-49b1-4520-bd37-3f791fb9c4b2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/08e720cb-49b1-4520-bd37-3f791fb9c4b2-metrics-certs\") pod \"08e720cb-49b1-4520-bd37-3f791fb9c4b2\" (UID: \"08e720cb-49b1-4520-bd37-3f791fb9c4b2\") " Oct 30 17:22:58.512447 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:58.512429 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08e720cb-49b1-4520-bd37-3f791fb9c4b2-openshift-service-ca" pod "08e720cb-49b1-4520-bd37-3f791fb9c4b2" (UID: "08e720cb-49b1-4520-bd37-3f791fb9c4b2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:22:58.520164 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:58.520117 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08e720cb-49b1-4520-bd37-3f791fb9c4b2-kube-api-access-mr7fg" pod "08e720cb-49b1-4520-bd37-3f791fb9c4b2" (UID: "08e720cb-49b1-4520-bd37-3f791fb9c4b2"). InnerVolumeSpecName "kube-api-access-mr7fg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:22:58.523938 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:58.523910 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08e720cb-49b1-4520-bd37-3f791fb9c4b2-metrics-certs" pod "08e720cb-49b1-4520-bd37-3f791fb9c4b2" (UID: "08e720cb-49b1-4520-bd37-3f791fb9c4b2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:22:58.524234 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:58.524216 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08e720cb-49b1-4520-bd37-3f791fb9c4b2-maas-api-tls" pod "08e720cb-49b1-4520-bd37-3f791fb9c4b2" (UID: "08e720cb-49b1-4520-bd37-3f791fb9c4b2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:22:58.622830 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:58.620088 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-11f480/sub-test-model-11f480-kserve-6946856dd5-j4hmg" Oct 30 17:22:58.622830 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:58.621843 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08e720cb-49b1-4520-bd37-3f791fb9c4b2-maas-api-tls\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:22:58.622830 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:58.621873 4522 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08e720cb-49b1-4520-bd37-3f791fb9c4b2-openshift-service-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:22:58.622830 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:58.621887 4522 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08e720cb-49b1-4520-bd37-3f791fb9c4b2-metrics-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:22:58.622830 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:58.621902 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mr7fg\" (UniqueName: \"kubernetes.io/projected/08e720cb-49b1-4520-bd37-3f791fb9c4b2-kube-api-access-mr7fg\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:22:58.650175 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:58.650140 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71437729-w8n4k" Oct 30 17:22:58.723254 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:58.723218 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad6583a4-0e86-4e83-8d27-5ec3f17060bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad6583a4-0e86-4e83-8d27-5ec3f17060bd-maas-api-xks-ca\") pod \"ad6583a4-0e86-4e83-8d27-5ec3f17060bd\" (UID: \"ad6583a4-0e86-4e83-8d27-5ec3f17060bd\") " Oct 30 17:22:58.723465 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:58.723409 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad6583a4-0e86-4e83-8d27-5ec3f17060bd-kube-api-access-9qdl9\" (UniqueName: \"kubernetes.io/projected/ad6583a4-0e86-4e83-8d27-5ec3f17060bd-kube-api-access-9qdl9\") pod \"ad6583a4-0e86-4e83-8d27-5ec3f17060bd\" (UID: \"ad6583a4-0e86-4e83-8d27-5ec3f17060bd\") " Oct 30 17:22:58.723567 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:58.723548 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad6583a4-0e86-4e83-8d27-5ec3f17060bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad6583a4-0e86-4e83-8d27-5ec3f17060bd-maas-api-openshift-ca\") pod \"ad6583a4-0e86-4e83-8d27-5ec3f17060bd\" (UID: \"ad6583a4-0e86-4e83-8d27-5ec3f17060bd\") " Oct 30 17:22:58.724209 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:58.723980 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad6583a4-0e86-4e83-8d27-5ec3f17060bd-maas-api-openshift-ca" pod "ad6583a4-0e86-4e83-8d27-5ec3f17060bd" (UID: "ad6583a4-0e86-4e83-8d27-5ec3f17060bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:22:58.724817 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:58.724783 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad6583a4-0e86-4e83-8d27-5ec3f17060bd-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:22:58.726910 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:58.726883 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad6583a4-0e86-4e83-8d27-5ec3f17060bd-maas-api-xks-ca" pod "ad6583a4-0e86-4e83-8d27-5ec3f17060bd" (UID: "ad6583a4-0e86-4e83-8d27-5ec3f17060bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:22:58.727019 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:58.726924 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad6583a4-0e86-4e83-8d27-5ec3f17060bd-kube-api-access-9qdl9" pod "ad6583a4-0e86-4e83-8d27-5ec3f17060bd" (UID: "ad6583a4-0e86-4e83-8d27-5ec3f17060bd"). InnerVolumeSpecName "kube-api-access-9qdl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:22:58.825920 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:58.825888 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qdl9\" (UniqueName: \"kubernetes.io/projected/ad6583a4-0e86-4e83-8d27-5ec3f17060bd-kube-api-access-9qdl9\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:22:58.825920 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:58.825917 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad6583a4-0e86-4e83-8d27-5ec3f17060bd-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:22:59.035018 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:59.034982 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c82666-t9m8f"] Oct 30 17:22:59.039707 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:59.039676 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c82666-t9m8f"] Oct 30 17:22:59.385723 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:59.385628 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71437729-w8n4k" Oct 30 17:22:59.385723 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:59.385660 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71437729-w8n4k" event={"ID":"ad6583a4-0e86-4e83-8d27-5ec3f17060bd","Type":"ContainerDied","Data":"fa3f7d882a5d1200a9d9adf11200b1a38cad143621d1771b6b8d3c997a95c00e"} Oct 30 17:22:59.385723 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:59.385708 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa3f7d882a5d1200a9d9adf11200b1a38cad143621d1771b6b8d3c997a95c00e" Oct 30 17:22:59.387266 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:59.387246 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c82666-56fc4d955c-29j8s" Oct 30 17:22:59.387463 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:59.387286 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c82666-56fc4d955c-29j8s" event={"ID":"08e720cb-49b1-4520-bd37-3f791fb9c4b2","Type":"ContainerDied","Data":"7d569920b50822779eb4548245af5b7bedf9f01ca2f7a72fc00c77a419eb493a"} Oct 30 17:22:59.387463 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:59.387366 4522 scope.go:122] "RemoveContainer" containerID="1cac4d23fa81e3dbbc96f5abceb3e0960d9629f83d6cdacfa6aba24be1592aa8" Oct 30 17:22:59.447941 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:59.447909 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c82666-56fc4d955c-29j8s"] Oct 30 17:22:59.459903 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:22:59.459873 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c82666-56fc4d955c-29j8s"] Oct 30 17:23:00.147502 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:00.147427 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08e720cb-49b1-4520-bd37-3f791fb9c4b2" path="/var/lib/kubelet/pods/08e720cb-49b1-4520-bd37-3f791fb9c4b2/volumes" Oct 30 17:23:00.147922 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:00.147907 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fa345e3-dbb2-4e58-bb92-6dcc9adcd63c" path="/var/lib/kubelet/pods/6fa345e3-dbb2-4e58-bb92-6dcc9adcd63c/volumes" Oct 30 17:23:03.888828 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:03.888793 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7c52f8-cf4f98854-2v744"] Oct 30 17:23:03.928440 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:03.928404 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7c52f8-c64c6769f-7m7qv"] Oct 30 17:23:06.328555 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:06.328509 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w5-78c9c2/rate-test-model-78c9c2-kserve-6fbd556d4f-6bbfp" Oct 30 17:23:06.341431 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:06.341398 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w5-78c9c2/rate-test-model-78c9c2-kserve-6fbd556d4f-6bbfp" Oct 30 17:23:06.458883 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:06.458852 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w5-78c9c2/rate-test-model-78c9c2-kserve-6fbd556d4f-6bbfp" Oct 30 17:23:06.511856 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:06.511819 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-j47tj"] Oct 30 17:23:06.512158 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:06.512129 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-j47tj" podUID="3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587" containerName="istio-proxy" containerID="cri-o://be87840f212e5810b5dfc76e4934a87cb9d7cb82442e4c126d6e5ba2653377b9" gracePeriod=30 Oct 30 17:23:07.617138 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:07.617104 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-11f480/sub-test-model-11f480-kserve-6946856dd5-j4hmg" Oct 30 17:23:07.629624 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:07.629593 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-11f480/sub-test-model-11f480-kserve-6946856dd5-j4hmg" Oct 30 17:23:08.478183 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:08.478155 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-11f480/sub-test-model-11f480-kserve-6946856dd5-j4hmg" Oct 30 17:23:08.890010 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:08.889914 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-7c52f8-cf4f98854-2v744" podUID="7bb27ee5-8209-41d1-bf37-b868a1dcc6a5" containerName="payload-processing" containerID="cri-o://d55657195295ec88d79a68b6a1341eeece3f844d66eadd92cde181a4f9292601" gracePeriod=25 Oct 30 17:23:08.929007 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:08.928962 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7c52f8-c64c6769f-7m7qv" podUID="3abcffab-3269-43fd-8d19-5fecae1eb0c1" containerName="payload-pre-processing" containerID="cri-o://569e9a604533dd6fb01d4644f3034eb5ff191563bb8cd262f6d24a158fd99ab3" gracePeriod=25 Oct 30 17:23:10.739955 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:10.739921 4522 patch_prober.go:29] interesting pod/e2e-inf-b-c82666-openshift-default-767d987659-j47tj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:23:10.740432 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:10.739975 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-j47tj" podUID="3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:23:11.808508 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:11.808479 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-j47tj" Oct 30 17:23:11.842797 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:11.842768 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587-credential-socket\") pod \"3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587\" (UID: \"3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587\") " Oct 30 17:23:11.842962 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:11.842832 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587-kube-api-access-bscjk\" (UniqueName: \"kubernetes.io/projected/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587-kube-api-access-bscjk\") pod \"3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587\" (UID: \"3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587\") " Oct 30 17:23:11.842962 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:11.842893 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587-istio-podinfo\") pod \"3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587\" (UID: \"3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587\") " Oct 30 17:23:11.843077 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:11.843043 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587-istio-envoy\") pod \"3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587\" (UID: \"3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587\") " Oct 30 17:23:11.843077 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:11.843061 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587-credential-socket" pod "3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587" (UID: "3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:11.843203 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:11.843094 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587-workload-certs\") pod \"3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587\" (UID: \"3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587\") " Oct 30 17:23:11.843203 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:11.843126 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587-istiod-ca-cert\") pod \"3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587\" (UID: \"3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587\") " Oct 30 17:23:11.843355 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:11.843255 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587-istio-data\") pod \"3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587\" (UID: \"3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587\") " Oct 30 17:23:11.843355 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:11.843291 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587-istio-token\" (UniqueName: \"kubernetes.io/projected/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587-istio-token\") pod \"3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587\" (UID: \"3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587\") " Oct 30 17:23:11.843485 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:11.843358 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587-workload-socket\") pod \"3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587\" (UID: \"3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587\") " Oct 30 17:23:11.843485 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:11.843426 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587-workload-certs" pod "3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587" (UID: "3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:11.843617 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:11.843593 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587-istio-data" pod "3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587" (UID: "3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:11.843776 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:11.843753 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587-istiod-ca-cert" pod "3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587" (UID: "3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:23:11.844042 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:11.843936 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587-workload-socket" pod "3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587" (UID: "3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:11.844294 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:11.844276 4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587-workload-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:23:11.844294 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:11.844295 4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587-istiod-ca-cert\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:23:11.844481 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:11.844318 4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587-istio-data\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:23:11.844481 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:11.844332 4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587-workload-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:23:11.844481 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:11.844346 4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587-credential-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:23:11.845392 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:11.845370 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587-istio-podinfo" pod "3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587" (UID: "3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:23:11.845545 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:11.845525 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587-istio-token" pod "3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587" (UID: "3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:23:11.845672 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:11.845653 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587-kube-api-access-bscjk" pod "3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587" (UID: "3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587"). InnerVolumeSpecName "kube-api-access-bscjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:23:11.845744 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:11.845717 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587-istio-envoy" pod "3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587" (UID: "3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:11.945813 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:11.945775 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bscjk\" (UniqueName: \"kubernetes.io/projected/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587-kube-api-access-bscjk\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:23:11.945813 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:11.945813 4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587-istio-podinfo\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:23:11.946032 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:11.945828 4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587-istio-envoy\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:23:11.946032 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:11.945839 4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587-istio-token\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:23:12.500215 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:12.500185 4522 generic.go:356] "Generic (PLEG): container finished" podID="3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587" containerID="be87840f212e5810b5dfc76e4934a87cb9d7cb82442e4c126d6e5ba2653377b9" exitCode=0 Oct 30 17:23:12.500429 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:12.500243 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-j47tj" Oct 30 17:23:12.500429 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:12.500268 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-j47tj" event={"ID":"3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587","Type":"ContainerDied","Data":"be87840f212e5810b5dfc76e4934a87cb9d7cb82442e4c126d6e5ba2653377b9"} Oct 30 17:23:12.500429 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:12.500350 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-j47tj" event={"ID":"3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587","Type":"ContainerDied","Data":"a49a51d774bb6635b2fac508d918a99f34edb0b2a80ac09522592367101bc1f8"} Oct 30 17:23:12.500429 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:12.500375 4522 scope.go:122] "RemoveContainer" containerID="be87840f212e5810b5dfc76e4934a87cb9d7cb82442e4c126d6e5ba2653377b9" Oct 30 17:23:12.511786 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:12.511768 4522 scope.go:122] "RemoveContainer" containerID="be87840f212e5810b5dfc76e4934a87cb9d7cb82442e4c126d6e5ba2653377b9" Oct 30 17:23:12.512067 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:23:12.512043 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be87840f212e5810b5dfc76e4934a87cb9d7cb82442e4c126d6e5ba2653377b9\": container with ID starting with be87840f212e5810b5dfc76e4934a87cb9d7cb82442e4c126d6e5ba2653377b9 not found: ID does not exist" containerID="be87840f212e5810b5dfc76e4934a87cb9d7cb82442e4c126d6e5ba2653377b9" Oct 30 17:23:12.512140 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:12.512078 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be87840f212e5810b5dfc76e4934a87cb9d7cb82442e4c126d6e5ba2653377b9"} err="failed to get container status \"be87840f212e5810b5dfc76e4934a87cb9d7cb82442e4c126d6e5ba2653377b9\": rpc error: code = NotFound desc = could not find container \"be87840f212e5810b5dfc76e4934a87cb9d7cb82442e4c126d6e5ba2653377b9\": container with ID starting with be87840f212e5810b5dfc76e4934a87cb9d7cb82442e4c126d6e5ba2653377b9 not found: ID does not exist" Oct 30 17:23:12.582335 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:12.582289 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-j47tj"] Oct 30 17:23:12.594728 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:12.594698 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-j47tj"] Oct 30 17:23:12.950378 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:23:12.950269 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:23:14.149401 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:14.149368 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587" path="/var/lib/kubelet/pods/3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587/volumes" Oct 30 17:23:16.789987 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:16.789947 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7c52f8-c64c6769f-7m7qv" podUID="3abcffab-3269-43fd-8d19-5fecae1eb0c1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:23:16.789987 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:16.789986 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-7c52f8-cf4f98854-2v744" podUID="7bb27ee5-8209-41d1-bf37-b868a1dcc6a5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:23:26.790707 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:26.790663 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-7c52f8-cf4f98854-2v744" podUID="7bb27ee5-8209-41d1-bf37-b868a1dcc6a5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:23:26.791225 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:26.790678 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7c52f8-c64c6769f-7m7qv" podUID="3abcffab-3269-43fd-8d19-5fecae1eb0c1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:23:27.805025 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:27.804987 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7c52f8-8hhsc"] Oct 30 17:23:27.817071 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:27.817039 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7c52f8-8hhsc"] Oct 30 17:23:28.147525 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:28.147434 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a423a8b4-3931-4958-8961-358be318996f" path="/var/lib/kubelet/pods/a423a8b4-3931-4958-8961-358be318996f/volumes" Oct 30 17:23:29.259330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:29.257071 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7c52f8-cf4f98854-2v744" Oct 30 17:23:29.365555 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:29.337876 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bb27ee5-8209-41d1-bf37-b868a1dcc6a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bb27ee5-8209-41d1-bf37-b868a1dcc6a5-plugins-config-volume\") pod \"7bb27ee5-8209-41d1-bf37-b868a1dcc6a5\" (UID: \"7bb27ee5-8209-41d1-bf37-b868a1dcc6a5\") " Oct 30 17:23:29.365555 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:29.338045 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bb27ee5-8209-41d1-bf37-b868a1dcc6a5-kube-api-access-jv2bs\" (UniqueName: \"kubernetes.io/projected/7bb27ee5-8209-41d1-bf37-b868a1dcc6a5-kube-api-access-jv2bs\") pod \"7bb27ee5-8209-41d1-bf37-b868a1dcc6a5\" (UID: \"7bb27ee5-8209-41d1-bf37-b868a1dcc6a5\") " Oct 30 17:23:29.365555 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:29.344040 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bb27ee5-8209-41d1-bf37-b868a1dcc6a5-plugins-config-volume" pod "7bb27ee5-8209-41d1-bf37-b868a1dcc6a5" (UID: "7bb27ee5-8209-41d1-bf37-b868a1dcc6a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:23:29.383398 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:29.366011 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bb27ee5-8209-41d1-bf37-b868a1dcc6a5-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:23:29.383398 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:29.383170 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bb27ee5-8209-41d1-bf37-b868a1dcc6a5-kube-api-access-jv2bs" pod "7bb27ee5-8209-41d1-bf37-b868a1dcc6a5" (UID: "7bb27ee5-8209-41d1-bf37-b868a1dcc6a5"). InnerVolumeSpecName "kube-api-access-jv2bs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:23:29.464292 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:29.464251 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7c52f8-c64c6769f-7m7qv" Oct 30 17:23:29.468863 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:29.468068 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jv2bs\" (UniqueName: \"kubernetes.io/projected/7bb27ee5-8209-41d1-bf37-b868a1dcc6a5-kube-api-access-jv2bs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:23:29.574038 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:29.573956 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3abcffab-3269-43fd-8d19-5fecae1eb0c1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3abcffab-3269-43fd-8d19-5fecae1eb0c1-plugins-config-volume\") pod \"3abcffab-3269-43fd-8d19-5fecae1eb0c1\" (UID: \"3abcffab-3269-43fd-8d19-5fecae1eb0c1\") " Oct 30 17:23:29.574854 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:29.574797 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3abcffab-3269-43fd-8d19-5fecae1eb0c1-kube-api-access-2n9n4\" (UniqueName: \"kubernetes.io/projected/3abcffab-3269-43fd-8d19-5fecae1eb0c1-kube-api-access-2n9n4\") pod \"3abcffab-3269-43fd-8d19-5fecae1eb0c1\" (UID: \"3abcffab-3269-43fd-8d19-5fecae1eb0c1\") " Oct 30 17:23:29.602344 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:29.594572 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3abcffab-3269-43fd-8d19-5fecae1eb0c1-plugins-config-volume" pod "3abcffab-3269-43fd-8d19-5fecae1eb0c1" (UID: "3abcffab-3269-43fd-8d19-5fecae1eb0c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:23:29.625942 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:29.625889 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3abcffab-3269-43fd-8d19-5fecae1eb0c1-kube-api-access-2n9n4" pod "3abcffab-3269-43fd-8d19-5fecae1eb0c1" (UID: "3abcffab-3269-43fd-8d19-5fecae1eb0c1"). InnerVolumeSpecName "kube-api-access-2n9n4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:23:29.678329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:29.673614 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7c52f8-c64c6769f-7m7qv" Oct 30 17:23:29.678329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:29.677855 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7c52f8-c64c6769f-7m7qv" event={"ID":"3abcffab-3269-43fd-8d19-5fecae1eb0c1","Type":"ContainerDied","Data":"569e9a604533dd6fb01d4644f3034eb5ff191563bb8cd262f6d24a158fd99ab3"} Oct 30 17:23:29.678329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:29.677929 4522 scope.go:122] "RemoveContainer" containerID="569e9a604533dd6fb01d4644f3034eb5ff191563bb8cd262f6d24a158fd99ab3" Oct 30 17:23:29.682512 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:29.682480 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n9n4\" (UniqueName: \"kubernetes.io/projected/3abcffab-3269-43fd-8d19-5fecae1eb0c1-kube-api-access-2n9n4\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:23:29.682646 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:29.682517 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3abcffab-3269-43fd-8d19-5fecae1eb0c1-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:23:29.687293 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:29.673510 4522 generic.go:356] "Generic (PLEG): container finished" podID="3abcffab-3269-43fd-8d19-5fecae1eb0c1" containerID="569e9a604533dd6fb01d4644f3034eb5ff191563bb8cd262f6d24a158fd99ab3" exitCode=0 Oct 30 17:23:29.687631 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:29.687602 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7c52f8-c64c6769f-7m7qv" event={"ID":"3abcffab-3269-43fd-8d19-5fecae1eb0c1","Type":"ContainerDied","Data":"a8ea693d497d84f02fb4cd0861e204c0ac18665a97e7af7795a6da1dfd814ce6"} Oct 30 17:23:29.696507 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:29.696483 4522 scope.go:122] "RemoveContainer" containerID="569e9a604533dd6fb01d4644f3034eb5ff191563bb8cd262f6d24a158fd99ab3" Oct 30 17:23:29.697661 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:23:29.697277 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"569e9a604533dd6fb01d4644f3034eb5ff191563bb8cd262f6d24a158fd99ab3\": container with ID starting with 569e9a604533dd6fb01d4644f3034eb5ff191563bb8cd262f6d24a158fd99ab3 not found: ID does not exist" containerID="569e9a604533dd6fb01d4644f3034eb5ff191563bb8cd262f6d24a158fd99ab3" Oct 30 17:23:29.697866 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:29.697836 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"569e9a604533dd6fb01d4644f3034eb5ff191563bb8cd262f6d24a158fd99ab3"} err="failed to get container status \"569e9a604533dd6fb01d4644f3034eb5ff191563bb8cd262f6d24a158fd99ab3\": rpc error: code = NotFound desc = could not find container \"569e9a604533dd6fb01d4644f3034eb5ff191563bb8cd262f6d24a158fd99ab3\": container with ID starting with 569e9a604533dd6fb01d4644f3034eb5ff191563bb8cd262f6d24a158fd99ab3 not found: ID does not exist" Oct 30 17:23:29.700329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:29.699556 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7c52f8-cf4f98854-2v744" Oct 30 17:23:29.704764 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:29.704597 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7c52f8-cf4f98854-2v744" event={"ID":"7bb27ee5-8209-41d1-bf37-b868a1dcc6a5","Type":"ContainerDied","Data":"d55657195295ec88d79a68b6a1341eeece3f844d66eadd92cde181a4f9292601"} Oct 30 17:23:29.704764 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:29.704669 4522 scope.go:122] "RemoveContainer" containerID="d55657195295ec88d79a68b6a1341eeece3f844d66eadd92cde181a4f9292601" Oct 30 17:23:29.704989 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:29.704965 4522 generic.go:356] "Generic (PLEG): container finished" podID="7bb27ee5-8209-41d1-bf37-b868a1dcc6a5" containerID="d55657195295ec88d79a68b6a1341eeece3f844d66eadd92cde181a4f9292601" exitCode=0 Oct 30 17:23:29.705186 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:29.705168 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7c52f8-cf4f98854-2v744" event={"ID":"7bb27ee5-8209-41d1-bf37-b868a1dcc6a5","Type":"ContainerDied","Data":"e8b6e8725113920cdccc4f3cc26009c426429c5ab2dd2708e20f030ed8900890"} Oct 30 17:23:29.826552 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:29.819452 4522 scope.go:122] "RemoveContainer" containerID="d55657195295ec88d79a68b6a1341eeece3f844d66eadd92cde181a4f9292601" Oct 30 17:23:29.857367 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:23:29.831880 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d55657195295ec88d79a68b6a1341eeece3f844d66eadd92cde181a4f9292601\": container with ID starting with d55657195295ec88d79a68b6a1341eeece3f844d66eadd92cde181a4f9292601 not found: ID does not exist" containerID="d55657195295ec88d79a68b6a1341eeece3f844d66eadd92cde181a4f9292601" Oct 30 17:23:29.857367 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:29.831959 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d55657195295ec88d79a68b6a1341eeece3f844d66eadd92cde181a4f9292601"} err="failed to get container status \"d55657195295ec88d79a68b6a1341eeece3f844d66eadd92cde181a4f9292601\": rpc error: code = NotFound desc = could not find container \"d55657195295ec88d79a68b6a1341eeece3f844d66eadd92cde181a4f9292601\": container with ID starting with d55657195295ec88d79a68b6a1341eeece3f844d66eadd92cde181a4f9292601 not found: ID does not exist" Oct 30 17:23:29.882326 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:29.880881 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7c52f8-c64c6769f-7m7qv"] Oct 30 17:23:29.887648 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:29.886669 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7c52f8-c64c6769f-7m7qv"] Oct 30 17:23:29.931760 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:29.930670 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7c52f8-cf4f98854-2v744"] Oct 30 17:23:29.953330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:29.943855 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7c52f8-cf4f98854-2v744"] Oct 30 17:23:30.148340 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:30.148253 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3abcffab-3269-43fd-8d19-5fecae1eb0c1" path="/var/lib/kubelet/pods/3abcffab-3269-43fd-8d19-5fecae1eb0c1/volumes" Oct 30 17:23:30.148943 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:30.148923 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bb27ee5-8209-41d1-bf37-b868a1dcc6a5" path="/var/lib/kubelet/pods/7bb27ee5-8209-41d1-bf37-b868a1dcc6a5/volumes" Oct 30 17:23:30.471124 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:30.471090 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e08b0a0b-6xr9w"] Oct 30 17:23:30.472285 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:30.472258 4522 cpu_manager.go:414] "containerMap: removing container" podUID="a423a8b4-3931-4958-8961-358be318996f" containerName="revoke-keys" Oct 30 17:23:30.472285 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:30.472283 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a423a8b4-3931-4958-8961-358be318996f" containerName="revoke-keys" Oct 30 17:23:30.472539 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:30.472328 4522 cpu_manager.go:414] "containerMap: removing container" podUID="3abcffab-3269-43fd-8d19-5fecae1eb0c1" containerName="payload-pre-processing" Oct 30 17:23:30.472539 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:30.472336 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3abcffab-3269-43fd-8d19-5fecae1eb0c1" containerName="payload-pre-processing" Oct 30 17:23:30.472539 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:30.472349 4522 cpu_manager.go:414] "containerMap: removing container" podUID="ad6583a4-0e86-4e83-8d27-5ec3f17060bd" containerName="revoke-keys" Oct 30 17:23:30.472539 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:30.472356 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad6583a4-0e86-4e83-8d27-5ec3f17060bd" containerName="revoke-keys" Oct 30 17:23:30.472539 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:30.472386 4522 cpu_manager.go:414] "containerMap: removing container" podUID="7bb27ee5-8209-41d1-bf37-b868a1dcc6a5" containerName="payload-processing" Oct 30 17:23:30.472539 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:30.472392 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bb27ee5-8209-41d1-bf37-b868a1dcc6a5" containerName="payload-processing" Oct 30 17:23:30.472539 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:30.472425 4522 cpu_manager.go:414] "containerMap: removing container" podUID="08e720cb-49b1-4520-bd37-3f791fb9c4b2" containerName="maas-api" Oct 30 17:23:30.472539 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:30.472433 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08e720cb-49b1-4520-bd37-3f791fb9c4b2" containerName="maas-api" Oct 30 17:23:30.472539 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:30.472441 4522 cpu_manager.go:414] "containerMap: removing container" podUID="3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587" containerName="istio-proxy" Oct 30 17:23:30.472539 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:30.472447 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587" containerName="istio-proxy" Oct 30 17:23:30.473101 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:30.472671 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="3abcffab-3269-43fd-8d19-5fecae1eb0c1" containerName="payload-pre-processing" Oct 30 17:23:30.473101 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:30.472690 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad6583a4-0e86-4e83-8d27-5ec3f17060bd" containerName="revoke-keys" Oct 30 17:23:30.473101 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:30.472704 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bb27ee5-8209-41d1-bf37-b868a1dcc6a5" containerName="payload-processing" Oct 30 17:23:30.473101 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:30.472717 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="08e720cb-49b1-4520-bd37-3f791fb9c4b2" containerName="maas-api" Oct 30 17:23:30.473101 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:30.472729 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f6ad9f3-0cb9-4e5b-96e1-42d9d0798587" containerName="istio-proxy" Oct 30 17:23:30.473101 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:30.472751 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="a423a8b4-3931-4958-8961-358be318996f" containerName="revoke-keys" Oct 30 17:23:30.478278 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:30.478257 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e08b0a0b-6xr9w" Oct 30 17:23:30.494981 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:30.494932 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:23:30.495182 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:30.495057 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:23:30.497617 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:30.497592 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e08b0a0b-6xr9w"] Oct 30 17:23:30.644109 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:30.644076 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5528580-d2f5-4201-859e-f9f2c1e6c588-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e08b0a0b-6xr9w\" (UID: \"f5528580-d2f5-4201-859e-f9f2c1e6c588\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e08b0a0b-6xr9w" Oct 30 17:23:30.644316 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:30.644227 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfv9g\" (UniqueName: \"kubernetes.io/projected/f5528580-d2f5-4201-859e-f9f2c1e6c588-kube-api-access-xfv9g\") pod \"maas-api-revoke-sub-e08b0a0b-6xr9w\" (UID: \"f5528580-d2f5-4201-859e-f9f2c1e6c588\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e08b0a0b-6xr9w" Oct 30 17:23:30.644388 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:30.644364 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5528580-d2f5-4201-859e-f9f2c1e6c588-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e08b0a0b-6xr9w\" (UID: \"f5528580-d2f5-4201-859e-f9f2c1e6c588\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e08b0a0b-6xr9w" Oct 30 17:23:30.665775 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:30.660727 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-8681b5/sub-test-model-8681b5-kserve-6dcbf55b89-hsdx9"] Oct 30 17:23:30.682610 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:30.682573 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-8681b5/sub-test-model-8681b5-kserve-6dcbf55b89-hsdx9" Oct 30 17:23:30.699042 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:30.699015 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8681b5/sub-test-model-8681b5-kserve-6dcbf55b89-hsdx9"] Oct 30 17:23:30.699752 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:30.699720 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-8681b5\"/\"sub-test-model-8681b5-kserve-self-signed-certs\"" Oct 30 17:23:30.699908 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:30.699835 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-8681b5\"/\"default-dockercfg-5twb5\"" Oct 30 17:23:30.699908 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:30.699869 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-8681b5\"/\"kube-root-ca.crt\"" Oct 30 17:23:30.700534 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:30.700509 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-8681b5\"/\"openshift-service-ca.crt\"" Oct 30 17:23:30.745887 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:30.745804 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5528580-d2f5-4201-859e-f9f2c1e6c588-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e08b0a0b-6xr9w\" (UID: \"f5528580-d2f5-4201-859e-f9f2c1e6c588\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e08b0a0b-6xr9w" Oct 30 17:23:30.746047 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:30.745906 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfv9g\" (UniqueName: \"kubernetes.io/projected/f5528580-d2f5-4201-859e-f9f2c1e6c588-kube-api-access-xfv9g\") pod \"maas-api-revoke-sub-e08b0a0b-6xr9w\" (UID: \"f5528580-d2f5-4201-859e-f9f2c1e6c588\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e08b0a0b-6xr9w" Oct 30 17:23:30.746047 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:30.746033 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5528580-d2f5-4201-859e-f9f2c1e6c588-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e08b0a0b-6xr9w\" (UID: \"f5528580-d2f5-4201-859e-f9f2c1e6c588\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e08b0a0b-6xr9w" Oct 30 17:23:30.746959 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:30.746933 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5528580-d2f5-4201-859e-f9f2c1e6c588-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e08b0a0b-6xr9w\" (UID: \"f5528580-d2f5-4201-859e-f9f2c1e6c588\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e08b0a0b-6xr9w" Oct 30 17:23:30.748357 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:30.748330 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5528580-d2f5-4201-859e-f9f2c1e6c588-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e08b0a0b-6xr9w\" (UID: \"f5528580-d2f5-4201-859e-f9f2c1e6c588\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e08b0a0b-6xr9w" Oct 30 17:23:30.776565 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:30.776529 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfv9g\" (UniqueName: \"kubernetes.io/projected/f5528580-d2f5-4201-859e-f9f2c1e6c588-kube-api-access-xfv9g\") pod \"maas-api-revoke-sub-e08b0a0b-6xr9w\" (UID: \"f5528580-d2f5-4201-859e-f9f2c1e6c588\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e08b0a0b-6xr9w" Oct 30 17:23:30.798290 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:30.798255 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e08b0a0b-6xr9w" Oct 30 17:23:30.848030 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:30.847994 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b-home\") pod \"sub-test-model-8681b5-kserve-6dcbf55b89-hsdx9\" (UID: \"3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b\") " pod="ai-tenant-e2e-shared-a-w3-8681b5/sub-test-model-8681b5-kserve-6dcbf55b89-hsdx9" Oct 30 17:23:30.848194 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:30.848038 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b-model-cache\") pod \"sub-test-model-8681b5-kserve-6dcbf55b89-hsdx9\" (UID: \"3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b\") " pod="ai-tenant-e2e-shared-a-w3-8681b5/sub-test-model-8681b5-kserve-6dcbf55b89-hsdx9" Oct 30 17:23:30.848194 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:30.848137 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b-tmp-dir\") pod \"sub-test-model-8681b5-kserve-6dcbf55b89-hsdx9\" (UID: \"3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b\") " pod="ai-tenant-e2e-shared-a-w3-8681b5/sub-test-model-8681b5-kserve-6dcbf55b89-hsdx9" Oct 30 17:23:30.848335 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:30.848269 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b-dshm\") pod \"sub-test-model-8681b5-kserve-6dcbf55b89-hsdx9\" (UID: \"3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b\") " pod="ai-tenant-e2e-shared-a-w3-8681b5/sub-test-model-8681b5-kserve-6dcbf55b89-hsdx9" Oct 30 17:23:30.848416 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:30.848361 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r84kk\" (UniqueName: \"kubernetes.io/projected/3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b-kube-api-access-r84kk\") pod \"sub-test-model-8681b5-kserve-6dcbf55b89-hsdx9\" (UID: \"3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b\") " pod="ai-tenant-e2e-shared-a-w3-8681b5/sub-test-model-8681b5-kserve-6dcbf55b89-hsdx9" Oct 30 17:23:30.848512 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:30.848497 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b-tls-certs\") pod \"sub-test-model-8681b5-kserve-6dcbf55b89-hsdx9\" (UID: \"3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b\") " pod="ai-tenant-e2e-shared-a-w3-8681b5/sub-test-model-8681b5-kserve-6dcbf55b89-hsdx9" Oct 30 17:23:30.954944 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:30.952052 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b-tls-certs\") pod \"sub-test-model-8681b5-kserve-6dcbf55b89-hsdx9\" (UID: \"3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b\") " pod="ai-tenant-e2e-shared-a-w3-8681b5/sub-test-model-8681b5-kserve-6dcbf55b89-hsdx9" Oct 30 17:23:30.954944 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:30.952402 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b-home\") pod \"sub-test-model-8681b5-kserve-6dcbf55b89-hsdx9\" (UID: \"3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b\") " pod="ai-tenant-e2e-shared-a-w3-8681b5/sub-test-model-8681b5-kserve-6dcbf55b89-hsdx9" Oct 30 17:23:30.954944 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:30.952831 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b-home\") pod \"sub-test-model-8681b5-kserve-6dcbf55b89-hsdx9\" (UID: \"3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b\") " pod="ai-tenant-e2e-shared-a-w3-8681b5/sub-test-model-8681b5-kserve-6dcbf55b89-hsdx9" Oct 30 17:23:30.954944 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:30.952953 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b-model-cache\") pod \"sub-test-model-8681b5-kserve-6dcbf55b89-hsdx9\" (UID: \"3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b\") " pod="ai-tenant-e2e-shared-a-w3-8681b5/sub-test-model-8681b5-kserve-6dcbf55b89-hsdx9" Oct 30 17:23:30.954944 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:30.952989 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b-tmp-dir\") pod \"sub-test-model-8681b5-kserve-6dcbf55b89-hsdx9\" (UID: \"3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b\") " pod="ai-tenant-e2e-shared-a-w3-8681b5/sub-test-model-8681b5-kserve-6dcbf55b89-hsdx9" Oct 30 17:23:30.954944 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:30.953060 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b-dshm\") pod \"sub-test-model-8681b5-kserve-6dcbf55b89-hsdx9\" (UID: \"3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b\") " pod="ai-tenant-e2e-shared-a-w3-8681b5/sub-test-model-8681b5-kserve-6dcbf55b89-hsdx9" Oct 30 17:23:30.954944 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:30.953113 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r84kk\" (UniqueName: \"kubernetes.io/projected/3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b-kube-api-access-r84kk\") pod \"sub-test-model-8681b5-kserve-6dcbf55b89-hsdx9\" (UID: \"3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b\") " pod="ai-tenant-e2e-shared-a-w3-8681b5/sub-test-model-8681b5-kserve-6dcbf55b89-hsdx9" Oct 30 17:23:30.954944 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:30.953290 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b-model-cache\") pod \"sub-test-model-8681b5-kserve-6dcbf55b89-hsdx9\" (UID: \"3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b\") " pod="ai-tenant-e2e-shared-a-w3-8681b5/sub-test-model-8681b5-kserve-6dcbf55b89-hsdx9" Oct 30 17:23:30.954944 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:30.953639 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b-tmp-dir\") pod \"sub-test-model-8681b5-kserve-6dcbf55b89-hsdx9\" (UID: \"3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b\") " pod="ai-tenant-e2e-shared-a-w3-8681b5/sub-test-model-8681b5-kserve-6dcbf55b89-hsdx9" Oct 30 17:23:30.957144 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:30.956518 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b-dshm\") pod \"sub-test-model-8681b5-kserve-6dcbf55b89-hsdx9\" (UID: \"3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b\") " pod="ai-tenant-e2e-shared-a-w3-8681b5/sub-test-model-8681b5-kserve-6dcbf55b89-hsdx9" Oct 30 17:23:30.958952 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:30.958906 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b-tls-certs\") pod \"sub-test-model-8681b5-kserve-6dcbf55b89-hsdx9\" (UID: \"3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b\") " pod="ai-tenant-e2e-shared-a-w3-8681b5/sub-test-model-8681b5-kserve-6dcbf55b89-hsdx9" Oct 30 17:23:31.040391 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:31.039940 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r84kk\" (UniqueName: \"kubernetes.io/projected/3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b-kube-api-access-r84kk\") pod \"sub-test-model-8681b5-kserve-6dcbf55b89-hsdx9\" (UID: \"3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b\") " pod="ai-tenant-e2e-shared-a-w3-8681b5/sub-test-model-8681b5-kserve-6dcbf55b89-hsdx9" Oct 30 17:23:31.209821 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:31.209732 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e08b0a0b-6xr9w"] Oct 30 17:23:31.304684 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:31.304599 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-8681b5/sub-test-model-8681b5-kserve-6dcbf55b89-hsdx9" Oct 30 17:23:31.712381 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:23:31.712352 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3977a0e9_bb6b_45e3_8b8c_ee5d4733a37b.slice/crio-c136c0d1d54e4c9bd2608eae7e715d80a0d70bae5f341b21b38d9f0693d926fe WatchSource:0}: Error finding container c136c0d1d54e4c9bd2608eae7e715d80a0d70bae5f341b21b38d9f0693d926fe: Status 404 returned error can't find the container with id c136c0d1d54e4c9bd2608eae7e715d80a0d70bae5f341b21b38d9f0693d926fe Oct 30 17:23:31.714003 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:31.713978 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8681b5/sub-test-model-8681b5-kserve-6dcbf55b89-hsdx9"] Oct 30 17:23:31.744371 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:31.744340 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e08b0a0b-6xr9w" event={"ID":"f5528580-d2f5-4201-859e-f9f2c1e6c588","Type":"ContainerStarted","Data":"7d89b7a3711e881e790e16efb4c3848d3124bd85ef98d31e7167cbf1c3e4e826"} Oct 30 17:23:31.744534 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:31.744405 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e08b0a0b-6xr9w" event={"ID":"f5528580-d2f5-4201-859e-f9f2c1e6c588","Type":"ContainerStarted","Data":"53894626a47021e1ac9571d94bf7c3a5e408a7b15a18a9373c1bb6da14f5f579"} Oct 30 17:23:31.745798 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:31.745778 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-8681b5/sub-test-model-8681b5-kserve-6dcbf55b89-hsdx9" event={"ID":"3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b","Type":"ContainerStarted","Data":"c136c0d1d54e4c9bd2608eae7e715d80a0d70bae5f341b21b38d9f0693d926fe"} Oct 30 17:23:31.787391 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:31.787348 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e08b0a0b-6xr9w" podStartSLOduration=1.787333155 podStartE2EDuration="1.787333155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:23:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:23:31.786873048 +0000 UTC m=+2185.787627969" watchObservedRunningTime="2026-10-30 17:23:31.787333155 +0000 UTC m=+2185.788088056" Oct 30 17:23:32.757937 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:32.757905 4522 generic.go:356] "Generic (PLEG): container finished" podID="f5528580-d2f5-4201-859e-f9f2c1e6c588" containerID="7d89b7a3711e881e790e16efb4c3848d3124bd85ef98d31e7167cbf1c3e4e826" exitCode=0 Oct 30 17:23:32.758448 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:32.757992 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e08b0a0b-6xr9w" event={"ID":"f5528580-d2f5-4201-859e-f9f2c1e6c588","Type":"ContainerDied","Data":"7d89b7a3711e881e790e16efb4c3848d3124bd85ef98d31e7167cbf1c3e4e826"} Oct 30 17:23:32.760127 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:32.760102 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-8681b5/sub-test-model-8681b5-kserve-6dcbf55b89-hsdx9" event={"ID":"3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b","Type":"ContainerStarted","Data":"928c0baa704782b9abf12e214a469d7a1ac9b97c602be85e7f5d8e1eb3530d08"} Oct 30 17:23:32.845433 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:32.845384 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-8681b5/sub-test-model-8681b5-kserve-6dcbf55b89-hsdx9" podStartSLOduration=2.8453723330000003 podStartE2EDuration="2.845372333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:23:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:23:32.844724084 +0000 UTC m=+2186.845479004" watchObservedRunningTime="2026-10-30 17:23:32.845372333 +0000 UTC m=+2186.846127231" Oct 30 17:23:33.305665 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:33.305625 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-8681b5/sub-test-model-8681b5-kserve-6dcbf55b89-hsdx9" Oct 30 17:23:33.942443 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:33.942410 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e08b0a0b-6xr9w" Oct 30 17:23:34.006415 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:34.006378 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5528580-d2f5-4201-859e-f9f2c1e6c588-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5528580-d2f5-4201-859e-f9f2c1e6c588-maas-api-xks-ca\") pod \"f5528580-d2f5-4201-859e-f9f2c1e6c588\" (UID: \"f5528580-d2f5-4201-859e-f9f2c1e6c588\") " Oct 30 17:23:34.006725 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:34.006703 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5528580-d2f5-4201-859e-f9f2c1e6c588-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5528580-d2f5-4201-859e-f9f2c1e6c588-maas-api-openshift-ca\") pod \"f5528580-d2f5-4201-859e-f9f2c1e6c588\" (UID: \"f5528580-d2f5-4201-859e-f9f2c1e6c588\") " Oct 30 17:23:34.006855 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:34.006745 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5528580-d2f5-4201-859e-f9f2c1e6c588-kube-api-access-xfv9g\" (UniqueName: \"kubernetes.io/projected/f5528580-d2f5-4201-859e-f9f2c1e6c588-kube-api-access-xfv9g\") pod \"f5528580-d2f5-4201-859e-f9f2c1e6c588\" (UID: \"f5528580-d2f5-4201-859e-f9f2c1e6c588\") " Oct 30 17:23:34.007139 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:34.007111 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5528580-d2f5-4201-859e-f9f2c1e6c588-maas-api-openshift-ca" pod "f5528580-d2f5-4201-859e-f9f2c1e6c588" (UID: "f5528580-d2f5-4201-859e-f9f2c1e6c588"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:23:34.007550 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:34.007524 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5528580-d2f5-4201-859e-f9f2c1e6c588-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:23:34.008725 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:34.008699 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5528580-d2f5-4201-859e-f9f2c1e6c588-maas-api-xks-ca" pod "f5528580-d2f5-4201-859e-f9f2c1e6c588" (UID: "f5528580-d2f5-4201-859e-f9f2c1e6c588"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:23:34.009378 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:34.009361 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5528580-d2f5-4201-859e-f9f2c1e6c588-kube-api-access-xfv9g" pod "f5528580-d2f5-4201-859e-f9f2c1e6c588" (UID: "f5528580-d2f5-4201-859e-f9f2c1e6c588"). InnerVolumeSpecName "kube-api-access-xfv9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:23:34.108575 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:34.108534 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5528580-d2f5-4201-859e-f9f2c1e6c588-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:23:34.108575 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:34.108567 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfv9g\" (UniqueName: \"kubernetes.io/projected/f5528580-d2f5-4201-859e-f9f2c1e6c588-kube-api-access-xfv9g\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:23:34.780296 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:34.780261 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e08b0a0b-6xr9w" Oct 30 17:23:34.780296 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:34.780281 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e08b0a0b-6xr9w" event={"ID":"f5528580-d2f5-4201-859e-f9f2c1e6c588","Type":"ContainerDied","Data":"53894626a47021e1ac9571d94bf7c3a5e408a7b15a18a9373c1bb6da14f5f579"} Oct 30 17:23:34.780580 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:34.780392 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53894626a47021e1ac9571d94bf7c3a5e408a7b15a18a9373c1bb6da14f5f579" Oct 30 17:23:35.263345 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:35.261191 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz"] Oct 30 17:23:35.263345 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:35.261791 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz" podUID="7a845206-e2dc-4996-8e24-4f59233146b2" containerName="istio-proxy" containerID="cri-o://ec67c1870c73c264be68b4cac46a1cfd30eece1dc7cc10658f9242901320fde5" gracePeriod=30 Oct 30 17:23:35.289341 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:35.285529 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v"] Oct 30 17:23:35.289341 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:35.285765 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v" podUID="056b7890-d03e-4f62-8585-b3f7a7777a24" containerName="istio-proxy" containerID="cri-o://6979ce90ee24f03a859277190837f2d9137302028de5e8d48011f33ef367a6c4" gracePeriod=30 Oct 30 17:23:35.792207 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:35.792162 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd5c8f-ccz6k"] Oct 30 17:23:35.792986 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:35.792968 4522 cpu_manager.go:414] "containerMap: removing container" podUID="f5528580-d2f5-4201-859e-f9f2c1e6c588" containerName="revoke-keys" Oct 30 17:23:35.792986 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:35.792986 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5528580-d2f5-4201-859e-f9f2c1e6c588" containerName="revoke-keys" Oct 30 17:23:35.793138 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:35.793128 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5528580-d2f5-4201-859e-f9f2c1e6c588" containerName="revoke-keys" Oct 30 17:23:35.797172 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:35.797155 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd5c8f-ccz6k" Oct 30 17:23:35.823324 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:35.820249 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd5c8f-ccz6k"] Oct 30 17:23:35.848711 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:35.848676 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvhvf\" (UniqueName: \"kubernetes.io/projected/b6fd669d-69c4-4552-b050-f07bc2664981-kube-api-access-mvhvf\") pod \"maas-api-revoke-keys-e2e-inf-b-dd5c8f-ccz6k\" (UID: \"b6fd669d-69c4-4552-b050-f07bc2664981\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd5c8f-ccz6k" Oct 30 17:23:35.848916 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:35.848812 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6fd669d-69c4-4552-b050-f07bc2664981-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-dd5c8f-ccz6k\" (UID: \"b6fd669d-69c4-4552-b050-f07bc2664981\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd5c8f-ccz6k" Oct 30 17:23:35.848984 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:35.848929 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6fd669d-69c4-4552-b050-f07bc2664981-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-dd5c8f-ccz6k\" (UID: \"b6fd669d-69c4-4552-b050-f07bc2664981\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd5c8f-ccz6k" Oct 30 17:23:35.855193 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:35.855160 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:23:35.855408 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:35.855276 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:23:35.950200 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:35.950166 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6fd669d-69c4-4552-b050-f07bc2664981-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-dd5c8f-ccz6k\" (UID: \"b6fd669d-69c4-4552-b050-f07bc2664981\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd5c8f-ccz6k" Oct 30 17:23:35.950507 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:35.950286 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvhvf\" (UniqueName: \"kubernetes.io/projected/b6fd669d-69c4-4552-b050-f07bc2664981-kube-api-access-mvhvf\") pod \"maas-api-revoke-keys-e2e-inf-b-dd5c8f-ccz6k\" (UID: \"b6fd669d-69c4-4552-b050-f07bc2664981\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd5c8f-ccz6k" Oct 30 17:23:35.950507 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:35.950394 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6fd669d-69c4-4552-b050-f07bc2664981-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-dd5c8f-ccz6k\" (UID: \"b6fd669d-69c4-4552-b050-f07bc2664981\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd5c8f-ccz6k" Oct 30 17:23:35.951023 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:35.950994 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6fd669d-69c4-4552-b050-f07bc2664981-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-dd5c8f-ccz6k\" (UID: \"b6fd669d-69c4-4552-b050-f07bc2664981\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd5c8f-ccz6k" Oct 30 17:23:35.952651 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:35.952626 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6fd669d-69c4-4552-b050-f07bc2664981-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-dd5c8f-ccz6k\" (UID: \"b6fd669d-69c4-4552-b050-f07bc2664981\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd5c8f-ccz6k" Oct 30 17:23:35.976665 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:35.976637 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvhvf\" (UniqueName: \"kubernetes.io/projected/b6fd669d-69c4-4552-b050-f07bc2664981-kube-api-access-mvhvf\") pod \"maas-api-revoke-keys-e2e-inf-b-dd5c8f-ccz6k\" (UID: \"b6fd669d-69c4-4552-b050-f07bc2664981\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd5c8f-ccz6k" Oct 30 17:23:36.116319 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:36.116209 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd5c8f-ccz6k" Oct 30 17:23:36.431508 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:23:36.431454 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6fd669d_69c4_4552_b050_f07bc2664981.slice/crio-d2b2a6c0659e69215de29f6f7830e25875fe4d73a6afe2d566af7cd90a443f61 WatchSource:0}: Error finding container d2b2a6c0659e69215de29f6f7830e25875fe4d73a6afe2d566af7cd90a443f61: Status 404 returned error can't find the container with id d2b2a6c0659e69215de29f6f7830e25875fe4d73a6afe2d566af7cd90a443f61 Oct 30 17:23:36.433373 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:36.433350 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd5c8f-ccz6k"] Oct 30 17:23:36.805615 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:36.805583 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd5c8f-ccz6k" event={"ID":"b6fd669d-69c4-4552-b050-f07bc2664981","Type":"ContainerStarted","Data":"d2b2a6c0659e69215de29f6f7830e25875fe4d73a6afe2d566af7cd90a443f61"} Oct 30 17:23:36.831725 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:36.831686 4522 patch_prober.go:29] interesting pod/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:23:36.831870 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:36.831750 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz" podUID="7a845206-e2dc-4996-8e24-4f59233146b2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:23:37.224139 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:37.224102 4522 patch_prober.go:29] interesting pod/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:23:37.224330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:37.224153 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v" podUID="056b7890-d03e-4f62-8585-b3f7a7777a24" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:23:37.817081 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:37.817052 4522 generic.go:356] "Generic (PLEG): container finished" podID="b6fd669d-69c4-4552-b050-f07bc2664981" containerID="fb32183e6794aec6fe7c21ebf4470d6c972c0d7b719617a41a33c3479897ddf6" exitCode=0 Oct 30 17:23:37.817493 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:37.817137 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd5c8f-ccz6k" event={"ID":"b6fd669d-69c4-4552-b050-f07bc2664981","Type":"ContainerDied","Data":"fb32183e6794aec6fe7c21ebf4470d6c972c0d7b719617a41a33c3479897ddf6"} Oct 30 17:23:38.990065 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:38.990035 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd5c8f-ccz6k" Oct 30 17:23:39.106109 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:39.106022 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6fd669d-69c4-4552-b050-f07bc2664981-kube-api-access-mvhvf\" (UniqueName: \"kubernetes.io/projected/b6fd669d-69c4-4552-b050-f07bc2664981-kube-api-access-mvhvf\") pod \"b6fd669d-69c4-4552-b050-f07bc2664981\" (UID: \"b6fd669d-69c4-4552-b050-f07bc2664981\") " Oct 30 17:23:39.106279 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:39.106156 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6fd669d-69c4-4552-b050-f07bc2664981-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6fd669d-69c4-4552-b050-f07bc2664981-maas-api-openshift-ca\") pod \"b6fd669d-69c4-4552-b050-f07bc2664981\" (UID: \"b6fd669d-69c4-4552-b050-f07bc2664981\") " Oct 30 17:23:39.106605 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:39.106583 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6fd669d-69c4-4552-b050-f07bc2664981-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6fd669d-69c4-4552-b050-f07bc2664981-maas-api-xks-ca\") pod \"b6fd669d-69c4-4552-b050-f07bc2664981\" (UID: \"b6fd669d-69c4-4552-b050-f07bc2664981\") " Oct 30 17:23:39.106677 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:39.106641 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6fd669d-69c4-4552-b050-f07bc2664981-maas-api-openshift-ca" pod "b6fd669d-69c4-4552-b050-f07bc2664981" (UID: "b6fd669d-69c4-4552-b050-f07bc2664981"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:23:39.107117 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:39.107096 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6fd669d-69c4-4552-b050-f07bc2664981-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:23:39.108198 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:39.108170 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6fd669d-69c4-4552-b050-f07bc2664981-kube-api-access-mvhvf" pod "b6fd669d-69c4-4552-b050-f07bc2664981" (UID: "b6fd669d-69c4-4552-b050-f07bc2664981"). InnerVolumeSpecName "kube-api-access-mvhvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:23:39.108818 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:39.108799 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6fd669d-69c4-4552-b050-f07bc2664981-maas-api-xks-ca" pod "b6fd669d-69c4-4552-b050-f07bc2664981" (UID: "b6fd669d-69c4-4552-b050-f07bc2664981"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:23:39.208895 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:39.208855 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6fd669d-69c4-4552-b050-f07bc2664981-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:23:39.208895 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:39.208893 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvhvf\" (UniqueName: \"kubernetes.io/projected/b6fd669d-69c4-4552-b050-f07bc2664981-kube-api-access-mvhvf\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:23:39.837075 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:39.837042 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd5c8f-ccz6k" Oct 30 17:23:39.837253 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:39.837076 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd5c8f-ccz6k" event={"ID":"b6fd669d-69c4-4552-b050-f07bc2664981","Type":"ContainerDied","Data":"d2b2a6c0659e69215de29f6f7830e25875fe4d73a6afe2d566af7cd90a443f61"} Oct 30 17:23:39.837253 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:39.837126 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2b2a6c0659e69215de29f6f7830e25875fe4d73a6afe2d566af7cd90a443f61" Oct 30 17:23:40.596806 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.596778 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v" Oct 30 17:23:40.653817 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.653789 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz" Oct 30 17:23:40.752554 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.752517 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/056b7890-d03e-4f62-8585-b3f7a7777a24-kube-api-access-9fh8k\" (UniqueName: \"kubernetes.io/projected/056b7890-d03e-4f62-8585-b3f7a7777a24-kube-api-access-9fh8k\") pod \"056b7890-d03e-4f62-8585-b3f7a7777a24\" (UID: \"056b7890-d03e-4f62-8585-b3f7a7777a24\") " Oct 30 17:23:40.752721 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.752571 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7a845206-e2dc-4996-8e24-4f59233146b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a845206-e2dc-4996-8e24-4f59233146b2-istio-podinfo\") pod \"7a845206-e2dc-4996-8e24-4f59233146b2\" (UID: \"7a845206-e2dc-4996-8e24-4f59233146b2\") " Oct 30 17:23:40.752721 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.752654 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a845206-e2dc-4996-8e24-4f59233146b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a845206-e2dc-4996-8e24-4f59233146b2-istio-data\") pod \"7a845206-e2dc-4996-8e24-4f59233146b2\" (UID: \"7a845206-e2dc-4996-8e24-4f59233146b2\") " Oct 30 17:23:40.752799 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.752749 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/056b7890-d03e-4f62-8585-b3f7a7777a24-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/056b7890-d03e-4f62-8585-b3f7a7777a24-credential-socket\") pod \"056b7890-d03e-4f62-8585-b3f7a7777a24\" (UID: \"056b7890-d03e-4f62-8585-b3f7a7777a24\") " Oct 30 17:23:40.752799 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.752786 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a845206-e2dc-4996-8e24-4f59233146b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a845206-e2dc-4996-8e24-4f59233146b2-istio-envoy\") pod \"7a845206-e2dc-4996-8e24-4f59233146b2\" (UID: \"7a845206-e2dc-4996-8e24-4f59233146b2\") " Oct 30 17:23:40.752894 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.752872 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/056b7890-d03e-4f62-8585-b3f7a7777a24-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/056b7890-d03e-4f62-8585-b3f7a7777a24-istio-data\") pod \"056b7890-d03e-4f62-8585-b3f7a7777a24\" (UID: \"056b7890-d03e-4f62-8585-b3f7a7777a24\") " Oct 30 17:23:40.753540 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.753518 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/056b7890-d03e-4f62-8585-b3f7a7777a24-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/056b7890-d03e-4f62-8585-b3f7a7777a24-istio-podinfo\") pod \"056b7890-d03e-4f62-8585-b3f7a7777a24\" (UID: \"056b7890-d03e-4f62-8585-b3f7a7777a24\") " Oct 30 17:23:40.753716 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.753697 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a845206-e2dc-4996-8e24-4f59233146b2-kube-api-access-mnfk5\" (UniqueName: \"kubernetes.io/projected/7a845206-e2dc-4996-8e24-4f59233146b2-kube-api-access-mnfk5\") pod \"7a845206-e2dc-4996-8e24-4f59233146b2\" (UID: \"7a845206-e2dc-4996-8e24-4f59233146b2\") " Oct 30 17:23:40.753875 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.753848 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/056b7890-d03e-4f62-8585-b3f7a7777a24-istio-token\" (UniqueName: \"kubernetes.io/projected/056b7890-d03e-4f62-8585-b3f7a7777a24-istio-token\") pod \"056b7890-d03e-4f62-8585-b3f7a7777a24\" (UID: \"056b7890-d03e-4f62-8585-b3f7a7777a24\") " Oct 30 17:23:40.754003 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.753893 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a845206-e2dc-4996-8e24-4f59233146b2-istio-token\" (UniqueName: \"kubernetes.io/projected/7a845206-e2dc-4996-8e24-4f59233146b2-istio-token\") pod \"7a845206-e2dc-4996-8e24-4f59233146b2\" (UID: \"7a845206-e2dc-4996-8e24-4f59233146b2\") " Oct 30 17:23:40.754003 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.753994 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a845206-e2dc-4996-8e24-4f59233146b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a845206-e2dc-4996-8e24-4f59233146b2-workload-certs\") pod \"7a845206-e2dc-4996-8e24-4f59233146b2\" (UID: \"7a845206-e2dc-4996-8e24-4f59233146b2\") " Oct 30 17:23:40.754190 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.754029 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a845206-e2dc-4996-8e24-4f59233146b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a845206-e2dc-4996-8e24-4f59233146b2-credential-socket\") pod \"7a845206-e2dc-4996-8e24-4f59233146b2\" (UID: \"7a845206-e2dc-4996-8e24-4f59233146b2\") " Oct 30 17:23:40.754190 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.754086 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/056b7890-d03e-4f62-8585-b3f7a7777a24-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/056b7890-d03e-4f62-8585-b3f7a7777a24-istiod-ca-cert\") pod \"056b7890-d03e-4f62-8585-b3f7a7777a24\" (UID: \"056b7890-d03e-4f62-8585-b3f7a7777a24\") " Oct 30 17:23:40.754190 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.754127 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/056b7890-d03e-4f62-8585-b3f7a7777a24-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/056b7890-d03e-4f62-8585-b3f7a7777a24-workload-certs\") pod \"056b7890-d03e-4f62-8585-b3f7a7777a24\" (UID: \"056b7890-d03e-4f62-8585-b3f7a7777a24\") " Oct 30 17:23:40.754420 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.754196 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a845206-e2dc-4996-8e24-4f59233146b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a845206-e2dc-4996-8e24-4f59233146b2-workload-socket\") pod \"7a845206-e2dc-4996-8e24-4f59233146b2\" (UID: \"7a845206-e2dc-4996-8e24-4f59233146b2\") " Oct 30 17:23:40.754420 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.754241 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/056b7890-d03e-4f62-8585-b3f7a7777a24-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/056b7890-d03e-4f62-8585-b3f7a7777a24-workload-socket\") pod \"056b7890-d03e-4f62-8585-b3f7a7777a24\" (UID: \"056b7890-d03e-4f62-8585-b3f7a7777a24\") " Oct 30 17:23:40.754420 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.754283 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/056b7890-d03e-4f62-8585-b3f7a7777a24-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/056b7890-d03e-4f62-8585-b3f7a7777a24-istio-envoy\") pod \"056b7890-d03e-4f62-8585-b3f7a7777a24\" (UID: \"056b7890-d03e-4f62-8585-b3f7a7777a24\") " Oct 30 17:23:40.754420 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.754333 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a845206-e2dc-4996-8e24-4f59233146b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a845206-e2dc-4996-8e24-4f59233146b2-istiod-ca-cert\") pod \"7a845206-e2dc-4996-8e24-4f59233146b2\" (UID: \"7a845206-e2dc-4996-8e24-4f59233146b2\") " Oct 30 17:23:40.755807 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.753358 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a845206-e2dc-4996-8e24-4f59233146b2-istio-data" pod "7a845206-e2dc-4996-8e24-4f59233146b2" (UID: "7a845206-e2dc-4996-8e24-4f59233146b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:40.755807 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.753456 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/056b7890-d03e-4f62-8585-b3f7a7777a24-credential-socket" pod "056b7890-d03e-4f62-8585-b3f7a7777a24" (UID: "056b7890-d03e-4f62-8585-b3f7a7777a24"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:40.755807 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.753547 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/056b7890-d03e-4f62-8585-b3f7a7777a24-istio-data" pod "056b7890-d03e-4f62-8585-b3f7a7777a24" (UID: "056b7890-d03e-4f62-8585-b3f7a7777a24"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:40.755807 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.755560 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/056b7890-d03e-4f62-8585-b3f7a7777a24-workload-certs" pod "056b7890-d03e-4f62-8585-b3f7a7777a24" (UID: "056b7890-d03e-4f62-8585-b3f7a7777a24"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:40.755807 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.755746 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a845206-e2dc-4996-8e24-4f59233146b2-workload-certs" pod "7a845206-e2dc-4996-8e24-4f59233146b2" (UID: "7a845206-e2dc-4996-8e24-4f59233146b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:40.756127 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.755945 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a845206-e2dc-4996-8e24-4f59233146b2-credential-socket" pod "7a845206-e2dc-4996-8e24-4f59233146b2" (UID: "7a845206-e2dc-4996-8e24-4f59233146b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:40.756299 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.756277 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/056b7890-d03e-4f62-8585-b3f7a7777a24-istiod-ca-cert" pod "056b7890-d03e-4f62-8585-b3f7a7777a24" (UID: "056b7890-d03e-4f62-8585-b3f7a7777a24"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:23:40.756523 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.756503 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a845206-e2dc-4996-8e24-4f59233146b2-istio-podinfo" pod "7a845206-e2dc-4996-8e24-4f59233146b2" (UID: "7a845206-e2dc-4996-8e24-4f59233146b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:23:40.756617 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.756563 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/056b7890-d03e-4f62-8585-b3f7a7777a24-workload-socket" pod "056b7890-d03e-4f62-8585-b3f7a7777a24" (UID: "056b7890-d03e-4f62-8585-b3f7a7777a24"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:40.756801 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.756767 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a845206-e2dc-4996-8e24-4f59233146b2-workload-socket" pod "7a845206-e2dc-4996-8e24-4f59233146b2" (UID: "7a845206-e2dc-4996-8e24-4f59233146b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:40.757125 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.757106 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a845206-e2dc-4996-8e24-4f59233146b2-istiod-ca-cert" pod "7a845206-e2dc-4996-8e24-4f59233146b2" (UID: "7a845206-e2dc-4996-8e24-4f59233146b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:23:40.758875 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.758287 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a845206-e2dc-4996-8e24-4f59233146b2-istio-token" pod "7a845206-e2dc-4996-8e24-4f59233146b2" (UID: "7a845206-e2dc-4996-8e24-4f59233146b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:23:40.758875 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.758579 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a845206-e2dc-4996-8e24-4f59233146b2-kube-api-access-mnfk5" pod "7a845206-e2dc-4996-8e24-4f59233146b2" (UID: "7a845206-e2dc-4996-8e24-4f59233146b2"). InnerVolumeSpecName "kube-api-access-mnfk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:23:40.761713 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.761684 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/056b7890-d03e-4f62-8585-b3f7a7777a24-istio-podinfo" pod "056b7890-d03e-4f62-8585-b3f7a7777a24" (UID: "056b7890-d03e-4f62-8585-b3f7a7777a24"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:23:40.761884 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.761729 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a845206-e2dc-4996-8e24-4f59233146b2-istio-envoy" pod "7a845206-e2dc-4996-8e24-4f59233146b2" (UID: "7a845206-e2dc-4996-8e24-4f59233146b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:40.761884 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.761762 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/056b7890-d03e-4f62-8585-b3f7a7777a24-istio-token" pod "056b7890-d03e-4f62-8585-b3f7a7777a24" (UID: "056b7890-d03e-4f62-8585-b3f7a7777a24"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:23:40.762032 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.762003 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/056b7890-d03e-4f62-8585-b3f7a7777a24-istio-envoy" pod "056b7890-d03e-4f62-8585-b3f7a7777a24" (UID: "056b7890-d03e-4f62-8585-b3f7a7777a24"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:40.762125 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.762057 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/056b7890-d03e-4f62-8585-b3f7a7777a24-kube-api-access-9fh8k" pod "056b7890-d03e-4f62-8585-b3f7a7777a24" (UID: "056b7890-d03e-4f62-8585-b3f7a7777a24"). InnerVolumeSpecName "kube-api-access-9fh8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:23:40.845979 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.845897 4522 generic.go:356] "Generic (PLEG): container finished" podID="056b7890-d03e-4f62-8585-b3f7a7777a24" containerID="6979ce90ee24f03a859277190837f2d9137302028de5e8d48011f33ef367a6c4" exitCode=0 Oct 30 17:23:40.845979 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.845948 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v" Oct 30 17:23:40.846177 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.845978 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v" event={"ID":"056b7890-d03e-4f62-8585-b3f7a7777a24","Type":"ContainerDied","Data":"6979ce90ee24f03a859277190837f2d9137302028de5e8d48011f33ef367a6c4"} Oct 30 17:23:40.846177 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.846057 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v" event={"ID":"056b7890-d03e-4f62-8585-b3f7a7777a24","Type":"ContainerDied","Data":"ac8d7f147344bfe4e84a6ae2f988dc6b28be7860f0bb09bf85b3943056af114a"} Oct 30 17:23:40.846177 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.846081 4522 scope.go:122] "RemoveContainer" containerID="6979ce90ee24f03a859277190837f2d9137302028de5e8d48011f33ef367a6c4" Oct 30 17:23:40.847806 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.847786 4522 generic.go:356] "Generic (PLEG): container finished" podID="7a845206-e2dc-4996-8e24-4f59233146b2" containerID="ec67c1870c73c264be68b4cac46a1cfd30eece1dc7cc10658f9242901320fde5" exitCode=0 Oct 30 17:23:40.847910 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.847829 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz" Oct 30 17:23:40.847910 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.847872 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz" event={"ID":"7a845206-e2dc-4996-8e24-4f59233146b2","Type":"ContainerDied","Data":"ec67c1870c73c264be68b4cac46a1cfd30eece1dc7cc10658f9242901320fde5"} Oct 30 17:23:40.847988 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.847920 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz" event={"ID":"7a845206-e2dc-4996-8e24-4f59233146b2","Type":"ContainerDied","Data":"0dac48efbe1758b2066c1f2fec414833373b840ba25d045780314314443ef21d"} Oct 30 17:23:40.856357 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.856297 4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/056b7890-d03e-4f62-8585-b3f7a7777a24-istio-podinfo\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:23:40.856357 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.856340 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnfk5\" (UniqueName: \"kubernetes.io/projected/7a845206-e2dc-4996-8e24-4f59233146b2-kube-api-access-mnfk5\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:23:40.856357 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.856355 4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/056b7890-d03e-4f62-8585-b3f7a7777a24-istio-token\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:23:40.856582 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.856369 4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a845206-e2dc-4996-8e24-4f59233146b2-istio-token\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:23:40.856582 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.856381 4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a845206-e2dc-4996-8e24-4f59233146b2-workload-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:23:40.856582 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.856392 4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a845206-e2dc-4996-8e24-4f59233146b2-credential-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:23:40.856582 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.856405 4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/056b7890-d03e-4f62-8585-b3f7a7777a24-istiod-ca-cert\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:23:40.856582 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.856418 4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/056b7890-d03e-4f62-8585-b3f7a7777a24-workload-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:23:40.856582 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.856431 4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a845206-e2dc-4996-8e24-4f59233146b2-workload-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:23:40.856582 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.856443 4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/056b7890-d03e-4f62-8585-b3f7a7777a24-workload-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:23:40.856582 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.856455 4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/056b7890-d03e-4f62-8585-b3f7a7777a24-istio-envoy\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:23:40.856582 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.856474 4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a845206-e2dc-4996-8e24-4f59233146b2-istiod-ca-cert\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:23:40.856582 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.856488 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fh8k\" (UniqueName: \"kubernetes.io/projected/056b7890-d03e-4f62-8585-b3f7a7777a24-kube-api-access-9fh8k\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:23:40.856582 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.856501 4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a845206-e2dc-4996-8e24-4f59233146b2-istio-podinfo\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:23:40.856582 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.856514 4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a845206-e2dc-4996-8e24-4f59233146b2-istio-data\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:23:40.856582 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.856527 4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/056b7890-d03e-4f62-8585-b3f7a7777a24-credential-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:23:40.856582 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.856540 4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a845206-e2dc-4996-8e24-4f59233146b2-istio-envoy\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:23:40.856582 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.856555 4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/056b7890-d03e-4f62-8585-b3f7a7777a24-istio-data\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:23:40.858948 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.858932 4522 scope.go:122] "RemoveContainer" containerID="6979ce90ee24f03a859277190837f2d9137302028de5e8d48011f33ef367a6c4" Oct 30 17:23:40.859235 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:23:40.859214 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6979ce90ee24f03a859277190837f2d9137302028de5e8d48011f33ef367a6c4\": container with ID starting with 6979ce90ee24f03a859277190837f2d9137302028de5e8d48011f33ef367a6c4 not found: ID does not exist" containerID="6979ce90ee24f03a859277190837f2d9137302028de5e8d48011f33ef367a6c4" Oct 30 17:23:40.859317 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.859241 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6979ce90ee24f03a859277190837f2d9137302028de5e8d48011f33ef367a6c4"} err="failed to get container status \"6979ce90ee24f03a859277190837f2d9137302028de5e8d48011f33ef367a6c4\": rpc error: code = NotFound desc = could not find container \"6979ce90ee24f03a859277190837f2d9137302028de5e8d48011f33ef367a6c4\": container with ID starting with 6979ce90ee24f03a859277190837f2d9137302028de5e8d48011f33ef367a6c4 not found: ID does not exist" Oct 30 17:23:40.859317 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.859257 4522 scope.go:122] "RemoveContainer" containerID="ec67c1870c73c264be68b4cac46a1cfd30eece1dc7cc10658f9242901320fde5" Oct 30 17:23:40.869838 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.869820 4522 scope.go:122] "RemoveContainer" containerID="ec67c1870c73c264be68b4cac46a1cfd30eece1dc7cc10658f9242901320fde5" Oct 30 17:23:40.870094 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:23:40.870076 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec67c1870c73c264be68b4cac46a1cfd30eece1dc7cc10658f9242901320fde5\": container with ID starting with ec67c1870c73c264be68b4cac46a1cfd30eece1dc7cc10658f9242901320fde5 not found: ID does not exist" containerID="ec67c1870c73c264be68b4cac46a1cfd30eece1dc7cc10658f9242901320fde5" Oct 30 17:23:40.870142 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.870101 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec67c1870c73c264be68b4cac46a1cfd30eece1dc7cc10658f9242901320fde5"} err="failed to get container status \"ec67c1870c73c264be68b4cac46a1cfd30eece1dc7cc10658f9242901320fde5\": rpc error: code = NotFound desc = could not find container \"ec67c1870c73c264be68b4cac46a1cfd30eece1dc7cc10658f9242901320fde5\": container with ID starting with ec67c1870c73c264be68b4cac46a1cfd30eece1dc7cc10658f9242901320fde5 not found: ID does not exist" Oct 30 17:23:40.906532 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.906497 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v"] Oct 30 17:23:40.910927 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.910901 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-cr46v"] Oct 30 17:23:40.948399 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.948370 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz"] Oct 30 17:23:40.952479 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:40.952452 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-dxzwz"] Oct 30 17:23:41.304941 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:41.304910 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-8681b5/sub-test-model-8681b5-kserve-6dcbf55b89-hsdx9" Oct 30 17:23:41.321803 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:41.321779 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-8681b5/sub-test-model-8681b5-kserve-6dcbf55b89-hsdx9" Oct 30 17:23:41.869435 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:41.869405 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-8681b5/sub-test-model-8681b5-kserve-6dcbf55b89-hsdx9" Oct 30 17:23:42.146973 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:42.146880 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="056b7890-d03e-4f62-8585-b3f7a7777a24" path="/var/lib/kubelet/pods/056b7890-d03e-4f62-8585-b3f7a7777a24/volumes" Oct 30 17:23:42.147591 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:42.147565 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a845206-e2dc-4996-8e24-4f59233146b2" path="/var/lib/kubelet/pods/7a845206-e2dc-4996-8e24-4f59233146b2/volumes" Oct 30 17:23:42.949751 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:23:42.949718 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:23:46.009863 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:46.009833 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-dd5c8f-796cddf7f4-w8wvf"] Oct 30 17:23:46.045968 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:46.045936 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-dd5c8f-6b45577955-57qp6"] Oct 30 17:23:51.011009 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:51.010923 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-dd5c8f-796cddf7f4-w8wvf" podUID="f897ea49-4823-457b-b3bc-940202d02d40" containerName="payload-processing" containerID="cri-o://23df8e36fd7f395e403fb6b80bb2f33a09a5a245ae0ee0695c25798d36f44509" gracePeriod=25 Oct 30 17:23:51.047205 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:51.047169 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd5c8f-6b45577955-57qp6" podUID="8226519e-a60e-4b3c-b4b3-83479839c8bb" containerName="payload-pre-processing" containerID="cri-o://05d34dc64a6d870192f7f10e27988f3ecd79d0874121e06516da2a4a9a6e3d9d" gracePeriod=25 Oct 30 17:23:51.300905 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:51.300881 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-dd5c8f-796cddf7f4-w8wvf" Oct 30 17:23:51.411735 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:51.411694 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f897ea49-4823-457b-b3bc-940202d02d40-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f897ea49-4823-457b-b3bc-940202d02d40-plugins-config-volume\") pod \"f897ea49-4823-457b-b3bc-940202d02d40\" (UID: \"f897ea49-4823-457b-b3bc-940202d02d40\") " Oct 30 17:23:51.411916 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:51.411833 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f897ea49-4823-457b-b3bc-940202d02d40-kube-api-access-x58xt\" (UniqueName: \"kubernetes.io/projected/f897ea49-4823-457b-b3bc-940202d02d40-kube-api-access-x58xt\") pod \"f897ea49-4823-457b-b3bc-940202d02d40\" (UID: \"f897ea49-4823-457b-b3bc-940202d02d40\") " Oct 30 17:23:51.412085 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:51.412063 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f897ea49-4823-457b-b3bc-940202d02d40-plugins-config-volume" pod "f897ea49-4823-457b-b3bc-940202d02d40" (UID: "f897ea49-4823-457b-b3bc-940202d02d40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:23:51.412461 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:51.412436 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f897ea49-4823-457b-b3bc-940202d02d40-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:23:51.413975 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:51.413946 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f897ea49-4823-457b-b3bc-940202d02d40-kube-api-access-x58xt" pod "f897ea49-4823-457b-b3bc-940202d02d40" (UID: "f897ea49-4823-457b-b3bc-940202d02d40"). InnerVolumeSpecName "kube-api-access-x58xt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:23:51.513685 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:51.513653 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x58xt\" (UniqueName: \"kubernetes.io/projected/f897ea49-4823-457b-b3bc-940202d02d40-kube-api-access-x58xt\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:23:51.958749 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:51.958709 4522 generic.go:356] "Generic (PLEG): container finished" podID="f897ea49-4823-457b-b3bc-940202d02d40" containerID="23df8e36fd7f395e403fb6b80bb2f33a09a5a245ae0ee0695c25798d36f44509" exitCode=0 Oct 30 17:23:51.958942 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:51.958775 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-dd5c8f-796cddf7f4-w8wvf" Oct 30 17:23:51.958942 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:51.958786 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-dd5c8f-796cddf7f4-w8wvf" event={"ID":"f897ea49-4823-457b-b3bc-940202d02d40","Type":"ContainerDied","Data":"23df8e36fd7f395e403fb6b80bb2f33a09a5a245ae0ee0695c25798d36f44509"} Oct 30 17:23:51.958942 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:51.958862 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-dd5c8f-796cddf7f4-w8wvf" event={"ID":"f897ea49-4823-457b-b3bc-940202d02d40","Type":"ContainerDied","Data":"e16e79b8d1873465de66b503b664c91612820ebacbf999d55d8676bb343d8341"} Oct 30 17:23:51.958942 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:51.958885 4522 scope.go:122] "RemoveContainer" containerID="23df8e36fd7f395e403fb6b80bb2f33a09a5a245ae0ee0695c25798d36f44509" Oct 30 17:23:51.970123 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:51.970104 4522 scope.go:122] "RemoveContainer" containerID="23df8e36fd7f395e403fb6b80bb2f33a09a5a245ae0ee0695c25798d36f44509" Oct 30 17:23:51.970419 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:23:51.970397 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23df8e36fd7f395e403fb6b80bb2f33a09a5a245ae0ee0695c25798d36f44509\": container with ID starting with 23df8e36fd7f395e403fb6b80bb2f33a09a5a245ae0ee0695c25798d36f44509 not found: ID does not exist" containerID="23df8e36fd7f395e403fb6b80bb2f33a09a5a245ae0ee0695c25798d36f44509" Oct 30 17:23:51.970467 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:51.970429 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23df8e36fd7f395e403fb6b80bb2f33a09a5a245ae0ee0695c25798d36f44509"} err="failed to get container status \"23df8e36fd7f395e403fb6b80bb2f33a09a5a245ae0ee0695c25798d36f44509\": rpc error: code = NotFound desc = could not find container \"23df8e36fd7f395e403fb6b80bb2f33a09a5a245ae0ee0695c25798d36f44509\": container with ID starting with 23df8e36fd7f395e403fb6b80bb2f33a09a5a245ae0ee0695c25798d36f44509 not found: ID does not exist" Oct 30 17:23:52.017790 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:52.017751 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-dd5c8f-796cddf7f4-w8wvf"] Oct 30 17:23:52.030971 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:52.030938 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-dd5c8f-796cddf7f4-w8wvf"] Oct 30 17:23:52.147215 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:52.147182 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f897ea49-4823-457b-b3bc-940202d02d40" path="/var/lib/kubelet/pods/f897ea49-4823-457b-b3bc-940202d02d40/volumes" Oct 30 17:23:54.917921 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:54.917887 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-094cd551-s65fn"] Oct 30 17:23:54.918649 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:54.918624 4522 cpu_manager.go:414] "containerMap: removing container" podUID="7a845206-e2dc-4996-8e24-4f59233146b2" containerName="istio-proxy" Oct 30 17:23:54.918649 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:54.918643 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a845206-e2dc-4996-8e24-4f59233146b2" containerName="istio-proxy" Oct 30 17:23:54.918832 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:54.918689 4522 cpu_manager.go:414] "containerMap: removing container" podUID="056b7890-d03e-4f62-8585-b3f7a7777a24" containerName="istio-proxy" Oct 30 17:23:54.918832 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:54.918694 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="056b7890-d03e-4f62-8585-b3f7a7777a24" containerName="istio-proxy" Oct 30 17:23:54.918832 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:54.918717 4522 cpu_manager.go:414] "containerMap: removing container" podUID="b6fd669d-69c4-4552-b050-f07bc2664981" containerName="revoke-keys" Oct 30 17:23:54.918832 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:54.918723 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6fd669d-69c4-4552-b050-f07bc2664981" containerName="revoke-keys" Oct 30 17:23:54.918832 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:54.918739 4522 cpu_manager.go:414] "containerMap: removing container" podUID="f897ea49-4823-457b-b3bc-940202d02d40" containerName="payload-processing" Oct 30 17:23:54.918832 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:54.918745 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f897ea49-4823-457b-b3bc-940202d02d40" containerName="payload-processing" Oct 30 17:23:54.919014 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:54.918844 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="056b7890-d03e-4f62-8585-b3f7a7777a24" containerName="istio-proxy" Oct 30 17:23:54.919014 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:54.918858 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6fd669d-69c4-4552-b050-f07bc2664981" containerName="revoke-keys" Oct 30 17:23:54.919014 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:54.918866 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="f897ea49-4823-457b-b3bc-940202d02d40" containerName="payload-processing" Oct 30 17:23:54.919014 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:54.918873 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a845206-e2dc-4996-8e24-4f59233146b2" containerName="istio-proxy" Oct 30 17:23:54.923666 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:54.923650 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-094cd551-s65fn" Oct 30 17:23:54.942526 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:54.942505 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:23:54.947444 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:54.947420 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-094cd551-s65fn"] Oct 30 17:23:54.959349 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:54.956066 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:23:55.066375 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:55.066345 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6caa6499-3ae1-4a95-9983-6c597d6e1436-maas-api-xks-ca\") pod \"maas-api-revoke-sub-094cd551-s65fn\" (UID: \"6caa6499-3ae1-4a95-9983-6c597d6e1436\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-094cd551-s65fn" Oct 30 17:23:55.066525 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:55.066486 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6caa6499-3ae1-4a95-9983-6c597d6e1436-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-094cd551-s65fn\" (UID: \"6caa6499-3ae1-4a95-9983-6c597d6e1436\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-094cd551-s65fn" Oct 30 17:23:55.066638 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:55.066619 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwkq9\" (UniqueName: \"kubernetes.io/projected/6caa6499-3ae1-4a95-9983-6c597d6e1436-kube-api-access-qwkq9\") pod \"maas-api-revoke-sub-094cd551-s65fn\" (UID: \"6caa6499-3ae1-4a95-9983-6c597d6e1436\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-094cd551-s65fn" Oct 30 17:23:55.167765 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:55.167730 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6caa6499-3ae1-4a95-9983-6c597d6e1436-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-094cd551-s65fn\" (UID: \"6caa6499-3ae1-4a95-9983-6c597d6e1436\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-094cd551-s65fn" Oct 30 17:23:55.167956 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:55.167854 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwkq9\" (UniqueName: \"kubernetes.io/projected/6caa6499-3ae1-4a95-9983-6c597d6e1436-kube-api-access-qwkq9\") pod \"maas-api-revoke-sub-094cd551-s65fn\" (UID: \"6caa6499-3ae1-4a95-9983-6c597d6e1436\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-094cd551-s65fn" Oct 30 17:23:55.167956 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:55.167946 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6caa6499-3ae1-4a95-9983-6c597d6e1436-maas-api-xks-ca\") pod \"maas-api-revoke-sub-094cd551-s65fn\" (UID: \"6caa6499-3ae1-4a95-9983-6c597d6e1436\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-094cd551-s65fn" Oct 30 17:23:55.168521 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:55.168497 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6caa6499-3ae1-4a95-9983-6c597d6e1436-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-094cd551-s65fn\" (UID: \"6caa6499-3ae1-4a95-9983-6c597d6e1436\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-094cd551-s65fn" Oct 30 17:23:55.170346 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:55.170328 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6caa6499-3ae1-4a95-9983-6c597d6e1436-maas-api-xks-ca\") pod \"maas-api-revoke-sub-094cd551-s65fn\" (UID: \"6caa6499-3ae1-4a95-9983-6c597d6e1436\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-094cd551-s65fn" Oct 30 17:23:55.192700 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:55.192674 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwkq9\" (UniqueName: \"kubernetes.io/projected/6caa6499-3ae1-4a95-9983-6c597d6e1436-kube-api-access-qwkq9\") pod \"maas-api-revoke-sub-094cd551-s65fn\" (UID: \"6caa6499-3ae1-4a95-9983-6c597d6e1436\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-094cd551-s65fn" Oct 30 17:23:55.238112 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:55.238082 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-094cd551-s65fn" Oct 30 17:23:55.557734 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:23:55.557703 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6caa6499_3ae1_4a95_9983_6c597d6e1436.slice/crio-a5846ee05d87b41165912c7d4ad5f88ba21ad799bcfd652fce935a5cbabca776 WatchSource:0}: Error finding container a5846ee05d87b41165912c7d4ad5f88ba21ad799bcfd652fce935a5cbabca776: Status 404 returned error can't find the container with id a5846ee05d87b41165912c7d4ad5f88ba21ad799bcfd652fce935a5cbabca776 Oct 30 17:23:55.560779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:55.560758 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-094cd551-s65fn"] Oct 30 17:23:56.002292 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:56.002242 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-094cd551-s65fn" event={"ID":"6caa6499-3ae1-4a95-9983-6c597d6e1436","Type":"ContainerStarted","Data":"ee9f4e7957593b8bf86c4fdf1d77b859eb688bfc941d13eea29a66f418c59acc"} Oct 30 17:23:56.003090 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:56.002382 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-094cd551-s65fn" event={"ID":"6caa6499-3ae1-4a95-9983-6c597d6e1436","Type":"ContainerStarted","Data":"a5846ee05d87b41165912c7d4ad5f88ba21ad799bcfd652fce935a5cbabca776"} Oct 30 17:23:56.070171 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:56.070056 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-094cd551-s65fn" podStartSLOduration=2.070044037 podStartE2EDuration="2.070044037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:23:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:23:56.065915914 +0000 UTC m=+2210.066670834" watchObservedRunningTime="2026-10-30 17:23:56.070044037 +0000 UTC m=+2210.070798938" Oct 30 17:23:57.011239 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:57.011210 4522 generic.go:356] "Generic (PLEG): container finished" podID="6caa6499-3ae1-4a95-9983-6c597d6e1436" containerID="ee9f4e7957593b8bf86c4fdf1d77b859eb688bfc941d13eea29a66f418c59acc" exitCode=0 Oct 30 17:23:57.011702 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:57.011296 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-094cd551-s65fn" event={"ID":"6caa6499-3ae1-4a95-9983-6c597d6e1436","Type":"ContainerDied","Data":"ee9f4e7957593b8bf86c4fdf1d77b859eb688bfc941d13eea29a66f418c59acc"} Oct 30 17:23:58.189652 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:58.189628 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-094cd551-s65fn" Oct 30 17:23:58.314527 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:58.314446 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6caa6499-3ae1-4a95-9983-6c597d6e1436-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6caa6499-3ae1-4a95-9983-6c597d6e1436-maas-api-xks-ca\") pod \"6caa6499-3ae1-4a95-9983-6c597d6e1436\" (UID: \"6caa6499-3ae1-4a95-9983-6c597d6e1436\") " Oct 30 17:23:58.314681 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:58.314541 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6caa6499-3ae1-4a95-9983-6c597d6e1436-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6caa6499-3ae1-4a95-9983-6c597d6e1436-maas-api-openshift-ca\") pod \"6caa6499-3ae1-4a95-9983-6c597d6e1436\" (UID: \"6caa6499-3ae1-4a95-9983-6c597d6e1436\") " Oct 30 17:23:58.314721 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:58.314684 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6caa6499-3ae1-4a95-9983-6c597d6e1436-kube-api-access-qwkq9\" (UniqueName: \"kubernetes.io/projected/6caa6499-3ae1-4a95-9983-6c597d6e1436-kube-api-access-qwkq9\") pod \"6caa6499-3ae1-4a95-9983-6c597d6e1436\" (UID: \"6caa6499-3ae1-4a95-9983-6c597d6e1436\") " Oct 30 17:23:58.314916 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:58.314891 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6caa6499-3ae1-4a95-9983-6c597d6e1436-maas-api-openshift-ca" pod "6caa6499-3ae1-4a95-9983-6c597d6e1436" (UID: "6caa6499-3ae1-4a95-9983-6c597d6e1436"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:23:58.315280 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:58.315262 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6caa6499-3ae1-4a95-9983-6c597d6e1436-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:23:58.316737 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:58.316712 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6caa6499-3ae1-4a95-9983-6c597d6e1436-maas-api-xks-ca" pod "6caa6499-3ae1-4a95-9983-6c597d6e1436" (UID: "6caa6499-3ae1-4a95-9983-6c597d6e1436"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:23:58.316840 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:58.316767 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6caa6499-3ae1-4a95-9983-6c597d6e1436-kube-api-access-qwkq9" pod "6caa6499-3ae1-4a95-9983-6c597d6e1436" (UID: "6caa6499-3ae1-4a95-9983-6c597d6e1436"). InnerVolumeSpecName "kube-api-access-qwkq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:23:58.416692 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:58.416660 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwkq9\" (UniqueName: \"kubernetes.io/projected/6caa6499-3ae1-4a95-9983-6c597d6e1436-kube-api-access-qwkq9\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:23:58.416692 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:58.416688 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6caa6499-3ae1-4a95-9983-6c597d6e1436-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:23:59.029041 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:59.029012 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-094cd551-s65fn" Oct 30 17:23:59.029041 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:59.029027 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-094cd551-s65fn" event={"ID":"6caa6499-3ae1-4a95-9983-6c597d6e1436","Type":"ContainerDied","Data":"a5846ee05d87b41165912c7d4ad5f88ba21ad799bcfd652fce935a5cbabca776"} Oct 30 17:23:59.029263 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:59.029074 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5846ee05d87b41165912c7d4ad5f88ba21ad799bcfd652fce935a5cbabca776" Oct 30 17:23:59.888419 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:23:59.888384 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd5c8f-6b45577955-57qp6" podUID="8226519e-a60e-4b3c-b4b3-83479839c8bb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:24:03.725808 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:03.725772 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-6d4f99/sub-test-model-6d4f99-kserve-9ddf765b5-r59ft"] Oct 30 17:24:03.727020 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:03.726997 4522 cpu_manager.go:414] "containerMap: removing container" podUID="6caa6499-3ae1-4a95-9983-6c597d6e1436" containerName="revoke-keys" Oct 30 17:24:03.727086 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:03.727022 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6caa6499-3ae1-4a95-9983-6c597d6e1436" containerName="revoke-keys" Oct 30 17:24:03.727215 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:03.727200 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="6caa6499-3ae1-4a95-9983-6c597d6e1436" containerName="revoke-keys" Oct 30 17:24:03.731989 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:03.731960 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-6d4f99/sub-test-model-6d4f99-kserve-9ddf765b5-r59ft" Oct 30 17:24:03.752789 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:03.752758 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-6d4f99/sub-test-model-6d4f99-kserve-9ddf765b5-r59ft"] Oct 30 17:24:03.759721 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:03.759691 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-6d4f99\"/\"kube-root-ca.crt\"" Oct 30 17:24:03.759721 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:03.759691 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-6d4f99\"/\"openshift-service-ca.crt\"" Oct 30 17:24:03.761809 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:03.761786 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-6d4f99\"/\"default-dockercfg-phpst\"" Oct 30 17:24:03.761922 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:03.761850 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-6d4f99\"/\"sub-test-model-6d4f99-kserve-self-signed-certs\"" Oct 30 17:24:03.795512 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:03.795468 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f146a44-f336-4ec0-9419-ffba69c846dc-dshm\") pod \"sub-test-model-6d4f99-kserve-9ddf765b5-r59ft\" (UID: \"8f146a44-f336-4ec0-9419-ffba69c846dc\") " pod="ai-tenant-e2e-shared-b-w2-6d4f99/sub-test-model-6d4f99-kserve-9ddf765b5-r59ft" Oct 30 17:24:03.795715 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:03.795656 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f146a44-f336-4ec0-9419-ffba69c846dc-home\") pod \"sub-test-model-6d4f99-kserve-9ddf765b5-r59ft\" (UID: \"8f146a44-f336-4ec0-9419-ffba69c846dc\") " pod="ai-tenant-e2e-shared-b-w2-6d4f99/sub-test-model-6d4f99-kserve-9ddf765b5-r59ft" Oct 30 17:24:03.795715 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:03.795684 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f146a44-f336-4ec0-9419-ffba69c846dc-tmp-dir\") pod \"sub-test-model-6d4f99-kserve-9ddf765b5-r59ft\" (UID: \"8f146a44-f336-4ec0-9419-ffba69c846dc\") " pod="ai-tenant-e2e-shared-b-w2-6d4f99/sub-test-model-6d4f99-kserve-9ddf765b5-r59ft" Oct 30 17:24:03.795906 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:03.795880 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6m6c\" (UniqueName: \"kubernetes.io/projected/8f146a44-f336-4ec0-9419-ffba69c846dc-kube-api-access-l6m6c\") pod \"sub-test-model-6d4f99-kserve-9ddf765b5-r59ft\" (UID: \"8f146a44-f336-4ec0-9419-ffba69c846dc\") " pod="ai-tenant-e2e-shared-b-w2-6d4f99/sub-test-model-6d4f99-kserve-9ddf765b5-r59ft" Oct 30 17:24:03.795976 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:03.795933 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f146a44-f336-4ec0-9419-ffba69c846dc-model-cache\") pod \"sub-test-model-6d4f99-kserve-9ddf765b5-r59ft\" (UID: \"8f146a44-f336-4ec0-9419-ffba69c846dc\") " pod="ai-tenant-e2e-shared-b-w2-6d4f99/sub-test-model-6d4f99-kserve-9ddf765b5-r59ft" Oct 30 17:24:03.796078 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:03.796053 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f146a44-f336-4ec0-9419-ffba69c846dc-tls-certs\") pod \"sub-test-model-6d4f99-kserve-9ddf765b5-r59ft\" (UID: \"8f146a44-f336-4ec0-9419-ffba69c846dc\") " pod="ai-tenant-e2e-shared-b-w2-6d4f99/sub-test-model-6d4f99-kserve-9ddf765b5-r59ft" Oct 30 17:24:03.897935 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:03.897898 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f146a44-f336-4ec0-9419-ffba69c846dc-home\") pod \"sub-test-model-6d4f99-kserve-9ddf765b5-r59ft\" (UID: \"8f146a44-f336-4ec0-9419-ffba69c846dc\") " pod="ai-tenant-e2e-shared-b-w2-6d4f99/sub-test-model-6d4f99-kserve-9ddf765b5-r59ft" Oct 30 17:24:03.897935 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:03.897934 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f146a44-f336-4ec0-9419-ffba69c846dc-tmp-dir\") pod \"sub-test-model-6d4f99-kserve-9ddf765b5-r59ft\" (UID: \"8f146a44-f336-4ec0-9419-ffba69c846dc\") " pod="ai-tenant-e2e-shared-b-w2-6d4f99/sub-test-model-6d4f99-kserve-9ddf765b5-r59ft" Oct 30 17:24:03.898198 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:03.898025 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6m6c\" (UniqueName: \"kubernetes.io/projected/8f146a44-f336-4ec0-9419-ffba69c846dc-kube-api-access-l6m6c\") pod \"sub-test-model-6d4f99-kserve-9ddf765b5-r59ft\" (UID: \"8f146a44-f336-4ec0-9419-ffba69c846dc\") " pod="ai-tenant-e2e-shared-b-w2-6d4f99/sub-test-model-6d4f99-kserve-9ddf765b5-r59ft" Oct 30 17:24:03.898198 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:03.898051 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f146a44-f336-4ec0-9419-ffba69c846dc-model-cache\") pod \"sub-test-model-6d4f99-kserve-9ddf765b5-r59ft\" (UID: \"8f146a44-f336-4ec0-9419-ffba69c846dc\") " pod="ai-tenant-e2e-shared-b-w2-6d4f99/sub-test-model-6d4f99-kserve-9ddf765b5-r59ft" Oct 30 17:24:03.898198 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:03.898092 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f146a44-f336-4ec0-9419-ffba69c846dc-tls-certs\") pod \"sub-test-model-6d4f99-kserve-9ddf765b5-r59ft\" (UID: \"8f146a44-f336-4ec0-9419-ffba69c846dc\") " pod="ai-tenant-e2e-shared-b-w2-6d4f99/sub-test-model-6d4f99-kserve-9ddf765b5-r59ft" Oct 30 17:24:03.898198 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:03.898153 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f146a44-f336-4ec0-9419-ffba69c846dc-dshm\") pod \"sub-test-model-6d4f99-kserve-9ddf765b5-r59ft\" (UID: \"8f146a44-f336-4ec0-9419-ffba69c846dc\") " pod="ai-tenant-e2e-shared-b-w2-6d4f99/sub-test-model-6d4f99-kserve-9ddf765b5-r59ft" Oct 30 17:24:03.898477 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:03.898431 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f146a44-f336-4ec0-9419-ffba69c846dc-home\") pod \"sub-test-model-6d4f99-kserve-9ddf765b5-r59ft\" (UID: \"8f146a44-f336-4ec0-9419-ffba69c846dc\") " pod="ai-tenant-e2e-shared-b-w2-6d4f99/sub-test-model-6d4f99-kserve-9ddf765b5-r59ft" Oct 30 17:24:03.898477 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:03.898446 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f146a44-f336-4ec0-9419-ffba69c846dc-tmp-dir\") pod \"sub-test-model-6d4f99-kserve-9ddf765b5-r59ft\" (UID: \"8f146a44-f336-4ec0-9419-ffba69c846dc\") " pod="ai-tenant-e2e-shared-b-w2-6d4f99/sub-test-model-6d4f99-kserve-9ddf765b5-r59ft" Oct 30 17:24:03.898550 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:03.898495 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f146a44-f336-4ec0-9419-ffba69c846dc-model-cache\") pod \"sub-test-model-6d4f99-kserve-9ddf765b5-r59ft\" (UID: \"8f146a44-f336-4ec0-9419-ffba69c846dc\") " pod="ai-tenant-e2e-shared-b-w2-6d4f99/sub-test-model-6d4f99-kserve-9ddf765b5-r59ft" Oct 30 17:24:03.900547 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:03.900521 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f146a44-f336-4ec0-9419-ffba69c846dc-dshm\") pod \"sub-test-model-6d4f99-kserve-9ddf765b5-r59ft\" (UID: \"8f146a44-f336-4ec0-9419-ffba69c846dc\") " pod="ai-tenant-e2e-shared-b-w2-6d4f99/sub-test-model-6d4f99-kserve-9ddf765b5-r59ft" Oct 30 17:24:03.900818 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:03.900798 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f146a44-f336-4ec0-9419-ffba69c846dc-tls-certs\") pod \"sub-test-model-6d4f99-kserve-9ddf765b5-r59ft\" (UID: \"8f146a44-f336-4ec0-9419-ffba69c846dc\") " pod="ai-tenant-e2e-shared-b-w2-6d4f99/sub-test-model-6d4f99-kserve-9ddf765b5-r59ft" Oct 30 17:24:03.923289 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:03.923258 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6m6c\" (UniqueName: \"kubernetes.io/projected/8f146a44-f336-4ec0-9419-ffba69c846dc-kube-api-access-l6m6c\") pod \"sub-test-model-6d4f99-kserve-9ddf765b5-r59ft\" (UID: \"8f146a44-f336-4ec0-9419-ffba69c846dc\") " pod="ai-tenant-e2e-shared-b-w2-6d4f99/sub-test-model-6d4f99-kserve-9ddf765b5-r59ft" Oct 30 17:24:04.050603 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:04.050517 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-6d4f99/sub-test-model-6d4f99-kserve-9ddf765b5-r59ft" Oct 30 17:24:04.368149 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:24:04.368114 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f146a44_f336_4ec0_9419_ffba69c846dc.slice/crio-ec2d0d1805a7b9a9dbcbe5019169aab10323d8ac4d5fe5a557939fc9df13189c WatchSource:0}: Error finding container ec2d0d1805a7b9a9dbcbe5019169aab10323d8ac4d5fe5a557939fc9df13189c: Status 404 returned error can't find the container with id ec2d0d1805a7b9a9dbcbe5019169aab10323d8ac4d5fe5a557939fc9df13189c Oct 30 17:24:04.369910 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:04.369886 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-6d4f99/sub-test-model-6d4f99-kserve-9ddf765b5-r59ft"] Oct 30 17:24:05.084768 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:05.084733 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-6d4f99/sub-test-model-6d4f99-kserve-9ddf765b5-r59ft" event={"ID":"8f146a44-f336-4ec0-9419-ffba69c846dc","Type":"ContainerStarted","Data":"383df1458a8cdda85ea2bff3085ca8f5b480248912730335b4f2d0fd20a1348b"} Oct 30 17:24:05.085184 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:05.084780 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-6d4f99/sub-test-model-6d4f99-kserve-9ddf765b5-r59ft" event={"ID":"8f146a44-f336-4ec0-9419-ffba69c846dc","Type":"ContainerStarted","Data":"ec2d0d1805a7b9a9dbcbe5019169aab10323d8ac4d5fe5a557939fc9df13189c"} Oct 30 17:24:05.130279 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:05.130224 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-6d4f99/sub-test-model-6d4f99-kserve-9ddf765b5-r59ft" podStartSLOduration=2.130210736 podStartE2EDuration="2.130210736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:24:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:24:05.13004754 +0000 UTC m=+2219.130802443" watchObservedRunningTime="2026-10-30 17:24:05.130210736 +0000 UTC m=+2219.130965636" Oct 30 17:24:06.051585 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:06.051553 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-6d4f99/sub-test-model-6d4f99-kserve-9ddf765b5-r59ft" Oct 30 17:24:07.610719 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:07.610682 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w5-82243b/rate-test-model-82243b-kserve-75d85889df-fd25g"] Oct 30 17:24:07.616955 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:07.616929 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-82243b/rate-test-model-82243b-kserve-75d85889df-fd25g" Oct 30 17:24:07.637706 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:07.637676 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-82243b/rate-test-model-82243b-kserve-75d85889df-fd25g"] Oct 30 17:24:07.645289 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:07.645261 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-82243b\"/\"kube-root-ca.crt\"" Oct 30 17:24:07.645289 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:07.645261 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-82243b\"/\"default-dockercfg-x4577\"" Oct 30 17:24:07.647353 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:07.647334 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-82243b\"/\"openshift-service-ca.crt\"" Oct 30 17:24:07.647438 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:07.647334 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-82243b\"/\"rate-test-model-82243b-kserve-self-signed-certs\"" Oct 30 17:24:07.654067 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:07.654040 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hqzt\" (UniqueName: \"kubernetes.io/projected/882b8f38-848d-4fe0-aec9-bd81b63ace42-kube-api-access-4hqzt\") pod \"rate-test-model-82243b-kserve-75d85889df-fd25g\" (UID: \"882b8f38-848d-4fe0-aec9-bd81b63ace42\") " pod="ai-tenant-e2e-shared-b-w5-82243b/rate-test-model-82243b-kserve-75d85889df-fd25g" Oct 30 17:24:07.654173 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:07.654085 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/882b8f38-848d-4fe0-aec9-bd81b63ace42-tmp-dir\") pod \"rate-test-model-82243b-kserve-75d85889df-fd25g\" (UID: \"882b8f38-848d-4fe0-aec9-bd81b63ace42\") " pod="ai-tenant-e2e-shared-b-w5-82243b/rate-test-model-82243b-kserve-75d85889df-fd25g" Oct 30 17:24:07.654220 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:07.654173 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/882b8f38-848d-4fe0-aec9-bd81b63ace42-home\") pod \"rate-test-model-82243b-kserve-75d85889df-fd25g\" (UID: \"882b8f38-848d-4fe0-aec9-bd81b63ace42\") " pod="ai-tenant-e2e-shared-b-w5-82243b/rate-test-model-82243b-kserve-75d85889df-fd25g" Oct 30 17:24:07.654274 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:07.654220 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/882b8f38-848d-4fe0-aec9-bd81b63ace42-model-cache\") pod \"rate-test-model-82243b-kserve-75d85889df-fd25g\" (UID: \"882b8f38-848d-4fe0-aec9-bd81b63ace42\") " pod="ai-tenant-e2e-shared-b-w5-82243b/rate-test-model-82243b-kserve-75d85889df-fd25g" Oct 30 17:24:07.654461 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:07.654441 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/882b8f38-848d-4fe0-aec9-bd81b63ace42-tls-certs\") pod \"rate-test-model-82243b-kserve-75d85889df-fd25g\" (UID: \"882b8f38-848d-4fe0-aec9-bd81b63ace42\") " pod="ai-tenant-e2e-shared-b-w5-82243b/rate-test-model-82243b-kserve-75d85889df-fd25g" Oct 30 17:24:07.654513 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:07.654471 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/882b8f38-848d-4fe0-aec9-bd81b63ace42-dshm\") pod \"rate-test-model-82243b-kserve-75d85889df-fd25g\" (UID: \"882b8f38-848d-4fe0-aec9-bd81b63ace42\") " pod="ai-tenant-e2e-shared-b-w5-82243b/rate-test-model-82243b-kserve-75d85889df-fd25g" Oct 30 17:24:07.755927 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:07.755888 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hqzt\" (UniqueName: \"kubernetes.io/projected/882b8f38-848d-4fe0-aec9-bd81b63ace42-kube-api-access-4hqzt\") pod \"rate-test-model-82243b-kserve-75d85889df-fd25g\" (UID: \"882b8f38-848d-4fe0-aec9-bd81b63ace42\") " pod="ai-tenant-e2e-shared-b-w5-82243b/rate-test-model-82243b-kserve-75d85889df-fd25g" Oct 30 17:24:07.756128 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:07.755941 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/882b8f38-848d-4fe0-aec9-bd81b63ace42-tmp-dir\") pod \"rate-test-model-82243b-kserve-75d85889df-fd25g\" (UID: \"882b8f38-848d-4fe0-aec9-bd81b63ace42\") " pod="ai-tenant-e2e-shared-b-w5-82243b/rate-test-model-82243b-kserve-75d85889df-fd25g" Oct 30 17:24:07.756128 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:07.755992 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/882b8f38-848d-4fe0-aec9-bd81b63ace42-home\") pod \"rate-test-model-82243b-kserve-75d85889df-fd25g\" (UID: \"882b8f38-848d-4fe0-aec9-bd81b63ace42\") " pod="ai-tenant-e2e-shared-b-w5-82243b/rate-test-model-82243b-kserve-75d85889df-fd25g" Oct 30 17:24:07.756128 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:07.756032 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/882b8f38-848d-4fe0-aec9-bd81b63ace42-model-cache\") pod \"rate-test-model-82243b-kserve-75d85889df-fd25g\" (UID: \"882b8f38-848d-4fe0-aec9-bd81b63ace42\") " pod="ai-tenant-e2e-shared-b-w5-82243b/rate-test-model-82243b-kserve-75d85889df-fd25g" Oct 30 17:24:07.756336 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:07.756211 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/882b8f38-848d-4fe0-aec9-bd81b63ace42-tls-certs\") pod \"rate-test-model-82243b-kserve-75d85889df-fd25g\" (UID: \"882b8f38-848d-4fe0-aec9-bd81b63ace42\") " pod="ai-tenant-e2e-shared-b-w5-82243b/rate-test-model-82243b-kserve-75d85889df-fd25g" Oct 30 17:24:07.756336 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:07.756240 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/882b8f38-848d-4fe0-aec9-bd81b63ace42-dshm\") pod \"rate-test-model-82243b-kserve-75d85889df-fd25g\" (UID: \"882b8f38-848d-4fe0-aec9-bd81b63ace42\") " pod="ai-tenant-e2e-shared-b-w5-82243b/rate-test-model-82243b-kserve-75d85889df-fd25g" Oct 30 17:24:07.756484 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:07.756444 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/882b8f38-848d-4fe0-aec9-bd81b63ace42-home\") pod \"rate-test-model-82243b-kserve-75d85889df-fd25g\" (UID: \"882b8f38-848d-4fe0-aec9-bd81b63ace42\") " pod="ai-tenant-e2e-shared-b-w5-82243b/rate-test-model-82243b-kserve-75d85889df-fd25g" Oct 30 17:24:07.756484 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:07.756461 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/882b8f38-848d-4fe0-aec9-bd81b63ace42-tmp-dir\") pod \"rate-test-model-82243b-kserve-75d85889df-fd25g\" (UID: \"882b8f38-848d-4fe0-aec9-bd81b63ace42\") " pod="ai-tenant-e2e-shared-b-w5-82243b/rate-test-model-82243b-kserve-75d85889df-fd25g" Oct 30 17:24:07.756589 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:07.756530 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/882b8f38-848d-4fe0-aec9-bd81b63ace42-model-cache\") pod \"rate-test-model-82243b-kserve-75d85889df-fd25g\" (UID: \"882b8f38-848d-4fe0-aec9-bd81b63ace42\") " pod="ai-tenant-e2e-shared-b-w5-82243b/rate-test-model-82243b-kserve-75d85889df-fd25g" Oct 30 17:24:07.758665 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:07.758637 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/882b8f38-848d-4fe0-aec9-bd81b63ace42-dshm\") pod \"rate-test-model-82243b-kserve-75d85889df-fd25g\" (UID: \"882b8f38-848d-4fe0-aec9-bd81b63ace42\") " pod="ai-tenant-e2e-shared-b-w5-82243b/rate-test-model-82243b-kserve-75d85889df-fd25g" Oct 30 17:24:07.758806 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:07.758790 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/882b8f38-848d-4fe0-aec9-bd81b63ace42-tls-certs\") pod \"rate-test-model-82243b-kserve-75d85889df-fd25g\" (UID: \"882b8f38-848d-4fe0-aec9-bd81b63ace42\") " pod="ai-tenant-e2e-shared-b-w5-82243b/rate-test-model-82243b-kserve-75d85889df-fd25g" Oct 30 17:24:07.782828 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:07.782793 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hqzt\" (UniqueName: \"kubernetes.io/projected/882b8f38-848d-4fe0-aec9-bd81b63ace42-kube-api-access-4hqzt\") pod \"rate-test-model-82243b-kserve-75d85889df-fd25g\" (UID: \"882b8f38-848d-4fe0-aec9-bd81b63ace42\") " pod="ai-tenant-e2e-shared-b-w5-82243b/rate-test-model-82243b-kserve-75d85889df-fd25g" Oct 30 17:24:07.936033 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:07.935947 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-82243b/rate-test-model-82243b-kserve-75d85889df-fd25g" Oct 30 17:24:08.247675 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:24:08.247643 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod882b8f38_848d_4fe0_aec9_bd81b63ace42.slice/crio-ef09d2261b7db2529d669f2513bdc5a474a2ed64a4a37c4153cc1dcbefd3b7af WatchSource:0}: Error finding container ef09d2261b7db2529d669f2513bdc5a474a2ed64a4a37c4153cc1dcbefd3b7af: Status 404 returned error can't find the container with id ef09d2261b7db2529d669f2513bdc5a474a2ed64a4a37c4153cc1dcbefd3b7af Oct 30 17:24:08.249737 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:08.249711 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-82243b/rate-test-model-82243b-kserve-75d85889df-fd25g"] Oct 30 17:24:09.119908 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:09.119869 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-82243b/rate-test-model-82243b-kserve-75d85889df-fd25g" event={"ID":"882b8f38-848d-4fe0-aec9-bd81b63ace42","Type":"ContainerStarted","Data":"35788251ba76cbd2479bd7a25f117bfb2c16954584f5982894c6356339ec77bb"} Oct 30 17:24:09.120368 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:09.119917 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-82243b/rate-test-model-82243b-kserve-75d85889df-fd25g" event={"ID":"882b8f38-848d-4fe0-aec9-bd81b63ace42","Type":"ContainerStarted","Data":"ef09d2261b7db2529d669f2513bdc5a474a2ed64a4a37c4153cc1dcbefd3b7af"} Oct 30 17:24:09.168954 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:09.168899 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w5-82243b/rate-test-model-82243b-kserve-75d85889df-fd25g" podStartSLOduration=2.16888367 podStartE2EDuration="2.16888367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:24:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:24:09.164324101 +0000 UTC m=+2223.165079004" watchObservedRunningTime="2026-10-30 17:24:09.16888367 +0000 UTC m=+2223.169638570" Oct 30 17:24:09.887786 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:09.887749 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd5c8f-6b45577955-57qp6" podUID="8226519e-a60e-4b3c-b4b3-83479839c8bb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:24:09.936944 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:09.936909 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w5-82243b/rate-test-model-82243b-kserve-75d85889df-fd25g" Oct 30 17:24:10.843724 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:10.843689 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc"] Oct 30 17:24:10.850619 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:10.850592 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc" Oct 30 17:24:10.868832 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:10.868802 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595"] Oct 30 17:24:10.873828 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:10.873809 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595" Oct 30 17:24:10.875080 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:10.875056 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc"] Oct 30 17:24:10.876103 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:10.876079 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-11f480-openshift-default-dockercfg-q54n7\"" Oct 30 17:24:10.896434 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:10.896403 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595"] Oct 30 17:24:10.917620 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:10.917589 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d44befd-705e-4589-b6d0-c944810a76bc-istio-data\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc\" (UID: \"4d44befd-705e-4589-b6d0-c944810a76bc\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc" Oct 30 17:24:10.917844 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:10.917814 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d44befd-705e-4589-b6d0-c944810a76bc-workload-certs\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc\" (UID: \"4d44befd-705e-4589-b6d0-c944810a76bc\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc" Oct 30 17:24:10.917985 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:10.917958 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d44befd-705e-4589-b6d0-c944810a76bc-istio-envoy\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc\" (UID: \"4d44befd-705e-4589-b6d0-c944810a76bc\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc" Oct 30 17:24:10.918060 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:10.918039 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d44befd-705e-4589-b6d0-c944810a76bc-workload-socket\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc\" (UID: \"4d44befd-705e-4589-b6d0-c944810a76bc\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc" Oct 30 17:24:10.918096 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:10.918078 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d44befd-705e-4589-b6d0-c944810a76bc-credential-socket\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc\" (UID: \"4d44befd-705e-4589-b6d0-c944810a76bc\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc" Oct 30 17:24:10.918188 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:10.918172 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d44befd-705e-4589-b6d0-c944810a76bc-istio-podinfo\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc\" (UID: \"4d44befd-705e-4589-b6d0-c944810a76bc\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc" Oct 30 17:24:10.918245 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:10.918230 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d44befd-705e-4589-b6d0-c944810a76bc-istiod-ca-cert\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc\" (UID: \"4d44befd-705e-4589-b6d0-c944810a76bc\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc" Oct 30 17:24:10.918380 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:10.918361 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d44befd-705e-4589-b6d0-c944810a76bc-istio-token\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc\" (UID: \"4d44befd-705e-4589-b6d0-c944810a76bc\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc" Oct 30 17:24:10.918461 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:10.918416 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sskxp\" (UniqueName: \"kubernetes.io/projected/4d44befd-705e-4589-b6d0-c944810a76bc-kube-api-access-sskxp\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc\" (UID: \"4d44befd-705e-4589-b6d0-c944810a76bc\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc" Oct 30 17:24:11.019554 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.019517 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d44befd-705e-4589-b6d0-c944810a76bc-istio-envoy\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc\" (UID: \"4d44befd-705e-4589-b6d0-c944810a76bc\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc" Oct 30 17:24:11.019554 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.019562 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d44befd-705e-4589-b6d0-c944810a76bc-workload-socket\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc\" (UID: \"4d44befd-705e-4589-b6d0-c944810a76bc\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc" Oct 30 17:24:11.019818 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.019583 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d44befd-705e-4589-b6d0-c944810a76bc-credential-socket\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc\" (UID: \"4d44befd-705e-4589-b6d0-c944810a76bc\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc" Oct 30 17:24:11.019818 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.019625 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d44befd-705e-4589-b6d0-c944810a76bc-istio-podinfo\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc\" (UID: \"4d44befd-705e-4589-b6d0-c944810a76bc\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc" Oct 30 17:24:11.019818 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.019650 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a5d5676-2fbf-4892-8b12-862c483b2d17-workload-certs\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595\" (UID: \"1a5d5676-2fbf-4892-8b12-862c483b2d17\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595" Oct 30 17:24:11.019818 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.019677 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d44befd-705e-4589-b6d0-c944810a76bc-istiod-ca-cert\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc\" (UID: \"4d44befd-705e-4589-b6d0-c944810a76bc\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc" Oct 30 17:24:11.020052 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.019888 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a5d5676-2fbf-4892-8b12-862c483b2d17-istio-data\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595\" (UID: \"1a5d5676-2fbf-4892-8b12-862c483b2d17\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595" Oct 30 17:24:11.020052 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.019950 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d44befd-705e-4589-b6d0-c944810a76bc-istio-token\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc\" (UID: \"4d44befd-705e-4589-b6d0-c944810a76bc\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc" Oct 30 17:24:11.020052 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.020010 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d44befd-705e-4589-b6d0-c944810a76bc-workload-socket\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc\" (UID: \"4d44befd-705e-4589-b6d0-c944810a76bc\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc" Oct 30 17:24:11.020052 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.020033 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sskxp\" (UniqueName: \"kubernetes.io/projected/4d44befd-705e-4589-b6d0-c944810a76bc-kube-api-access-sskxp\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc\" (UID: \"4d44befd-705e-4589-b6d0-c944810a76bc\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc" Oct 30 17:24:11.020261 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.020152 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d44befd-705e-4589-b6d0-c944810a76bc-istio-data\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc\" (UID: \"4d44befd-705e-4589-b6d0-c944810a76bc\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc" Oct 30 17:24:11.020336 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.020258 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a5d5676-2fbf-4892-8b12-862c483b2d17-istio-envoy\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595\" (UID: \"1a5d5676-2fbf-4892-8b12-862c483b2d17\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595" Oct 30 17:24:11.020336 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.020291 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gczm\" (UniqueName: \"kubernetes.io/projected/1a5d5676-2fbf-4892-8b12-862c483b2d17-kube-api-access-4gczm\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595\" (UID: \"1a5d5676-2fbf-4892-8b12-862c483b2d17\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595" Oct 30 17:24:11.020476 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.020373 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a5d5676-2fbf-4892-8b12-862c483b2d17-workload-socket\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595\" (UID: \"1a5d5676-2fbf-4892-8b12-862c483b2d17\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595" Oct 30 17:24:11.020476 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.020431 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a5d5676-2fbf-4892-8b12-862c483b2d17-istiod-ca-cert\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595\" (UID: \"1a5d5676-2fbf-4892-8b12-862c483b2d17\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595" Oct 30 17:24:11.020589 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.020462 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d44befd-705e-4589-b6d0-c944810a76bc-credential-socket\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc\" (UID: \"4d44befd-705e-4589-b6d0-c944810a76bc\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc" Oct 30 17:24:11.020589 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.020528 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d44befd-705e-4589-b6d0-c944810a76bc-workload-certs\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc\" (UID: \"4d44befd-705e-4589-b6d0-c944810a76bc\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc" Oct 30 17:24:11.020705 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.020588 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a5d5676-2fbf-4892-8b12-862c483b2d17-istio-podinfo\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595\" (UID: \"1a5d5676-2fbf-4892-8b12-862c483b2d17\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595" Oct 30 17:24:11.020705 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.020633 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a5d5676-2fbf-4892-8b12-862c483b2d17-istio-token\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595\" (UID: \"1a5d5676-2fbf-4892-8b12-862c483b2d17\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595" Oct 30 17:24:11.020705 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.020661 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d44befd-705e-4589-b6d0-c944810a76bc-istio-data\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc\" (UID: \"4d44befd-705e-4589-b6d0-c944810a76bc\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc" Oct 30 17:24:11.020882 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.020711 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a5d5676-2fbf-4892-8b12-862c483b2d17-credential-socket\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595\" (UID: \"1a5d5676-2fbf-4892-8b12-862c483b2d17\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595" Oct 30 17:24:11.020882 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.020849 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d44befd-705e-4589-b6d0-c944810a76bc-workload-certs\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc\" (UID: \"4d44befd-705e-4589-b6d0-c944810a76bc\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc" Oct 30 17:24:11.021005 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.020899 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d44befd-705e-4589-b6d0-c944810a76bc-istiod-ca-cert\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc\" (UID: \"4d44befd-705e-4589-b6d0-c944810a76bc\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc" Oct 30 17:24:11.022217 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.022192 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d44befd-705e-4589-b6d0-c944810a76bc-istio-envoy\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc\" (UID: \"4d44befd-705e-4589-b6d0-c944810a76bc\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc" Oct 30 17:24:11.022409 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.022394 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d44befd-705e-4589-b6d0-c944810a76bc-istio-podinfo\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc\" (UID: \"4d44befd-705e-4589-b6d0-c944810a76bc\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc" Oct 30 17:24:11.047638 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.047604 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d44befd-705e-4589-b6d0-c944810a76bc-istio-token\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc\" (UID: \"4d44befd-705e-4589-b6d0-c944810a76bc\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc" Oct 30 17:24:11.052754 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.052731 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sskxp\" (UniqueName: \"kubernetes.io/projected/4d44befd-705e-4589-b6d0-c944810a76bc-kube-api-access-sskxp\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc\" (UID: \"4d44befd-705e-4589-b6d0-c944810a76bc\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc" Oct 30 17:24:11.122692 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.122611 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a5d5676-2fbf-4892-8b12-862c483b2d17-workload-certs\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595\" (UID: \"1a5d5676-2fbf-4892-8b12-862c483b2d17\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595" Oct 30 17:24:11.122692 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.122680 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a5d5676-2fbf-4892-8b12-862c483b2d17-istio-data\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595\" (UID: \"1a5d5676-2fbf-4892-8b12-862c483b2d17\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595" Oct 30 17:24:11.122868 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.122851 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a5d5676-2fbf-4892-8b12-862c483b2d17-istio-envoy\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595\" (UID: \"1a5d5676-2fbf-4892-8b12-862c483b2d17\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595" Oct 30 17:24:11.122909 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.122886 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gczm\" (UniqueName: \"kubernetes.io/projected/1a5d5676-2fbf-4892-8b12-862c483b2d17-kube-api-access-4gczm\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595\" (UID: \"1a5d5676-2fbf-4892-8b12-862c483b2d17\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595" Oct 30 17:24:11.122964 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.122924 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a5d5676-2fbf-4892-8b12-862c483b2d17-workload-socket\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595\" (UID: \"1a5d5676-2fbf-4892-8b12-862c483b2d17\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595" Oct 30 17:24:11.123031 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.122971 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a5d5676-2fbf-4892-8b12-862c483b2d17-istiod-ca-cert\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595\" (UID: \"1a5d5676-2fbf-4892-8b12-862c483b2d17\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595" Oct 30 17:24:11.123085 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.123047 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a5d5676-2fbf-4892-8b12-862c483b2d17-istio-podinfo\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595\" (UID: \"1a5d5676-2fbf-4892-8b12-862c483b2d17\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595" Oct 30 17:24:11.123085 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.123054 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a5d5676-2fbf-4892-8b12-862c483b2d17-istio-data\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595\" (UID: \"1a5d5676-2fbf-4892-8b12-862c483b2d17\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595" Oct 30 17:24:11.123085 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.122967 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a5d5676-2fbf-4892-8b12-862c483b2d17-workload-certs\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595\" (UID: \"1a5d5676-2fbf-4892-8b12-862c483b2d17\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595" Oct 30 17:24:11.123239 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.123083 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a5d5676-2fbf-4892-8b12-862c483b2d17-istio-token\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595\" (UID: \"1a5d5676-2fbf-4892-8b12-862c483b2d17\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595" Oct 30 17:24:11.123239 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.123129 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a5d5676-2fbf-4892-8b12-862c483b2d17-credential-socket\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595\" (UID: \"1a5d5676-2fbf-4892-8b12-862c483b2d17\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595" Oct 30 17:24:11.123393 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.123241 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a5d5676-2fbf-4892-8b12-862c483b2d17-workload-socket\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595\" (UID: \"1a5d5676-2fbf-4892-8b12-862c483b2d17\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595" Oct 30 17:24:11.123514 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.123493 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a5d5676-2fbf-4892-8b12-862c483b2d17-credential-socket\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595\" (UID: \"1a5d5676-2fbf-4892-8b12-862c483b2d17\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595" Oct 30 17:24:11.123636 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.123616 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a5d5676-2fbf-4892-8b12-862c483b2d17-istiod-ca-cert\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595\" (UID: \"1a5d5676-2fbf-4892-8b12-862c483b2d17\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595" Oct 30 17:24:11.125162 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.125142 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a5d5676-2fbf-4892-8b12-862c483b2d17-istio-envoy\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595\" (UID: \"1a5d5676-2fbf-4892-8b12-862c483b2d17\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595" Oct 30 17:24:11.125520 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.125501 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a5d5676-2fbf-4892-8b12-862c483b2d17-istio-podinfo\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595\" (UID: \"1a5d5676-2fbf-4892-8b12-862c483b2d17\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595" Oct 30 17:24:11.147776 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.147754 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a5d5676-2fbf-4892-8b12-862c483b2d17-istio-token\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595\" (UID: \"1a5d5676-2fbf-4892-8b12-862c483b2d17\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595" Oct 30 17:24:11.147908 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.147839 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gczm\" (UniqueName: \"kubernetes.io/projected/1a5d5676-2fbf-4892-8b12-862c483b2d17-kube-api-access-4gczm\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595\" (UID: \"1a5d5676-2fbf-4892-8b12-862c483b2d17\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595" Oct 30 17:24:11.167044 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.166981 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc" Oct 30 17:24:11.195668 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.195636 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595" Oct 30 17:24:11.353905 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.353851 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2d01ec-zmpbj"] Oct 30 17:24:11.361138 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.361111 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2d01ec-zmpbj" Oct 30 17:24:11.380238 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.380203 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:24:11.380503 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.380483 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:24:11.388874 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.388439 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2d01ec-zmpbj"] Oct 30 17:24:11.392635 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.392605 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd5c8f-6b45577955-57qp6" Oct 30 17:24:11.439458 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.439428 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68186e31-c661-41ea-bf07-6f97fb8d39b6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2d01ec-zmpbj\" (UID: \"68186e31-c661-41ea-bf07-6f97fb8d39b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2d01ec-zmpbj" Oct 30 17:24:11.439693 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.439664 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sksb6\" (UniqueName: \"kubernetes.io/projected/68186e31-c661-41ea-bf07-6f97fb8d39b6-kube-api-access-sksb6\") pod \"maas-api-revoke-keys-e2e-inf-a-2d01ec-zmpbj\" (UID: \"68186e31-c661-41ea-bf07-6f97fb8d39b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2d01ec-zmpbj" Oct 30 17:24:11.439777 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.439760 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68186e31-c661-41ea-bf07-6f97fb8d39b6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2d01ec-zmpbj\" (UID: \"68186e31-c661-41ea-bf07-6f97fb8d39b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2d01ec-zmpbj" Oct 30 17:24:11.504951 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:24:11.504915 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d44befd_705e_4589_b6d0_c944810a76bc.slice/crio-ce0c4e7ff0af068eacad45ef0e0802884de0a23fe10009a8ab0d9a2358fbcb3c WatchSource:0}: Error finding container ce0c4e7ff0af068eacad45ef0e0802884de0a23fe10009a8ab0d9a2358fbcb3c: Status 404 returned error can't find the container with id ce0c4e7ff0af068eacad45ef0e0802884de0a23fe10009a8ab0d9a2358fbcb3c Oct 30 17:24:11.508291 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.508258 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc"] Oct 30 17:24:11.509188 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.509154 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:24:11.509301 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.509231 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:24:11.509301 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.509280 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:24:11.542129 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.542095 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8226519e-a60e-4b3c-b4b3-83479839c8bb-kube-api-access-sd4ns\" (UniqueName: \"kubernetes.io/projected/8226519e-a60e-4b3c-b4b3-83479839c8bb-kube-api-access-sd4ns\") pod \"8226519e-a60e-4b3c-b4b3-83479839c8bb\" (UID: \"8226519e-a60e-4b3c-b4b3-83479839c8bb\") " Oct 30 17:24:11.542296 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.542279 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8226519e-a60e-4b3c-b4b3-83479839c8bb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8226519e-a60e-4b3c-b4b3-83479839c8bb-plugins-config-volume\") pod \"8226519e-a60e-4b3c-b4b3-83479839c8bb\" (UID: \"8226519e-a60e-4b3c-b4b3-83479839c8bb\") " Oct 30 17:24:11.543175 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.543145 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68186e31-c661-41ea-bf07-6f97fb8d39b6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2d01ec-zmpbj\" (UID: \"68186e31-c661-41ea-bf07-6f97fb8d39b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2d01ec-zmpbj" Oct 30 17:24:11.543293 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.543242 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sksb6\" (UniqueName: \"kubernetes.io/projected/68186e31-c661-41ea-bf07-6f97fb8d39b6-kube-api-access-sksb6\") pod \"maas-api-revoke-keys-e2e-inf-a-2d01ec-zmpbj\" (UID: \"68186e31-c661-41ea-bf07-6f97fb8d39b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2d01ec-zmpbj" Oct 30 17:24:11.543293 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.543275 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68186e31-c661-41ea-bf07-6f97fb8d39b6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2d01ec-zmpbj\" (UID: \"68186e31-c661-41ea-bf07-6f97fb8d39b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2d01ec-zmpbj" Oct 30 17:24:11.543732 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.543704 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8226519e-a60e-4b3c-b4b3-83479839c8bb-plugins-config-volume" pod "8226519e-a60e-4b3c-b4b3-83479839c8bb" (UID: "8226519e-a60e-4b3c-b4b3-83479839c8bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:24:11.544093 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.544075 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68186e31-c661-41ea-bf07-6f97fb8d39b6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2d01ec-zmpbj\" (UID: \"68186e31-c661-41ea-bf07-6f97fb8d39b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2d01ec-zmpbj" Oct 30 17:24:11.568590 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.568553 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8226519e-a60e-4b3c-b4b3-83479839c8bb-kube-api-access-sd4ns" pod "8226519e-a60e-4b3c-b4b3-83479839c8bb" (UID: "8226519e-a60e-4b3c-b4b3-83479839c8bb"). InnerVolumeSpecName "kube-api-access-sd4ns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:24:11.570091 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.569886 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68186e31-c661-41ea-bf07-6f97fb8d39b6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2d01ec-zmpbj\" (UID: \"68186e31-c661-41ea-bf07-6f97fb8d39b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2d01ec-zmpbj" Oct 30 17:24:11.577833 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.577802 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595"] Oct 30 17:24:11.584700 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.584655 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:24:11.585661 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.585634 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sksb6\" (UniqueName: \"kubernetes.io/projected/68186e31-c661-41ea-bf07-6f97fb8d39b6-kube-api-access-sksb6\") pod \"maas-api-revoke-keys-e2e-inf-a-2d01ec-zmpbj\" (UID: \"68186e31-c661-41ea-bf07-6f97fb8d39b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2d01ec-zmpbj" Oct 30 17:24:11.586574 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.586538 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:24:11.586835 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.586800 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:24:11.647240 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.647208 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8226519e-a60e-4b3c-b4b3-83479839c8bb-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:24:11.647380 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.647246 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd4ns\" (UniqueName: \"kubernetes.io/projected/8226519e-a60e-4b3c-b4b3-83479839c8bb-kube-api-access-sd4ns\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:24:11.703848 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:11.692029 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2d01ec-zmpbj" Oct 30 17:24:12.142516 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:24:12.142484 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68186e31_c661_41ea_bf07_6f97fb8d39b6.slice/crio-9b64575b313e0cb68a7fe122a06321060ee61e61e858552bf2e12594fbd7a099 WatchSource:0}: Error finding container 9b64575b313e0cb68a7fe122a06321060ee61e61e858552bf2e12594fbd7a099: Status 404 returned error can't find the container with id 9b64575b313e0cb68a7fe122a06321060ee61e61e858552bf2e12594fbd7a099 Oct 30 17:24:12.154580 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:12.152395 4522 generic.go:356] "Generic (PLEG): container finished" podID="8226519e-a60e-4b3c-b4b3-83479839c8bb" containerID="05d34dc64a6d870192f7f10e27988f3ecd79d0874121e06516da2a4a9a6e3d9d" exitCode=0 Oct 30 17:24:12.154580 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:12.152543 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd5c8f-6b45577955-57qp6" Oct 30 17:24:12.154580 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:12.153510 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2d01ec-zmpbj"] Oct 30 17:24:12.154580 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:12.153562 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595" event={"ID":"1a5d5676-2fbf-4892-8b12-862c483b2d17","Type":"ContainerStarted","Data":"4bfd3354e7a42dae397dd62b3824c2129b5c1b1bf65841352b13313c5eed0a4c"} Oct 30 17:24:12.154580 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:12.153592 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595" event={"ID":"1a5d5676-2fbf-4892-8b12-862c483b2d17","Type":"ContainerStarted","Data":"d24e113a8041fdbde7528723ed7a807a4a5208036d7a0b3e89bfa86277999443"} Oct 30 17:24:12.154580 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:12.153613 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd5c8f-6b45577955-57qp6" event={"ID":"8226519e-a60e-4b3c-b4b3-83479839c8bb","Type":"ContainerDied","Data":"05d34dc64a6d870192f7f10e27988f3ecd79d0874121e06516da2a4a9a6e3d9d"} Oct 30 17:24:12.154580 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:12.153640 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd5c8f-6b45577955-57qp6" event={"ID":"8226519e-a60e-4b3c-b4b3-83479839c8bb","Type":"ContainerDied","Data":"3f9f84696bd368284ea1e0d8ca2625589e9b5a811b41bc416f1dc94c9069bf76"} Oct 30 17:24:12.154580 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:12.153670 4522 scope.go:122] "RemoveContainer" containerID="05d34dc64a6d870192f7f10e27988f3ecd79d0874121e06516da2a4a9a6e3d9d" Oct 30 17:24:12.155606 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:12.155576 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc" event={"ID":"4d44befd-705e-4589-b6d0-c944810a76bc","Type":"ContainerStarted","Data":"8e2803f8b7c3d8f4026c1c9193383ed799fb717c5629d680431eb3a3e6190c06"} Oct 30 17:24:12.155729 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:12.155627 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc" event={"ID":"4d44befd-705e-4589-b6d0-c944810a76bc","Type":"ContainerStarted","Data":"ce0c4e7ff0af068eacad45ef0e0802884de0a23fe10009a8ab0d9a2358fbcb3c"} Oct 30 17:24:12.168170 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:12.168150 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc" Oct 30 17:24:12.170174 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:12.170147 4522 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body= Oct 30 17:24:12.170443 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:12.170199 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc" podUID="4d44befd-705e-4589-b6d0-c944810a76bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" Oct 30 17:24:12.170940 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:12.170917 4522 scope.go:122] "RemoveContainer" containerID="05d34dc64a6d870192f7f10e27988f3ecd79d0874121e06516da2a4a9a6e3d9d" Oct 30 17:24:12.171291 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:24:12.171263 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05d34dc64a6d870192f7f10e27988f3ecd79d0874121e06516da2a4a9a6e3d9d\": container with ID starting with 05d34dc64a6d870192f7f10e27988f3ecd79d0874121e06516da2a4a9a6e3d9d not found: ID does not exist" containerID="05d34dc64a6d870192f7f10e27988f3ecd79d0874121e06516da2a4a9a6e3d9d" Oct 30 17:24:12.171420 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:12.171300 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05d34dc64a6d870192f7f10e27988f3ecd79d0874121e06516da2a4a9a6e3d9d"} err="failed to get container status \"05d34dc64a6d870192f7f10e27988f3ecd79d0874121e06516da2a4a9a6e3d9d\": rpc error: code = NotFound desc = could not find container \"05d34dc64a6d870192f7f10e27988f3ecd79d0874121e06516da2a4a9a6e3d9d\": container with ID starting with 05d34dc64a6d870192f7f10e27988f3ecd79d0874121e06516da2a4a9a6e3d9d not found: ID does not exist" Oct 30 17:24:12.195989 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:12.195960 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595" Oct 30 17:24:12.197680 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:12.197648 4522 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body= Oct 30 17:24:12.197804 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:12.197705 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595" podUID="1a5d5676-2fbf-4892-8b12-862c483b2d17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" Oct 30 17:24:12.206177 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:12.206134 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595" podStartSLOduration=2.206120977 podStartE2EDuration="2.206120977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:24:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:24:12.203871564 +0000 UTC m=+2226.204626486" watchObservedRunningTime="2026-10-30 17:24:12.206120977 +0000 UTC m=+2226.206875877" Oct 30 17:24:12.259030 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:12.258977 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc" podStartSLOduration=2.258963451 podStartE2EDuration="2.258963451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:24:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:24:12.256264702 +0000 UTC m=+2226.257019605" watchObservedRunningTime="2026-10-30 17:24:12.258963451 +0000 UTC m=+2226.259718351" Oct 30 17:24:12.298785 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:12.298706 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-dd5c8f-6b45577955-57qp6"] Oct 30 17:24:12.303164 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:12.303135 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-dd5c8f-6b45577955-57qp6"] Oct 30 17:24:12.980500 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:24:12.980414 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:24:13.159864 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:13.159179 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-db8417/sub-test-model-db8417-kserve-fb7fdb549-6xvgx"] Oct 30 17:24:13.160561 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:13.160535 4522 cpu_manager.go:414] "containerMap: removing container" podUID="8226519e-a60e-4b3c-b4b3-83479839c8bb" containerName="payload-pre-processing" Oct 30 17:24:13.160561 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:13.160560 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8226519e-a60e-4b3c-b4b3-83479839c8bb" containerName="payload-pre-processing" Oct 30 17:24:13.160764 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:13.160748 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="8226519e-a60e-4b3c-b4b3-83479839c8bb" containerName="payload-pre-processing" Oct 30 17:24:13.166735 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:13.166708 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-db8417/sub-test-model-db8417-kserve-fb7fdb549-6xvgx"] Oct 30 17:24:13.166896 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:13.166846 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-db8417/sub-test-model-db8417-kserve-fb7fdb549-6xvgx" Oct 30 17:24:13.168825 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:13.168289 4522 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body= Oct 30 17:24:13.168825 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:13.168362 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc" podUID="4d44befd-705e-4589-b6d0-c944810a76bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" Oct 30 17:24:13.177995 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:13.177964 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2d01ec-zmpbj" event={"ID":"68186e31-c661-41ea-bf07-6f97fb8d39b6","Type":"ContainerStarted","Data":"6e11a96e9bedc3c5a36e71649fcca22a349778bc515d1851c3a678bcc6e340c1"} Oct 30 17:24:13.178359 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:13.178332 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2d01ec-zmpbj" event={"ID":"68186e31-c661-41ea-bf07-6f97fb8d39b6","Type":"ContainerStarted","Data":"9b64575b313e0cb68a7fe122a06321060ee61e61e858552bf2e12594fbd7a099"} Oct 30 17:24:13.185968 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:13.185198 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-db8417\"/\"kube-root-ca.crt\"" Oct 30 17:24:13.187655 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:13.187632 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-db8417\"/\"sub-test-model-db8417-kserve-self-signed-certs\"" Oct 30 17:24:13.187779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:13.187759 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-db8417\"/\"openshift-service-ca.crt\"" Oct 30 17:24:13.249389 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:13.245123 4522 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body= Oct 30 17:24:13.249389 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:13.245172 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595" podUID="1a5d5676-2fbf-4892-8b12-862c483b2d17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" Oct 30 17:24:13.249389 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:13.249113 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-db8417\"/\"default-dockercfg-dtqp2\"" Oct 30 17:24:13.271999 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:13.271847 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f922e81e-fa9c-46ad-af32-5e0266311e85-tls-certs\") pod \"sub-test-model-db8417-kserve-fb7fdb549-6xvgx\" (UID: \"f922e81e-fa9c-46ad-af32-5e0266311e85\") " pod="ai-tenant-e2e-shared-b-w3-db8417/sub-test-model-db8417-kserve-fb7fdb549-6xvgx" Oct 30 17:24:13.271999 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:13.271887 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl227\" (UniqueName: \"kubernetes.io/projected/f922e81e-fa9c-46ad-af32-5e0266311e85-kube-api-access-zl227\") pod \"sub-test-model-db8417-kserve-fb7fdb549-6xvgx\" (UID: \"f922e81e-fa9c-46ad-af32-5e0266311e85\") " pod="ai-tenant-e2e-shared-b-w3-db8417/sub-test-model-db8417-kserve-fb7fdb549-6xvgx" Oct 30 17:24:13.271999 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:13.271947 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f922e81e-fa9c-46ad-af32-5e0266311e85-dshm\") pod \"sub-test-model-db8417-kserve-fb7fdb549-6xvgx\" (UID: \"f922e81e-fa9c-46ad-af32-5e0266311e85\") " pod="ai-tenant-e2e-shared-b-w3-db8417/sub-test-model-db8417-kserve-fb7fdb549-6xvgx" Oct 30 17:24:13.272382 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:13.272120 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f922e81e-fa9c-46ad-af32-5e0266311e85-home\") pod \"sub-test-model-db8417-kserve-fb7fdb549-6xvgx\" (UID: \"f922e81e-fa9c-46ad-af32-5e0266311e85\") " pod="ai-tenant-e2e-shared-b-w3-db8417/sub-test-model-db8417-kserve-fb7fdb549-6xvgx" Oct 30 17:24:13.272382 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:13.272188 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f922e81e-fa9c-46ad-af32-5e0266311e85-tmp-dir\") pod \"sub-test-model-db8417-kserve-fb7fdb549-6xvgx\" (UID: \"f922e81e-fa9c-46ad-af32-5e0266311e85\") " pod="ai-tenant-e2e-shared-b-w3-db8417/sub-test-model-db8417-kserve-fb7fdb549-6xvgx" Oct 30 17:24:13.272527 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:13.272394 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f922e81e-fa9c-46ad-af32-5e0266311e85-model-cache\") pod \"sub-test-model-db8417-kserve-fb7fdb549-6xvgx\" (UID: \"f922e81e-fa9c-46ad-af32-5e0266311e85\") " pod="ai-tenant-e2e-shared-b-w3-db8417/sub-test-model-db8417-kserve-fb7fdb549-6xvgx" Oct 30 17:24:13.368329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:13.363053 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2d01ec-zmpbj" podStartSLOduration=2.363025481 podStartE2EDuration="2.363025481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:24:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:24:13.347587729 +0000 UTC m=+2227.348342625" watchObservedRunningTime="2026-10-30 17:24:13.363025481 +0000 UTC m=+2227.363780383" Oct 30 17:24:13.396330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:13.395437 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f922e81e-fa9c-46ad-af32-5e0266311e85-home\") pod \"sub-test-model-db8417-kserve-fb7fdb549-6xvgx\" (UID: \"f922e81e-fa9c-46ad-af32-5e0266311e85\") " pod="ai-tenant-e2e-shared-b-w3-db8417/sub-test-model-db8417-kserve-fb7fdb549-6xvgx" Oct 30 17:24:13.396330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:13.395593 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f922e81e-fa9c-46ad-af32-5e0266311e85-tmp-dir\") pod \"sub-test-model-db8417-kserve-fb7fdb549-6xvgx\" (UID: \"f922e81e-fa9c-46ad-af32-5e0266311e85\") " pod="ai-tenant-e2e-shared-b-w3-db8417/sub-test-model-db8417-kserve-fb7fdb549-6xvgx" Oct 30 17:24:13.396330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:13.395827 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f922e81e-fa9c-46ad-af32-5e0266311e85-model-cache\") pod \"sub-test-model-db8417-kserve-fb7fdb549-6xvgx\" (UID: \"f922e81e-fa9c-46ad-af32-5e0266311e85\") " pod="ai-tenant-e2e-shared-b-w3-db8417/sub-test-model-db8417-kserve-fb7fdb549-6xvgx" Oct 30 17:24:13.396330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:13.395932 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f922e81e-fa9c-46ad-af32-5e0266311e85-tls-certs\") pod \"sub-test-model-db8417-kserve-fb7fdb549-6xvgx\" (UID: \"f922e81e-fa9c-46ad-af32-5e0266311e85\") " pod="ai-tenant-e2e-shared-b-w3-db8417/sub-test-model-db8417-kserve-fb7fdb549-6xvgx" Oct 30 17:24:13.396330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:13.395961 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl227\" (UniqueName: \"kubernetes.io/projected/f922e81e-fa9c-46ad-af32-5e0266311e85-kube-api-access-zl227\") pod \"sub-test-model-db8417-kserve-fb7fdb549-6xvgx\" (UID: \"f922e81e-fa9c-46ad-af32-5e0266311e85\") " pod="ai-tenant-e2e-shared-b-w3-db8417/sub-test-model-db8417-kserve-fb7fdb549-6xvgx" Oct 30 17:24:13.396330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:13.396056 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f922e81e-fa9c-46ad-af32-5e0266311e85-dshm\") pod \"sub-test-model-db8417-kserve-fb7fdb549-6xvgx\" (UID: \"f922e81e-fa9c-46ad-af32-5e0266311e85\") " pod="ai-tenant-e2e-shared-b-w3-db8417/sub-test-model-db8417-kserve-fb7fdb549-6xvgx" Oct 30 17:24:13.397948 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:13.397918 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f922e81e-fa9c-46ad-af32-5e0266311e85-model-cache\") pod \"sub-test-model-db8417-kserve-fb7fdb549-6xvgx\" (UID: \"f922e81e-fa9c-46ad-af32-5e0266311e85\") " pod="ai-tenant-e2e-shared-b-w3-db8417/sub-test-model-db8417-kserve-fb7fdb549-6xvgx" Oct 30 17:24:13.401548 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:13.398813 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f922e81e-fa9c-46ad-af32-5e0266311e85-home\") pod \"sub-test-model-db8417-kserve-fb7fdb549-6xvgx\" (UID: \"f922e81e-fa9c-46ad-af32-5e0266311e85\") " pod="ai-tenant-e2e-shared-b-w3-db8417/sub-test-model-db8417-kserve-fb7fdb549-6xvgx" Oct 30 17:24:13.401724 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:13.399531 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f922e81e-fa9c-46ad-af32-5e0266311e85-tmp-dir\") pod \"sub-test-model-db8417-kserve-fb7fdb549-6xvgx\" (UID: \"f922e81e-fa9c-46ad-af32-5e0266311e85\") " pod="ai-tenant-e2e-shared-b-w3-db8417/sub-test-model-db8417-kserve-fb7fdb549-6xvgx" Oct 30 17:24:13.528448 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:13.516253 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f922e81e-fa9c-46ad-af32-5e0266311e85-tls-certs\") pod \"sub-test-model-db8417-kserve-fb7fdb549-6xvgx\" (UID: \"f922e81e-fa9c-46ad-af32-5e0266311e85\") " pod="ai-tenant-e2e-shared-b-w3-db8417/sub-test-model-db8417-kserve-fb7fdb549-6xvgx" Oct 30 17:24:13.528448 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:13.518220 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f922e81e-fa9c-46ad-af32-5e0266311e85-dshm\") pod \"sub-test-model-db8417-kserve-fb7fdb549-6xvgx\" (UID: \"f922e81e-fa9c-46ad-af32-5e0266311e85\") " pod="ai-tenant-e2e-shared-b-w3-db8417/sub-test-model-db8417-kserve-fb7fdb549-6xvgx" Oct 30 17:24:13.565355 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:13.564021 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl227\" (UniqueName: \"kubernetes.io/projected/f922e81e-fa9c-46ad-af32-5e0266311e85-kube-api-access-zl227\") pod \"sub-test-model-db8417-kserve-fb7fdb549-6xvgx\" (UID: \"f922e81e-fa9c-46ad-af32-5e0266311e85\") " pod="ai-tenant-e2e-shared-b-w3-db8417/sub-test-model-db8417-kserve-fb7fdb549-6xvgx" Oct 30 17:24:13.790226 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:13.790149 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-db8417/sub-test-model-db8417-kserve-fb7fdb549-6xvgx" Oct 30 17:24:14.051392 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:14.051037 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-6d4f99/sub-test-model-6d4f99-kserve-9ddf765b5-r59ft" Oct 30 17:24:14.063938 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:14.063910 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-6d4f99/sub-test-model-6d4f99-kserve-9ddf765b5-r59ft" Oct 30 17:24:14.158286 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:14.158250 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8226519e-a60e-4b3c-b4b3-83479839c8bb" path="/var/lib/kubelet/pods/8226519e-a60e-4b3c-b4b3-83479839c8bb/volumes" Oct 30 17:24:14.168379 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:14.168355 4522 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body= Oct 30 17:24:14.168788 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:14.168767 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc" podUID="4d44befd-705e-4589-b6d0-c944810a76bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" Oct 30 17:24:14.241333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:14.240504 4522 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body= Oct 30 17:24:14.241333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:14.240586 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595" podUID="1a5d5676-2fbf-4892-8b12-862c483b2d17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" Oct 30 17:24:14.248639 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:14.248594 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-db8417/sub-test-model-db8417-kserve-fb7fdb549-6xvgx"] Oct 30 17:24:14.251030 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:24:14.251001 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf922e81e_fa9c_46ad_af32_5e0266311e85.slice/crio-1a8889ae68bfcfedd3890085018cef5be2dcc6246f59e8f6bd5397c37b8382c3 WatchSource:0}: Error finding container 1a8889ae68bfcfedd3890085018cef5be2dcc6246f59e8f6bd5397c37b8382c3: Status 404 returned error can't find the container with id 1a8889ae68bfcfedd3890085018cef5be2dcc6246f59e8f6bd5397c37b8382c3 Oct 30 17:24:14.255257 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:14.255234 4522 generic.go:356] "Generic (PLEG): container finished" podID="68186e31-c661-41ea-bf07-6f97fb8d39b6" containerID="6e11a96e9bedc3c5a36e71649fcca22a349778bc515d1851c3a678bcc6e340c1" exitCode=0 Oct 30 17:24:14.255707 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:14.255687 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2d01ec-zmpbj" event={"ID":"68186e31-c661-41ea-bf07-6f97fb8d39b6","Type":"ContainerDied","Data":"6e11a96e9bedc3c5a36e71649fcca22a349778bc515d1851c3a678bcc6e340c1"} Oct 30 17:24:14.276632 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:14.273704 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-6d4f99/sub-test-model-6d4f99-kserve-9ddf765b5-r59ft" Oct 30 17:24:15.167957 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:15.167921 4522 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body= Oct 30 17:24:15.168164 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:15.167986 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc" podUID="4d44befd-705e-4589-b6d0-c944810a76bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" Oct 30 17:24:15.215341 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:15.214424 4522 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body= Oct 30 17:24:15.215341 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:15.214486 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595" podUID="1a5d5676-2fbf-4892-8b12-862c483b2d17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" Oct 30 17:24:15.270937 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:15.269446 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-db8417/sub-test-model-db8417-kserve-fb7fdb549-6xvgx" event={"ID":"f922e81e-fa9c-46ad-af32-5e0266311e85","Type":"ContainerStarted","Data":"8428cadbae9a3eea3d9c784cda8d128aa1daa7cacb5447b3ec7719581847cfcd"} Oct 30 17:24:15.270937 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:15.269496 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-db8417/sub-test-model-db8417-kserve-fb7fdb549-6xvgx" event={"ID":"f922e81e-fa9c-46ad-af32-5e0266311e85","Type":"ContainerStarted","Data":"1a8889ae68bfcfedd3890085018cef5be2dcc6246f59e8f6bd5397c37b8382c3"} Oct 30 17:24:15.342896 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:15.342828 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-db8417/sub-test-model-db8417-kserve-fb7fdb549-6xvgx" podStartSLOduration=2.3428081499999998 podStartE2EDuration="2.34280815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:24:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:24:15.31579466 +0000 UTC m=+2229.316549562" watchObservedRunningTime="2026-10-30 17:24:15.34280815 +0000 UTC m=+2229.343563052" Oct 30 17:24:15.658546 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:15.658512 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2d01ec-zmpbj" Oct 30 17:24:15.689874 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:15.689843 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68186e31-c661-41ea-bf07-6f97fb8d39b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68186e31-c661-41ea-bf07-6f97fb8d39b6-maas-api-openshift-ca\") pod \"68186e31-c661-41ea-bf07-6f97fb8d39b6\" (UID: \"68186e31-c661-41ea-bf07-6f97fb8d39b6\") " Oct 30 17:24:15.690051 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:15.690016 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68186e31-c661-41ea-bf07-6f97fb8d39b6-kube-api-access-sksb6\" (UniqueName: \"kubernetes.io/projected/68186e31-c661-41ea-bf07-6f97fb8d39b6-kube-api-access-sksb6\") pod \"68186e31-c661-41ea-bf07-6f97fb8d39b6\" (UID: \"68186e31-c661-41ea-bf07-6f97fb8d39b6\") " Oct 30 17:24:15.690186 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:15.690167 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68186e31-c661-41ea-bf07-6f97fb8d39b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68186e31-c661-41ea-bf07-6f97fb8d39b6-maas-api-xks-ca\") pod \"68186e31-c661-41ea-bf07-6f97fb8d39b6\" (UID: \"68186e31-c661-41ea-bf07-6f97fb8d39b6\") " Oct 30 17:24:15.690248 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:15.690231 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68186e31-c661-41ea-bf07-6f97fb8d39b6-maas-api-openshift-ca" pod "68186e31-c661-41ea-bf07-6f97fb8d39b6" (UID: "68186e31-c661-41ea-bf07-6f97fb8d39b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:24:15.690942 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:15.690921 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68186e31-c661-41ea-bf07-6f97fb8d39b6-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:24:15.747998 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:15.747020 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68186e31-c661-41ea-bf07-6f97fb8d39b6-maas-api-xks-ca" pod "68186e31-c661-41ea-bf07-6f97fb8d39b6" (UID: "68186e31-c661-41ea-bf07-6f97fb8d39b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:24:15.748729 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:15.748697 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68186e31-c661-41ea-bf07-6f97fb8d39b6-kube-api-access-sksb6" pod "68186e31-c661-41ea-bf07-6f97fb8d39b6" (UID: "68186e31-c661-41ea-bf07-6f97fb8d39b6"). InnerVolumeSpecName "kube-api-access-sksb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:24:15.791350 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:15.791262 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-db8417/sub-test-model-db8417-kserve-fb7fdb549-6xvgx" Oct 30 17:24:15.846612 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:15.843484 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sksb6\" (UniqueName: \"kubernetes.io/projected/68186e31-c661-41ea-bf07-6f97fb8d39b6-kube-api-access-sksb6\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:24:15.846612 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:15.843518 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68186e31-c661-41ea-bf07-6f97fb8d39b6-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:24:16.167736 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:16.167384 4522 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body= Oct 30 17:24:16.167736 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:16.167432 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc" podUID="4d44befd-705e-4589-b6d0-c944810a76bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" Oct 30 17:24:16.216333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:16.212168 4522 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body= Oct 30 17:24:16.216333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:16.212230 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595" podUID="1a5d5676-2fbf-4892-8b12-862c483b2d17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" Oct 30 17:24:16.286890 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:16.286862 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2d01ec-zmpbj" Oct 30 17:24:16.286890 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:16.286884 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2d01ec-zmpbj" event={"ID":"68186e31-c661-41ea-bf07-6f97fb8d39b6","Type":"ContainerDied","Data":"9b64575b313e0cb68a7fe122a06321060ee61e61e858552bf2e12594fbd7a099"} Oct 30 17:24:16.287137 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:16.286942 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b64575b313e0cb68a7fe122a06321060ee61e61e858552bf2e12594fbd7a099" Oct 30 17:24:17.167796 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:17.167757 4522 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body= Oct 30 17:24:17.167975 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:17.167843 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc" podUID="4d44befd-705e-4589-b6d0-c944810a76bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" Oct 30 17:24:17.200751 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:17.200713 4522 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body= Oct 30 17:24:17.201018 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:17.200991 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595" podUID="1a5d5676-2fbf-4892-8b12-862c483b2d17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" Oct 30 17:24:17.947501 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:17.947469 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w5-82243b/rate-test-model-82243b-kserve-75d85889df-fd25g" Oct 30 17:24:17.970771 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:17.970729 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w5-82243b/rate-test-model-82243b-kserve-75d85889df-fd25g" Oct 30 17:24:18.167602 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:18.167563 4522 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body= Oct 30 17:24:18.167805 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:18.167626 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc" podUID="4d44befd-705e-4589-b6d0-c944810a76bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" Oct 30 17:24:18.197025 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:18.196991 4522 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body= Oct 30 17:24:18.197177 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:18.197043 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595" podUID="1a5d5676-2fbf-4892-8b12-862c483b2d17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" Oct 30 17:24:18.387098 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:18.387061 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w5-82243b/rate-test-model-82243b-kserve-75d85889df-fd25g" Oct 30 17:24:19.172720 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:19.172689 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc" Oct 30 17:24:19.173138 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:19.172971 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc" Oct 30 17:24:19.173812 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:19.173790 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc" Oct 30 17:24:19.201008 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:19.200979 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595" Oct 30 17:24:19.201716 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:19.201694 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595" Oct 30 17:24:19.202571 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:19.202552 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595" Oct 30 17:24:20.211953 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:20.211916 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd5c8f-ccz6k"] Oct 30 17:24:20.224883 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:20.224840 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd5c8f-ccz6k"] Oct 30 17:24:21.457803 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:21.457760 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2d01ec-785fcdd98-cz2mk"] Oct 30 17:24:22.147452 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:22.147423 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6fd669d-69c4-4552-b050-f07bc2664981" path="/var/lib/kubelet/pods/b6fd669d-69c4-4552-b050-f07bc2664981/volumes" Oct 30 17:24:23.790979 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:23.790945 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-db8417/sub-test-model-db8417-kserve-fb7fdb549-6xvgx" Oct 30 17:24:23.804177 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:23.804151 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-db8417/sub-test-model-db8417-kserve-fb7fdb549-6xvgx" Oct 30 17:24:24.444580 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:24.444551 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-db8417/sub-test-model-db8417-kserve-fb7fdb549-6xvgx" Oct 30 17:24:25.822103 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:25.822066 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6"] Oct 30 17:24:25.826161 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:25.826135 4522 cpu_manager.go:414] "containerMap: removing container" podUID="68186e31-c661-41ea-bf07-6f97fb8d39b6" containerName="revoke-keys" Oct 30 17:24:25.826738 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:25.826716 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68186e31-c661-41ea-bf07-6f97fb8d39b6" containerName="revoke-keys" Oct 30 17:24:25.827079 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:25.827058 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="68186e31-c661-41ea-bf07-6f97fb8d39b6" containerName="revoke-keys" Oct 30 17:24:25.879837 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:25.879809 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6"] Oct 30 17:24:25.880079 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:25.880064 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8656-755369"] Oct 30 17:24:25.885796 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:25.884475 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6" Oct 30 17:24:25.888368 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:25.887080 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8656-755369" Oct 30 17:24:25.888368 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:25.887431 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8656-755369"] Oct 30 17:24:25.903604 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:25.903581 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-wddsm\"" Oct 30 17:24:25.987138 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:25.987096 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmnk2\" (UniqueName: \"kubernetes.io/projected/d534a90d-d839-4d06-b709-83d44c667b86-kube-api-access-fmnk2\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6\" (UID: \"d534a90d-d839-4d06-b709-83d44c667b86\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6" Oct 30 17:24:25.987419 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:25.987394 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d534a90d-d839-4d06-b709-83d44c667b86-workload-certs\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6\" (UID: \"d534a90d-d839-4d06-b709-83d44c667b86\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6" Oct 30 17:24:25.987482 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:25.987464 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d534a90d-d839-4d06-b709-83d44c667b86-istio-podinfo\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6\" (UID: \"d534a90d-d839-4d06-b709-83d44c667b86\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6" Oct 30 17:24:25.987585 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:25.987565 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d534a90d-d839-4d06-b709-83d44c667b86-istio-token\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6\" (UID: \"d534a90d-d839-4d06-b709-83d44c667b86\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6" Oct 30 17:24:25.987651 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:25.987593 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d534a90d-d839-4d06-b709-83d44c667b86-istiod-ca-cert\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6\" (UID: \"d534a90d-d839-4d06-b709-83d44c667b86\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6" Oct 30 17:24:25.987692 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:25.987669 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d534a90d-d839-4d06-b709-83d44c667b86-istio-data\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6\" (UID: \"d534a90d-d839-4d06-b709-83d44c667b86\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6" Oct 30 17:24:25.987733 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:25.987711 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmn96\" (UniqueName: \"kubernetes.io/projected/f41acc1b-28dc-45e7-90bc-9c0b575c6467-kube-api-access-vmn96\") pod \"test-curl-8656-755369\" (UID: \"f41acc1b-28dc-45e7-90bc-9c0b575c6467\") " pod="opendatahub/test-curl-8656-755369" Oct 30 17:24:25.987769 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:25.987740 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d534a90d-d839-4d06-b709-83d44c667b86-workload-socket\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6\" (UID: \"d534a90d-d839-4d06-b709-83d44c667b86\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6" Oct 30 17:24:25.987892 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:25.987873 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d534a90d-d839-4d06-b709-83d44c667b86-istio-envoy\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6\" (UID: \"d534a90d-d839-4d06-b709-83d44c667b86\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6" Oct 30 17:24:25.987934 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:25.987919 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d534a90d-d839-4d06-b709-83d44c667b86-credential-socket\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6\" (UID: \"d534a90d-d839-4d06-b709-83d44c667b86\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6" Oct 30 17:24:26.089352 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:26.089227 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d534a90d-d839-4d06-b709-83d44c667b86-workload-certs\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6\" (UID: \"d534a90d-d839-4d06-b709-83d44c667b86\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6" Oct 30 17:24:26.089352 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:26.089281 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d534a90d-d839-4d06-b709-83d44c667b86-istio-podinfo\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6\" (UID: \"d534a90d-d839-4d06-b709-83d44c667b86\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6" Oct 30 17:24:26.089616 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:26.089369 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d534a90d-d839-4d06-b709-83d44c667b86-istio-token\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6\" (UID: \"d534a90d-d839-4d06-b709-83d44c667b86\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6" Oct 30 17:24:26.089616 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:26.089385 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d534a90d-d839-4d06-b709-83d44c667b86-istiod-ca-cert\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6\" (UID: \"d534a90d-d839-4d06-b709-83d44c667b86\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6" Oct 30 17:24:26.089616 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:26.089407 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d534a90d-d839-4d06-b709-83d44c667b86-istio-data\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6\" (UID: \"d534a90d-d839-4d06-b709-83d44c667b86\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6" Oct 30 17:24:26.089616 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:26.089433 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmn96\" (UniqueName: \"kubernetes.io/projected/f41acc1b-28dc-45e7-90bc-9c0b575c6467-kube-api-access-vmn96\") pod \"test-curl-8656-755369\" (UID: \"f41acc1b-28dc-45e7-90bc-9c0b575c6467\") " pod="opendatahub/test-curl-8656-755369" Oct 30 17:24:26.089616 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:26.089468 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d534a90d-d839-4d06-b709-83d44c667b86-workload-socket\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6\" (UID: \"d534a90d-d839-4d06-b709-83d44c667b86\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6" Oct 30 17:24:26.089616 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:26.089559 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d534a90d-d839-4d06-b709-83d44c667b86-istio-envoy\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6\" (UID: \"d534a90d-d839-4d06-b709-83d44c667b86\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6" Oct 30 17:24:26.089616 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:26.089594 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d534a90d-d839-4d06-b709-83d44c667b86-credential-socket\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6\" (UID: \"d534a90d-d839-4d06-b709-83d44c667b86\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6" Oct 30 17:24:26.089961 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:26.089648 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmnk2\" (UniqueName: \"kubernetes.io/projected/d534a90d-d839-4d06-b709-83d44c667b86-kube-api-access-fmnk2\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6\" (UID: \"d534a90d-d839-4d06-b709-83d44c667b86\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6" Oct 30 17:24:26.089961 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:26.089705 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d534a90d-d839-4d06-b709-83d44c667b86-workload-certs\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6\" (UID: \"d534a90d-d839-4d06-b709-83d44c667b86\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6" Oct 30 17:24:26.089961 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:26.089756 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d534a90d-d839-4d06-b709-83d44c667b86-istio-data\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6\" (UID: \"d534a90d-d839-4d06-b709-83d44c667b86\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6" Oct 30 17:24:26.090086 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:26.090072 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d534a90d-d839-4d06-b709-83d44c667b86-credential-socket\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6\" (UID: \"d534a90d-d839-4d06-b709-83d44c667b86\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6" Oct 30 17:24:26.090131 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:26.090116 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d534a90d-d839-4d06-b709-83d44c667b86-workload-socket\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6\" (UID: \"d534a90d-d839-4d06-b709-83d44c667b86\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6" Oct 30 17:24:26.090356 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:26.090292 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d534a90d-d839-4d06-b709-83d44c667b86-istiod-ca-cert\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6\" (UID: \"d534a90d-d839-4d06-b709-83d44c667b86\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6" Oct 30 17:24:26.091830 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:26.091805 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d534a90d-d839-4d06-b709-83d44c667b86-istio-envoy\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6\" (UID: \"d534a90d-d839-4d06-b709-83d44c667b86\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6" Oct 30 17:24:26.091962 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:26.091944 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d534a90d-d839-4d06-b709-83d44c667b86-istio-podinfo\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6\" (UID: \"d534a90d-d839-4d06-b709-83d44c667b86\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6" Oct 30 17:24:26.114906 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:26.114876 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmn96\" (UniqueName: \"kubernetes.io/projected/f41acc1b-28dc-45e7-90bc-9c0b575c6467-kube-api-access-vmn96\") pod \"test-curl-8656-755369\" (UID: \"f41acc1b-28dc-45e7-90bc-9c0b575c6467\") " pod="opendatahub/test-curl-8656-755369" Oct 30 17:24:26.114906 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:26.114891 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d534a90d-d839-4d06-b709-83d44c667b86-istio-token\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6\" (UID: \"d534a90d-d839-4d06-b709-83d44c667b86\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6" Oct 30 17:24:26.115162 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:26.115145 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmnk2\" (UniqueName: \"kubernetes.io/projected/d534a90d-d839-4d06-b709-83d44c667b86-kube-api-access-fmnk2\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6\" (UID: \"d534a90d-d839-4d06-b709-83d44c667b86\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6" Oct 30 17:24:26.207752 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:26.207718 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6" Oct 30 17:24:26.211409 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:26.211380 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8656-755369" Oct 30 17:24:26.459140 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:26.459104 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-2d01ec-785fcdd98-cz2mk" podUID="e7778b16-47c7-4892-a0e3-368175354cf4" containerName="payload-processing" containerID="cri-o://9ce7ea259599b11a545e1623e1ddbe10fcb1ff962a67b0d1c8135440bcc38287" gracePeriod=25 Oct 30 17:24:26.558541 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:24:26.558509 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf41acc1b_28dc_45e7_90bc_9c0b575c6467.slice/crio-5b9caf0fcb3ff70913d1810bd5ff4c30b0169702e12a3a09dadf58f704b211b9 WatchSource:0}: Error finding container 5b9caf0fcb3ff70913d1810bd5ff4c30b0169702e12a3a09dadf58f704b211b9: Status 404 returned error can't find the container with id 5b9caf0fcb3ff70913d1810bd5ff4c30b0169702e12a3a09dadf58f704b211b9 Oct 30 17:24:26.560677 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:26.560657 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8656-755369"] Oct 30 17:24:26.565833 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:26.565808 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6"] Oct 30 17:24:26.571875 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:24:26.571851 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd534a90d_d839_4d06_b709_83d44c667b86.slice/crio-e14af1cff3d036e4c86d37d3403deadcecf4d7f02aa64fd0b51f950ced60b6ee WatchSource:0}: Error finding container e14af1cff3d036e4c86d37d3403deadcecf4d7f02aa64fd0b51f950ced60b6ee: Status 404 returned error can't find the container with id e14af1cff3d036e4c86d37d3403deadcecf4d7f02aa64fd0b51f950ced60b6ee Oct 30 17:24:26.574642 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:26.574611 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:24:26.574709 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:26.574674 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:24:26.574709 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:26.574703 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:24:27.464908 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:27.464869 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8656-755369" event={"ID":"f41acc1b-28dc-45e7-90bc-9c0b575c6467","Type":"ContainerStarted","Data":"503cdff555ed57d1bed95c50695a6ee731dca5fbb12c65bbd49134754d7a8b98"} Oct 30 17:24:27.465450 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:27.464919 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8656-755369" event={"ID":"f41acc1b-28dc-45e7-90bc-9c0b575c6467","Type":"ContainerStarted","Data":"5b9caf0fcb3ff70913d1810bd5ff4c30b0169702e12a3a09dadf58f704b211b9"} Oct 30 17:24:27.466936 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:27.466907 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6" event={"ID":"d534a90d-d839-4d06-b709-83d44c667b86","Type":"ContainerStarted","Data":"2c27612d534dc3ed63ca17ee6308591836c5dfdbd5e3da315c91c0b7b043f34e"} Oct 30 17:24:27.467082 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:27.466963 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6" event={"ID":"d534a90d-d839-4d06-b709-83d44c667b86","Type":"ContainerStarted","Data":"e14af1cff3d036e4c86d37d3403deadcecf4d7f02aa64fd0b51f950ced60b6ee"} Oct 30 17:24:27.519114 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:27.519057 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8656-755369" podStartSLOduration=2.519038591 podStartE2EDuration="2.519038591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:24:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:24:27.509483015 +0000 UTC m=+2241.510237918" watchObservedRunningTime="2026-10-30 17:24:27.519038591 +0000 UTC m=+2241.519793492" Oct 30 17:24:27.560830 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:27.560776 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6" podStartSLOduration=2.560762349 podStartE2EDuration="2.560762349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:24:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:24:27.558131096 +0000 UTC m=+2241.558886020" watchObservedRunningTime="2026-10-30 17:24:27.560762349 +0000 UTC m=+2241.561517249" Oct 30 17:24:27.752555 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:27.752464 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9af8e32c-642v5"] Oct 30 17:24:27.757222 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:27.757204 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af8e32c-642v5" Oct 30 17:24:27.776839 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:27.776811 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:24:27.776976 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:27.776927 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:24:27.778043 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:27.778019 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9af8e32c-642v5"] Oct 30 17:24:27.853912 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:27.853873 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d25a3aba-e336-4d85-9b43-edb6aa07b38d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9af8e32c-642v5\" (UID: \"d25a3aba-e336-4d85-9b43-edb6aa07b38d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af8e32c-642v5" Oct 30 17:24:27.853912 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:27.853911 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d25a3aba-e336-4d85-9b43-edb6aa07b38d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9af8e32c-642v5\" (UID: \"d25a3aba-e336-4d85-9b43-edb6aa07b38d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af8e32c-642v5" Oct 30 17:24:27.854107 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:27.853933 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxnfr\" (UniqueName: \"kubernetes.io/projected/d25a3aba-e336-4d85-9b43-edb6aa07b38d-kube-api-access-mxnfr\") pod \"maas-api-revoke-sub-9af8e32c-642v5\" (UID: \"d25a3aba-e336-4d85-9b43-edb6aa07b38d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af8e32c-642v5" Oct 30 17:24:27.955399 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:27.955364 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d25a3aba-e336-4d85-9b43-edb6aa07b38d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9af8e32c-642v5\" (UID: \"d25a3aba-e336-4d85-9b43-edb6aa07b38d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af8e32c-642v5" Oct 30 17:24:27.955587 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:27.955407 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d25a3aba-e336-4d85-9b43-edb6aa07b38d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9af8e32c-642v5\" (UID: \"d25a3aba-e336-4d85-9b43-edb6aa07b38d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af8e32c-642v5" Oct 30 17:24:27.955587 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:27.955502 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxnfr\" (UniqueName: \"kubernetes.io/projected/d25a3aba-e336-4d85-9b43-edb6aa07b38d-kube-api-access-mxnfr\") pod \"maas-api-revoke-sub-9af8e32c-642v5\" (UID: \"d25a3aba-e336-4d85-9b43-edb6aa07b38d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af8e32c-642v5" Oct 30 17:24:27.956050 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:27.956030 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d25a3aba-e336-4d85-9b43-edb6aa07b38d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9af8e32c-642v5\" (UID: \"d25a3aba-e336-4d85-9b43-edb6aa07b38d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af8e32c-642v5" Oct 30 17:24:27.957751 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:27.957729 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d25a3aba-e336-4d85-9b43-edb6aa07b38d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9af8e32c-642v5\" (UID: \"d25a3aba-e336-4d85-9b43-edb6aa07b38d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af8e32c-642v5" Oct 30 17:24:27.980651 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:27.980625 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxnfr\" (UniqueName: \"kubernetes.io/projected/d25a3aba-e336-4d85-9b43-edb6aa07b38d-kube-api-access-mxnfr\") pod \"maas-api-revoke-sub-9af8e32c-642v5\" (UID: \"d25a3aba-e336-4d85-9b43-edb6aa07b38d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af8e32c-642v5" Oct 30 17:24:28.072882 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:28.072796 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af8e32c-642v5" Oct 30 17:24:28.241568 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:28.241535 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6" Oct 30 17:24:28.243067 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:28.243020 4522 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body= Oct 30 17:24:28.243230 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:28.243090 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6" podUID="d534a90d-d839-4d06-b709-83d44c667b86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" Oct 30 17:24:28.421912 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:24:28.421881 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd25a3aba_e336_4d85_9b43_edb6aa07b38d.slice/crio-9c660edeae6e83aab01acef6a48880f8dfac8be4b00184e9e3fd61a1b9c5fb69 WatchSource:0}: Error finding container 9c660edeae6e83aab01acef6a48880f8dfac8be4b00184e9e3fd61a1b9c5fb69: Status 404 returned error can't find the container with id 9c660edeae6e83aab01acef6a48880f8dfac8be4b00184e9e3fd61a1b9c5fb69 Oct 30 17:24:28.422044 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:28.421951 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9af8e32c-642v5"] Oct 30 17:24:28.503224 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:28.498665 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af8e32c-642v5" event={"ID":"d25a3aba-e336-4d85-9b43-edb6aa07b38d","Type":"ContainerStarted","Data":"9c660edeae6e83aab01acef6a48880f8dfac8be4b00184e9e3fd61a1b9c5fb69"} Oct 30 17:24:28.701382 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:28.691652 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8656-755369"] Oct 30 17:24:28.730970 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:28.730227 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8656-755369"] Oct 30 17:24:28.745065 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:28.744057 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl"] Oct 30 17:24:28.745065 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:28.744326 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl" podUID="0e42c53c-946a-40fc-b126-5cd8a39b532d" containerName="istio-proxy" containerID="cri-o://2e763d6c5d0687d027d9c909c38d2c9655886d3b45fb6eb611720bb3278c4423" gracePeriod=30 Oct 30 17:24:28.978343 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:28.976767 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fba7356e-m8gjf"] Oct 30 17:24:28.978343 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:28.978052 4522 cpu_manager.go:414] "containerMap: removing container" podUID="f41acc1b-28dc-45e7-90bc-9c0b575c6467" containerName="test-curl-8656-755369" Oct 30 17:24:28.978343 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:28.978078 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f41acc1b-28dc-45e7-90bc-9c0b575c6467" containerName="test-curl-8656-755369" Oct 30 17:24:28.978734 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:28.978367 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="f41acc1b-28dc-45e7-90bc-9c0b575c6467" containerName="test-curl-8656-755369" Oct 30 17:24:29.052596 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:29.051596 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba7356e-m8gjf" Oct 30 17:24:29.052596 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:29.051503 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fba7356e-m8gjf"] Oct 30 17:24:29.148814 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:29.148781 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxxx7\" (UniqueName: \"kubernetes.io/projected/22c82bab-c6f4-48ba-8bb9-06b49eed06be-kube-api-access-rxxx7\") pod \"maas-api-revoke-sub-fba7356e-m8gjf\" (UID: \"22c82bab-c6f4-48ba-8bb9-06b49eed06be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba7356e-m8gjf" Oct 30 17:24:29.149002 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:29.148891 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22c82bab-c6f4-48ba-8bb9-06b49eed06be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fba7356e-m8gjf\" (UID: \"22c82bab-c6f4-48ba-8bb9-06b49eed06be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba7356e-m8gjf" Oct 30 17:24:29.149091 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:29.149075 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22c82bab-c6f4-48ba-8bb9-06b49eed06be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fba7356e-m8gjf\" (UID: \"22c82bab-c6f4-48ba-8bb9-06b49eed06be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba7356e-m8gjf" Oct 30 17:24:29.219376 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:29.219273 4522 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body= Oct 30 17:24:29.219376 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:29.219356 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6" podUID="d534a90d-d839-4d06-b709-83d44c667b86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" Oct 30 17:24:29.250207 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:29.250173 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22c82bab-c6f4-48ba-8bb9-06b49eed06be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fba7356e-m8gjf\" (UID: \"22c82bab-c6f4-48ba-8bb9-06b49eed06be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba7356e-m8gjf" Oct 30 17:24:29.250395 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:29.250343 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxxx7\" (UniqueName: \"kubernetes.io/projected/22c82bab-c6f4-48ba-8bb9-06b49eed06be-kube-api-access-rxxx7\") pod \"maas-api-revoke-sub-fba7356e-m8gjf\" (UID: \"22c82bab-c6f4-48ba-8bb9-06b49eed06be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba7356e-m8gjf" Oct 30 17:24:29.250485 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:29.250428 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22c82bab-c6f4-48ba-8bb9-06b49eed06be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fba7356e-m8gjf\" (UID: \"22c82bab-c6f4-48ba-8bb9-06b49eed06be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba7356e-m8gjf" Oct 30 17:24:29.251226 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:29.251201 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22c82bab-c6f4-48ba-8bb9-06b49eed06be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fba7356e-m8gjf\" (UID: \"22c82bab-c6f4-48ba-8bb9-06b49eed06be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba7356e-m8gjf" Oct 30 17:24:29.253611 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:29.253586 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22c82bab-c6f4-48ba-8bb9-06b49eed06be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fba7356e-m8gjf\" (UID: \"22c82bab-c6f4-48ba-8bb9-06b49eed06be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba7356e-m8gjf" Oct 30 17:24:29.280960 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:29.280930 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxxx7\" (UniqueName: \"kubernetes.io/projected/22c82bab-c6f4-48ba-8bb9-06b49eed06be-kube-api-access-rxxx7\") pod \"maas-api-revoke-sub-fba7356e-m8gjf\" (UID: \"22c82bab-c6f4-48ba-8bb9-06b49eed06be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba7356e-m8gjf" Oct 30 17:24:29.370444 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:29.370408 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba7356e-m8gjf" Oct 30 17:24:29.532884 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:29.532846 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8656-755369" podUID="f41acc1b-28dc-45e7-90bc-9c0b575c6467" containerName="test-curl-8656-755369" containerID="cri-o://503cdff555ed57d1bed95c50695a6ee731dca5fbb12c65bbd49134754d7a8b98" gracePeriod=2 Oct 30 17:24:29.533415 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:29.533364 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af8e32c-642v5" event={"ID":"d25a3aba-e336-4d85-9b43-edb6aa07b38d","Type":"ContainerStarted","Data":"2fa6f955f412fa5b607f41e20027e3a922bdb06c152a16cc6abc20416e257648"} Oct 30 17:24:29.549608 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:24:29.548619 4522 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8656-755369\" is forbidden: User \"system:node:ip-10-0-128-218.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-218.ec2.internal' and this object" podUID="f41acc1b-28dc-45e7-90bc-9c0b575c6467" pod="opendatahub/test-curl-8656-755369" Oct 30 17:24:29.620545 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:29.620492 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af8e32c-642v5" podStartSLOduration=2.620477737 podStartE2EDuration="2.620477737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:24:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:24:29.617501968 +0000 UTC m=+2243.618256871" watchObservedRunningTime="2026-10-30 17:24:29.620477737 +0000 UTC m=+2243.621232632" Oct 30 17:24:29.632484 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:24:29.632444 4522 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8656-755369\" is forbidden: User \"system:node:ip-10-0-128-218.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-218.ec2.internal' and this object" podUID="f41acc1b-28dc-45e7-90bc-9c0b575c6467" pod="opendatahub/test-curl-8656-755369" Oct 30 17:24:29.720975 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:29.720944 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fba7356e-m8gjf"] Oct 30 17:24:29.750122 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:24:29.750046 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22c82bab_c6f4_48ba_8bb9_06b49eed06be.slice/crio-3a51347f3ef19a849b096551b6a1317d00519aa996cb6c156444fe754452c2c9 WatchSource:0}: Error finding container 3a51347f3ef19a849b096551b6a1317d00519aa996cb6c156444fe754452c2c9: Status 404 returned error can't find the container with id 3a51347f3ef19a849b096551b6a1317d00519aa996cb6c156444fe754452c2c9 Oct 30 17:24:30.212331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:30.211762 4522 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body= Oct 30 17:24:30.212331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:30.211826 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6" podUID="d534a90d-d839-4d06-b709-83d44c667b86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" Oct 30 17:24:30.542888 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:30.542845 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba7356e-m8gjf" event={"ID":"22c82bab-c6f4-48ba-8bb9-06b49eed06be","Type":"ContainerStarted","Data":"abe94397698f6cd8accd710d3e38f3e159517610dba84494000bfeeb5eb937e4"} Oct 30 17:24:30.543269 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:30.542912 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba7356e-m8gjf" event={"ID":"22c82bab-c6f4-48ba-8bb9-06b49eed06be","Type":"ContainerStarted","Data":"3a51347f3ef19a849b096551b6a1317d00519aa996cb6c156444fe754452c2c9"} Oct 30 17:24:30.545139 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:30.545114 4522 generic.go:356] "Generic (PLEG): container finished" podID="d25a3aba-e336-4d85-9b43-edb6aa07b38d" containerID="2fa6f955f412fa5b607f41e20027e3a922bdb06c152a16cc6abc20416e257648" exitCode=0 Oct 30 17:24:30.545266 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:30.545191 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af8e32c-642v5" event={"ID":"d25a3aba-e336-4d85-9b43-edb6aa07b38d","Type":"ContainerDied","Data":"2fa6f955f412fa5b607f41e20027e3a922bdb06c152a16cc6abc20416e257648"} Oct 30 17:24:30.591029 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:30.590978 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba7356e-m8gjf" podStartSLOduration=2.590965921 podStartE2EDuration="2.590965921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:24:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:24:30.588097206 +0000 UTC m=+2244.588852106" watchObservedRunningTime="2026-10-30 17:24:30.590965921 +0000 UTC m=+2244.591720821" Oct 30 17:24:31.216329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:31.211755 4522 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body= Oct 30 17:24:31.216329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:31.211825 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6" podUID="d534a90d-d839-4d06-b709-83d44c667b86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" Oct 30 17:24:31.563851 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:31.563823 4522 generic.go:356] "Generic (PLEG): container finished" podID="22c82bab-c6f4-48ba-8bb9-06b49eed06be" containerID="abe94397698f6cd8accd710d3e38f3e159517610dba84494000bfeeb5eb937e4" exitCode=0 Oct 30 17:24:31.564237 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:31.563905 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba7356e-m8gjf" event={"ID":"22c82bab-c6f4-48ba-8bb9-06b49eed06be","Type":"ContainerDied","Data":"abe94397698f6cd8accd710d3e38f3e159517610dba84494000bfeeb5eb937e4"} Oct 30 17:24:31.919574 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:31.919523 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af8e32c-642v5" Oct 30 17:24:31.957230 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:31.957166 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8656-755369" Oct 30 17:24:32.028915 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:32.028873 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d25a3aba-e336-4d85-9b43-edb6aa07b38d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d25a3aba-e336-4d85-9b43-edb6aa07b38d-maas-api-openshift-ca\") pod \"d25a3aba-e336-4d85-9b43-edb6aa07b38d\" (UID: \"d25a3aba-e336-4d85-9b43-edb6aa07b38d\") " Oct 30 17:24:32.029094 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:32.029009 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d25a3aba-e336-4d85-9b43-edb6aa07b38d-kube-api-access-mxnfr\" (UniqueName: \"kubernetes.io/projected/d25a3aba-e336-4d85-9b43-edb6aa07b38d-kube-api-access-mxnfr\") pod \"d25a3aba-e336-4d85-9b43-edb6aa07b38d\" (UID: \"d25a3aba-e336-4d85-9b43-edb6aa07b38d\") " Oct 30 17:24:32.029375 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:32.029248 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d25a3aba-e336-4d85-9b43-edb6aa07b38d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d25a3aba-e336-4d85-9b43-edb6aa07b38d-maas-api-xks-ca\") pod \"d25a3aba-e336-4d85-9b43-edb6aa07b38d\" (UID: \"d25a3aba-e336-4d85-9b43-edb6aa07b38d\") " Oct 30 17:24:32.029761 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:32.029721 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d25a3aba-e336-4d85-9b43-edb6aa07b38d-maas-api-openshift-ca" pod "d25a3aba-e336-4d85-9b43-edb6aa07b38d" (UID: "d25a3aba-e336-4d85-9b43-edb6aa07b38d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:24:32.030092 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:32.030065 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d25a3aba-e336-4d85-9b43-edb6aa07b38d-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:24:32.031824 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:32.031791 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d25a3aba-e336-4d85-9b43-edb6aa07b38d-maas-api-xks-ca" pod "d25a3aba-e336-4d85-9b43-edb6aa07b38d" (UID: "d25a3aba-e336-4d85-9b43-edb6aa07b38d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:24:32.032140 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:32.032126 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d25a3aba-e336-4d85-9b43-edb6aa07b38d-kube-api-access-mxnfr" pod "d25a3aba-e336-4d85-9b43-edb6aa07b38d" (UID: "d25a3aba-e336-4d85-9b43-edb6aa07b38d"). InnerVolumeSpecName "kube-api-access-mxnfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:24:32.131317 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:32.131225 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f41acc1b-28dc-45e7-90bc-9c0b575c6467-kube-api-access-vmn96\" (UniqueName: \"kubernetes.io/projected/f41acc1b-28dc-45e7-90bc-9c0b575c6467-kube-api-access-vmn96\") pod \"f41acc1b-28dc-45e7-90bc-9c0b575c6467\" (UID: \"f41acc1b-28dc-45e7-90bc-9c0b575c6467\") " Oct 30 17:24:32.131981 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:32.131955 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxnfr\" (UniqueName: \"kubernetes.io/projected/d25a3aba-e336-4d85-9b43-edb6aa07b38d-kube-api-access-mxnfr\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:24:32.131981 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:32.131979 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d25a3aba-e336-4d85-9b43-edb6aa07b38d-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:24:32.133709 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:32.133681 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f41acc1b-28dc-45e7-90bc-9c0b575c6467-kube-api-access-vmn96" pod "f41acc1b-28dc-45e7-90bc-9c0b575c6467" (UID: "f41acc1b-28dc-45e7-90bc-9c0b575c6467"). InnerVolumeSpecName "kube-api-access-vmn96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:24:32.148107 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:32.148073 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f41acc1b-28dc-45e7-90bc-9c0b575c6467" path="/var/lib/kubelet/pods/f41acc1b-28dc-45e7-90bc-9c0b575c6467/volumes" Oct 30 17:24:32.216365 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:32.214746 4522 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body= Oct 30 17:24:32.216365 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:32.214819 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6" podUID="d534a90d-d839-4d06-b709-83d44c667b86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" Oct 30 17:24:32.233111 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:32.233081 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmn96\" (UniqueName: \"kubernetes.io/projected/f41acc1b-28dc-45e7-90bc-9c0b575c6467-kube-api-access-vmn96\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:24:32.573909 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:32.573877 4522 generic.go:356] "Generic (PLEG): container finished" podID="f41acc1b-28dc-45e7-90bc-9c0b575c6467" containerID="503cdff555ed57d1bed95c50695a6ee731dca5fbb12c65bbd49134754d7a8b98" exitCode=137 Oct 30 17:24:32.574339 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:32.573919 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8656-755369" Oct 30 17:24:32.574339 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:32.573978 4522 scope.go:122] "RemoveContainer" containerID="503cdff555ed57d1bed95c50695a6ee731dca5fbb12c65bbd49134754d7a8b98" Oct 30 17:24:32.576069 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:32.575677 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af8e32c-642v5" event={"ID":"d25a3aba-e336-4d85-9b43-edb6aa07b38d","Type":"ContainerDied","Data":"9c660edeae6e83aab01acef6a48880f8dfac8be4b00184e9e3fd61a1b9c5fb69"} Oct 30 17:24:32.576294 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:32.576278 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c660edeae6e83aab01acef6a48880f8dfac8be4b00184e9e3fd61a1b9c5fb69" Oct 30 17:24:32.576447 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:32.575638 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af8e32c-642v5" Oct 30 17:24:32.586668 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:32.586650 4522 scope.go:122] "RemoveContainer" containerID="503cdff555ed57d1bed95c50695a6ee731dca5fbb12c65bbd49134754d7a8b98" Oct 30 17:24:32.586940 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:24:32.586923 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"503cdff555ed57d1bed95c50695a6ee731dca5fbb12c65bbd49134754d7a8b98\": container with ID starting with 503cdff555ed57d1bed95c50695a6ee731dca5fbb12c65bbd49134754d7a8b98 not found: ID does not exist" containerID="503cdff555ed57d1bed95c50695a6ee731dca5fbb12c65bbd49134754d7a8b98" Oct 30 17:24:32.586990 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:32.586948 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"503cdff555ed57d1bed95c50695a6ee731dca5fbb12c65bbd49134754d7a8b98"} err="failed to get container status \"503cdff555ed57d1bed95c50695a6ee731dca5fbb12c65bbd49134754d7a8b98\": rpc error: code = NotFound desc = could not find container \"503cdff555ed57d1bed95c50695a6ee731dca5fbb12c65bbd49134754d7a8b98\": container with ID starting with 503cdff555ed57d1bed95c50695a6ee731dca5fbb12c65bbd49134754d7a8b98 not found: ID does not exist" Oct 30 17:24:32.766185 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:32.766159 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba7356e-m8gjf" Oct 30 17:24:32.944979 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:32.944878 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22c82bab-c6f4-48ba-8bb9-06b49eed06be-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22c82bab-c6f4-48ba-8bb9-06b49eed06be-maas-api-openshift-ca\") pod \"22c82bab-c6f4-48ba-8bb9-06b49eed06be\" (UID: \"22c82bab-c6f4-48ba-8bb9-06b49eed06be\") " Oct 30 17:24:32.945162 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:32.945092 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22c82bab-c6f4-48ba-8bb9-06b49eed06be-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22c82bab-c6f4-48ba-8bb9-06b49eed06be-maas-api-xks-ca\") pod \"22c82bab-c6f4-48ba-8bb9-06b49eed06be\" (UID: \"22c82bab-c6f4-48ba-8bb9-06b49eed06be\") " Oct 30 17:24:32.945162 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:32.945149 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22c82bab-c6f4-48ba-8bb9-06b49eed06be-kube-api-access-rxxx7\" (UniqueName: \"kubernetes.io/projected/22c82bab-c6f4-48ba-8bb9-06b49eed06be-kube-api-access-rxxx7\") pod \"22c82bab-c6f4-48ba-8bb9-06b49eed06be\" (UID: \"22c82bab-c6f4-48ba-8bb9-06b49eed06be\") " Oct 30 17:24:32.945368 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:32.945344 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22c82bab-c6f4-48ba-8bb9-06b49eed06be-maas-api-openshift-ca" pod "22c82bab-c6f4-48ba-8bb9-06b49eed06be" (UID: "22c82bab-c6f4-48ba-8bb9-06b49eed06be"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:24:32.946055 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:32.946029 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22c82bab-c6f4-48ba-8bb9-06b49eed06be-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:24:32.947299 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:32.947273 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22c82bab-c6f4-48ba-8bb9-06b49eed06be-maas-api-xks-ca" pod "22c82bab-c6f4-48ba-8bb9-06b49eed06be" (UID: "22c82bab-c6f4-48ba-8bb9-06b49eed06be"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:24:32.947421 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:32.947368 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22c82bab-c6f4-48ba-8bb9-06b49eed06be-kube-api-access-rxxx7" pod "22c82bab-c6f4-48ba-8bb9-06b49eed06be" (UID: "22c82bab-c6f4-48ba-8bb9-06b49eed06be"). InnerVolumeSpecName "kube-api-access-rxxx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:24:33.048113 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:33.048083 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22c82bab-c6f4-48ba-8bb9-06b49eed06be-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:24:33.048113 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:33.048113 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxxx7\" (UniqueName: \"kubernetes.io/projected/22c82bab-c6f4-48ba-8bb9-06b49eed06be-kube-api-access-rxxx7\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:24:33.313845 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:33.313807 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6" Oct 30 17:24:33.314354 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:33.314328 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6" Oct 30 17:24:33.316171 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:33.316150 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6" Oct 30 17:24:33.597111 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:33.597028 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba7356e-m8gjf" Oct 30 17:24:33.598261 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:33.598229 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba7356e-m8gjf" event={"ID":"22c82bab-c6f4-48ba-8bb9-06b49eed06be","Type":"ContainerDied","Data":"3a51347f3ef19a849b096551b6a1317d00519aa996cb6c156444fe754452c2c9"} Oct 30 17:24:33.598382 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:33.598288 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a51347f3ef19a849b096551b6a1317d00519aa996cb6c156444fe754452c2c9" Oct 30 17:24:34.060679 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:34.060651 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl" Oct 30 17:24:34.075088 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:34.075062 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e42c53c-946a-40fc-b126-5cd8a39b532d-kube-api-access-lvfrv\" (UniqueName: \"kubernetes.io/projected/0e42c53c-946a-40fc-b126-5cd8a39b532d-kube-api-access-lvfrv\") pod \"0e42c53c-946a-40fc-b126-5cd8a39b532d\" (UID: \"0e42c53c-946a-40fc-b126-5cd8a39b532d\") " Oct 30 17:24:34.075221 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:34.075157 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e42c53c-946a-40fc-b126-5cd8a39b532d-istio-token\" (UniqueName: \"kubernetes.io/projected/0e42c53c-946a-40fc-b126-5cd8a39b532d-istio-token\") pod \"0e42c53c-946a-40fc-b126-5cd8a39b532d\" (UID: \"0e42c53c-946a-40fc-b126-5cd8a39b532d\") " Oct 30 17:24:34.075221 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:34.075207 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e42c53c-946a-40fc-b126-5cd8a39b532d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e42c53c-946a-40fc-b126-5cd8a39b532d-istio-envoy\") pod \"0e42c53c-946a-40fc-b126-5cd8a39b532d\" (UID: \"0e42c53c-946a-40fc-b126-5cd8a39b532d\") " Oct 30 17:24:34.075350 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:34.075266 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e42c53c-946a-40fc-b126-5cd8a39b532d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e42c53c-946a-40fc-b126-5cd8a39b532d-istiod-ca-cert\") pod \"0e42c53c-946a-40fc-b126-5cd8a39b532d\" (UID: \"0e42c53c-946a-40fc-b126-5cd8a39b532d\") " Oct 30 17:24:34.075350 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:34.075299 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e42c53c-946a-40fc-b126-5cd8a39b532d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e42c53c-946a-40fc-b126-5cd8a39b532d-istio-podinfo\") pod \"0e42c53c-946a-40fc-b126-5cd8a39b532d\" (UID: \"0e42c53c-946a-40fc-b126-5cd8a39b532d\") " Oct 30 17:24:34.075484 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:34.075397 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e42c53c-946a-40fc-b126-5cd8a39b532d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e42c53c-946a-40fc-b126-5cd8a39b532d-workload-socket\") pod \"0e42c53c-946a-40fc-b126-5cd8a39b532d\" (UID: \"0e42c53c-946a-40fc-b126-5cd8a39b532d\") " Oct 30 17:24:34.075484 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:34.075451 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e42c53c-946a-40fc-b126-5cd8a39b532d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e42c53c-946a-40fc-b126-5cd8a39b532d-workload-certs\") pod \"0e42c53c-946a-40fc-b126-5cd8a39b532d\" (UID: \"0e42c53c-946a-40fc-b126-5cd8a39b532d\") " Oct 30 17:24:34.075586 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:34.075494 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e42c53c-946a-40fc-b126-5cd8a39b532d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e42c53c-946a-40fc-b126-5cd8a39b532d-istio-data\") pod \"0e42c53c-946a-40fc-b126-5cd8a39b532d\" (UID: \"0e42c53c-946a-40fc-b126-5cd8a39b532d\") " Oct 30 17:24:34.075707 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:34.075691 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e42c53c-946a-40fc-b126-5cd8a39b532d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e42c53c-946a-40fc-b126-5cd8a39b532d-credential-socket\") pod \"0e42c53c-946a-40fc-b126-5cd8a39b532d\" (UID: \"0e42c53c-946a-40fc-b126-5cd8a39b532d\") " Oct 30 17:24:34.075789 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:34.075733 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e42c53c-946a-40fc-b126-5cd8a39b532d-workload-certs" pod "0e42c53c-946a-40fc-b126-5cd8a39b532d" (UID: "0e42c53c-946a-40fc-b126-5cd8a39b532d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:24:34.075789 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:34.075668 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e42c53c-946a-40fc-b126-5cd8a39b532d-istiod-ca-cert" pod "0e42c53c-946a-40fc-b126-5cd8a39b532d" (UID: "0e42c53c-946a-40fc-b126-5cd8a39b532d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:24:34.075991 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:34.075969 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e42c53c-946a-40fc-b126-5cd8a39b532d-credential-socket" pod "0e42c53c-946a-40fc-b126-5cd8a39b532d" (UID: "0e42c53c-946a-40fc-b126-5cd8a39b532d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:24:34.076087 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:34.075997 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e42c53c-946a-40fc-b126-5cd8a39b532d-istio-data" pod "0e42c53c-946a-40fc-b126-5cd8a39b532d" (UID: "0e42c53c-946a-40fc-b126-5cd8a39b532d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:24:34.076246 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:34.076225 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e42c53c-946a-40fc-b126-5cd8a39b532d-workload-socket" pod "0e42c53c-946a-40fc-b126-5cd8a39b532d" (UID: "0e42c53c-946a-40fc-b126-5cd8a39b532d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:24:34.076799 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:34.076776 4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e42c53c-946a-40fc-b126-5cd8a39b532d-credential-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:24:34.076884 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:34.076806 4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e42c53c-946a-40fc-b126-5cd8a39b532d-istiod-ca-cert\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:24:34.076884 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:34.076822 4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e42c53c-946a-40fc-b126-5cd8a39b532d-workload-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:24:34.076884 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:34.076836 4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e42c53c-946a-40fc-b126-5cd8a39b532d-workload-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:24:34.076884 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:34.076850 4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e42c53c-946a-40fc-b126-5cd8a39b532d-istio-data\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:24:34.077688 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:34.077656 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e42c53c-946a-40fc-b126-5cd8a39b532d-istio-podinfo" pod "0e42c53c-946a-40fc-b126-5cd8a39b532d" (UID: "0e42c53c-946a-40fc-b126-5cd8a39b532d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:24:34.077812 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:34.077728 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e42c53c-946a-40fc-b126-5cd8a39b532d-istio-envoy" pod "0e42c53c-946a-40fc-b126-5cd8a39b532d" (UID: "0e42c53c-946a-40fc-b126-5cd8a39b532d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:24:34.077886 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:34.077870 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e42c53c-946a-40fc-b126-5cd8a39b532d-istio-token" pod "0e42c53c-946a-40fc-b126-5cd8a39b532d" (UID: "0e42c53c-946a-40fc-b126-5cd8a39b532d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:24:34.077944 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:34.077894 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e42c53c-946a-40fc-b126-5cd8a39b532d-kube-api-access-lvfrv" pod "0e42c53c-946a-40fc-b126-5cd8a39b532d" (UID: "0e42c53c-946a-40fc-b126-5cd8a39b532d"). InnerVolumeSpecName "kube-api-access-lvfrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:24:34.178715 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:34.178612 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvfrv\" (UniqueName: \"kubernetes.io/projected/0e42c53c-946a-40fc-b126-5cd8a39b532d-kube-api-access-lvfrv\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:24:34.178715 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:34.178635 4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e42c53c-946a-40fc-b126-5cd8a39b532d-istio-token\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:24:34.178715 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:34.178644 4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e42c53c-946a-40fc-b126-5cd8a39b532d-istio-envoy\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:24:34.178715 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:34.178652 4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e42c53c-946a-40fc-b126-5cd8a39b532d-istio-podinfo\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:24:34.269734 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:34.269691 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2d01ec-zmpbj"] Oct 30 17:24:34.304851 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:34.304814 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2d01ec-zmpbj"] Oct 30 17:24:34.605925 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:34.605889 4522 generic.go:356] "Generic (PLEG): container finished" podID="0e42c53c-946a-40fc-b126-5cd8a39b532d" containerID="2e763d6c5d0687d027d9c909c38d2c9655886d3b45fb6eb611720bb3278c4423" exitCode=0 Oct 30 17:24:34.606354 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:34.606020 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl" event={"ID":"0e42c53c-946a-40fc-b126-5cd8a39b532d","Type":"ContainerDied","Data":"2e763d6c5d0687d027d9c909c38d2c9655886d3b45fb6eb611720bb3278c4423"} Oct 30 17:24:34.606354 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:34.606101 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl" event={"ID":"0e42c53c-946a-40fc-b126-5cd8a39b532d","Type":"ContainerDied","Data":"414c6c9d2272248abe38aeca19782937499358eecdd920ef1e7532aa59a4e25c"} Oct 30 17:24:34.606354 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:34.606028 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl" Oct 30 17:24:34.606354 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:34.606124 4522 scope.go:122] "RemoveContainer" containerID="2e763d6c5d0687d027d9c909c38d2c9655886d3b45fb6eb611720bb3278c4423" Oct 30 17:24:34.617491 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:34.617472 4522 scope.go:122] "RemoveContainer" containerID="e34ffbc603e71d8b1737bd3ffa4ea74e7266c5f8a4ae4c6d447947ba2255893b" Oct 30 17:24:34.627968 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:34.627952 4522 scope.go:122] "RemoveContainer" containerID="2e763d6c5d0687d027d9c909c38d2c9655886d3b45fb6eb611720bb3278c4423" Oct 30 17:24:34.628236 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:24:34.628219 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e763d6c5d0687d027d9c909c38d2c9655886d3b45fb6eb611720bb3278c4423\": container with ID starting with 2e763d6c5d0687d027d9c909c38d2c9655886d3b45fb6eb611720bb3278c4423 not found: ID does not exist" containerID="2e763d6c5d0687d027d9c909c38d2c9655886d3b45fb6eb611720bb3278c4423" Oct 30 17:24:34.628282 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:34.628246 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e763d6c5d0687d027d9c909c38d2c9655886d3b45fb6eb611720bb3278c4423"} err="failed to get container status \"2e763d6c5d0687d027d9c909c38d2c9655886d3b45fb6eb611720bb3278c4423\": rpc error: code = NotFound desc = could not find container \"2e763d6c5d0687d027d9c909c38d2c9655886d3b45fb6eb611720bb3278c4423\": container with ID starting with 2e763d6c5d0687d027d9c909c38d2c9655886d3b45fb6eb611720bb3278c4423 not found: ID does not exist" Oct 30 17:24:34.628282 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:34.628260 4522 scope.go:122] "RemoveContainer" containerID="e34ffbc603e71d8b1737bd3ffa4ea74e7266c5f8a4ae4c6d447947ba2255893b" Oct 30 17:24:34.628522 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:24:34.628504 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e34ffbc603e71d8b1737bd3ffa4ea74e7266c5f8a4ae4c6d447947ba2255893b\": container with ID starting with e34ffbc603e71d8b1737bd3ffa4ea74e7266c5f8a4ae4c6d447947ba2255893b not found: ID does not exist" containerID="e34ffbc603e71d8b1737bd3ffa4ea74e7266c5f8a4ae4c6d447947ba2255893b" Oct 30 17:24:34.628576 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:34.628528 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e34ffbc603e71d8b1737bd3ffa4ea74e7266c5f8a4ae4c6d447947ba2255893b"} err="failed to get container status \"e34ffbc603e71d8b1737bd3ffa4ea74e7266c5f8a4ae4c6d447947ba2255893b\": rpc error: code = NotFound desc = could not find container \"e34ffbc603e71d8b1737bd3ffa4ea74e7266c5f8a4ae4c6d447947ba2255893b\": container with ID starting with e34ffbc603e71d8b1737bd3ffa4ea74e7266c5f8a4ae4c6d447947ba2255893b not found: ID does not exist" Oct 30 17:24:34.666748 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:34.666718 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl"] Oct 30 17:24:34.679102 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:34.679068 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-2d01ec-785fcdd98-cz2mk" podUID="e7778b16-47c7-4892-a0e3-368175354cf4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:24:34.679765 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:34.679746 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-dd5c8f-openshift-default-84cc5ff49-nqwhl"] Oct 30 17:24:36.149472 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:36.149441 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e42c53c-946a-40fc-b126-5cd8a39b532d" path="/var/lib/kubelet/pods/0e42c53c-946a-40fc-b126-5cd8a39b532d/volumes" Oct 30 17:24:36.149990 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:36.149942 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68186e31-c661-41ea-bf07-6f97fb8d39b6" path="/var/lib/kubelet/pods/68186e31-c661-41ea-bf07-6f97fb8d39b6/volumes" Oct 30 17:24:39.304111 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:39.304077 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de890356-fdmx7"] Oct 30 17:24:39.304919 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:39.304900 4522 cpu_manager.go:414] "containerMap: removing container" podUID="d25a3aba-e336-4d85-9b43-edb6aa07b38d" containerName="revoke-keys" Oct 30 17:24:39.304966 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:39.304920 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d25a3aba-e336-4d85-9b43-edb6aa07b38d" containerName="revoke-keys" Oct 30 17:24:39.304966 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:39.304936 4522 cpu_manager.go:414] "containerMap: removing container" podUID="0e42c53c-946a-40fc-b126-5cd8a39b532d" containerName="istio-proxy" Oct 30 17:24:39.304966 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:39.304941 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e42c53c-946a-40fc-b126-5cd8a39b532d" containerName="istio-proxy" Oct 30 17:24:39.304966 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:39.304957 4522 cpu_manager.go:414] "containerMap: removing container" podUID="0e42c53c-946a-40fc-b126-5cd8a39b532d" containerName="istio-proxy" Oct 30 17:24:39.304966 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:39.304963 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e42c53c-946a-40fc-b126-5cd8a39b532d" containerName="istio-proxy" Oct 30 17:24:39.305133 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:39.304984 4522 cpu_manager.go:414] "containerMap: removing container" podUID="22c82bab-c6f4-48ba-8bb9-06b49eed06be" containerName="revoke-keys" Oct 30 17:24:39.305133 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:39.304989 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22c82bab-c6f4-48ba-8bb9-06b49eed06be" containerName="revoke-keys" Oct 30 17:24:39.305133 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:39.305121 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e42c53c-946a-40fc-b126-5cd8a39b532d" containerName="istio-proxy" Oct 30 17:24:39.305133 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:39.305132 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="22c82bab-c6f4-48ba-8bb9-06b49eed06be" containerName="revoke-keys" Oct 30 17:24:39.305260 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:39.305145 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="d25a3aba-e336-4d85-9b43-edb6aa07b38d" containerName="revoke-keys" Oct 30 17:24:39.309517 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:39.309500 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de890356-fdmx7" Oct 30 17:24:39.328829 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:39.328798 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de890356-fdmx7"] Oct 30 17:24:39.336171 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:39.336150 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:24:39.336300 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:39.336287 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:24:39.364730 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:39.364678 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e08ea37-fe40-45d2-a36d-d4904537e910-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de890356-fdmx7\" (UID: \"4e08ea37-fe40-45d2-a36d-d4904537e910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de890356-fdmx7" Oct 30 17:24:39.364916 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:39.364742 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnknx\" (UniqueName: \"kubernetes.io/projected/4e08ea37-fe40-45d2-a36d-d4904537e910-kube-api-access-rnknx\") pod \"maas-api-revoke-sub-de890356-fdmx7\" (UID: \"4e08ea37-fe40-45d2-a36d-d4904537e910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de890356-fdmx7" Oct 30 17:24:39.365092 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:39.365072 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e08ea37-fe40-45d2-a36d-d4904537e910-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de890356-fdmx7\" (UID: \"4e08ea37-fe40-45d2-a36d-d4904537e910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de890356-fdmx7" Oct 30 17:24:39.466251 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:39.466219 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e08ea37-fe40-45d2-a36d-d4904537e910-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de890356-fdmx7\" (UID: \"4e08ea37-fe40-45d2-a36d-d4904537e910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de890356-fdmx7" Oct 30 17:24:39.466448 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:39.466344 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e08ea37-fe40-45d2-a36d-d4904537e910-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de890356-fdmx7\" (UID: \"4e08ea37-fe40-45d2-a36d-d4904537e910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de890356-fdmx7" Oct 30 17:24:39.466448 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:39.466381 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnknx\" (UniqueName: \"kubernetes.io/projected/4e08ea37-fe40-45d2-a36d-d4904537e910-kube-api-access-rnknx\") pod \"maas-api-revoke-sub-de890356-fdmx7\" (UID: \"4e08ea37-fe40-45d2-a36d-d4904537e910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de890356-fdmx7" Oct 30 17:24:39.467001 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:39.466974 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e08ea37-fe40-45d2-a36d-d4904537e910-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de890356-fdmx7\" (UID: \"4e08ea37-fe40-45d2-a36d-d4904537e910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de890356-fdmx7" Oct 30 17:24:39.468743 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:39.468723 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e08ea37-fe40-45d2-a36d-d4904537e910-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de890356-fdmx7\" (UID: \"4e08ea37-fe40-45d2-a36d-d4904537e910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de890356-fdmx7" Oct 30 17:24:39.491808 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:39.491781 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnknx\" (UniqueName: \"kubernetes.io/projected/4e08ea37-fe40-45d2-a36d-d4904537e910-kube-api-access-rnknx\") pod \"maas-api-revoke-sub-de890356-fdmx7\" (UID: \"4e08ea37-fe40-45d2-a36d-d4904537e910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de890356-fdmx7" Oct 30 17:24:39.623498 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:39.623409 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de890356-fdmx7" Oct 30 17:24:39.931716 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:24:39.931678 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e08ea37_fe40_45d2_a36d_d4904537e910.slice/crio-32ecb68637b9cbbf68aaf3830888704d9d273929ee3d67a396587debacf851fa WatchSource:0}: Error finding container 32ecb68637b9cbbf68aaf3830888704d9d273929ee3d67a396587debacf851fa: Status 404 returned error can't find the container with id 32ecb68637b9cbbf68aaf3830888704d9d273929ee3d67a396587debacf851fa Oct 30 17:24:39.934063 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:39.934038 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de890356-fdmx7"] Oct 30 17:24:40.674013 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:40.673919 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de890356-fdmx7" event={"ID":"4e08ea37-fe40-45d2-a36d-d4904537e910","Type":"ContainerStarted","Data":"ef0b13f2d389e6f7147a1e0b294bdbbc16bc6f5c7d1fed436ceb169e7d1b4410"} Oct 30 17:24:40.674013 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:40.674000 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de890356-fdmx7" event={"ID":"4e08ea37-fe40-45d2-a36d-d4904537e910","Type":"ContainerStarted","Data":"32ecb68637b9cbbf68aaf3830888704d9d273929ee3d67a396587debacf851fa"} Oct 30 17:24:40.720128 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:40.720077 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de890356-fdmx7" podStartSLOduration=1.7200630019999998 podStartE2EDuration="1.720063002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:24:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:24:40.717146215 +0000 UTC m=+2254.717901136" watchObservedRunningTime="2026-10-30 17:24:40.720063002 +0000 UTC m=+2254.720817919" Oct 30 17:24:41.694077 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:41.694037 4522 generic.go:356] "Generic (PLEG): container finished" podID="4e08ea37-fe40-45d2-a36d-d4904537e910" containerID="ef0b13f2d389e6f7147a1e0b294bdbbc16bc6f5c7d1fed436ceb169e7d1b4410" exitCode=0 Oct 30 17:24:41.694077 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:41.694091 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de890356-fdmx7" event={"ID":"4e08ea37-fe40-45d2-a36d-d4904537e910","Type":"ContainerDied","Data":"ef0b13f2d389e6f7147a1e0b294bdbbc16bc6f5c7d1fed436ceb169e7d1b4410"} Oct 30 17:24:42.163547 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:42.163513 4522 pod_container_manager_linux.go:218] "Failed to delete cgroup paths" cgroupName=["kubepods","burstable","pod8226519e-a60e-4b3c-b4b3-83479839c8bb"] err="unable to destroy cgroup paths for cgroup [kubepods burstable pod8226519e-a60e-4b3c-b4b3-83479839c8bb] : Timed out while waiting for systemd to remove kubepods-burstable-pod8226519e_a60e_4b3c_b4b3_83479839c8bb.slice" Oct 30 17:24:42.163547 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:42.163544 4522 kubelet_pods.go:2797] "Failed to destroy orphaned pod cgroup" podUID="8226519e-a60e-4b3c-b4b3-83479839c8bb" err="failed to delete cgroup paths for [kubepods burstable pod8226519e-a60e-4b3c-b4b3-83479839c8bb] : unable to destroy cgroup paths for cgroup [kubepods burstable pod8226519e-a60e-4b3c-b4b3-83479839c8bb] : Timed out while waiting for systemd to remove kubepods-burstable-pod8226519e_a60e_4b3c_b4b3_83479839c8bb.slice" Oct 30 17:24:42.371665 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:42.371625 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e5ac8f-lcmfr"] Oct 30 17:24:42.372575 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:42.372558 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e42c53c-946a-40fc-b126-5cd8a39b532d" containerName="istio-proxy" Oct 30 17:24:42.376567 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:42.376549 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e5ac8f-lcmfr" Oct 30 17:24:42.396885 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:42.396842 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e5ac8f-lcmfr"] Oct 30 17:24:42.519573 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:42.519529 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d77de96-a093-4b65-b38a-6b66002dfd59-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e5ac8f-lcmfr\" (UID: \"7d77de96-a093-4b65-b38a-6b66002dfd59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e5ac8f-lcmfr" Oct 30 17:24:42.519573 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:42.519573 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d77de96-a093-4b65-b38a-6b66002dfd59-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e5ac8f-lcmfr\" (UID: \"7d77de96-a093-4b65-b38a-6b66002dfd59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e5ac8f-lcmfr" Oct 30 17:24:42.519933 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:42.519915 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jdxm\" (UniqueName: \"kubernetes.io/projected/7d77de96-a093-4b65-b38a-6b66002dfd59-kube-api-access-5jdxm\") pod \"maas-api-revoke-keys-e2e-inf-b-e5ac8f-lcmfr\" (UID: \"7d77de96-a093-4b65-b38a-6b66002dfd59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e5ac8f-lcmfr" Oct 30 17:24:42.621349 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:42.621301 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jdxm\" (UniqueName: \"kubernetes.io/projected/7d77de96-a093-4b65-b38a-6b66002dfd59-kube-api-access-5jdxm\") pod \"maas-api-revoke-keys-e2e-inf-b-e5ac8f-lcmfr\" (UID: \"7d77de96-a093-4b65-b38a-6b66002dfd59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e5ac8f-lcmfr" Oct 30 17:24:42.621522 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:42.621427 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d77de96-a093-4b65-b38a-6b66002dfd59-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e5ac8f-lcmfr\" (UID: \"7d77de96-a093-4b65-b38a-6b66002dfd59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e5ac8f-lcmfr" Oct 30 17:24:42.621522 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:42.621458 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d77de96-a093-4b65-b38a-6b66002dfd59-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e5ac8f-lcmfr\" (UID: \"7d77de96-a093-4b65-b38a-6b66002dfd59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e5ac8f-lcmfr" Oct 30 17:24:42.622087 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:42.622064 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d77de96-a093-4b65-b38a-6b66002dfd59-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e5ac8f-lcmfr\" (UID: \"7d77de96-a093-4b65-b38a-6b66002dfd59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e5ac8f-lcmfr" Oct 30 17:24:42.623934 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:42.623905 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d77de96-a093-4b65-b38a-6b66002dfd59-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e5ac8f-lcmfr\" (UID: \"7d77de96-a093-4b65-b38a-6b66002dfd59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e5ac8f-lcmfr" Oct 30 17:24:42.645896 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:42.645864 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jdxm\" (UniqueName: \"kubernetes.io/projected/7d77de96-a093-4b65-b38a-6b66002dfd59-kube-api-access-5jdxm\") pod \"maas-api-revoke-keys-e2e-inf-b-e5ac8f-lcmfr\" (UID: \"7d77de96-a093-4b65-b38a-6b66002dfd59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e5ac8f-lcmfr" Oct 30 17:24:42.692278 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:42.692250 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e5ac8f-lcmfr" Oct 30 17:24:42.913754 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:42.913729 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de890356-fdmx7" Oct 30 17:24:42.948124 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:24:42.948094 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:24:43.008161 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:24:43.008130 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d77de96_a093_4b65_b38a_6b66002dfd59.slice/crio-a3826b3e0f63928db775dcdf78adc04ba6562daeaad460c87dd550dff20b4e77 WatchSource:0}: Error finding container a3826b3e0f63928db775dcdf78adc04ba6562daeaad460c87dd550dff20b4e77: Status 404 returned error can't find the container with id a3826b3e0f63928db775dcdf78adc04ba6562daeaad460c87dd550dff20b4e77 Oct 30 17:24:43.010477 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:43.010451 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e5ac8f-lcmfr"] Oct 30 17:24:43.028750 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:43.028718 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e08ea37-fe40-45d2-a36d-d4904537e910-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e08ea37-fe40-45d2-a36d-d4904537e910-maas-api-openshift-ca\") pod \"4e08ea37-fe40-45d2-a36d-d4904537e910\" (UID: \"4e08ea37-fe40-45d2-a36d-d4904537e910\") " Oct 30 17:24:43.028837 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:43.028797 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e08ea37-fe40-45d2-a36d-d4904537e910-kube-api-access-rnknx\" (UniqueName: \"kubernetes.io/projected/4e08ea37-fe40-45d2-a36d-d4904537e910-kube-api-access-rnknx\") pod \"4e08ea37-fe40-45d2-a36d-d4904537e910\" (UID: \"4e08ea37-fe40-45d2-a36d-d4904537e910\") " Oct 30 17:24:43.028892 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:43.028838 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e08ea37-fe40-45d2-a36d-d4904537e910-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e08ea37-fe40-45d2-a36d-d4904537e910-maas-api-xks-ca\") pod \"4e08ea37-fe40-45d2-a36d-d4904537e910\" (UID: \"4e08ea37-fe40-45d2-a36d-d4904537e910\") " Oct 30 17:24:43.029075 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:43.029050 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e08ea37-fe40-45d2-a36d-d4904537e910-maas-api-openshift-ca" pod "4e08ea37-fe40-45d2-a36d-d4904537e910" (UID: "4e08ea37-fe40-45d2-a36d-d4904537e910"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:24:43.029629 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:43.029602 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e08ea37-fe40-45d2-a36d-d4904537e910-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:24:43.030915 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:43.030886 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e08ea37-fe40-45d2-a36d-d4904537e910-maas-api-xks-ca" pod "4e08ea37-fe40-45d2-a36d-d4904537e910" (UID: "4e08ea37-fe40-45d2-a36d-d4904537e910"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:24:43.030997 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:43.030975 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e08ea37-fe40-45d2-a36d-d4904537e910-kube-api-access-rnknx" pod "4e08ea37-fe40-45d2-a36d-d4904537e910" (UID: "4e08ea37-fe40-45d2-a36d-d4904537e910"). InnerVolumeSpecName "kube-api-access-rnknx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:24:43.131619 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:43.131541 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnknx\" (UniqueName: \"kubernetes.io/projected/4e08ea37-fe40-45d2-a36d-d4904537e910-kube-api-access-rnknx\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:24:43.131619 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:43.131569 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e08ea37-fe40-45d2-a36d-d4904537e910-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:24:43.720747 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:43.720711 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e5ac8f-lcmfr" event={"ID":"7d77de96-a093-4b65-b38a-6b66002dfd59","Type":"ContainerStarted","Data":"372e3ea64bea5d0e271c12341eaeee27d682347f403ef0f3828d3da9c528bf12"} Oct 30 17:24:43.720965 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:43.720767 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e5ac8f-lcmfr" event={"ID":"7d77de96-a093-4b65-b38a-6b66002dfd59","Type":"ContainerStarted","Data":"a3826b3e0f63928db775dcdf78adc04ba6562daeaad460c87dd550dff20b4e77"} Oct 30 17:24:43.722231 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:43.722211 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de890356-fdmx7" Oct 30 17:24:43.722367 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:43.722206 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de890356-fdmx7" event={"ID":"4e08ea37-fe40-45d2-a36d-d4904537e910","Type":"ContainerDied","Data":"32ecb68637b9cbbf68aaf3830888704d9d273929ee3d67a396587debacf851fa"} Oct 30 17:24:43.722367 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:43.722352 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32ecb68637b9cbbf68aaf3830888704d9d273929ee3d67a396587debacf851fa" Oct 30 17:24:43.770676 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:43.770620 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e5ac8f-lcmfr" podStartSLOduration=1.770605295 podStartE2EDuration="1.770605295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:24:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:24:43.76610617 +0000 UTC m=+2257.766861090" watchObservedRunningTime="2026-10-30 17:24:43.770605295 +0000 UTC m=+2257.771360195" Oct 30 17:24:44.678728 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:44.678693 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-2d01ec-785fcdd98-cz2mk" podUID="e7778b16-47c7-4892-a0e3-368175354cf4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:24:44.732488 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:44.732449 4522 generic.go:356] "Generic (PLEG): container finished" podID="7d77de96-a093-4b65-b38a-6b66002dfd59" containerID="372e3ea64bea5d0e271c12341eaeee27d682347f403ef0f3828d3da9c528bf12" exitCode=0 Oct 30 17:24:44.732672 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:44.732531 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e5ac8f-lcmfr" event={"ID":"7d77de96-a093-4b65-b38a-6b66002dfd59","Type":"ContainerDied","Data":"372e3ea64bea5d0e271c12341eaeee27d682347f403ef0f3828d3da9c528bf12"} Oct 30 17:24:45.931730 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:45.931697 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e5ac8f-lcmfr" Oct 30 17:24:46.079574 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:46.079485 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d77de96-a093-4b65-b38a-6b66002dfd59-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d77de96-a093-4b65-b38a-6b66002dfd59-maas-api-xks-ca\") pod \"7d77de96-a093-4b65-b38a-6b66002dfd59\" (UID: \"7d77de96-a093-4b65-b38a-6b66002dfd59\") " Oct 30 17:24:46.079574 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:46.079534 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d77de96-a093-4b65-b38a-6b66002dfd59-kube-api-access-5jdxm\" (UniqueName: \"kubernetes.io/projected/7d77de96-a093-4b65-b38a-6b66002dfd59-kube-api-access-5jdxm\") pod \"7d77de96-a093-4b65-b38a-6b66002dfd59\" (UID: \"7d77de96-a093-4b65-b38a-6b66002dfd59\") " Oct 30 17:24:46.079817 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:46.079671 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d77de96-a093-4b65-b38a-6b66002dfd59-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d77de96-a093-4b65-b38a-6b66002dfd59-maas-api-openshift-ca\") pod \"7d77de96-a093-4b65-b38a-6b66002dfd59\" (UID: \"7d77de96-a093-4b65-b38a-6b66002dfd59\") " Oct 30 17:24:46.080035 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:46.079992 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d77de96-a093-4b65-b38a-6b66002dfd59-maas-api-openshift-ca" pod "7d77de96-a093-4b65-b38a-6b66002dfd59" (UID: "7d77de96-a093-4b65-b38a-6b66002dfd59"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:24:46.080535 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:46.080510 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d77de96-a093-4b65-b38a-6b66002dfd59-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:24:46.081774 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:46.081748 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d77de96-a093-4b65-b38a-6b66002dfd59-maas-api-xks-ca" pod "7d77de96-a093-4b65-b38a-6b66002dfd59" (UID: "7d77de96-a093-4b65-b38a-6b66002dfd59"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:24:46.081910 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:46.081894 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d77de96-a093-4b65-b38a-6b66002dfd59-kube-api-access-5jdxm" pod "7d77de96-a093-4b65-b38a-6b66002dfd59" (UID: "7d77de96-a093-4b65-b38a-6b66002dfd59"). InnerVolumeSpecName "kube-api-access-5jdxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:24:46.181740 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:46.181713 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d77de96-a093-4b65-b38a-6b66002dfd59-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:24:46.181740 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:46.181739 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jdxm\" (UniqueName: \"kubernetes.io/projected/7d77de96-a093-4b65-b38a-6b66002dfd59-kube-api-access-5jdxm\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:24:46.719608 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:46.719546 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2d01ec-785fcdd98-cz2mk" Oct 30 17:24:46.749583 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:46.749552 4522 generic.go:356] "Generic (PLEG): container finished" podID="e7778b16-47c7-4892-a0e3-368175354cf4" containerID="9ce7ea259599b11a545e1623e1ddbe10fcb1ff962a67b0d1c8135440bcc38287" exitCode=0 Oct 30 17:24:46.749754 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:46.749601 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2d01ec-785fcdd98-cz2mk" Oct 30 17:24:46.749754 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:46.749643 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2d01ec-785fcdd98-cz2mk" event={"ID":"e7778b16-47c7-4892-a0e3-368175354cf4","Type":"ContainerDied","Data":"9ce7ea259599b11a545e1623e1ddbe10fcb1ff962a67b0d1c8135440bcc38287"} Oct 30 17:24:46.749754 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:46.749702 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2d01ec-785fcdd98-cz2mk" event={"ID":"e7778b16-47c7-4892-a0e3-368175354cf4","Type":"ContainerDied","Data":"c8477b25dfff824d89aa79bbcb293c2d4c759dd3a883754882eb9fa135b9f42c"} Oct 30 17:24:46.749754 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:46.749730 4522 scope.go:122] "RemoveContainer" containerID="9ce7ea259599b11a545e1623e1ddbe10fcb1ff962a67b0d1c8135440bcc38287" Oct 30 17:24:46.751079 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:46.751061 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e5ac8f-lcmfr" Oct 30 17:24:46.751143 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:46.751097 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e5ac8f-lcmfr" event={"ID":"7d77de96-a093-4b65-b38a-6b66002dfd59","Type":"ContainerDied","Data":"a3826b3e0f63928db775dcdf78adc04ba6562daeaad460c87dd550dff20b4e77"} Oct 30 17:24:46.751143 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:46.751137 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3826b3e0f63928db775dcdf78adc04ba6562daeaad460c87dd550dff20b4e77" Oct 30 17:24:46.761421 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:46.761404 4522 scope.go:122] "RemoveContainer" containerID="9ce7ea259599b11a545e1623e1ddbe10fcb1ff962a67b0d1c8135440bcc38287" Oct 30 17:24:46.761681 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:24:46.761661 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ce7ea259599b11a545e1623e1ddbe10fcb1ff962a67b0d1c8135440bcc38287\": container with ID starting with 9ce7ea259599b11a545e1623e1ddbe10fcb1ff962a67b0d1c8135440bcc38287 not found: ID does not exist" containerID="9ce7ea259599b11a545e1623e1ddbe10fcb1ff962a67b0d1c8135440bcc38287" Oct 30 17:24:46.761732 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:46.761687 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ce7ea259599b11a545e1623e1ddbe10fcb1ff962a67b0d1c8135440bcc38287"} err="failed to get container status \"9ce7ea259599b11a545e1623e1ddbe10fcb1ff962a67b0d1c8135440bcc38287\": rpc error: code = NotFound desc = could not find container \"9ce7ea259599b11a545e1623e1ddbe10fcb1ff962a67b0d1c8135440bcc38287\": container with ID starting with 9ce7ea259599b11a545e1623e1ddbe10fcb1ff962a67b0d1c8135440bcc38287 not found: ID does not exist" Oct 30 17:24:46.900243 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:46.900210 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7778b16-47c7-4892-a0e3-368175354cf4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7778b16-47c7-4892-a0e3-368175354cf4-plugins-config-volume\") pod \"e7778b16-47c7-4892-a0e3-368175354cf4\" (UID: \"e7778b16-47c7-4892-a0e3-368175354cf4\") " Oct 30 17:24:46.900458 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:46.900442 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7778b16-47c7-4892-a0e3-368175354cf4-kube-api-access-n44nj\" (UniqueName: \"kubernetes.io/projected/e7778b16-47c7-4892-a0e3-368175354cf4-kube-api-access-n44nj\") pod \"e7778b16-47c7-4892-a0e3-368175354cf4\" (UID: \"e7778b16-47c7-4892-a0e3-368175354cf4\") " Oct 30 17:24:46.900676 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:46.900651 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7778b16-47c7-4892-a0e3-368175354cf4-plugins-config-volume" pod "e7778b16-47c7-4892-a0e3-368175354cf4" (UID: "e7778b16-47c7-4892-a0e3-368175354cf4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:24:46.901026 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:46.901005 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7778b16-47c7-4892-a0e3-368175354cf4-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:24:46.902574 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:46.902554 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7778b16-47c7-4892-a0e3-368175354cf4-kube-api-access-n44nj" pod "e7778b16-47c7-4892-a0e3-368175354cf4" (UID: "e7778b16-47c7-4892-a0e3-368175354cf4"). InnerVolumeSpecName "kube-api-access-n44nj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:24:47.002907 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:47.002832 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n44nj\" (UniqueName: \"kubernetes.io/projected/e7778b16-47c7-4892-a0e3-368175354cf4-kube-api-access-n44nj\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:24:47.104301 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:47.104268 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2d01ec-785fcdd98-cz2mk"] Oct 30 17:24:47.116896 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:47.116869 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2d01ec-785fcdd98-cz2mk"] Oct 30 17:24:48.147128 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:48.147088 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7778b16-47c7-4892-a0e3-368175354cf4" path="/var/lib/kubelet/pods/e7778b16-47c7-4892-a0e3-368175354cf4/volumes" Oct 30 17:24:54.093163 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:54.093123 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e5ac8f-98c6fb647-mlcx8"] Oct 30 17:24:59.093851 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:24:59.093791 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e5ac8f-98c6fb647-mlcx8" podUID="35026397-efc0-4b45-b548-01ce4d2ac26e" containerName="payload-pre-processing" containerID="cri-o://08fbfa4e57ab86881fde7563462c59344b02946269addf1e9bf72520ced2f371" gracePeriod=25 Oct 30 17:25:04.148200 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:04.148170 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-42d381-qb5sp"] Oct 30 17:25:04.148925 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:04.148905 4522 cpu_manager.go:414] "containerMap: removing container" podUID="e7778b16-47c7-4892-a0e3-368175354cf4" containerName="payload-processing" Oct 30 17:25:04.148994 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:04.148926 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7778b16-47c7-4892-a0e3-368175354cf4" containerName="payload-processing" Oct 30 17:25:04.148994 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:04.148958 4522 cpu_manager.go:414] "containerMap: removing container" podUID="4e08ea37-fe40-45d2-a36d-d4904537e910" containerName="revoke-keys" Oct 30 17:25:04.148994 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:04.148963 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e08ea37-fe40-45d2-a36d-d4904537e910" containerName="revoke-keys" Oct 30 17:25:04.148994 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:04.148982 4522 cpu_manager.go:414] "containerMap: removing container" podUID="7d77de96-a093-4b65-b38a-6b66002dfd59" containerName="revoke-keys" Oct 30 17:25:04.148994 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:04.148986 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d77de96-a093-4b65-b38a-6b66002dfd59" containerName="revoke-keys" Oct 30 17:25:04.149172 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:04.149099 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e08ea37-fe40-45d2-a36d-d4904537e910" containerName="revoke-keys" Oct 30 17:25:04.149172 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:04.149110 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d77de96-a093-4b65-b38a-6b66002dfd59" containerName="revoke-keys" Oct 30 17:25:04.149172 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:04.149122 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7778b16-47c7-4892-a0e3-368175354cf4" containerName="payload-processing" Oct 30 17:25:04.153428 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:04.153409 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-42d381-qb5sp" Oct 30 17:25:04.168487 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:04.168448 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-42d381-qb5sp"] Oct 30 17:25:04.175447 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:04.175414 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:25:04.175611 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:04.175532 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:25:04.304486 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:04.304438 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/522c89f3-afd3-4819-9024-220111395373-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-42d381-qb5sp\" (UID: \"522c89f3-afd3-4819-9024-220111395373\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-42d381-qb5sp" Oct 30 17:25:04.304735 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:04.304652 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtr4n\" (UniqueName: \"kubernetes.io/projected/522c89f3-afd3-4819-9024-220111395373-kube-api-access-jtr4n\") pod \"maas-api-revoke-keys-e2e-inf-a-42d381-qb5sp\" (UID: \"522c89f3-afd3-4819-9024-220111395373\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-42d381-qb5sp" Oct 30 17:25:04.304887 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:04.304859 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/522c89f3-afd3-4819-9024-220111395373-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-42d381-qb5sp\" (UID: \"522c89f3-afd3-4819-9024-220111395373\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-42d381-qb5sp" Oct 30 17:25:04.406045 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:04.405948 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/522c89f3-afd3-4819-9024-220111395373-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-42d381-qb5sp\" (UID: \"522c89f3-afd3-4819-9024-220111395373\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-42d381-qb5sp" Oct 30 17:25:04.406224 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:04.406118 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/522c89f3-afd3-4819-9024-220111395373-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-42d381-qb5sp\" (UID: \"522c89f3-afd3-4819-9024-220111395373\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-42d381-qb5sp" Oct 30 17:25:04.406337 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:04.406271 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtr4n\" (UniqueName: \"kubernetes.io/projected/522c89f3-afd3-4819-9024-220111395373-kube-api-access-jtr4n\") pod \"maas-api-revoke-keys-e2e-inf-a-42d381-qb5sp\" (UID: \"522c89f3-afd3-4819-9024-220111395373\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-42d381-qb5sp" Oct 30 17:25:04.406826 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:04.406799 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/522c89f3-afd3-4819-9024-220111395373-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-42d381-qb5sp\" (UID: \"522c89f3-afd3-4819-9024-220111395373\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-42d381-qb5sp" Oct 30 17:25:04.408432 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:04.408413 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/522c89f3-afd3-4819-9024-220111395373-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-42d381-qb5sp\" (UID: \"522c89f3-afd3-4819-9024-220111395373\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-42d381-qb5sp" Oct 30 17:25:04.431368 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:04.431339 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtr4n\" (UniqueName: \"kubernetes.io/projected/522c89f3-afd3-4819-9024-220111395373-kube-api-access-jtr4n\") pod \"maas-api-revoke-keys-e2e-inf-a-42d381-qb5sp\" (UID: \"522c89f3-afd3-4819-9024-220111395373\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-42d381-qb5sp" Oct 30 17:25:04.469543 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:04.469514 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-42d381-qb5sp" Oct 30 17:25:04.777668 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:25:04.777613 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod522c89f3_afd3_4819_9024_220111395373.slice/crio-d47e973e6eb353f96d6582be6c71169d833cfb12dc3b239144b54130d1b0a9e3 WatchSource:0}: Error finding container d47e973e6eb353f96d6582be6c71169d833cfb12dc3b239144b54130d1b0a9e3: Status 404 returned error can't find the container with id d47e973e6eb353f96d6582be6c71169d833cfb12dc3b239144b54130d1b0a9e3 Oct 30 17:25:04.782053 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:04.782019 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-42d381-qb5sp"] Oct 30 17:25:04.939343 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:04.939291 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-42d381-qb5sp" event={"ID":"522c89f3-afd3-4819-9024-220111395373","Type":"ContainerStarted","Data":"d47e973e6eb353f96d6582be6c71169d833cfb12dc3b239144b54130d1b0a9e3"} Oct 30 17:25:05.950561 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:05.950524 4522 generic.go:356] "Generic (PLEG): container finished" podID="522c89f3-afd3-4819-9024-220111395373" containerID="0d9c839ee5d51fe5a54b648864216adaa78a312ddcdd403e991e9c77b52e3f2c" exitCode=0 Oct 30 17:25:05.950973 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:05.950570 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-42d381-qb5sp" event={"ID":"522c89f3-afd3-4819-9024-220111395373","Type":"ContainerDied","Data":"0d9c839ee5d51fe5a54b648864216adaa78a312ddcdd403e991e9c77b52e3f2c"} Oct 30 17:25:06.148900 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:06.148820 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h5k2r"] Oct 30 17:25:06.149361 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:06.149328 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h5k2r" podUID="23c04722-0747-49fe-a166-0969959367bd" containerName="istio-proxy" containerID="cri-o://44d81bdc413b0e8823bda116b585b0611ebd70226426347fa26fbbbd78dfae77" gracePeriod=30 Oct 30 17:25:06.639941 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:06.639906 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e5ac8f-98c6fb647-mlcx8" podUID="35026397-efc0-4b45-b548-01ce4d2ac26e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:25:07.128763 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:07.128732 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-42d381-qb5sp" Oct 30 17:25:07.249101 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:07.249013 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/522c89f3-afd3-4819-9024-220111395373-kube-api-access-jtr4n\" (UniqueName: \"kubernetes.io/projected/522c89f3-afd3-4819-9024-220111395373-kube-api-access-jtr4n\") pod \"522c89f3-afd3-4819-9024-220111395373\" (UID: \"522c89f3-afd3-4819-9024-220111395373\") " Oct 30 17:25:07.249101 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:07.249100 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/522c89f3-afd3-4819-9024-220111395373-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/522c89f3-afd3-4819-9024-220111395373-maas-api-openshift-ca\") pod \"522c89f3-afd3-4819-9024-220111395373\" (UID: \"522c89f3-afd3-4819-9024-220111395373\") " Oct 30 17:25:07.249491 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:07.249466 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/522c89f3-afd3-4819-9024-220111395373-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/522c89f3-afd3-4819-9024-220111395373-maas-api-xks-ca\") pod \"522c89f3-afd3-4819-9024-220111395373\" (UID: \"522c89f3-afd3-4819-9024-220111395373\") " Oct 30 17:25:07.249648 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:07.249496 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/522c89f3-afd3-4819-9024-220111395373-maas-api-openshift-ca" pod "522c89f3-afd3-4819-9024-220111395373" (UID: "522c89f3-afd3-4819-9024-220111395373"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:25:07.250048 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:07.250028 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/522c89f3-afd3-4819-9024-220111395373-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:25:07.251213 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:07.251188 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/522c89f3-afd3-4819-9024-220111395373-kube-api-access-jtr4n" pod "522c89f3-afd3-4819-9024-220111395373" (UID: "522c89f3-afd3-4819-9024-220111395373"). InnerVolumeSpecName "kube-api-access-jtr4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:25:07.251359 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:07.251336 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/522c89f3-afd3-4819-9024-220111395373-maas-api-xks-ca" pod "522c89f3-afd3-4819-9024-220111395373" (UID: "522c89f3-afd3-4819-9024-220111395373"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:25:07.352144 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:07.352103 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/522c89f3-afd3-4819-9024-220111395373-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:25:07.352144 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:07.352141 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtr4n\" (UniqueName: \"kubernetes.io/projected/522c89f3-afd3-4819-9024-220111395373-kube-api-access-jtr4n\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:25:07.969752 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:07.969722 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-42d381-qb5sp" Oct 30 17:25:07.969938 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:07.969720 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-42d381-qb5sp" event={"ID":"522c89f3-afd3-4819-9024-220111395373","Type":"ContainerDied","Data":"d47e973e6eb353f96d6582be6c71169d833cfb12dc3b239144b54130d1b0a9e3"} Oct 30 17:25:07.969938 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:07.969842 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d47e973e6eb353f96d6582be6c71169d833cfb12dc3b239144b54130d1b0a9e3" Oct 30 17:25:10.042905 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:10.039329 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0feca821-pqs6z"] Oct 30 17:25:10.058607 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:10.058543 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0feca821-pqs6z"] Oct 30 17:25:10.149020 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:10.148990 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9d0a693-f2b8-4036-8f23-a3e2c304eb12" path="/var/lib/kubelet/pods/e9d0a693-f2b8-4036-8f23-a3e2c304eb12/volumes" Oct 30 17:25:11.448316 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:11.448280 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h5k2r" Oct 30 17:25:11.518397 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:11.518058 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23c04722-0747-49fe-a166-0969959367bd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23c04722-0747-49fe-a166-0969959367bd-credential-socket\") pod \"23c04722-0747-49fe-a166-0969959367bd\" (UID: \"23c04722-0747-49fe-a166-0969959367bd\") " Oct 30 17:25:11.518397 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:11.518149 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23c04722-0747-49fe-a166-0969959367bd-kube-api-access-wcgcd\" (UniqueName: \"kubernetes.io/projected/23c04722-0747-49fe-a166-0969959367bd-kube-api-access-wcgcd\") pod \"23c04722-0747-49fe-a166-0969959367bd\" (UID: \"23c04722-0747-49fe-a166-0969959367bd\") " Oct 30 17:25:11.518397 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:11.518352 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23c04722-0747-49fe-a166-0969959367bd-credential-socket" pod "23c04722-0747-49fe-a166-0969959367bd" (UID: "23c04722-0747-49fe-a166-0969959367bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:25:11.518397 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:11.518371 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23c04722-0747-49fe-a166-0969959367bd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23c04722-0747-49fe-a166-0969959367bd-workload-socket\") pod \"23c04722-0747-49fe-a166-0969959367bd\" (UID: \"23c04722-0747-49fe-a166-0969959367bd\") " Oct 30 17:25:11.518807 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:11.518480 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23c04722-0747-49fe-a166-0969959367bd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23c04722-0747-49fe-a166-0969959367bd-istio-data\") pod \"23c04722-0747-49fe-a166-0969959367bd\" (UID: \"23c04722-0747-49fe-a166-0969959367bd\") " Oct 30 17:25:11.518807 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:11.518617 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23c04722-0747-49fe-a166-0969959367bd-workload-socket" pod "23c04722-0747-49fe-a166-0969959367bd" (UID: "23c04722-0747-49fe-a166-0969959367bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:25:11.518807 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:11.518628 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23c04722-0747-49fe-a166-0969959367bd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23c04722-0747-49fe-a166-0969959367bd-istiod-ca-cert\") pod \"23c04722-0747-49fe-a166-0969959367bd\" (UID: \"23c04722-0747-49fe-a166-0969959367bd\") " Oct 30 17:25:11.518807 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:11.518685 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23c04722-0747-49fe-a166-0969959367bd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23c04722-0747-49fe-a166-0969959367bd-workload-certs\") pod \"23c04722-0747-49fe-a166-0969959367bd\" (UID: \"23c04722-0747-49fe-a166-0969959367bd\") " Oct 30 17:25:11.518807 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:11.518726 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23c04722-0747-49fe-a166-0969959367bd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23c04722-0747-49fe-a166-0969959367bd-istio-envoy\") pod \"23c04722-0747-49fe-a166-0969959367bd\" (UID: \"23c04722-0747-49fe-a166-0969959367bd\") " Oct 30 17:25:11.518807 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:11.518774 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/23c04722-0747-49fe-a166-0969959367bd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23c04722-0747-49fe-a166-0969959367bd-istio-podinfo\") pod \"23c04722-0747-49fe-a166-0969959367bd\" (UID: \"23c04722-0747-49fe-a166-0969959367bd\") " Oct 30 17:25:11.519152 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:11.518984 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23c04722-0747-49fe-a166-0969959367bd-istiod-ca-cert" pod "23c04722-0747-49fe-a166-0969959367bd" (UID: "23c04722-0747-49fe-a166-0969959367bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:25:11.519152 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:11.519074 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23c04722-0747-49fe-a166-0969959367bd-istio-data" pod "23c04722-0747-49fe-a166-0969959367bd" (UID: "23c04722-0747-49fe-a166-0969959367bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:25:11.519352 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:11.519291 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23c04722-0747-49fe-a166-0969959367bd-workload-certs" pod "23c04722-0747-49fe-a166-0969959367bd" (UID: "23c04722-0747-49fe-a166-0969959367bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:25:11.520342 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:11.519468 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23c04722-0747-49fe-a166-0969959367bd-istio-token\" (UniqueName: \"kubernetes.io/projected/23c04722-0747-49fe-a166-0969959367bd-istio-token\") pod \"23c04722-0747-49fe-a166-0969959367bd\" (UID: \"23c04722-0747-49fe-a166-0969959367bd\") " Oct 30 17:25:11.520342 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:11.520233 4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23c04722-0747-49fe-a166-0969959367bd-istiod-ca-cert\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:25:11.520342 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:11.520255 4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23c04722-0747-49fe-a166-0969959367bd-workload-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:25:11.520342 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:11.520268 4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23c04722-0747-49fe-a166-0969959367bd-credential-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:25:11.520342 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:11.520281 4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23c04722-0747-49fe-a166-0969959367bd-workload-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:25:11.520342 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:11.520294 4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23c04722-0747-49fe-a166-0969959367bd-istio-data\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:25:11.521921 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:11.521887 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23c04722-0747-49fe-a166-0969959367bd-kube-api-access-wcgcd" pod "23c04722-0747-49fe-a166-0969959367bd" (UID: "23c04722-0747-49fe-a166-0969959367bd"). InnerVolumeSpecName "kube-api-access-wcgcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:25:11.526724 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:11.526698 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23c04722-0747-49fe-a166-0969959367bd-istio-token" pod "23c04722-0747-49fe-a166-0969959367bd" (UID: "23c04722-0747-49fe-a166-0969959367bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:25:11.531285 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:11.531247 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23c04722-0747-49fe-a166-0969959367bd-istio-envoy" pod "23c04722-0747-49fe-a166-0969959367bd" (UID: "23c04722-0747-49fe-a166-0969959367bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:25:11.531285 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:11.531264 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23c04722-0747-49fe-a166-0969959367bd-istio-podinfo" pod "23c04722-0747-49fe-a166-0969959367bd" (UID: "23c04722-0747-49fe-a166-0969959367bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:25:11.622429 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:11.622387 4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23c04722-0747-49fe-a166-0969959367bd-istio-envoy\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:25:11.622429 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:11.622425 4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23c04722-0747-49fe-a166-0969959367bd-istio-podinfo\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:25:11.622429 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:11.622438 4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23c04722-0747-49fe-a166-0969959367bd-istio-token\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:25:11.622768 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:11.622454 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcgcd\" (UniqueName: \"kubernetes.io/projected/23c04722-0747-49fe-a166-0969959367bd-kube-api-access-wcgcd\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:25:12.008576 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:12.008540 4522 generic.go:356] "Generic (PLEG): container finished" podID="23c04722-0747-49fe-a166-0969959367bd" containerID="44d81bdc413b0e8823bda116b585b0611ebd70226426347fa26fbbbd78dfae77" exitCode=0 Oct 30 17:25:12.008765 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:12.008596 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h5k2r" Oct 30 17:25:12.008765 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:12.008629 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h5k2r" event={"ID":"23c04722-0747-49fe-a166-0969959367bd","Type":"ContainerDied","Data":"44d81bdc413b0e8823bda116b585b0611ebd70226426347fa26fbbbd78dfae77"} Oct 30 17:25:12.008765 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:12.008698 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h5k2r" event={"ID":"23c04722-0747-49fe-a166-0969959367bd","Type":"ContainerDied","Data":"8f5fb68eab981f952b22ed8ee0925ef5e722c37d7aef92ae3cee46db700df2fd"} Oct 30 17:25:12.008765 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:12.008727 4522 scope.go:122] "RemoveContainer" containerID="44d81bdc413b0e8823bda116b585b0611ebd70226426347fa26fbbbd78dfae77" Oct 30 17:25:12.021510 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:12.021492 4522 scope.go:122] "RemoveContainer" containerID="44d81bdc413b0e8823bda116b585b0611ebd70226426347fa26fbbbd78dfae77" Oct 30 17:25:12.021772 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:25:12.021754 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44d81bdc413b0e8823bda116b585b0611ebd70226426347fa26fbbbd78dfae77\": container with ID starting with 44d81bdc413b0e8823bda116b585b0611ebd70226426347fa26fbbbd78dfae77 not found: ID does not exist" containerID="44d81bdc413b0e8823bda116b585b0611ebd70226426347fa26fbbbd78dfae77" Oct 30 17:25:12.021843 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:12.021784 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44d81bdc413b0e8823bda116b585b0611ebd70226426347fa26fbbbd78dfae77"} err="failed to get container status \"44d81bdc413b0e8823bda116b585b0611ebd70226426347fa26fbbbd78dfae77\": rpc error: code = NotFound desc = could not find container \"44d81bdc413b0e8823bda116b585b0611ebd70226426347fa26fbbbd78dfae77\": container with ID starting with 44d81bdc413b0e8823bda116b585b0611ebd70226426347fa26fbbbd78dfae77 not found: ID does not exist" Oct 30 17:25:12.043058 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:12.043027 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef03b6bc-rzdkj"] Oct 30 17:25:12.047580 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:12.047555 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef03b6bc-rzdkj"] Oct 30 17:25:12.068048 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:12.068023 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h5k2r"] Oct 30 17:25:12.073247 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:12.073221 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-h5k2r"] Oct 30 17:25:12.147594 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:12.147563 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13d8213f-ae96-45a6-99c3-26d15d56313e" path="/var/lib/kubelet/pods/13d8213f-ae96-45a6-99c3-26d15d56313e/volumes" Oct 30 17:25:12.148108 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:12.148094 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23c04722-0747-49fe-a166-0969959367bd" path="/var/lib/kubelet/pods/23c04722-0747-49fe-a166-0969959367bd/volumes" Oct 30 17:25:12.953061 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:25:12.953024 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:25:14.293478 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:14.293444 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-42d381-85f9dccd5f-fkl4k"] Oct 30 17:25:14.755407 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:14.752764 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq"] Oct 30 17:25:14.755407 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:14.754168 4522 cpu_manager.go:414] "containerMap: removing container" podUID="522c89f3-afd3-4819-9024-220111395373" containerName="revoke-keys" Oct 30 17:25:14.755407 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:14.754190 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="522c89f3-afd3-4819-9024-220111395373" containerName="revoke-keys" Oct 30 17:25:14.755407 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:14.754215 4522 cpu_manager.go:414] "containerMap: removing container" podUID="23c04722-0747-49fe-a166-0969959367bd" containerName="istio-proxy" Oct 30 17:25:14.755407 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:14.754221 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23c04722-0747-49fe-a166-0969959367bd" containerName="istio-proxy" Oct 30 17:25:14.755407 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:14.754756 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="522c89f3-afd3-4819-9024-220111395373" containerName="revoke-keys" Oct 30 17:25:14.755407 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:14.754782 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="23c04722-0747-49fe-a166-0969959367bd" containerName="istio-proxy" Oct 30 17:25:14.761420 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:14.761377 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq" Oct 30 17:25:14.779739 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:14.779712 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq"] Oct 30 17:25:14.780329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:14.780292 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-db8417-openshift-default-dockercfg-94mls\"" Oct 30 17:25:14.882250 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:14.882218 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06d654a9-d7c3-4b45-8bcc-abfad4c49970-workload-certs\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq\" (UID: \"06d654a9-d7c3-4b45-8bcc-abfad4c49970\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq" Oct 30 17:25:14.882470 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:14.882259 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06d654a9-d7c3-4b45-8bcc-abfad4c49970-istiod-ca-cert\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq\" (UID: \"06d654a9-d7c3-4b45-8bcc-abfad4c49970\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq" Oct 30 17:25:14.882470 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:14.882361 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06d654a9-d7c3-4b45-8bcc-abfad4c49970-istio-token\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq\" (UID: \"06d654a9-d7c3-4b45-8bcc-abfad4c49970\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq" Oct 30 17:25:14.882470 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:14.882404 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwl69\" (UniqueName: \"kubernetes.io/projected/06d654a9-d7c3-4b45-8bcc-abfad4c49970-kube-api-access-qwl69\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq\" (UID: \"06d654a9-d7c3-4b45-8bcc-abfad4c49970\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq" Oct 30 17:25:14.882590 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:14.882499 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06d654a9-d7c3-4b45-8bcc-abfad4c49970-istio-podinfo\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq\" (UID: \"06d654a9-d7c3-4b45-8bcc-abfad4c49970\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq" Oct 30 17:25:14.882590 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:14.882535 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06d654a9-d7c3-4b45-8bcc-abfad4c49970-istio-data\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq\" (UID: \"06d654a9-d7c3-4b45-8bcc-abfad4c49970\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq" Oct 30 17:25:14.882657 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:14.882595 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06d654a9-d7c3-4b45-8bcc-abfad4c49970-istio-envoy\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq\" (UID: \"06d654a9-d7c3-4b45-8bcc-abfad4c49970\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq" Oct 30 17:25:14.882657 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:14.882617 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06d654a9-d7c3-4b45-8bcc-abfad4c49970-workload-socket\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq\" (UID: \"06d654a9-d7c3-4b45-8bcc-abfad4c49970\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq" Oct 30 17:25:14.882724 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:14.882703 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06d654a9-d7c3-4b45-8bcc-abfad4c49970-credential-socket\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq\" (UID: \"06d654a9-d7c3-4b45-8bcc-abfad4c49970\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq" Oct 30 17:25:14.984668 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:14.984628 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06d654a9-d7c3-4b45-8bcc-abfad4c49970-workload-certs\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq\" (UID: \"06d654a9-d7c3-4b45-8bcc-abfad4c49970\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq" Oct 30 17:25:14.984668 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:14.984674 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06d654a9-d7c3-4b45-8bcc-abfad4c49970-istiod-ca-cert\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq\" (UID: \"06d654a9-d7c3-4b45-8bcc-abfad4c49970\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq" Oct 30 17:25:14.984945 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:14.984717 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06d654a9-d7c3-4b45-8bcc-abfad4c49970-istio-token\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq\" (UID: \"06d654a9-d7c3-4b45-8bcc-abfad4c49970\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq" Oct 30 17:25:14.984945 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:14.984764 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwl69\" (UniqueName: \"kubernetes.io/projected/06d654a9-d7c3-4b45-8bcc-abfad4c49970-kube-api-access-qwl69\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq\" (UID: \"06d654a9-d7c3-4b45-8bcc-abfad4c49970\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq" Oct 30 17:25:14.984945 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:14.984807 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06d654a9-d7c3-4b45-8bcc-abfad4c49970-istio-podinfo\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq\" (UID: \"06d654a9-d7c3-4b45-8bcc-abfad4c49970\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq" Oct 30 17:25:14.984945 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:14.984826 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06d654a9-d7c3-4b45-8bcc-abfad4c49970-istio-data\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq\" (UID: \"06d654a9-d7c3-4b45-8bcc-abfad4c49970\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq" Oct 30 17:25:14.984945 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:14.984854 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06d654a9-d7c3-4b45-8bcc-abfad4c49970-istio-envoy\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq\" (UID: \"06d654a9-d7c3-4b45-8bcc-abfad4c49970\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq" Oct 30 17:25:14.984945 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:14.984877 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06d654a9-d7c3-4b45-8bcc-abfad4c49970-workload-socket\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq\" (UID: \"06d654a9-d7c3-4b45-8bcc-abfad4c49970\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq" Oct 30 17:25:14.984945 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:14.984922 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06d654a9-d7c3-4b45-8bcc-abfad4c49970-credential-socket\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq\" (UID: \"06d654a9-d7c3-4b45-8bcc-abfad4c49970\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq" Oct 30 17:25:14.985444 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:14.985424 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06d654a9-d7c3-4b45-8bcc-abfad4c49970-credential-socket\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq\" (UID: \"06d654a9-d7c3-4b45-8bcc-abfad4c49970\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq" Oct 30 17:25:14.985689 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:14.985669 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06d654a9-d7c3-4b45-8bcc-abfad4c49970-istiod-ca-cert\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq\" (UID: \"06d654a9-d7c3-4b45-8bcc-abfad4c49970\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq" Oct 30 17:25:14.986324 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:14.986286 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06d654a9-d7c3-4b45-8bcc-abfad4c49970-workload-socket\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq\" (UID: \"06d654a9-d7c3-4b45-8bcc-abfad4c49970\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq" Oct 30 17:25:14.986825 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:14.986632 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06d654a9-d7c3-4b45-8bcc-abfad4c49970-workload-certs\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq\" (UID: \"06d654a9-d7c3-4b45-8bcc-abfad4c49970\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq" Oct 30 17:25:14.987074 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:14.986735 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06d654a9-d7c3-4b45-8bcc-abfad4c49970-istio-data\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq\" (UID: \"06d654a9-d7c3-4b45-8bcc-abfad4c49970\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq" Oct 30 17:25:14.988860 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:14.988835 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06d654a9-d7c3-4b45-8bcc-abfad4c49970-istio-podinfo\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq\" (UID: \"06d654a9-d7c3-4b45-8bcc-abfad4c49970\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq" Oct 30 17:25:14.988860 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:14.988852 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06d654a9-d7c3-4b45-8bcc-abfad4c49970-istio-envoy\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq\" (UID: \"06d654a9-d7c3-4b45-8bcc-abfad4c49970\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq" Oct 30 17:25:15.010973 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:15.010898 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06d654a9-d7c3-4b45-8bcc-abfad4c49970-istio-token\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq\" (UID: \"06d654a9-d7c3-4b45-8bcc-abfad4c49970\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq" Oct 30 17:25:15.011113 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:15.011090 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwl69\" (UniqueName: \"kubernetes.io/projected/06d654a9-d7c3-4b45-8bcc-abfad4c49970-kube-api-access-qwl69\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq\" (UID: \"06d654a9-d7c3-4b45-8bcc-abfad4c49970\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq" Oct 30 17:25:15.082815 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:15.082783 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq" Oct 30 17:25:15.414548 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:25:15.414519 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06d654a9_d7c3_4b45_8bcc_abfad4c49970.slice/crio-17e1b0060e50944d33d43d9d4284a696e33758cd9817359e09eab9c069e91602 WatchSource:0}: Error finding container 17e1b0060e50944d33d43d9d4284a696e33758cd9817359e09eab9c069e91602: Status 404 returned error can't find the container with id 17e1b0060e50944d33d43d9d4284a696e33758cd9817359e09eab9c069e91602 Oct 30 17:25:15.416800 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:15.416774 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq"] Oct 30 17:25:15.417220 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:15.417189 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:25:15.417272 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:15.417257 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:25:15.417327 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:15.417288 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:25:16.066341 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:16.065849 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq" event={"ID":"06d654a9-d7c3-4b45-8bcc-abfad4c49970","Type":"ContainerStarted","Data":"be25d5e29c60fb7663216b9c320b6eb4d8d0e1b02e79d41efaf3a572b1552977"} Oct 30 17:25:16.066341 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:16.065921 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq" event={"ID":"06d654a9-d7c3-4b45-8bcc-abfad4c49970","Type":"ContainerStarted","Data":"17e1b0060e50944d33d43d9d4284a696e33758cd9817359e09eab9c069e91602"} Oct 30 17:25:16.085772 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:16.085486 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq" Oct 30 17:25:16.085772 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:16.085612 4522 patch_prober.go:29] interesting pod/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body= Oct 30 17:25:16.085772 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:16.085657 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq" podUID="06d654a9-d7c3-4b45-8bcc-abfad4c49970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" Oct 30 17:25:16.152445 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:16.152383 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq" podStartSLOduration=2.152365814 podStartE2EDuration="2.152365814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:25:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:25:16.14899252 +0000 UTC m=+2290.149747422" watchObservedRunningTime="2026-10-30 17:25:16.152365814 +0000 UTC m=+2290.153120722" Oct 30 17:25:16.639705 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:16.639668 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e5ac8f-98c6fb647-mlcx8" podUID="35026397-efc0-4b45-b548-01ce4d2ac26e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:25:17.083863 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:17.083830 4522 patch_prober.go:29] interesting pod/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body= Oct 30 17:25:17.084049 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:17.083882 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq" podUID="06d654a9-d7c3-4b45-8bcc-abfad4c49970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" Oct 30 17:25:18.083172 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:18.083137 4522 patch_prober.go:29] interesting pod/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body= Oct 30 17:25:18.083617 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:18.083186 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq" podUID="06d654a9-d7c3-4b45-8bcc-abfad4c49970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" Oct 30 17:25:19.083429 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:19.083338 4522 patch_prober.go:29] interesting pod/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body= Oct 30 17:25:19.083429 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:19.083399 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq" podUID="06d654a9-d7c3-4b45-8bcc-abfad4c49970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" Oct 30 17:25:19.358404 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:19.312664 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-42d381-85f9dccd5f-fkl4k" podUID="9710d6d6-5a90-4b5d-b7fe-5835b6323136" containerName="payload-pre-processing" containerID="cri-o://57380519ba24aab22c522722eae8974a4907ec55450e7d27c9fe9b979e774459" gracePeriod=25 Oct 30 17:25:19.431965 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:19.431934 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e5ac8f-98c6fb647-mlcx8" Oct 30 17:25:19.651383 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:19.650956 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35026397-efc0-4b45-b548-01ce4d2ac26e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35026397-efc0-4b45-b548-01ce4d2ac26e-plugins-config-volume\") pod \"35026397-efc0-4b45-b548-01ce4d2ac26e\" (UID: \"35026397-efc0-4b45-b548-01ce4d2ac26e\") " Oct 30 17:25:19.651383 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:19.651105 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35026397-efc0-4b45-b548-01ce4d2ac26e-kube-api-access-cc54c\" (UniqueName: \"kubernetes.io/projected/35026397-efc0-4b45-b548-01ce4d2ac26e-kube-api-access-cc54c\") pod \"35026397-efc0-4b45-b548-01ce4d2ac26e\" (UID: \"35026397-efc0-4b45-b548-01ce4d2ac26e\") " Oct 30 17:25:19.659935 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:19.653794 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35026397-efc0-4b45-b548-01ce4d2ac26e-plugins-config-volume" pod "35026397-efc0-4b45-b548-01ce4d2ac26e" (UID: "35026397-efc0-4b45-b548-01ce4d2ac26e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:25:19.672343 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:19.669521 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35026397-efc0-4b45-b548-01ce4d2ac26e-kube-api-access-cc54c" pod "35026397-efc0-4b45-b548-01ce4d2ac26e" (UID: "35026397-efc0-4b45-b548-01ce4d2ac26e"). InnerVolumeSpecName "kube-api-access-cc54c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:25:19.753941 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:19.753709 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35026397-efc0-4b45-b548-01ce4d2ac26e-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:25:19.753941 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:19.753787 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cc54c\" (UniqueName: \"kubernetes.io/projected/35026397-efc0-4b45-b548-01ce4d2ac26e-kube-api-access-cc54c\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:25:20.083996 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:20.083964 4522 patch_prober.go:29] interesting pod/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body= Oct 30 17:25:20.084451 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:20.084419 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq" podUID="06d654a9-d7c3-4b45-8bcc-abfad4c49970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" Oct 30 17:25:20.127446 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:20.127412 4522 generic.go:356] "Generic (PLEG): container finished" podID="35026397-efc0-4b45-b548-01ce4d2ac26e" containerID="08fbfa4e57ab86881fde7563462c59344b02946269addf1e9bf72520ced2f371" exitCode=0 Oct 30 17:25:20.127666 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:20.127465 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e5ac8f-98c6fb647-mlcx8" Oct 30 17:25:20.127666 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:20.127498 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e5ac8f-98c6fb647-mlcx8" event={"ID":"35026397-efc0-4b45-b548-01ce4d2ac26e","Type":"ContainerDied","Data":"08fbfa4e57ab86881fde7563462c59344b02946269addf1e9bf72520ced2f371"} Oct 30 17:25:20.127666 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:20.127566 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e5ac8f-98c6fb647-mlcx8" event={"ID":"35026397-efc0-4b45-b548-01ce4d2ac26e","Type":"ContainerDied","Data":"830fc9c467af787afb4c157c4af3754200e101516c9d4d72c2f2826f04fc0466"} Oct 30 17:25:20.127666 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:20.127598 4522 scope.go:122] "RemoveContainer" containerID="08fbfa4e57ab86881fde7563462c59344b02946269addf1e9bf72520ced2f371" Oct 30 17:25:20.140105 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:20.140087 4522 scope.go:122] "RemoveContainer" containerID="08fbfa4e57ab86881fde7563462c59344b02946269addf1e9bf72520ced2f371" Oct 30 17:25:20.140423 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:25:20.140399 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08fbfa4e57ab86881fde7563462c59344b02946269addf1e9bf72520ced2f371\": container with ID starting with 08fbfa4e57ab86881fde7563462c59344b02946269addf1e9bf72520ced2f371 not found: ID does not exist" containerID="08fbfa4e57ab86881fde7563462c59344b02946269addf1e9bf72520ced2f371" Oct 30 17:25:20.140507 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:20.140430 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08fbfa4e57ab86881fde7563462c59344b02946269addf1e9bf72520ced2f371"} err="failed to get container status \"08fbfa4e57ab86881fde7563462c59344b02946269addf1e9bf72520ced2f371\": rpc error: code = NotFound desc = could not find container \"08fbfa4e57ab86881fde7563462c59344b02946269addf1e9bf72520ced2f371\": container with ID starting with 08fbfa4e57ab86881fde7563462c59344b02946269addf1e9bf72520ced2f371 not found: ID does not exist" Oct 30 17:25:20.177417 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:20.177379 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e5ac8f-lcmfr"] Oct 30 17:25:20.232355 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:20.193792 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e5ac8f-lcmfr"] Oct 30 17:25:20.237138 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:20.237101 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e5ac8f-98c6fb647-mlcx8"] Oct 30 17:25:20.245988 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:20.245955 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e5ac8f-98c6fb647-mlcx8"] Oct 30 17:25:21.083830 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:21.083793 4522 patch_prober.go:29] interesting pod/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body= Oct 30 17:25:21.084022 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:21.083851 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq" podUID="06d654a9-d7c3-4b45-8bcc-abfad4c49970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" Oct 30 17:25:22.083930 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:22.083886 4522 patch_prober.go:29] interesting pod/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body= Oct 30 17:25:22.084129 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:22.083952 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq" podUID="06d654a9-d7c3-4b45-8bcc-abfad4c49970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" Oct 30 17:25:22.154850 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:22.154809 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35026397-efc0-4b45-b548-01ce4d2ac26e" path="/var/lib/kubelet/pods/35026397-efc0-4b45-b548-01ce4d2ac26e/volumes" Oct 30 17:25:22.155482 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:22.155363 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d77de96-a093-4b65-b38a-6b66002dfd59" path="/var/lib/kubelet/pods/7d77de96-a093-4b65-b38a-6b66002dfd59/volumes" Oct 30 17:25:23.089190 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:23.089158 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq" Oct 30 17:25:23.089656 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:23.089643 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq" Oct 30 17:25:23.090474 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:23.090450 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq" Oct 30 17:25:26.051136 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:26.051099 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-42d381-85f9dccd5f-fkl4k" podUID="9710d6d6-5a90-4b5d-b7fe-5835b6323136" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:25:36.051404 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:36.051368 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-42d381-85f9dccd5f-fkl4k" podUID="9710d6d6-5a90-4b5d-b7fe-5835b6323136" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:25:40.066082 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:40.066054 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-42d381-85f9dccd5f-fkl4k" Oct 30 17:25:40.165391 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:40.165269 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9710d6d6-5a90-4b5d-b7fe-5835b6323136-kube-api-access-rfwvx\" (UniqueName: \"kubernetes.io/projected/9710d6d6-5a90-4b5d-b7fe-5835b6323136-kube-api-access-rfwvx\") pod \"9710d6d6-5a90-4b5d-b7fe-5835b6323136\" (UID: \"9710d6d6-5a90-4b5d-b7fe-5835b6323136\") " Oct 30 17:25:40.165579 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:40.165558 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9710d6d6-5a90-4b5d-b7fe-5835b6323136-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9710d6d6-5a90-4b5d-b7fe-5835b6323136-plugins-config-volume\") pod \"9710d6d6-5a90-4b5d-b7fe-5835b6323136\" (UID: \"9710d6d6-5a90-4b5d-b7fe-5835b6323136\") " Oct 30 17:25:40.165899 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:40.165877 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9710d6d6-5a90-4b5d-b7fe-5835b6323136-plugins-config-volume" pod "9710d6d6-5a90-4b5d-b7fe-5835b6323136" (UID: "9710d6d6-5a90-4b5d-b7fe-5835b6323136"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:25:40.166177 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:40.166157 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9710d6d6-5a90-4b5d-b7fe-5835b6323136-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:25:40.167535 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:40.167514 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9710d6d6-5a90-4b5d-b7fe-5835b6323136-kube-api-access-rfwvx" pod "9710d6d6-5a90-4b5d-b7fe-5835b6323136" (UID: "9710d6d6-5a90-4b5d-b7fe-5835b6323136"). InnerVolumeSpecName "kube-api-access-rfwvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:25:40.268283 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:40.268252 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfwvx\" (UniqueName: \"kubernetes.io/projected/9710d6d6-5a90-4b5d-b7fe-5835b6323136-kube-api-access-rfwvx\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:25:40.345073 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:40.345037 4522 generic.go:356] "Generic (PLEG): container finished" podID="9710d6d6-5a90-4b5d-b7fe-5835b6323136" containerID="57380519ba24aab22c522722eae8974a4907ec55450e7d27c9fe9b979e774459" exitCode=0 Oct 30 17:25:40.345272 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:40.345088 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-42d381-85f9dccd5f-fkl4k" Oct 30 17:25:40.345272 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:40.345123 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-42d381-85f9dccd5f-fkl4k" event={"ID":"9710d6d6-5a90-4b5d-b7fe-5835b6323136","Type":"ContainerDied","Data":"57380519ba24aab22c522722eae8974a4907ec55450e7d27c9fe9b979e774459"} Oct 30 17:25:40.345272 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:40.345184 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-42d381-85f9dccd5f-fkl4k" event={"ID":"9710d6d6-5a90-4b5d-b7fe-5835b6323136","Type":"ContainerDied","Data":"f1f078a6fb778018a58bb7a0963ba3c6e9783f264cc912f30036a651b65ebf36"} Oct 30 17:25:40.345272 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:40.345208 4522 scope.go:122] "RemoveContainer" containerID="57380519ba24aab22c522722eae8974a4907ec55450e7d27c9fe9b979e774459" Oct 30 17:25:40.356972 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:40.356948 4522 scope.go:122] "RemoveContainer" containerID="57380519ba24aab22c522722eae8974a4907ec55450e7d27c9fe9b979e774459" Oct 30 17:25:40.357266 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:25:40.357241 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57380519ba24aab22c522722eae8974a4907ec55450e7d27c9fe9b979e774459\": container with ID starting with 57380519ba24aab22c522722eae8974a4907ec55450e7d27c9fe9b979e774459 not found: ID does not exist" containerID="57380519ba24aab22c522722eae8974a4907ec55450e7d27c9fe9b979e774459" Oct 30 17:25:40.357361 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:40.357279 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57380519ba24aab22c522722eae8974a4907ec55450e7d27c9fe9b979e774459"} err="failed to get container status \"57380519ba24aab22c522722eae8974a4907ec55450e7d27c9fe9b979e774459\": rpc error: code = NotFound desc = could not find container \"57380519ba24aab22c522722eae8974a4907ec55450e7d27c9fe9b979e774459\": container with ID starting with 57380519ba24aab22c522722eae8974a4907ec55450e7d27c9fe9b979e774459 not found: ID does not exist" Oct 30 17:25:40.408794 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:40.408761 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-42d381-85f9dccd5f-fkl4k"] Oct 30 17:25:40.421502 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:40.421432 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-42d381-85f9dccd5f-fkl4k"] Oct 30 17:25:42.148012 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:42.147975 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9710d6d6-5a90-4b5d-b7fe-5835b6323136" path="/var/lib/kubelet/pods/9710d6d6-5a90-4b5d-b7fe-5835b6323136/volumes" Oct 30 17:25:42.951160 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:25:42.951121 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:25:43.847737 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:43.847705 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-42d381-qb5sp"] Oct 30 17:25:43.862546 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:43.862507 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-42d381-qb5sp"] Oct 30 17:25:44.146575 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:44.146502 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="522c89f3-afd3-4819-9024-220111395373" path="/var/lib/kubelet/pods/522c89f3-afd3-4819-9024-220111395373/volumes" Oct 30 17:25:48.038717 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:48.038679 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbd06511-gw24m"] Oct 30 17:25:48.043155 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:48.043127 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbd06511-gw24m"] Oct 30 17:25:48.048381 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:48.048357 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bf44438-kpfmd"] Oct 30 17:25:48.055012 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:48.054989 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bf44438-kpfmd"] Oct 30 17:25:48.147122 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:48.147089 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ec65b68-1b8a-49f0-aca2-3c9fd0fcaa56" path="/var/lib/kubelet/pods/7ec65b68-1b8a-49f0-aca2-3c9fd0fcaa56/volumes" Oct 30 17:25:48.147653 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:48.147637 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6196f49-61cf-45eb-bf47-21621604e662" path="/var/lib/kubelet/pods/e6196f49-61cf-45eb-bf47-21621604e662/volumes" Oct 30 17:25:51.038449 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:51.038407 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7"] Oct 30 17:25:51.040125 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:51.040071 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7" podUID="eb91a3a6-25a0-4c9b-9564-8221f24efa30" containerName="istio-proxy" containerID="cri-o://6d84f39c8ce66309ebc1baa1a57661e955e8f2e7f098d192d1a008b7c0c50443" gracePeriod=30 Oct 30 17:25:51.750036 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:51.750001 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c45877-kncn2"] Oct 30 17:25:51.750742 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:51.750726 4522 cpu_manager.go:414] "containerMap: removing container" podUID="35026397-efc0-4b45-b548-01ce4d2ac26e" containerName="payload-pre-processing" Oct 30 17:25:51.750807 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:51.750744 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35026397-efc0-4b45-b548-01ce4d2ac26e" containerName="payload-pre-processing" Oct 30 17:25:51.750807 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:51.750768 4522 cpu_manager.go:414] "containerMap: removing container" podUID="9710d6d6-5a90-4b5d-b7fe-5835b6323136" containerName="payload-pre-processing" Oct 30 17:25:51.750807 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:51.750774 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9710d6d6-5a90-4b5d-b7fe-5835b6323136" containerName="payload-pre-processing" Oct 30 17:25:51.750914 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:51.750905 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="35026397-efc0-4b45-b548-01ce4d2ac26e" containerName="payload-pre-processing" Oct 30 17:25:51.750949 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:51.750919 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="9710d6d6-5a90-4b5d-b7fe-5835b6323136" containerName="payload-pre-processing" Oct 30 17:25:51.756088 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:51.756071 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c45877-kncn2" Oct 30 17:25:51.775677 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:51.775647 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c45877-kncn2"] Oct 30 17:25:51.837943 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:51.837909 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:25:51.838274 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:51.838255 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:25:51.876903 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:51.876860 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7354dfa5-d620-41cd-ab0b-d754e15235cd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c45877-kncn2\" (UID: \"7354dfa5-d620-41cd-ab0b-d754e15235cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c45877-kncn2" Oct 30 17:25:51.877114 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:51.877093 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7354dfa5-d620-41cd-ab0b-d754e15235cd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c45877-kncn2\" (UID: \"7354dfa5-d620-41cd-ab0b-d754e15235cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c45877-kncn2" Oct 30 17:25:51.877176 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:51.877142 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqlsz\" (UniqueName: \"kubernetes.io/projected/7354dfa5-d620-41cd-ab0b-d754e15235cd-kube-api-access-rqlsz\") pod \"maas-api-revoke-keys-e2e-inf-b-c45877-kncn2\" (UID: \"7354dfa5-d620-41cd-ab0b-d754e15235cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c45877-kncn2" Oct 30 17:25:51.979207 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:51.979165 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7354dfa5-d620-41cd-ab0b-d754e15235cd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c45877-kncn2\" (UID: \"7354dfa5-d620-41cd-ab0b-d754e15235cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c45877-kncn2" Oct 30 17:25:51.979422 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:51.979372 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7354dfa5-d620-41cd-ab0b-d754e15235cd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c45877-kncn2\" (UID: \"7354dfa5-d620-41cd-ab0b-d754e15235cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c45877-kncn2" Oct 30 17:25:51.979501 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:51.979425 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqlsz\" (UniqueName: \"kubernetes.io/projected/7354dfa5-d620-41cd-ab0b-d754e15235cd-kube-api-access-rqlsz\") pod \"maas-api-revoke-keys-e2e-inf-b-c45877-kncn2\" (UID: \"7354dfa5-d620-41cd-ab0b-d754e15235cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c45877-kncn2" Oct 30 17:25:51.980097 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:51.980067 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7354dfa5-d620-41cd-ab0b-d754e15235cd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c45877-kncn2\" (UID: \"7354dfa5-d620-41cd-ab0b-d754e15235cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c45877-kncn2" Oct 30 17:25:51.983690 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:51.983666 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7354dfa5-d620-41cd-ab0b-d754e15235cd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c45877-kncn2\" (UID: \"7354dfa5-d620-41cd-ab0b-d754e15235cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c45877-kncn2" Oct 30 17:25:52.004692 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:52.004624 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqlsz\" (UniqueName: \"kubernetes.io/projected/7354dfa5-d620-41cd-ab0b-d754e15235cd-kube-api-access-rqlsz\") pod \"maas-api-revoke-keys-e2e-inf-b-c45877-kncn2\" (UID: \"7354dfa5-d620-41cd-ab0b-d754e15235cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c45877-kncn2" Oct 30 17:25:52.070615 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:52.070577 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c45877-kncn2" Oct 30 17:25:52.383713 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:25:52.383673 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7354dfa5_d620_41cd_ab0b_d754e15235cd.slice/crio-2d7d3f4f84fcf154b47f49f9836930438fac1e0f83f843fe1486fb9ea65025ec WatchSource:0}: Error finding container 2d7d3f4f84fcf154b47f49f9836930438fac1e0f83f843fe1486fb9ea65025ec: Status 404 returned error can't find the container with id 2d7d3f4f84fcf154b47f49f9836930438fac1e0f83f843fe1486fb9ea65025ec Oct 30 17:25:52.386350 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:52.386323 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c45877-kncn2"] Oct 30 17:25:52.456918 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:52.456881 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c45877-kncn2" event={"ID":"7354dfa5-d620-41cd-ab0b-d754e15235cd","Type":"ContainerStarted","Data":"2d7d3f4f84fcf154b47f49f9836930438fac1e0f83f843fe1486fb9ea65025ec"} Oct 30 17:25:53.476656 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:53.476622 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c45877-kncn2" event={"ID":"7354dfa5-d620-41cd-ab0b-d754e15235cd","Type":"ContainerStarted","Data":"338dc6ef06443409e504a4e8f56126f8300f866ec5c80f28c15bd34aa3083ad3"} Oct 30 17:25:53.521587 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:53.521538 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c45877-kncn2" podStartSLOduration=2.521524715 podStartE2EDuration="2.521524715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:25:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:25:53.520197264 +0000 UTC m=+2327.520952189" watchObservedRunningTime="2026-10-30 17:25:53.521524715 +0000 UTC m=+2327.522279616" Oct 30 17:25:54.196950 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:54.195033 4522 patch_prober.go:29] interesting pod/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:25:54.196950 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:54.195093 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7" podUID="eb91a3a6-25a0-4c9b-9564-8221f24efa30" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:25:54.652143 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:54.652108 4522 generic.go:356] "Generic (PLEG): container finished" podID="7354dfa5-d620-41cd-ab0b-d754e15235cd" containerID="338dc6ef06443409e504a4e8f56126f8300f866ec5c80f28c15bd34aa3083ad3" exitCode=0 Oct 30 17:25:54.652892 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:54.652869 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c45877-kncn2" event={"ID":"7354dfa5-d620-41cd-ab0b-d754e15235cd","Type":"ContainerDied","Data":"338dc6ef06443409e504a4e8f56126f8300f866ec5c80f28c15bd34aa3083ad3"} Oct 30 17:25:55.878171 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:55.878141 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c45877-kncn2" Oct 30 17:25:56.046557 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:56.046519 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7354dfa5-d620-41cd-ab0b-d754e15235cd-kube-api-access-rqlsz\" (UniqueName: \"kubernetes.io/projected/7354dfa5-d620-41cd-ab0b-d754e15235cd-kube-api-access-rqlsz\") pod \"7354dfa5-d620-41cd-ab0b-d754e15235cd\" (UID: \"7354dfa5-d620-41cd-ab0b-d754e15235cd\") " Oct 30 17:25:56.046756 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:56.046623 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7354dfa5-d620-41cd-ab0b-d754e15235cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7354dfa5-d620-41cd-ab0b-d754e15235cd-maas-api-xks-ca\") pod \"7354dfa5-d620-41cd-ab0b-d754e15235cd\" (UID: \"7354dfa5-d620-41cd-ab0b-d754e15235cd\") " Oct 30 17:25:56.046756 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:56.046693 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7354dfa5-d620-41cd-ab0b-d754e15235cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7354dfa5-d620-41cd-ab0b-d754e15235cd-maas-api-openshift-ca\") pod \"7354dfa5-d620-41cd-ab0b-d754e15235cd\" (UID: \"7354dfa5-d620-41cd-ab0b-d754e15235cd\") " Oct 30 17:25:56.047140 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:56.047105 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7354dfa5-d620-41cd-ab0b-d754e15235cd-maas-api-openshift-ca" pod "7354dfa5-d620-41cd-ab0b-d754e15235cd" (UID: "7354dfa5-d620-41cd-ab0b-d754e15235cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:25:56.047429 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:56.047411 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7354dfa5-d620-41cd-ab0b-d754e15235cd-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:25:56.048841 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:56.048814 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7354dfa5-d620-41cd-ab0b-d754e15235cd-maas-api-xks-ca" pod "7354dfa5-d620-41cd-ab0b-d754e15235cd" (UID: "7354dfa5-d620-41cd-ab0b-d754e15235cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:25:56.048915 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:56.048854 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7354dfa5-d620-41cd-ab0b-d754e15235cd-kube-api-access-rqlsz" pod "7354dfa5-d620-41cd-ab0b-d754e15235cd" (UID: "7354dfa5-d620-41cd-ab0b-d754e15235cd"). InnerVolumeSpecName "kube-api-access-rqlsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:25:56.148709 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:56.148671 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqlsz\" (UniqueName: \"kubernetes.io/projected/7354dfa5-d620-41cd-ab0b-d754e15235cd-kube-api-access-rqlsz\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:25:56.148709 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:56.148697 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7354dfa5-d620-41cd-ab0b-d754e15235cd-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:25:56.336401 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:56.336374 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7" Oct 30 17:25:56.454054 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:56.454020 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb91a3a6-25a0-4c9b-9564-8221f24efa30-kube-api-access-fvkf5\" (UniqueName: \"kubernetes.io/projected/eb91a3a6-25a0-4c9b-9564-8221f24efa30-kube-api-access-fvkf5\") pod \"eb91a3a6-25a0-4c9b-9564-8221f24efa30\" (UID: \"eb91a3a6-25a0-4c9b-9564-8221f24efa30\") " Oct 30 17:25:56.454260 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:56.454065 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb91a3a6-25a0-4c9b-9564-8221f24efa30-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb91a3a6-25a0-4c9b-9564-8221f24efa30-workload-certs\") pod \"eb91a3a6-25a0-4c9b-9564-8221f24efa30\" (UID: \"eb91a3a6-25a0-4c9b-9564-8221f24efa30\") " Oct 30 17:25:56.454260 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:56.454118 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb91a3a6-25a0-4c9b-9564-8221f24efa30-istio-token\" (UniqueName: \"kubernetes.io/projected/eb91a3a6-25a0-4c9b-9564-8221f24efa30-istio-token\") pod \"eb91a3a6-25a0-4c9b-9564-8221f24efa30\" (UID: \"eb91a3a6-25a0-4c9b-9564-8221f24efa30\") " Oct 30 17:25:56.454260 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:56.454174 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb91a3a6-25a0-4c9b-9564-8221f24efa30-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb91a3a6-25a0-4c9b-9564-8221f24efa30-istio-podinfo\") pod \"eb91a3a6-25a0-4c9b-9564-8221f24efa30\" (UID: \"eb91a3a6-25a0-4c9b-9564-8221f24efa30\") " Oct 30 17:25:56.454260 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:56.454200 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb91a3a6-25a0-4c9b-9564-8221f24efa30-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb91a3a6-25a0-4c9b-9564-8221f24efa30-istio-envoy\") pod \"eb91a3a6-25a0-4c9b-9564-8221f24efa30\" (UID: \"eb91a3a6-25a0-4c9b-9564-8221f24efa30\") " Oct 30 17:25:56.454535 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:56.454361 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb91a3a6-25a0-4c9b-9564-8221f24efa30-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb91a3a6-25a0-4c9b-9564-8221f24efa30-istiod-ca-cert\") pod \"eb91a3a6-25a0-4c9b-9564-8221f24efa30\" (UID: \"eb91a3a6-25a0-4c9b-9564-8221f24efa30\") " Oct 30 17:25:56.454535 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:56.454463 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb91a3a6-25a0-4c9b-9564-8221f24efa30-workload-certs" pod "eb91a3a6-25a0-4c9b-9564-8221f24efa30" (UID: "eb91a3a6-25a0-4c9b-9564-8221f24efa30"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:25:56.454535 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:56.454492 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb91a3a6-25a0-4c9b-9564-8221f24efa30-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb91a3a6-25a0-4c9b-9564-8221f24efa30-credential-socket\") pod \"eb91a3a6-25a0-4c9b-9564-8221f24efa30\" (UID: \"eb91a3a6-25a0-4c9b-9564-8221f24efa30\") " Oct 30 17:25:56.454728 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:56.454547 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb91a3a6-25a0-4c9b-9564-8221f24efa30-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb91a3a6-25a0-4c9b-9564-8221f24efa30-istio-data\") pod \"eb91a3a6-25a0-4c9b-9564-8221f24efa30\" (UID: \"eb91a3a6-25a0-4c9b-9564-8221f24efa30\") " Oct 30 17:25:56.454728 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:56.454623 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb91a3a6-25a0-4c9b-9564-8221f24efa30-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb91a3a6-25a0-4c9b-9564-8221f24efa30-workload-socket\") pod \"eb91a3a6-25a0-4c9b-9564-8221f24efa30\" (UID: \"eb91a3a6-25a0-4c9b-9564-8221f24efa30\") " Oct 30 17:25:56.454843 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:56.454733 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb91a3a6-25a0-4c9b-9564-8221f24efa30-credential-socket" pod "eb91a3a6-25a0-4c9b-9564-8221f24efa30" (UID: "eb91a3a6-25a0-4c9b-9564-8221f24efa30"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:25:56.454843 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:56.454743 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb91a3a6-25a0-4c9b-9564-8221f24efa30-istiod-ca-cert" pod "eb91a3a6-25a0-4c9b-9564-8221f24efa30" (UID: "eb91a3a6-25a0-4c9b-9564-8221f24efa30"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:25:56.454843 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:56.454819 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb91a3a6-25a0-4c9b-9564-8221f24efa30-istio-data" pod "eb91a3a6-25a0-4c9b-9564-8221f24efa30" (UID: "eb91a3a6-25a0-4c9b-9564-8221f24efa30"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:25:56.455092 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:56.455067 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb91a3a6-25a0-4c9b-9564-8221f24efa30-workload-socket" pod "eb91a3a6-25a0-4c9b-9564-8221f24efa30" (UID: "eb91a3a6-25a0-4c9b-9564-8221f24efa30"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:25:56.455449 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:56.455426 4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb91a3a6-25a0-4c9b-9564-8221f24efa30-istiod-ca-cert\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:25:56.455553 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:56.455453 4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb91a3a6-25a0-4c9b-9564-8221f24efa30-credential-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:25:56.455553 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:56.455469 4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb91a3a6-25a0-4c9b-9564-8221f24efa30-istio-data\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:25:56.455553 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:56.455483 4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb91a3a6-25a0-4c9b-9564-8221f24efa30-workload-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:25:56.455553 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:56.455498 4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb91a3a6-25a0-4c9b-9564-8221f24efa30-workload-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:25:56.456712 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:56.456684 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb91a3a6-25a0-4c9b-9564-8221f24efa30-istio-podinfo" pod "eb91a3a6-25a0-4c9b-9564-8221f24efa30" (UID: "eb91a3a6-25a0-4c9b-9564-8221f24efa30"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:25:56.456831 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:56.456744 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb91a3a6-25a0-4c9b-9564-8221f24efa30-istio-token" pod "eb91a3a6-25a0-4c9b-9564-8221f24efa30" (UID: "eb91a3a6-25a0-4c9b-9564-8221f24efa30"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:25:56.456831 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:56.456773 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb91a3a6-25a0-4c9b-9564-8221f24efa30-kube-api-access-fvkf5" pod "eb91a3a6-25a0-4c9b-9564-8221f24efa30" (UID: "eb91a3a6-25a0-4c9b-9564-8221f24efa30"). InnerVolumeSpecName "kube-api-access-fvkf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:25:56.456926 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:56.456884 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb91a3a6-25a0-4c9b-9564-8221f24efa30-istio-envoy" pod "eb91a3a6-25a0-4c9b-9564-8221f24efa30" (UID: "eb91a3a6-25a0-4c9b-9564-8221f24efa30"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:25:56.556999 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:56.556967 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvkf5\" (UniqueName: \"kubernetes.io/projected/eb91a3a6-25a0-4c9b-9564-8221f24efa30-kube-api-access-fvkf5\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:25:56.556999 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:56.556996 4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb91a3a6-25a0-4c9b-9564-8221f24efa30-istio-token\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:25:56.557209 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:56.557010 4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb91a3a6-25a0-4c9b-9564-8221f24efa30-istio-podinfo\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:25:56.557209 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:56.557023 4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb91a3a6-25a0-4c9b-9564-8221f24efa30-istio-envoy\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:25:56.673878 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:56.673800 4522 generic.go:356] "Generic (PLEG): container finished" podID="eb91a3a6-25a0-4c9b-9564-8221f24efa30" containerID="6d84f39c8ce66309ebc1baa1a57661e955e8f2e7f098d192d1a008b7c0c50443" exitCode=0 Oct 30 17:25:56.673878 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:56.673852 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7" Oct 30 17:25:56.674087 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:56.673877 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7" event={"ID":"eb91a3a6-25a0-4c9b-9564-8221f24efa30","Type":"ContainerDied","Data":"6d84f39c8ce66309ebc1baa1a57661e955e8f2e7f098d192d1a008b7c0c50443"} Oct 30 17:25:56.674087 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:56.673931 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7" event={"ID":"eb91a3a6-25a0-4c9b-9564-8221f24efa30","Type":"ContainerDied","Data":"5b7ecc8ff20f732d9d7620bbf154d0fb2eb3ff44d0b2392791953c4aef64562d"} Oct 30 17:25:56.674087 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:56.673955 4522 scope.go:122] "RemoveContainer" containerID="6d84f39c8ce66309ebc1baa1a57661e955e8f2e7f098d192d1a008b7c0c50443" Oct 30 17:25:56.675672 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:56.675653 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c45877-kncn2" event={"ID":"7354dfa5-d620-41cd-ab0b-d754e15235cd","Type":"ContainerDied","Data":"2d7d3f4f84fcf154b47f49f9836930438fac1e0f83f843fe1486fb9ea65025ec"} Oct 30 17:25:56.675771 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:56.675688 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d7d3f4f84fcf154b47f49f9836930438fac1e0f83f843fe1486fb9ea65025ec" Oct 30 17:25:56.675771 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:56.675659 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c45877-kncn2" Oct 30 17:25:56.686600 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:56.686579 4522 scope.go:122] "RemoveContainer" containerID="67a9433870f1ece7c693445db5d4e4ec6ac14a3d8448ef0a3cfcfc178b256f36" Oct 30 17:25:56.698522 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:56.698483 4522 scope.go:122] "RemoveContainer" containerID="6d84f39c8ce66309ebc1baa1a57661e955e8f2e7f098d192d1a008b7c0c50443" Oct 30 17:25:56.698849 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:25:56.698827 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d84f39c8ce66309ebc1baa1a57661e955e8f2e7f098d192d1a008b7c0c50443\": container with ID starting with 6d84f39c8ce66309ebc1baa1a57661e955e8f2e7f098d192d1a008b7c0c50443 not found: ID does not exist" containerID="6d84f39c8ce66309ebc1baa1a57661e955e8f2e7f098d192d1a008b7c0c50443" Oct 30 17:25:56.698952 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:56.698855 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d84f39c8ce66309ebc1baa1a57661e955e8f2e7f098d192d1a008b7c0c50443"} err="failed to get container status \"6d84f39c8ce66309ebc1baa1a57661e955e8f2e7f098d192d1a008b7c0c50443\": rpc error: code = NotFound desc = could not find container \"6d84f39c8ce66309ebc1baa1a57661e955e8f2e7f098d192d1a008b7c0c50443\": container with ID starting with 6d84f39c8ce66309ebc1baa1a57661e955e8f2e7f098d192d1a008b7c0c50443 not found: ID does not exist" Oct 30 17:25:56.698952 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:56.698873 4522 scope.go:122] "RemoveContainer" containerID="67a9433870f1ece7c693445db5d4e4ec6ac14a3d8448ef0a3cfcfc178b256f36" Oct 30 17:25:56.699176 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:25:56.699155 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67a9433870f1ece7c693445db5d4e4ec6ac14a3d8448ef0a3cfcfc178b256f36\": container with ID starting with 67a9433870f1ece7c693445db5d4e4ec6ac14a3d8448ef0a3cfcfc178b256f36 not found: ID does not exist" containerID="67a9433870f1ece7c693445db5d4e4ec6ac14a3d8448ef0a3cfcfc178b256f36" Oct 30 17:25:56.699244 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:56.699182 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67a9433870f1ece7c693445db5d4e4ec6ac14a3d8448ef0a3cfcfc178b256f36"} err="failed to get container status \"67a9433870f1ece7c693445db5d4e4ec6ac14a3d8448ef0a3cfcfc178b256f36\": rpc error: code = NotFound desc = could not find container \"67a9433870f1ece7c693445db5d4e4ec6ac14a3d8448ef0a3cfcfc178b256f36\": container with ID starting with 67a9433870f1ece7c693445db5d4e4ec6ac14a3d8448ef0a3cfcfc178b256f36 not found: ID does not exist" Oct 30 17:25:56.770890 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:56.770855 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7"] Oct 30 17:25:56.775276 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:56.775244 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-7rdv7"] Oct 30 17:25:58.147159 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:25:58.147124 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb91a3a6-25a0-4c9b-9564-8221f24efa30" path="/var/lib/kubelet/pods/eb91a3a6-25a0-4c9b-9564-8221f24efa30/volumes" Oct 30 17:26:00.442271 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:00.442236 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d5f5a1e-n8wrb"] Oct 30 17:26:00.443016 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:00.442998 4522 cpu_manager.go:414] "containerMap: removing container" podUID="eb91a3a6-25a0-4c9b-9564-8221f24efa30" containerName="istio-proxy" Oct 30 17:26:00.443016 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:00.443016 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb91a3a6-25a0-4c9b-9564-8221f24efa30" containerName="istio-proxy" Oct 30 17:26:00.443156 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:00.443037 4522 cpu_manager.go:414] "containerMap: removing container" podUID="eb91a3a6-25a0-4c9b-9564-8221f24efa30" containerName="istio-proxy" Oct 30 17:26:00.443156 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:00.443043 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb91a3a6-25a0-4c9b-9564-8221f24efa30" containerName="istio-proxy" Oct 30 17:26:00.443156 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:00.443065 4522 cpu_manager.go:414] "containerMap: removing container" podUID="7354dfa5-d620-41cd-ab0b-d754e15235cd" containerName="revoke-keys" Oct 30 17:26:00.443156 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:00.443070 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7354dfa5-d620-41cd-ab0b-d754e15235cd" containerName="revoke-keys" Oct 30 17:26:00.443401 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:00.443177 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="7354dfa5-d620-41cd-ab0b-d754e15235cd" containerName="revoke-keys" Oct 30 17:26:00.443401 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:00.443193 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb91a3a6-25a0-4c9b-9564-8221f24efa30" containerName="istio-proxy" Oct 30 17:26:00.443401 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:00.443200 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb91a3a6-25a0-4c9b-9564-8221f24efa30" containerName="istio-proxy" Oct 30 17:26:00.448900 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:00.448880 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5f5a1e-n8wrb" Oct 30 17:26:00.469991 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:00.469960 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d5f5a1e-n8wrb"] Oct 30 17:26:00.477331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:00.476981 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:26:00.477331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:00.477026 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:26:00.628779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:00.628748 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eea2f1a1-5fe2-4cf3-a643-12e22b0c69ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d5f5a1e-n8wrb\" (UID: \"eea2f1a1-5fe2-4cf3-a643-12e22b0c69ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5f5a1e-n8wrb" Oct 30 17:26:00.629027 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:00.629002 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8qbj\" (UniqueName: \"kubernetes.io/projected/eea2f1a1-5fe2-4cf3-a643-12e22b0c69ca-kube-api-access-d8qbj\") pod \"maas-api-revoke-sub-2d5f5a1e-n8wrb\" (UID: \"eea2f1a1-5fe2-4cf3-a643-12e22b0c69ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5f5a1e-n8wrb" Oct 30 17:26:00.629114 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:00.629096 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eea2f1a1-5fe2-4cf3-a643-12e22b0c69ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d5f5a1e-n8wrb\" (UID: \"eea2f1a1-5fe2-4cf3-a643-12e22b0c69ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5f5a1e-n8wrb" Oct 30 17:26:00.730413 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:00.730381 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8qbj\" (UniqueName: \"kubernetes.io/projected/eea2f1a1-5fe2-4cf3-a643-12e22b0c69ca-kube-api-access-d8qbj\") pod \"maas-api-revoke-sub-2d5f5a1e-n8wrb\" (UID: \"eea2f1a1-5fe2-4cf3-a643-12e22b0c69ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5f5a1e-n8wrb" Oct 30 17:26:00.730607 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:00.730420 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eea2f1a1-5fe2-4cf3-a643-12e22b0c69ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d5f5a1e-n8wrb\" (UID: \"eea2f1a1-5fe2-4cf3-a643-12e22b0c69ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5f5a1e-n8wrb" Oct 30 17:26:00.730607 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:00.730516 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eea2f1a1-5fe2-4cf3-a643-12e22b0c69ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d5f5a1e-n8wrb\" (UID: \"eea2f1a1-5fe2-4cf3-a643-12e22b0c69ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5f5a1e-n8wrb" Oct 30 17:26:00.731232 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:00.731202 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eea2f1a1-5fe2-4cf3-a643-12e22b0c69ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d5f5a1e-n8wrb\" (UID: \"eea2f1a1-5fe2-4cf3-a643-12e22b0c69ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5f5a1e-n8wrb" Oct 30 17:26:00.733023 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:00.733001 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eea2f1a1-5fe2-4cf3-a643-12e22b0c69ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d5f5a1e-n8wrb\" (UID: \"eea2f1a1-5fe2-4cf3-a643-12e22b0c69ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5f5a1e-n8wrb" Oct 30 17:26:00.757155 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:00.757124 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8qbj\" (UniqueName: \"kubernetes.io/projected/eea2f1a1-5fe2-4cf3-a643-12e22b0c69ca-kube-api-access-d8qbj\") pod \"maas-api-revoke-sub-2d5f5a1e-n8wrb\" (UID: \"eea2f1a1-5fe2-4cf3-a643-12e22b0c69ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5f5a1e-n8wrb" Oct 30 17:26:00.764042 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:00.764017 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5f5a1e-n8wrb" Oct 30 17:26:01.076167 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:26:01.076131 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeea2f1a1_5fe2_4cf3_a643_12e22b0c69ca.slice/crio-52dd2ec43f2c9a0df81eff1c9ed24d632faedd6b38f9193492e682f7592a39d7 WatchSource:0}: Error finding container 52dd2ec43f2c9a0df81eff1c9ed24d632faedd6b38f9193492e682f7592a39d7: Status 404 returned error can't find the container with id 52dd2ec43f2c9a0df81eff1c9ed24d632faedd6b38f9193492e682f7592a39d7 Oct 30 17:26:01.078586 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:01.078557 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d5f5a1e-n8wrb"] Oct 30 17:26:01.745455 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:01.745417 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5f5a1e-n8wrb" event={"ID":"eea2f1a1-5fe2-4cf3-a643-12e22b0c69ca","Type":"ContainerStarted","Data":"969248328e2cccc45171c4e22cf438f16d1e4902170b4798aceac0f547959c13"} Oct 30 17:26:01.746111 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:01.745504 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5f5a1e-n8wrb" event={"ID":"eea2f1a1-5fe2-4cf3-a643-12e22b0c69ca","Type":"ContainerStarted","Data":"52dd2ec43f2c9a0df81eff1c9ed24d632faedd6b38f9193492e682f7592a39d7"} Oct 30 17:26:01.792150 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:01.792089 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5f5a1e-n8wrb" podStartSLOduration=1.792073694 podStartE2EDuration="1.792073694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:26:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:26:01.789701905 +0000 UTC m=+2335.790456824" watchObservedRunningTime="2026-10-30 17:26:01.792073694 +0000 UTC m=+2335.792828594" Oct 30 17:26:01.888316 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:01.888280 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c45877-787d5595b7-8mhf5"] Oct 30 17:26:02.595359 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:02.593042 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6144f753-gfncd"] Oct 30 17:26:02.602793 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:02.602761 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6144f753-gfncd" Oct 30 17:26:02.664965 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:02.664910 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6144f753-gfncd"] Oct 30 17:26:02.755488 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:02.755451 4522 generic.go:356] "Generic (PLEG): container finished" podID="eea2f1a1-5fe2-4cf3-a643-12e22b0c69ca" containerID="969248328e2cccc45171c4e22cf438f16d1e4902170b4798aceac0f547959c13" exitCode=0 Oct 30 17:26:02.755943 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:02.755543 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5f5a1e-n8wrb" event={"ID":"eea2f1a1-5fe2-4cf3-a643-12e22b0c69ca","Type":"ContainerDied","Data":"969248328e2cccc45171c4e22cf438f16d1e4902170b4798aceac0f547959c13"} Oct 30 17:26:02.769384 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:02.769353 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a5af931-bbe4-4516-b808-86ca7efb37ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6144f753-gfncd\" (UID: \"7a5af931-bbe4-4516-b808-86ca7efb37ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6144f753-gfncd" Oct 30 17:26:02.769541 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:02.769511 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j88ft\" (UniqueName: \"kubernetes.io/projected/7a5af931-bbe4-4516-b808-86ca7efb37ad-kube-api-access-j88ft\") pod \"maas-api-revoke-sub-6144f753-gfncd\" (UID: \"7a5af931-bbe4-4516-b808-86ca7efb37ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6144f753-gfncd" Oct 30 17:26:02.769681 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:02.769664 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a5af931-bbe4-4516-b808-86ca7efb37ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6144f753-gfncd\" (UID: \"7a5af931-bbe4-4516-b808-86ca7efb37ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6144f753-gfncd" Oct 30 17:26:02.871771 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:02.871678 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a5af931-bbe4-4516-b808-86ca7efb37ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6144f753-gfncd\" (UID: \"7a5af931-bbe4-4516-b808-86ca7efb37ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6144f753-gfncd" Oct 30 17:26:02.871916 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:02.871820 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j88ft\" (UniqueName: \"kubernetes.io/projected/7a5af931-bbe4-4516-b808-86ca7efb37ad-kube-api-access-j88ft\") pod \"maas-api-revoke-sub-6144f753-gfncd\" (UID: \"7a5af931-bbe4-4516-b808-86ca7efb37ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6144f753-gfncd" Oct 30 17:26:02.871969 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:02.871948 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a5af931-bbe4-4516-b808-86ca7efb37ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6144f753-gfncd\" (UID: \"7a5af931-bbe4-4516-b808-86ca7efb37ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6144f753-gfncd" Oct 30 17:26:02.872495 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:02.872459 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a5af931-bbe4-4516-b808-86ca7efb37ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6144f753-gfncd\" (UID: \"7a5af931-bbe4-4516-b808-86ca7efb37ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6144f753-gfncd" Oct 30 17:26:02.874533 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:02.874507 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a5af931-bbe4-4516-b808-86ca7efb37ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6144f753-gfncd\" (UID: \"7a5af931-bbe4-4516-b808-86ca7efb37ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6144f753-gfncd" Oct 30 17:26:02.897268 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:02.897240 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j88ft\" (UniqueName: \"kubernetes.io/projected/7a5af931-bbe4-4516-b808-86ca7efb37ad-kube-api-access-j88ft\") pod \"maas-api-revoke-sub-6144f753-gfncd\" (UID: \"7a5af931-bbe4-4516-b808-86ca7efb37ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6144f753-gfncd" Oct 30 17:26:02.946441 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:02.946410 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6144f753-gfncd" Oct 30 17:26:03.265203 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:26:03.265158 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a5af931_bbe4_4516_b808_86ca7efb37ad.slice/crio-cb23d411ff9cbe8da8db9e0acb5142fcb0d833385c069552fcce85c293abd76f WatchSource:0}: Error finding container cb23d411ff9cbe8da8db9e0acb5142fcb0d833385c069552fcce85c293abd76f: Status 404 returned error can't find the container with id cb23d411ff9cbe8da8db9e0acb5142fcb0d833385c069552fcce85c293abd76f Oct 30 17:26:03.267385 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:03.267359 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6144f753-gfncd"] Oct 30 17:26:03.765730 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:03.765692 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6144f753-gfncd" event={"ID":"7a5af931-bbe4-4516-b808-86ca7efb37ad","Type":"ContainerStarted","Data":"1ecbce7b32cec1d9a0765cbb1b184dbd7938601f874e5a38efd39cac2a1a1d39"} Oct 30 17:26:03.766192 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:03.765758 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6144f753-gfncd" event={"ID":"7a5af931-bbe4-4516-b808-86ca7efb37ad","Type":"ContainerStarted","Data":"cb23d411ff9cbe8da8db9e0acb5142fcb0d833385c069552fcce85c293abd76f"} Oct 30 17:26:03.812447 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:03.812387 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6144f753-gfncd" podStartSLOduration=1.812366753 podStartE2EDuration="1.812366753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:26:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:26:03.811160114 +0000 UTC m=+2337.811915033" watchObservedRunningTime="2026-10-30 17:26:03.812366753 +0000 UTC m=+2337.813121655" Oct 30 17:26:03.940801 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:03.940773 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5f5a1e-n8wrb" Oct 30 17:26:03.991290 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:03.991253 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eea2f1a1-5fe2-4cf3-a643-12e22b0c69ca-kube-api-access-d8qbj\" (UniqueName: \"kubernetes.io/projected/eea2f1a1-5fe2-4cf3-a643-12e22b0c69ca-kube-api-access-d8qbj\") pod \"eea2f1a1-5fe2-4cf3-a643-12e22b0c69ca\" (UID: \"eea2f1a1-5fe2-4cf3-a643-12e22b0c69ca\") " Oct 30 17:26:03.991491 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:03.991469 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eea2f1a1-5fe2-4cf3-a643-12e22b0c69ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eea2f1a1-5fe2-4cf3-a643-12e22b0c69ca-maas-api-openshift-ca\") pod \"eea2f1a1-5fe2-4cf3-a643-12e22b0c69ca\" (UID: \"eea2f1a1-5fe2-4cf3-a643-12e22b0c69ca\") " Oct 30 17:26:03.991575 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:03.991558 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eea2f1a1-5fe2-4cf3-a643-12e22b0c69ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eea2f1a1-5fe2-4cf3-a643-12e22b0c69ca-maas-api-xks-ca\") pod \"eea2f1a1-5fe2-4cf3-a643-12e22b0c69ca\" (UID: \"eea2f1a1-5fe2-4cf3-a643-12e22b0c69ca\") " Oct 30 17:26:03.991898 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:03.991868 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eea2f1a1-5fe2-4cf3-a643-12e22b0c69ca-maas-api-openshift-ca" pod "eea2f1a1-5fe2-4cf3-a643-12e22b0c69ca" (UID: "eea2f1a1-5fe2-4cf3-a643-12e22b0c69ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:26:03.992260 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:03.992233 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eea2f1a1-5fe2-4cf3-a643-12e22b0c69ca-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:26:03.993595 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:03.993565 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eea2f1a1-5fe2-4cf3-a643-12e22b0c69ca-kube-api-access-d8qbj" pod "eea2f1a1-5fe2-4cf3-a643-12e22b0c69ca" (UID: "eea2f1a1-5fe2-4cf3-a643-12e22b0c69ca"). InnerVolumeSpecName "kube-api-access-d8qbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:26:03.993689 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:03.993665 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eea2f1a1-5fe2-4cf3-a643-12e22b0c69ca-maas-api-xks-ca" pod "eea2f1a1-5fe2-4cf3-a643-12e22b0c69ca" (UID: "eea2f1a1-5fe2-4cf3-a643-12e22b0c69ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:26:04.094105 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:04.094064 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eea2f1a1-5fe2-4cf3-a643-12e22b0c69ca-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:26:04.094105 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:04.094094 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8qbj\" (UniqueName: \"kubernetes.io/projected/eea2f1a1-5fe2-4cf3-a643-12e22b0c69ca-kube-api-access-d8qbj\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:26:04.776343 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:04.776295 4522 generic.go:356] "Generic (PLEG): container finished" podID="7a5af931-bbe4-4516-b808-86ca7efb37ad" containerID="1ecbce7b32cec1d9a0765cbb1b184dbd7938601f874e5a38efd39cac2a1a1d39" exitCode=0 Oct 30 17:26:04.776786 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:04.776375 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6144f753-gfncd" event={"ID":"7a5af931-bbe4-4516-b808-86ca7efb37ad","Type":"ContainerDied","Data":"1ecbce7b32cec1d9a0765cbb1b184dbd7938601f874e5a38efd39cac2a1a1d39"} Oct 30 17:26:04.777982 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:04.777965 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5f5a1e-n8wrb" Oct 30 17:26:04.778081 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:04.778021 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5f5a1e-n8wrb" event={"ID":"eea2f1a1-5fe2-4cf3-a643-12e22b0c69ca","Type":"ContainerDied","Data":"52dd2ec43f2c9a0df81eff1c9ed24d632faedd6b38f9193492e682f7592a39d7"} Oct 30 17:26:04.778081 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:04.778060 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52dd2ec43f2c9a0df81eff1c9ed24d632faedd6b38f9193492e682f7592a39d7" Oct 30 17:26:05.952925 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:05.952898 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6144f753-gfncd" Oct 30 17:26:06.023122 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:06.023091 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a5af931-bbe4-4516-b808-86ca7efb37ad-kube-api-access-j88ft\" (UniqueName: \"kubernetes.io/projected/7a5af931-bbe4-4516-b808-86ca7efb37ad-kube-api-access-j88ft\") pod \"7a5af931-bbe4-4516-b808-86ca7efb37ad\" (UID: \"7a5af931-bbe4-4516-b808-86ca7efb37ad\") " Oct 30 17:26:06.023297 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:06.023251 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a5af931-bbe4-4516-b808-86ca7efb37ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a5af931-bbe4-4516-b808-86ca7efb37ad-maas-api-xks-ca\") pod \"7a5af931-bbe4-4516-b808-86ca7efb37ad\" (UID: \"7a5af931-bbe4-4516-b808-86ca7efb37ad\") " Oct 30 17:26:06.023457 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:06.023441 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a5af931-bbe4-4516-b808-86ca7efb37ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a5af931-bbe4-4516-b808-86ca7efb37ad-maas-api-openshift-ca\") pod \"7a5af931-bbe4-4516-b808-86ca7efb37ad\" (UID: \"7a5af931-bbe4-4516-b808-86ca7efb37ad\") " Oct 30 17:26:06.023832 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:06.023803 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a5af931-bbe4-4516-b808-86ca7efb37ad-maas-api-openshift-ca" pod "7a5af931-bbe4-4516-b808-86ca7efb37ad" (UID: "7a5af931-bbe4-4516-b808-86ca7efb37ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:26:06.024074 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:06.024056 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a5af931-bbe4-4516-b808-86ca7efb37ad-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:26:06.031465 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:06.031437 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a5af931-bbe4-4516-b808-86ca7efb37ad-maas-api-xks-ca" pod "7a5af931-bbe4-4516-b808-86ca7efb37ad" (UID: "7a5af931-bbe4-4516-b808-86ca7efb37ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:26:06.032110 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:06.032084 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a5af931-bbe4-4516-b808-86ca7efb37ad-kube-api-access-j88ft" pod "7a5af931-bbe4-4516-b808-86ca7efb37ad" (UID: "7a5af931-bbe4-4516-b808-86ca7efb37ad"). InnerVolumeSpecName "kube-api-access-j88ft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:26:06.125491 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:06.125453 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j88ft\" (UniqueName: \"kubernetes.io/projected/7a5af931-bbe4-4516-b808-86ca7efb37ad-kube-api-access-j88ft\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:26:06.125491 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:06.125488 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a5af931-bbe4-4516-b808-86ca7efb37ad-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:26:06.797820 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:06.797792 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6144f753-gfncd" Oct 30 17:26:06.798006 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:06.797836 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6144f753-gfncd" event={"ID":"7a5af931-bbe4-4516-b808-86ca7efb37ad","Type":"ContainerDied","Data":"cb23d411ff9cbe8da8db9e0acb5142fcb0d833385c069552fcce85c293abd76f"} Oct 30 17:26:06.798006 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:06.797894 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb23d411ff9cbe8da8db9e0acb5142fcb0d833385c069552fcce85c293abd76f" Oct 30 17:26:06.889396 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:06.889345 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c45877-787d5595b7-8mhf5" podUID="ab382b04-4fd2-4172-bc26-8f10347da05c" containerName="payload-pre-processing" containerID="cri-o://9570aca6851573754f290ca5602901a159ee6aa98b9281b81dd46851fe49567e" gracePeriod=25 Oct 30 17:26:09.309404 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:09.309362 4522 scope.go:122] "RemoveContainer" containerID="18792109e2ce3e9b48b0bb3564c835bfe6b384f2fb0209b1a911de94311485b7" Oct 30 17:26:09.320655 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:09.320633 4522 scope.go:122] "RemoveContainer" containerID="a96de25733c9705950c19e3cf107472e4de3b4c502ecbab04d85c017126e5918" Oct 30 17:26:10.774538 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:10.774502 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d2f4ecb-wfqwc"] Oct 30 17:26:10.775358 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:10.775335 4522 cpu_manager.go:414] "containerMap: removing container" podUID="eea2f1a1-5fe2-4cf3-a643-12e22b0c69ca" containerName="revoke-keys" Oct 30 17:26:10.775459 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:10.775358 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eea2f1a1-5fe2-4cf3-a643-12e22b0c69ca" containerName="revoke-keys" Oct 30 17:26:10.775459 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:10.775415 4522 cpu_manager.go:414] "containerMap: removing container" podUID="7a5af931-bbe4-4516-b808-86ca7efb37ad" containerName="revoke-keys" Oct 30 17:26:10.775459 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:10.775421 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a5af931-bbe4-4516-b808-86ca7efb37ad" containerName="revoke-keys" Oct 30 17:26:10.775584 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:10.775574 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a5af931-bbe4-4516-b808-86ca7efb37ad" containerName="revoke-keys" Oct 30 17:26:10.775623 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:10.775589 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="eea2f1a1-5fe2-4cf3-a643-12e22b0c69ca" containerName="revoke-keys" Oct 30 17:26:10.781782 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:10.781760 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d2f4ecb-wfqwc" Oct 30 17:26:10.798319 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:10.798273 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d2f4ecb-wfqwc"] Oct 30 17:26:10.805911 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:10.805879 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:26:10.806117 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:10.805979 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:26:10.921979 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:10.921946 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/418d079d-aedd-408e-9d92-b354789e72fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d2f4ecb-wfqwc\" (UID: \"418d079d-aedd-408e-9d92-b354789e72fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d2f4ecb-wfqwc" Oct 30 17:26:10.922222 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:10.922129 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nggpk\" (UniqueName: \"kubernetes.io/projected/418d079d-aedd-408e-9d92-b354789e72fc-kube-api-access-nggpk\") pod \"maas-api-revoke-sub-8d2f4ecb-wfqwc\" (UID: \"418d079d-aedd-408e-9d92-b354789e72fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d2f4ecb-wfqwc" Oct 30 17:26:10.922394 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:10.922375 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/418d079d-aedd-408e-9d92-b354789e72fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d2f4ecb-wfqwc\" (UID: \"418d079d-aedd-408e-9d92-b354789e72fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d2f4ecb-wfqwc" Oct 30 17:26:11.024111 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:11.024074 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/418d079d-aedd-408e-9d92-b354789e72fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d2f4ecb-wfqwc\" (UID: \"418d079d-aedd-408e-9d92-b354789e72fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d2f4ecb-wfqwc" Oct 30 17:26:11.024315 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:11.024196 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nggpk\" (UniqueName: \"kubernetes.io/projected/418d079d-aedd-408e-9d92-b354789e72fc-kube-api-access-nggpk\") pod \"maas-api-revoke-sub-8d2f4ecb-wfqwc\" (UID: \"418d079d-aedd-408e-9d92-b354789e72fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d2f4ecb-wfqwc" Oct 30 17:26:11.024446 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:11.024426 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/418d079d-aedd-408e-9d92-b354789e72fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d2f4ecb-wfqwc\" (UID: \"418d079d-aedd-408e-9d92-b354789e72fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d2f4ecb-wfqwc" Oct 30 17:26:11.025097 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:11.025039 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/418d079d-aedd-408e-9d92-b354789e72fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d2f4ecb-wfqwc\" (UID: \"418d079d-aedd-408e-9d92-b354789e72fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d2f4ecb-wfqwc" Oct 30 17:26:11.026567 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:11.026549 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/418d079d-aedd-408e-9d92-b354789e72fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d2f4ecb-wfqwc\" (UID: \"418d079d-aedd-408e-9d92-b354789e72fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d2f4ecb-wfqwc" Oct 30 17:26:11.049240 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:11.049208 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nggpk\" (UniqueName: \"kubernetes.io/projected/418d079d-aedd-408e-9d92-b354789e72fc-kube-api-access-nggpk\") pod \"maas-api-revoke-sub-8d2f4ecb-wfqwc\" (UID: \"418d079d-aedd-408e-9d92-b354789e72fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d2f4ecb-wfqwc" Oct 30 17:26:11.097500 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:11.097468 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d2f4ecb-wfqwc" Oct 30 17:26:11.410128 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:26:11.410074 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod418d079d_aedd_408e_9d92_b354789e72fc.slice/crio-2d2c95beb440372a92287924d4adf46f5b2bba4e9f9d03466d6af94b8ec2825a WatchSource:0}: Error finding container 2d2c95beb440372a92287924d4adf46f5b2bba4e9f9d03466d6af94b8ec2825a: Status 404 returned error can't find the container with id 2d2c95beb440372a92287924d4adf46f5b2bba4e9f9d03466d6af94b8ec2825a Oct 30 17:26:11.412665 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:11.412150 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d2f4ecb-wfqwc"] Oct 30 17:26:11.849963 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:11.849930 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d2f4ecb-wfqwc" event={"ID":"418d079d-aedd-408e-9d92-b354789e72fc","Type":"ContainerStarted","Data":"f26db7a2a5a0c1026063ed919bedccbd2b5e524e47512553dfd4dd02d1e37a99"} Oct 30 17:26:11.850382 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:11.849992 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d2f4ecb-wfqwc" event={"ID":"418d079d-aedd-408e-9d92-b354789e72fc","Type":"ContainerStarted","Data":"2d2c95beb440372a92287924d4adf46f5b2bba4e9f9d03466d6af94b8ec2825a"} Oct 30 17:26:11.918148 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:11.918092 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d2f4ecb-wfqwc" podStartSLOduration=1.918076669 podStartE2EDuration="1.918076669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:26:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:26:11.912790976 +0000 UTC m=+2345.913545876" watchObservedRunningTime="2026-10-30 17:26:11.918076669 +0000 UTC m=+2345.918831571" Oct 30 17:26:12.695744 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:12.695656 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c45877-787d5595b7-8mhf5" podUID="ab382b04-4fd2-4172-bc26-8f10347da05c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:26:12.859905 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:12.859871 4522 generic.go:356] "Generic (PLEG): container finished" podID="418d079d-aedd-408e-9d92-b354789e72fc" containerID="f26db7a2a5a0c1026063ed919bedccbd2b5e524e47512553dfd4dd02d1e37a99" exitCode=0 Oct 30 17:26:12.860435 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:12.859934 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d2f4ecb-wfqwc" event={"ID":"418d079d-aedd-408e-9d92-b354789e72fc","Type":"ContainerDied","Data":"f26db7a2a5a0c1026063ed919bedccbd2b5e524e47512553dfd4dd02d1e37a99"} Oct 30 17:26:12.949354 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:12.949246 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60c5652f-xch6m"] Oct 30 17:26:12.954154 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:12.954130 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c5652f-xch6m" Oct 30 17:26:12.957921 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:26:12.957886 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:26:12.968412 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:12.968377 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60c5652f-xch6m"] Oct 30 17:26:13.055210 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:13.055171 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcgpg\" (UniqueName: \"kubernetes.io/projected/f4afca48-3a36-4718-a83f-314a67da8061-kube-api-access-dcgpg\") pod \"maas-api-revoke-sub-60c5652f-xch6m\" (UID: \"f4afca48-3a36-4718-a83f-314a67da8061\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c5652f-xch6m" Oct 30 17:26:13.055424 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:13.055404 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4afca48-3a36-4718-a83f-314a67da8061-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60c5652f-xch6m\" (UID: \"f4afca48-3a36-4718-a83f-314a67da8061\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c5652f-xch6m" Oct 30 17:26:13.055534 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:13.055519 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4afca48-3a36-4718-a83f-314a67da8061-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60c5652f-xch6m\" (UID: \"f4afca48-3a36-4718-a83f-314a67da8061\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c5652f-xch6m" Oct 30 17:26:13.162298 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:13.162249 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4afca48-3a36-4718-a83f-314a67da8061-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60c5652f-xch6m\" (UID: \"f4afca48-3a36-4718-a83f-314a67da8061\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c5652f-xch6m" Oct 30 17:26:13.162576 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:13.162415 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4afca48-3a36-4718-a83f-314a67da8061-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60c5652f-xch6m\" (UID: \"f4afca48-3a36-4718-a83f-314a67da8061\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c5652f-xch6m" Oct 30 17:26:13.162576 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:13.162569 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcgpg\" (UniqueName: \"kubernetes.io/projected/f4afca48-3a36-4718-a83f-314a67da8061-kube-api-access-dcgpg\") pod \"maas-api-revoke-sub-60c5652f-xch6m\" (UID: \"f4afca48-3a36-4718-a83f-314a67da8061\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c5652f-xch6m" Oct 30 17:26:13.163812 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:13.163775 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4afca48-3a36-4718-a83f-314a67da8061-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60c5652f-xch6m\" (UID: \"f4afca48-3a36-4718-a83f-314a67da8061\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c5652f-xch6m" Oct 30 17:26:13.175241 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:13.175204 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4afca48-3a36-4718-a83f-314a67da8061-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60c5652f-xch6m\" (UID: \"f4afca48-3a36-4718-a83f-314a67da8061\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c5652f-xch6m" Oct 30 17:26:13.187806 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:13.187775 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcgpg\" (UniqueName: \"kubernetes.io/projected/f4afca48-3a36-4718-a83f-314a67da8061-kube-api-access-dcgpg\") pod \"maas-api-revoke-sub-60c5652f-xch6m\" (UID: \"f4afca48-3a36-4718-a83f-314a67da8061\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c5652f-xch6m" Oct 30 17:26:13.280717 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:13.276759 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c5652f-xch6m" Oct 30 17:26:13.612992 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:26:13.612965 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4afca48_3a36_4718_a83f_314a67da8061.slice/crio-f5b13425184b077c4da742d44ca994854b3d11a10246e349aaed197f192f51ef WatchSource:0}: Error finding container f5b13425184b077c4da742d44ca994854b3d11a10246e349aaed197f192f51ef: Status 404 returned error can't find the container with id f5b13425184b077c4da742d44ca994854b3d11a10246e349aaed197f192f51ef Oct 30 17:26:13.614889 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:13.614866 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60c5652f-xch6m"] Oct 30 17:26:13.876833 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:13.876745 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c5652f-xch6m" event={"ID":"f4afca48-3a36-4718-a83f-314a67da8061","Type":"ContainerStarted","Data":"f5b13425184b077c4da742d44ca994854b3d11a10246e349aaed197f192f51ef"} Oct 30 17:26:14.016620 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:14.016592 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d2f4ecb-wfqwc" Oct 30 17:26:14.176486 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:14.176404 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/418d079d-aedd-408e-9d92-b354789e72fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/418d079d-aedd-408e-9d92-b354789e72fc-maas-api-openshift-ca\") pod \"418d079d-aedd-408e-9d92-b354789e72fc\" (UID: \"418d079d-aedd-408e-9d92-b354789e72fc\") " Oct 30 17:26:14.176642 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:14.176583 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/418d079d-aedd-408e-9d92-b354789e72fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/418d079d-aedd-408e-9d92-b354789e72fc-maas-api-xks-ca\") pod \"418d079d-aedd-408e-9d92-b354789e72fc\" (UID: \"418d079d-aedd-408e-9d92-b354789e72fc\") " Oct 30 17:26:14.176709 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:14.176691 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/418d079d-aedd-408e-9d92-b354789e72fc-kube-api-access-nggpk\" (UniqueName: \"kubernetes.io/projected/418d079d-aedd-408e-9d92-b354789e72fc-kube-api-access-nggpk\") pod \"418d079d-aedd-408e-9d92-b354789e72fc\" (UID: \"418d079d-aedd-408e-9d92-b354789e72fc\") " Oct 30 17:26:14.176821 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:14.176801 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/418d079d-aedd-408e-9d92-b354789e72fc-maas-api-openshift-ca" pod "418d079d-aedd-408e-9d92-b354789e72fc" (UID: "418d079d-aedd-408e-9d92-b354789e72fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:26:14.177283 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:14.177263 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/418d079d-aedd-408e-9d92-b354789e72fc-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:26:14.178812 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:14.178785 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/418d079d-aedd-408e-9d92-b354789e72fc-maas-api-xks-ca" pod "418d079d-aedd-408e-9d92-b354789e72fc" (UID: "418d079d-aedd-408e-9d92-b354789e72fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:26:14.178906 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:14.178870 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/418d079d-aedd-408e-9d92-b354789e72fc-kube-api-access-nggpk" pod "418d079d-aedd-408e-9d92-b354789e72fc" (UID: "418d079d-aedd-408e-9d92-b354789e72fc"). InnerVolumeSpecName "kube-api-access-nggpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:26:14.279945 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:14.279162 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/418d079d-aedd-408e-9d92-b354789e72fc-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:26:14.279945 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:14.279195 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nggpk\" (UniqueName: \"kubernetes.io/projected/418d079d-aedd-408e-9d92-b354789e72fc-kube-api-access-nggpk\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:26:14.888056 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:14.888023 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d2f4ecb-wfqwc" Oct 30 17:26:14.888608 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:14.888070 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d2f4ecb-wfqwc" event={"ID":"418d079d-aedd-408e-9d92-b354789e72fc","Type":"ContainerDied","Data":"2d2c95beb440372a92287924d4adf46f5b2bba4e9f9d03466d6af94b8ec2825a"} Oct 30 17:26:14.888608 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:14.888158 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d2c95beb440372a92287924d4adf46f5b2bba4e9f9d03466d6af94b8ec2825a" Oct 30 17:26:14.890056 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:14.890030 4522 generic.go:356] "Generic (PLEG): container finished" podID="f4afca48-3a36-4718-a83f-314a67da8061" containerID="f1701401ed6d9598cd6669dc3fa3369cebe01b01e9db40fc027dd8272841ba51" exitCode=0 Oct 30 17:26:14.890199 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:14.890105 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c5652f-xch6m" event={"ID":"f4afca48-3a36-4718-a83f-314a67da8061","Type":"ContainerDied","Data":"f1701401ed6d9598cd6669dc3fa3369cebe01b01e9db40fc027dd8272841ba51"} Oct 30 17:26:16.057821 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:16.057794 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c5652f-xch6m" Oct 30 17:26:16.104845 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:16.104808 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4afca48-3a36-4718-a83f-314a67da8061-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4afca48-3a36-4718-a83f-314a67da8061-maas-api-openshift-ca\") pod \"f4afca48-3a36-4718-a83f-314a67da8061\" (UID: \"f4afca48-3a36-4718-a83f-314a67da8061\") " Oct 30 17:26:16.105077 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:16.105029 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4afca48-3a36-4718-a83f-314a67da8061-kube-api-access-dcgpg\" (UniqueName: \"kubernetes.io/projected/f4afca48-3a36-4718-a83f-314a67da8061-kube-api-access-dcgpg\") pod \"f4afca48-3a36-4718-a83f-314a67da8061\" (UID: \"f4afca48-3a36-4718-a83f-314a67da8061\") " Oct 30 17:26:16.105168 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:16.105148 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4afca48-3a36-4718-a83f-314a67da8061-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4afca48-3a36-4718-a83f-314a67da8061-maas-api-xks-ca\") pod \"f4afca48-3a36-4718-a83f-314a67da8061\" (UID: \"f4afca48-3a36-4718-a83f-314a67da8061\") " Oct 30 17:26:16.107380 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:16.105696 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4afca48-3a36-4718-a83f-314a67da8061-maas-api-openshift-ca" pod "f4afca48-3a36-4718-a83f-314a67da8061" (UID: "f4afca48-3a36-4718-a83f-314a67da8061"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:26:16.107380 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:16.106002 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4afca48-3a36-4718-a83f-314a67da8061-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:26:16.110347 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:16.108073 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4afca48-3a36-4718-a83f-314a67da8061-kube-api-access-dcgpg" pod "f4afca48-3a36-4718-a83f-314a67da8061" (UID: "f4afca48-3a36-4718-a83f-314a67da8061"). InnerVolumeSpecName "kube-api-access-dcgpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:26:16.110347 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:16.108104 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4afca48-3a36-4718-a83f-314a67da8061-maas-api-xks-ca" pod "f4afca48-3a36-4718-a83f-314a67da8061" (UID: "f4afca48-3a36-4718-a83f-314a67da8061"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:26:16.207268 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:16.207228 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcgpg\" (UniqueName: \"kubernetes.io/projected/f4afca48-3a36-4718-a83f-314a67da8061-kube-api-access-dcgpg\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:26:16.207268 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:16.207260 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4afca48-3a36-4718-a83f-314a67da8061-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:26:16.908593 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:16.908564 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c5652f-xch6m" Oct 30 17:26:16.908807 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:16.908600 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c5652f-xch6m" event={"ID":"f4afca48-3a36-4718-a83f-314a67da8061","Type":"ContainerDied","Data":"f5b13425184b077c4da742d44ca994854b3d11a10246e349aaed197f192f51ef"} Oct 30 17:26:16.908807 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:16.908665 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5b13425184b077c4da742d44ca994854b3d11a10246e349aaed197f192f51ef" Oct 30 17:26:17.164911 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:17.164831 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc"] Oct 30 17:26:17.165625 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:17.165609 4522 cpu_manager.go:414] "containerMap: removing container" podUID="f4afca48-3a36-4718-a83f-314a67da8061" containerName="revoke-keys" Oct 30 17:26:17.165698 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:17.165626 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4afca48-3a36-4718-a83f-314a67da8061" containerName="revoke-keys" Oct 30 17:26:17.165698 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:17.165645 4522 cpu_manager.go:414] "containerMap: removing container" podUID="418d079d-aedd-408e-9d92-b354789e72fc" containerName="revoke-keys" Oct 30 17:26:17.165698 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:17.165651 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="418d079d-aedd-408e-9d92-b354789e72fc" containerName="revoke-keys" Oct 30 17:26:17.165806 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:17.165796 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4afca48-3a36-4718-a83f-314a67da8061" containerName="revoke-keys" Oct 30 17:26:17.165841 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:17.165811 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="418d079d-aedd-408e-9d92-b354789e72fc" containerName="revoke-keys" Oct 30 17:26:17.169709 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:17.169690 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc" Oct 30 17:26:17.200436 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:17.200392 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-6d4f99-openshift-default-dockercfg-chllj\"" Oct 30 17:26:17.209046 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:17.209011 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc"] Oct 30 17:26:17.225342 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:17.225290 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7-credential-socket\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc\" (UID: \"a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc" Oct 30 17:26:17.225543 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:17.225440 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7-istio-envoy\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc\" (UID: \"a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc" Oct 30 17:26:17.225613 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:17.225538 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7-istio-token\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc\" (UID: \"a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc" Oct 30 17:26:17.225613 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:17.225593 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7-istio-data\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc\" (UID: \"a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc" Oct 30 17:26:17.225762 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:17.225737 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7-workload-socket\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc\" (UID: \"a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc" Oct 30 17:26:17.226217 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:17.226196 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7-workload-certs\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc\" (UID: \"a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc" Oct 30 17:26:17.226372 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:17.226225 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q827x\" (UniqueName: \"kubernetes.io/projected/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7-kube-api-access-q827x\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc\" (UID: \"a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc" Oct 30 17:26:17.226372 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:17.226242 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7-istiod-ca-cert\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc\" (UID: \"a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc" Oct 30 17:26:17.226372 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:17.226299 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7-istio-podinfo\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc\" (UID: \"a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc" Oct 30 17:26:17.328028 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:17.327992 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7-istio-podinfo\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc\" (UID: \"a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc" Oct 30 17:26:17.328244 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:17.328076 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7-credential-socket\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc\" (UID: \"a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc" Oct 30 17:26:17.328244 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:17.328135 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7-istio-envoy\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc\" (UID: \"a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc" Oct 30 17:26:17.328244 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:17.328185 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7-istio-token\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc\" (UID: \"a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc" Oct 30 17:26:17.328244 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:17.328216 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7-istio-data\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc\" (UID: \"a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc" Oct 30 17:26:17.328542 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:17.328321 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7-workload-socket\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc\" (UID: \"a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc" Oct 30 17:26:17.328542 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:17.328392 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7-workload-certs\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc\" (UID: \"a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc" Oct 30 17:26:17.328542 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:17.328422 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q827x\" (UniqueName: \"kubernetes.io/projected/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7-kube-api-access-q827x\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc\" (UID: \"a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc" Oct 30 17:26:17.328542 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:17.328448 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7-istiod-ca-cert\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc\" (UID: \"a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc" Oct 30 17:26:17.328772 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:17.328569 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7-credential-socket\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc\" (UID: \"a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc" Oct 30 17:26:17.328835 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:17.328825 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7-workload-certs\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc\" (UID: \"a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc" Oct 30 17:26:17.328896 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:17.328839 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7-workload-socket\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc\" (UID: \"a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc" Oct 30 17:26:17.329050 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:17.328998 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7-istio-data\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc\" (UID: \"a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc" Oct 30 17:26:17.329194 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:17.329151 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7-istiod-ca-cert\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc\" (UID: \"a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc" Oct 30 17:26:17.330708 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:17.330684 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7-istio-envoy\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc\" (UID: \"a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc" Oct 30 17:26:17.330890 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:17.330868 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7-istio-podinfo\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc\" (UID: \"a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc" Oct 30 17:26:17.353184 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:17.353154 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7-istio-token\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc\" (UID: \"a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc" Oct 30 17:26:17.353354 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:17.353235 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q827x\" (UniqueName: \"kubernetes.io/projected/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7-kube-api-access-q827x\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc\" (UID: \"a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc" Oct 30 17:26:17.483790 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:17.483754 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc" Oct 30 17:26:17.811390 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:26:17.811356 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2f5cba4_9318_4bd1_ba06_f77f17ca1ae7.slice/crio-5ccd40da17c1de11e972b06de2e3ad98335e25bf69def48367ba50ab37779357 WatchSource:0}: Error finding container 5ccd40da17c1de11e972b06de2e3ad98335e25bf69def48367ba50ab37779357: Status 404 returned error can't find the container with id 5ccd40da17c1de11e972b06de2e3ad98335e25bf69def48367ba50ab37779357 Oct 30 17:26:17.813678 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:17.813656 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc"] Oct 30 17:26:17.814747 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:17.814715 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:26:17.814819 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:17.814793 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:26:17.814875 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:17.814821 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:26:17.921236 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:17.921195 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc" event={"ID":"a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7","Type":"ContainerStarted","Data":"5ccd40da17c1de11e972b06de2e3ad98335e25bf69def48367ba50ab37779357"} Oct 30 17:26:18.931662 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:18.931628 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc" event={"ID":"a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7","Type":"ContainerStarted","Data":"f8e750719d86c8219dd9e97846d47e2a982963c63adda49e8d9b17a09e3df519"} Oct 30 17:26:19.484271 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:19.484239 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc" Oct 30 17:26:19.486337 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:19.486296 4522 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" start-of-body= Oct 30 17:26:19.486487 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:19.486361 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc" podUID="a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" Oct 30 17:26:20.484424 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:20.484383 4522 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" start-of-body= Oct 30 17:26:20.484818 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:20.484436 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc" podUID="a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" Oct 30 17:26:21.514345 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:21.484141 4522 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" start-of-body= Oct 30 17:26:21.514345 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:21.484191 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc" podUID="a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" Oct 30 17:26:22.484071 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:22.484026 4522 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" start-of-body= Oct 30 17:26:22.484283 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:22.484102 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc" podUID="a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" Oct 30 17:26:22.717107 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:22.717072 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c45877-787d5595b7-8mhf5" podUID="ab382b04-4fd2-4172-bc26-8f10347da05c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:26:23.484208 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:23.484172 4522 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" start-of-body= Oct 30 17:26:23.484458 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:23.484244 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc" podUID="a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" Oct 30 17:26:23.877501 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:23.877405 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc" podStartSLOduration=6.877390884 podStartE2EDuration="6.877390884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:26:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:26:19.01069282 +0000 UTC m=+2353.011447747" watchObservedRunningTime="2026-10-30 17:26:23.877390884 +0000 UTC m=+2357.878145781" Oct 30 17:26:23.877929 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:23.877913 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9656b4a-fmbjm"] Oct 30 17:26:23.882788 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:23.882767 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9656b4a-fmbjm" Oct 30 17:26:23.908160 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:23.908124 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9656b4a-fmbjm"] Oct 30 17:26:23.913793 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:23.913760 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:26:23.913963 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:23.913904 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:26:23.963204 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:23.963167 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ae3b4c0-03b3-4f19-b7e1-892f1dee51f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9656b4a-fmbjm\" (UID: \"5ae3b4c0-03b3-4f19-b7e1-892f1dee51f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9656b4a-fmbjm" Oct 30 17:26:23.963430 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:23.963329 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ae3b4c0-03b3-4f19-b7e1-892f1dee51f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9656b4a-fmbjm\" (UID: \"5ae3b4c0-03b3-4f19-b7e1-892f1dee51f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9656b4a-fmbjm" Oct 30 17:26:23.963532 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:23.963511 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz8cr\" (UniqueName: \"kubernetes.io/projected/5ae3b4c0-03b3-4f19-b7e1-892f1dee51f3-kube-api-access-qz8cr\") pod \"maas-api-revoke-sub-a9656b4a-fmbjm\" (UID: \"5ae3b4c0-03b3-4f19-b7e1-892f1dee51f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9656b4a-fmbjm" Oct 30 17:26:24.064739 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:24.064706 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ae3b4c0-03b3-4f19-b7e1-892f1dee51f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9656b4a-fmbjm\" (UID: \"5ae3b4c0-03b3-4f19-b7e1-892f1dee51f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9656b4a-fmbjm" Oct 30 17:26:24.064929 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:24.064820 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz8cr\" (UniqueName: \"kubernetes.io/projected/5ae3b4c0-03b3-4f19-b7e1-892f1dee51f3-kube-api-access-qz8cr\") pod \"maas-api-revoke-sub-a9656b4a-fmbjm\" (UID: \"5ae3b4c0-03b3-4f19-b7e1-892f1dee51f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9656b4a-fmbjm" Oct 30 17:26:24.065008 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:24.064959 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ae3b4c0-03b3-4f19-b7e1-892f1dee51f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9656b4a-fmbjm\" (UID: \"5ae3b4c0-03b3-4f19-b7e1-892f1dee51f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9656b4a-fmbjm" Oct 30 17:26:24.065474 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:24.065450 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ae3b4c0-03b3-4f19-b7e1-892f1dee51f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9656b4a-fmbjm\" (UID: \"5ae3b4c0-03b3-4f19-b7e1-892f1dee51f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9656b4a-fmbjm" Oct 30 17:26:24.067397 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:24.067370 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ae3b4c0-03b3-4f19-b7e1-892f1dee51f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9656b4a-fmbjm\" (UID: \"5ae3b4c0-03b3-4f19-b7e1-892f1dee51f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9656b4a-fmbjm" Oct 30 17:26:24.090681 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:24.090654 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz8cr\" (UniqueName: \"kubernetes.io/projected/5ae3b4c0-03b3-4f19-b7e1-892f1dee51f3-kube-api-access-qz8cr\") pod \"maas-api-revoke-sub-a9656b4a-fmbjm\" (UID: \"5ae3b4c0-03b3-4f19-b7e1-892f1dee51f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9656b4a-fmbjm" Oct 30 17:26:24.201999 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:24.201967 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9656b4a-fmbjm" Oct 30 17:26:24.521689 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:26:24.521654 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ae3b4c0_03b3_4f19_b7e1_892f1dee51f3.slice/crio-cff3646ea39fc2e9c942d34763285d3894e71cb351614a601f4c9423a846ee63 WatchSource:0}: Error finding container cff3646ea39fc2e9c942d34763285d3894e71cb351614a601f4c9423a846ee63: Status 404 returned error can't find the container with id cff3646ea39fc2e9c942d34763285d3894e71cb351614a601f4c9423a846ee63 Oct 30 17:26:24.529665 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:24.529640 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9656b4a-fmbjm"] Oct 30 17:26:24.582209 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:24.582174 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc" Oct 30 17:26:24.582731 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:24.582704 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc" Oct 30 17:26:24.584824 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:24.584799 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc" Oct 30 17:26:24.987283 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:24.987251 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9656b4a-fmbjm" event={"ID":"5ae3b4c0-03b3-4f19-b7e1-892f1dee51f3","Type":"ContainerStarted","Data":"cff3646ea39fc2e9c942d34763285d3894e71cb351614a601f4c9423a846ee63"} Oct 30 17:26:25.996733 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:25.996700 4522 generic.go:356] "Generic (PLEG): container finished" podID="5ae3b4c0-03b3-4f19-b7e1-892f1dee51f3" containerID="fa6fd7a5e4882d23fe0a196263449de591104a39ce48281617bd78108fca7616" exitCode=0 Oct 30 17:26:25.997224 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:25.996784 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9656b4a-fmbjm" event={"ID":"5ae3b4c0-03b3-4f19-b7e1-892f1dee51f3","Type":"ContainerDied","Data":"fa6fd7a5e4882d23fe0a196263449de591104a39ce48281617bd78108fca7616"} Oct 30 17:26:27.183223 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:27.183181 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9656b4a-fmbjm" Oct 30 17:26:27.226481 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:27.226458 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c45877-787d5595b7-8mhf5" Oct 30 17:26:27.332413 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:27.332380 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ae3b4c0-03b3-4f19-b7e1-892f1dee51f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ae3b4c0-03b3-4f19-b7e1-892f1dee51f3-maas-api-xks-ca\") pod \"5ae3b4c0-03b3-4f19-b7e1-892f1dee51f3\" (UID: \"5ae3b4c0-03b3-4f19-b7e1-892f1dee51f3\") " Oct 30 17:26:27.332413 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:27.332416 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab382b04-4fd2-4172-bc26-8f10347da05c-kube-api-access-v2sjj\" (UniqueName: \"kubernetes.io/projected/ab382b04-4fd2-4172-bc26-8f10347da05c-kube-api-access-v2sjj\") pod \"ab382b04-4fd2-4172-bc26-8f10347da05c\" (UID: \"ab382b04-4fd2-4172-bc26-8f10347da05c\") " Oct 30 17:26:27.332686 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:27.332648 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ae3b4c0-03b3-4f19-b7e1-892f1dee51f3-kube-api-access-qz8cr\" (UniqueName: \"kubernetes.io/projected/5ae3b4c0-03b3-4f19-b7e1-892f1dee51f3-kube-api-access-qz8cr\") pod \"5ae3b4c0-03b3-4f19-b7e1-892f1dee51f3\" (UID: \"5ae3b4c0-03b3-4f19-b7e1-892f1dee51f3\") " Oct 30 17:26:27.332758 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:27.332702 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ae3b4c0-03b3-4f19-b7e1-892f1dee51f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ae3b4c0-03b3-4f19-b7e1-892f1dee51f3-maas-api-openshift-ca\") pod \"5ae3b4c0-03b3-4f19-b7e1-892f1dee51f3\" (UID: \"5ae3b4c0-03b3-4f19-b7e1-892f1dee51f3\") " Oct 30 17:26:27.332758 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:27.332741 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab382b04-4fd2-4172-bc26-8f10347da05c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab382b04-4fd2-4172-bc26-8f10347da05c-plugins-config-volume\") pod \"ab382b04-4fd2-4172-bc26-8f10347da05c\" (UID: \"ab382b04-4fd2-4172-bc26-8f10347da05c\") " Oct 30 17:26:27.333172 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:27.333142 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ae3b4c0-03b3-4f19-b7e1-892f1dee51f3-maas-api-openshift-ca" pod "5ae3b4c0-03b3-4f19-b7e1-892f1dee51f3" (UID: "5ae3b4c0-03b3-4f19-b7e1-892f1dee51f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:26:27.333462 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:27.333432 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab382b04-4fd2-4172-bc26-8f10347da05c-plugins-config-volume" pod "ab382b04-4fd2-4172-bc26-8f10347da05c" (UID: "ab382b04-4fd2-4172-bc26-8f10347da05c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:26:27.333745 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:27.333721 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ae3b4c0-03b3-4f19-b7e1-892f1dee51f3-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:26:27.333819 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:27.333751 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab382b04-4fd2-4172-bc26-8f10347da05c-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:26:27.335164 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:27.335144 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ae3b4c0-03b3-4f19-b7e1-892f1dee51f3-maas-api-xks-ca" pod "5ae3b4c0-03b3-4f19-b7e1-892f1dee51f3" (UID: "5ae3b4c0-03b3-4f19-b7e1-892f1dee51f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:26:27.335448 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:27.335419 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ae3b4c0-03b3-4f19-b7e1-892f1dee51f3-kube-api-access-qz8cr" pod "5ae3b4c0-03b3-4f19-b7e1-892f1dee51f3" (UID: "5ae3b4c0-03b3-4f19-b7e1-892f1dee51f3"). InnerVolumeSpecName "kube-api-access-qz8cr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:26:27.335541 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:27.335516 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab382b04-4fd2-4172-bc26-8f10347da05c-kube-api-access-v2sjj" pod "ab382b04-4fd2-4172-bc26-8f10347da05c" (UID: "ab382b04-4fd2-4172-bc26-8f10347da05c"). InnerVolumeSpecName "kube-api-access-v2sjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:26:27.435866 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:27.435783 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qz8cr\" (UniqueName: \"kubernetes.io/projected/5ae3b4c0-03b3-4f19-b7e1-892f1dee51f3-kube-api-access-qz8cr\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:26:27.435866 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:27.435815 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ae3b4c0-03b3-4f19-b7e1-892f1dee51f3-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:26:27.435866 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:27.435826 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2sjj\" (UniqueName: \"kubernetes.io/projected/ab382b04-4fd2-4172-bc26-8f10347da05c-kube-api-access-v2sjj\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:26:28.014836 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:28.014802 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9656b4a-fmbjm" Oct 30 17:26:28.015035 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:28.014833 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9656b4a-fmbjm" event={"ID":"5ae3b4c0-03b3-4f19-b7e1-892f1dee51f3","Type":"ContainerDied","Data":"cff3646ea39fc2e9c942d34763285d3894e71cb351614a601f4c9423a846ee63"} Oct 30 17:26:28.015035 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:28.014891 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cff3646ea39fc2e9c942d34763285d3894e71cb351614a601f4c9423a846ee63" Oct 30 17:26:28.016414 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:28.016390 4522 generic.go:356] "Generic (PLEG): container finished" podID="ab382b04-4fd2-4172-bc26-8f10347da05c" containerID="9570aca6851573754f290ca5602901a159ee6aa98b9281b81dd46851fe49567e" exitCode=0 Oct 30 17:26:28.016720 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:28.016445 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c45877-787d5595b7-8mhf5" Oct 30 17:26:28.016720 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:28.016487 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c45877-787d5595b7-8mhf5" event={"ID":"ab382b04-4fd2-4172-bc26-8f10347da05c","Type":"ContainerDied","Data":"9570aca6851573754f290ca5602901a159ee6aa98b9281b81dd46851fe49567e"} Oct 30 17:26:28.016720 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:28.016543 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c45877-787d5595b7-8mhf5" event={"ID":"ab382b04-4fd2-4172-bc26-8f10347da05c","Type":"ContainerDied","Data":"2e1d9d327a593c1312c5ace768ae064f59cf2d9dc6be1f9cc03796be06822a43"} Oct 30 17:26:28.016720 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:28.016570 4522 scope.go:122] "RemoveContainer" containerID="9570aca6851573754f290ca5602901a159ee6aa98b9281b81dd46851fe49567e" Oct 30 17:26:28.029470 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:28.029447 4522 scope.go:122] "RemoveContainer" containerID="9570aca6851573754f290ca5602901a159ee6aa98b9281b81dd46851fe49567e" Oct 30 17:26:28.029733 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:26:28.029718 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9570aca6851573754f290ca5602901a159ee6aa98b9281b81dd46851fe49567e\": container with ID starting with 9570aca6851573754f290ca5602901a159ee6aa98b9281b81dd46851fe49567e not found: ID does not exist" containerID="9570aca6851573754f290ca5602901a159ee6aa98b9281b81dd46851fe49567e" Oct 30 17:26:28.029791 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:28.029739 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9570aca6851573754f290ca5602901a159ee6aa98b9281b81dd46851fe49567e"} err="failed to get container status \"9570aca6851573754f290ca5602901a159ee6aa98b9281b81dd46851fe49567e\": rpc error: code = NotFound desc = could not find container \"9570aca6851573754f290ca5602901a159ee6aa98b9281b81dd46851fe49567e\": container with ID starting with 9570aca6851573754f290ca5602901a159ee6aa98b9281b81dd46851fe49567e not found: ID does not exist" Oct 30 17:26:28.091391 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:28.091361 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c45877-787d5595b7-8mhf5"] Oct 30 17:26:28.104432 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:28.104403 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c45877-787d5595b7-8mhf5"] Oct 30 17:26:28.147050 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:28.147019 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab382b04-4fd2-4172-bc26-8f10347da05c" path="/var/lib/kubelet/pods/ab382b04-4fd2-4172-bc26-8f10347da05c/volumes" Oct 30 17:26:29.662135 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:29.662097 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c45877-kncn2"] Oct 30 17:26:29.675032 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:29.675000 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c45877-kncn2"] Oct 30 17:26:30.214491 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:30.201215 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7354dfa5-d620-41cd-ab0b-d754e15235cd" path="/var/lib/kubelet/pods/7354dfa5-d620-41cd-ab0b-d754e15235cd/volumes" Oct 30 17:26:34.463389 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:34.463354 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81a19108-gwbtm"] Oct 30 17:26:34.464082 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:34.464068 4522 cpu_manager.go:414] "containerMap: removing container" podUID="ab382b04-4fd2-4172-bc26-8f10347da05c" containerName="payload-pre-processing" Oct 30 17:26:34.464128 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:34.464083 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab382b04-4fd2-4172-bc26-8f10347da05c" containerName="payload-pre-processing" Oct 30 17:26:34.464128 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:34.464102 4522 cpu_manager.go:414] "containerMap: removing container" podUID="5ae3b4c0-03b3-4f19-b7e1-892f1dee51f3" containerName="revoke-keys" Oct 30 17:26:34.464128 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:34.464108 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ae3b4c0-03b3-4f19-b7e1-892f1dee51f3" containerName="revoke-keys" Oct 30 17:26:34.464253 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:34.464239 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab382b04-4fd2-4172-bc26-8f10347da05c" containerName="payload-pre-processing" Oct 30 17:26:34.464253 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:34.464252 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ae3b4c0-03b3-4f19-b7e1-892f1dee51f3" containerName="revoke-keys" Oct 30 17:26:34.468062 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:34.468045 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a19108-gwbtm" Oct 30 17:26:34.489473 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:34.489447 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81a19108-gwbtm"] Oct 30 17:26:34.495661 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:34.495637 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:26:34.495870 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:34.495854 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:26:34.573778 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:34.573751 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65354bf3-e5b6-4039-8dd5-c6573ac71059-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81a19108-gwbtm\" (UID: \"65354bf3-e5b6-4039-8dd5-c6573ac71059\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a19108-gwbtm" Oct 30 17:26:34.573950 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:34.573865 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwxwg\" (UniqueName: \"kubernetes.io/projected/65354bf3-e5b6-4039-8dd5-c6573ac71059-kube-api-access-wwxwg\") pod \"maas-api-revoke-sub-81a19108-gwbtm\" (UID: \"65354bf3-e5b6-4039-8dd5-c6573ac71059\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a19108-gwbtm" Oct 30 17:26:34.573950 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:34.573937 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65354bf3-e5b6-4039-8dd5-c6573ac71059-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81a19108-gwbtm\" (UID: \"65354bf3-e5b6-4039-8dd5-c6573ac71059\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a19108-gwbtm" Oct 30 17:26:34.675805 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:34.675777 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwxwg\" (UniqueName: \"kubernetes.io/projected/65354bf3-e5b6-4039-8dd5-c6573ac71059-kube-api-access-wwxwg\") pod \"maas-api-revoke-sub-81a19108-gwbtm\" (UID: \"65354bf3-e5b6-4039-8dd5-c6573ac71059\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a19108-gwbtm" Oct 30 17:26:34.676012 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:34.675842 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65354bf3-e5b6-4039-8dd5-c6573ac71059-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81a19108-gwbtm\" (UID: \"65354bf3-e5b6-4039-8dd5-c6573ac71059\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a19108-gwbtm" Oct 30 17:26:34.676092 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:34.676015 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65354bf3-e5b6-4039-8dd5-c6573ac71059-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81a19108-gwbtm\" (UID: \"65354bf3-e5b6-4039-8dd5-c6573ac71059\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a19108-gwbtm" Oct 30 17:26:34.676709 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:34.676683 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65354bf3-e5b6-4039-8dd5-c6573ac71059-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81a19108-gwbtm\" (UID: \"65354bf3-e5b6-4039-8dd5-c6573ac71059\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a19108-gwbtm" Oct 30 17:26:34.678167 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:34.678144 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65354bf3-e5b6-4039-8dd5-c6573ac71059-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81a19108-gwbtm\" (UID: \"65354bf3-e5b6-4039-8dd5-c6573ac71059\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a19108-gwbtm" Oct 30 17:26:34.700846 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:34.700825 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwxwg\" (UniqueName: \"kubernetes.io/projected/65354bf3-e5b6-4039-8dd5-c6573ac71059-kube-api-access-wwxwg\") pod \"maas-api-revoke-sub-81a19108-gwbtm\" (UID: \"65354bf3-e5b6-4039-8dd5-c6573ac71059\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a19108-gwbtm" Oct 30 17:26:34.785006 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:34.784934 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a19108-gwbtm" Oct 30 17:26:35.145818 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:26:35.145786 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65354bf3_e5b6_4039_8dd5_c6573ac71059.slice/crio-ebf1806cbe8d327107ae9228a661b8ca25dd6e9ce50ca7baa866b58d60166cec WatchSource:0}: Error finding container ebf1806cbe8d327107ae9228a661b8ca25dd6e9ce50ca7baa866b58d60166cec: Status 404 returned error can't find the container with id ebf1806cbe8d327107ae9228a661b8ca25dd6e9ce50ca7baa866b58d60166cec Oct 30 17:26:35.148981 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:35.148959 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81a19108-gwbtm"] Oct 30 17:26:36.187103 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:36.187067 4522 generic.go:356] "Generic (PLEG): container finished" podID="65354bf3-e5b6-4039-8dd5-c6573ac71059" containerID="64223a986e42ea78327a08be763269195539b3f95570a50478fa0c6fc479d26b" exitCode=0 Oct 30 17:26:36.187103 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:36.187082 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a19108-gwbtm" event={"ID":"65354bf3-e5b6-4039-8dd5-c6573ac71059","Type":"ContainerDied","Data":"64223a986e42ea78327a08be763269195539b3f95570a50478fa0c6fc479d26b"} Oct 30 17:26:36.187810 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:36.187139 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a19108-gwbtm" event={"ID":"65354bf3-e5b6-4039-8dd5-c6573ac71059","Type":"ContainerStarted","Data":"ebf1806cbe8d327107ae9228a661b8ca25dd6e9ce50ca7baa866b58d60166cec"} Oct 30 17:26:37.426855 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:37.426826 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a19108-gwbtm" Oct 30 17:26:37.545425 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:37.545350 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65354bf3-e5b6-4039-8dd5-c6573ac71059-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65354bf3-e5b6-4039-8dd5-c6573ac71059-maas-api-openshift-ca\") pod \"65354bf3-e5b6-4039-8dd5-c6573ac71059\" (UID: \"65354bf3-e5b6-4039-8dd5-c6573ac71059\") " Oct 30 17:26:37.545578 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:37.545466 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65354bf3-e5b6-4039-8dd5-c6573ac71059-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65354bf3-e5b6-4039-8dd5-c6573ac71059-maas-api-xks-ca\") pod \"65354bf3-e5b6-4039-8dd5-c6573ac71059\" (UID: \"65354bf3-e5b6-4039-8dd5-c6573ac71059\") " Oct 30 17:26:37.545656 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:37.545645 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65354bf3-e5b6-4039-8dd5-c6573ac71059-kube-api-access-wwxwg\" (UniqueName: \"kubernetes.io/projected/65354bf3-e5b6-4039-8dd5-c6573ac71059-kube-api-access-wwxwg\") pod \"65354bf3-e5b6-4039-8dd5-c6573ac71059\" (UID: \"65354bf3-e5b6-4039-8dd5-c6573ac71059\") " Oct 30 17:26:37.545701 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:37.545678 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65354bf3-e5b6-4039-8dd5-c6573ac71059-maas-api-openshift-ca" pod "65354bf3-e5b6-4039-8dd5-c6573ac71059" (UID: "65354bf3-e5b6-4039-8dd5-c6573ac71059"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:26:37.546333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:37.546296 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65354bf3-e5b6-4039-8dd5-c6573ac71059-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:26:37.547732 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:37.547707 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65354bf3-e5b6-4039-8dd5-c6573ac71059-maas-api-xks-ca" pod "65354bf3-e5b6-4039-8dd5-c6573ac71059" (UID: "65354bf3-e5b6-4039-8dd5-c6573ac71059"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:26:37.548009 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:37.547990 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65354bf3-e5b6-4039-8dd5-c6573ac71059-kube-api-access-wwxwg" pod "65354bf3-e5b6-4039-8dd5-c6573ac71059" (UID: "65354bf3-e5b6-4039-8dd5-c6573ac71059"). InnerVolumeSpecName "kube-api-access-wwxwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:26:37.647767 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:37.647725 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65354bf3-e5b6-4039-8dd5-c6573ac71059-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:26:37.647767 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:37.647759 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwxwg\" (UniqueName: \"kubernetes.io/projected/65354bf3-e5b6-4039-8dd5-c6573ac71059-kube-api-access-wwxwg\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:26:38.092605 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:38.092564 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh"] Oct 30 17:26:38.092855 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:38.092830 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh" podUID="40f8b984-d0d8-4938-859e-268ae92cf8f2" containerName="istio-proxy" containerID="cri-o://e77d008790cb6f74989e7fc89e4510f0b2b0dd107c2e75415d2fec9e5e4d0b66" gracePeriod=30 Oct 30 17:26:38.252516 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:38.252480 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a19108-gwbtm" Oct 30 17:26:38.252699 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:38.252486 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a19108-gwbtm" event={"ID":"65354bf3-e5b6-4039-8dd5-c6573ac71059","Type":"ContainerDied","Data":"ebf1806cbe8d327107ae9228a661b8ca25dd6e9ce50ca7baa866b58d60166cec"} Oct 30 17:26:38.252699 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:38.252618 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebf1806cbe8d327107ae9228a661b8ca25dd6e9ce50ca7baa866b58d60166cec" Oct 30 17:26:42.948818 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:26:42.948784 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:26:43.315933 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:43.315901 4522 generic.go:356] "Generic (PLEG): container finished" podID="40f8b984-d0d8-4938-859e-268ae92cf8f2" containerID="e77d008790cb6f74989e7fc89e4510f0b2b0dd107c2e75415d2fec9e5e4d0b66" exitCode=0 Oct 30 17:26:43.316139 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:43.315971 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh" event={"ID":"40f8b984-d0d8-4938-859e-268ae92cf8f2","Type":"ContainerDied","Data":"e77d008790cb6f74989e7fc89e4510f0b2b0dd107c2e75415d2fec9e5e4d0b66"} Oct 30 17:26:43.316139 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:43.316044 4522 scope.go:122] "RemoveContainer" containerID="19340ec98a9c74003f96585f0347e896a9703170056e3ececf339bf3431d812c" Oct 30 17:26:43.425563 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:43.425534 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh" Oct 30 17:26:43.573553 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:43.571797 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40f8b984-d0d8-4938-859e-268ae92cf8f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40f8b984-d0d8-4938-859e-268ae92cf8f2-workload-certs\") pod \"40f8b984-d0d8-4938-859e-268ae92cf8f2\" (UID: \"40f8b984-d0d8-4938-859e-268ae92cf8f2\") " Oct 30 17:26:43.573553 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:43.571919 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/40f8b984-d0d8-4938-859e-268ae92cf8f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40f8b984-d0d8-4938-859e-268ae92cf8f2-istio-podinfo\") pod \"40f8b984-d0d8-4938-859e-268ae92cf8f2\" (UID: \"40f8b984-d0d8-4938-859e-268ae92cf8f2\") " Oct 30 17:26:43.573553 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:43.572010 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40f8b984-d0d8-4938-859e-268ae92cf8f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40f8b984-d0d8-4938-859e-268ae92cf8f2-istiod-ca-cert\") pod \"40f8b984-d0d8-4938-859e-268ae92cf8f2\" (UID: \"40f8b984-d0d8-4938-859e-268ae92cf8f2\") " Oct 30 17:26:43.573553 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:43.572118 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40f8b984-d0d8-4938-859e-268ae92cf8f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40f8b984-d0d8-4938-859e-268ae92cf8f2-istio-data\") pod \"40f8b984-d0d8-4938-859e-268ae92cf8f2\" (UID: \"40f8b984-d0d8-4938-859e-268ae92cf8f2\") " Oct 30 17:26:43.573553 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:43.572149 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40f8b984-d0d8-4938-859e-268ae92cf8f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40f8b984-d0d8-4938-859e-268ae92cf8f2-credential-socket\") pod \"40f8b984-d0d8-4938-859e-268ae92cf8f2\" (UID: \"40f8b984-d0d8-4938-859e-268ae92cf8f2\") " Oct 30 17:26:43.573553 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:43.572169 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40f8b984-d0d8-4938-859e-268ae92cf8f2-istio-token\" (UniqueName: \"kubernetes.io/projected/40f8b984-d0d8-4938-859e-268ae92cf8f2-istio-token\") pod \"40f8b984-d0d8-4938-859e-268ae92cf8f2\" (UID: \"40f8b984-d0d8-4938-859e-268ae92cf8f2\") " Oct 30 17:26:43.573553 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:43.572191 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40f8b984-d0d8-4938-859e-268ae92cf8f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40f8b984-d0d8-4938-859e-268ae92cf8f2-istio-envoy\") pod \"40f8b984-d0d8-4938-859e-268ae92cf8f2\" (UID: \"40f8b984-d0d8-4938-859e-268ae92cf8f2\") " Oct 30 17:26:43.573553 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:43.572244 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40f8b984-d0d8-4938-859e-268ae92cf8f2-kube-api-access-gx92p\" (UniqueName: \"kubernetes.io/projected/40f8b984-d0d8-4938-859e-268ae92cf8f2-kube-api-access-gx92p\") pod \"40f8b984-d0d8-4938-859e-268ae92cf8f2\" (UID: \"40f8b984-d0d8-4938-859e-268ae92cf8f2\") " Oct 30 17:26:43.573553 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:43.572470 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40f8b984-d0d8-4938-859e-268ae92cf8f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40f8b984-d0d8-4938-859e-268ae92cf8f2-workload-socket\") pod \"40f8b984-d0d8-4938-859e-268ae92cf8f2\" (UID: \"40f8b984-d0d8-4938-859e-268ae92cf8f2\") " Oct 30 17:26:43.573553 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:43.573413 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40f8b984-d0d8-4938-859e-268ae92cf8f2-workload-socket" pod "40f8b984-d0d8-4938-859e-268ae92cf8f2" (UID: "40f8b984-d0d8-4938-859e-268ae92cf8f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:26:43.574795 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:43.573586 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40f8b984-d0d8-4938-859e-268ae92cf8f2-workload-certs" pod "40f8b984-d0d8-4938-859e-268ae92cf8f2" (UID: "40f8b984-d0d8-4938-859e-268ae92cf8f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:26:43.574795 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:43.574145 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40f8b984-d0d8-4938-859e-268ae92cf8f2-istio-data" pod "40f8b984-d0d8-4938-859e-268ae92cf8f2" (UID: "40f8b984-d0d8-4938-859e-268ae92cf8f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:26:43.574795 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:43.574171 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40f8b984-d0d8-4938-859e-268ae92cf8f2-istiod-ca-cert" pod "40f8b984-d0d8-4938-859e-268ae92cf8f2" (UID: "40f8b984-d0d8-4938-859e-268ae92cf8f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:26:43.574795 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:43.574736 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40f8b984-d0d8-4938-859e-268ae92cf8f2-credential-socket" pod "40f8b984-d0d8-4938-859e-268ae92cf8f2" (UID: "40f8b984-d0d8-4938-859e-268ae92cf8f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:26:43.579853 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:43.579810 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40f8b984-d0d8-4938-859e-268ae92cf8f2-istio-envoy" pod "40f8b984-d0d8-4938-859e-268ae92cf8f2" (UID: "40f8b984-d0d8-4938-859e-268ae92cf8f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:26:43.580342 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:43.579971 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40f8b984-d0d8-4938-859e-268ae92cf8f2-istio-podinfo" pod "40f8b984-d0d8-4938-859e-268ae92cf8f2" (UID: "40f8b984-d0d8-4938-859e-268ae92cf8f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:26:43.580590 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:43.580334 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40f8b984-d0d8-4938-859e-268ae92cf8f2-istio-token" pod "40f8b984-d0d8-4938-859e-268ae92cf8f2" (UID: "40f8b984-d0d8-4938-859e-268ae92cf8f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:26:43.582155 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:43.582129 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40f8b984-d0d8-4938-859e-268ae92cf8f2-kube-api-access-gx92p" pod "40f8b984-d0d8-4938-859e-268ae92cf8f2" (UID: "40f8b984-d0d8-4938-859e-268ae92cf8f2"). InnerVolumeSpecName "kube-api-access-gx92p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:26:43.674535 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:43.674502 4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40f8b984-d0d8-4938-859e-268ae92cf8f2-istio-token\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:26:43.674535 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:43.674529 4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40f8b984-d0d8-4938-859e-268ae92cf8f2-istio-envoy\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:26:43.674535 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:43.674539 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx92p\" (UniqueName: \"kubernetes.io/projected/40f8b984-d0d8-4938-859e-268ae92cf8f2-kube-api-access-gx92p\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:26:43.674784 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:43.674549 4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40f8b984-d0d8-4938-859e-268ae92cf8f2-workload-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:26:43.674784 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:43.674558 4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40f8b984-d0d8-4938-859e-268ae92cf8f2-workload-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:26:43.674784 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:43.674566 4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40f8b984-d0d8-4938-859e-268ae92cf8f2-istio-podinfo\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:26:43.674784 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:43.674574 4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40f8b984-d0d8-4938-859e-268ae92cf8f2-istiod-ca-cert\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:26:43.674784 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:43.674582 4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40f8b984-d0d8-4938-859e-268ae92cf8f2-istio-data\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:26:43.674784 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:43.674590 4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40f8b984-d0d8-4938-859e-268ae92cf8f2-credential-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:26:44.325762 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:44.325731 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh" Oct 30 17:26:44.326167 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:44.325769 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh" event={"ID":"40f8b984-d0d8-4938-859e-268ae92cf8f2","Type":"ContainerDied","Data":"13cbd6dbcd8167cc9de63f43e3b789ba1ccbc49f92d3bccd8bb3bbc67605a210"} Oct 30 17:26:44.326167 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:44.325847 4522 scope.go:122] "RemoveContainer" containerID="e77d008790cb6f74989e7fc89e4510f0b2b0dd107c2e75415d2fec9e5e4d0b66" Oct 30 17:26:44.398410 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:44.398376 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh"] Oct 30 17:26:44.411342 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:44.411256 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-77hlh"] Oct 30 17:26:44.697391 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:44.697317 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fea415b0-z26b7"] Oct 30 17:26:44.698053 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:44.698037 4522 cpu_manager.go:414] "containerMap: removing container" podUID="40f8b984-d0d8-4938-859e-268ae92cf8f2" containerName="istio-proxy" Oct 30 17:26:44.698099 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:44.698055 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40f8b984-d0d8-4938-859e-268ae92cf8f2" containerName="istio-proxy" Oct 30 17:26:44.698099 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:44.698094 4522 cpu_manager.go:414] "containerMap: removing container" podUID="65354bf3-e5b6-4039-8dd5-c6573ac71059" containerName="revoke-keys" Oct 30 17:26:44.698168 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:44.698099 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65354bf3-e5b6-4039-8dd5-c6573ac71059" containerName="revoke-keys" Oct 30 17:26:44.698232 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:44.698223 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="40f8b984-d0d8-4938-859e-268ae92cf8f2" containerName="istio-proxy" Oct 30 17:26:44.698270 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:44.698234 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="65354bf3-e5b6-4039-8dd5-c6573ac71059" containerName="revoke-keys" Oct 30 17:26:44.698270 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:44.698248 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="40f8b984-d0d8-4938-859e-268ae92cf8f2" containerName="istio-proxy" Oct 30 17:26:44.703521 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:44.703499 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea415b0-z26b7" Oct 30 17:26:44.719962 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:44.719935 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:26:44.720122 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:44.720015 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:26:44.722062 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:44.722036 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fea415b0-z26b7"] Oct 30 17:26:44.792975 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:44.792937 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e3b947c-7748-4cf7-a326-e3187e2b2b1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fea415b0-z26b7\" (UID: \"5e3b947c-7748-4cf7-a326-e3187e2b2b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea415b0-z26b7" Oct 30 17:26:44.793200 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:44.793172 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e3b947c-7748-4cf7-a326-e3187e2b2b1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fea415b0-z26b7\" (UID: \"5e3b947c-7748-4cf7-a326-e3187e2b2b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea415b0-z26b7" Oct 30 17:26:44.793335 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:44.793294 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxrm2\" (UniqueName: \"kubernetes.io/projected/5e3b947c-7748-4cf7-a326-e3187e2b2b1d-kube-api-access-jxrm2\") pod \"maas-api-revoke-sub-fea415b0-z26b7\" (UID: \"5e3b947c-7748-4cf7-a326-e3187e2b2b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea415b0-z26b7" Oct 30 17:26:44.895138 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:44.895105 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e3b947c-7748-4cf7-a326-e3187e2b2b1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fea415b0-z26b7\" (UID: \"5e3b947c-7748-4cf7-a326-e3187e2b2b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea415b0-z26b7" Oct 30 17:26:44.895373 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:44.895214 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxrm2\" (UniqueName: \"kubernetes.io/projected/5e3b947c-7748-4cf7-a326-e3187e2b2b1d-kube-api-access-jxrm2\") pod \"maas-api-revoke-sub-fea415b0-z26b7\" (UID: \"5e3b947c-7748-4cf7-a326-e3187e2b2b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea415b0-z26b7" Oct 30 17:26:44.895373 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:44.895346 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e3b947c-7748-4cf7-a326-e3187e2b2b1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fea415b0-z26b7\" (UID: \"5e3b947c-7748-4cf7-a326-e3187e2b2b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea415b0-z26b7" Oct 30 17:26:44.895985 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:44.895964 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e3b947c-7748-4cf7-a326-e3187e2b2b1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fea415b0-z26b7\" (UID: \"5e3b947c-7748-4cf7-a326-e3187e2b2b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea415b0-z26b7" Oct 30 17:26:44.897559 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:44.897543 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e3b947c-7748-4cf7-a326-e3187e2b2b1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fea415b0-z26b7\" (UID: \"5e3b947c-7748-4cf7-a326-e3187e2b2b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea415b0-z26b7" Oct 30 17:26:44.923733 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:44.923707 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxrm2\" (UniqueName: \"kubernetes.io/projected/5e3b947c-7748-4cf7-a326-e3187e2b2b1d-kube-api-access-jxrm2\") pod \"maas-api-revoke-sub-fea415b0-z26b7\" (UID: \"5e3b947c-7748-4cf7-a326-e3187e2b2b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea415b0-z26b7" Oct 30 17:26:45.019136 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:45.019104 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea415b0-z26b7" Oct 30 17:26:45.049113 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:45.049039 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16339e0f-bvkvm"] Oct 30 17:26:45.053806 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:45.053781 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16339e0f-bvkvm"] Oct 30 17:26:45.335991 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:26:45.335956 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e3b947c_7748_4cf7_a326_e3187e2b2b1d.slice/crio-922e3979eafb102a4ccafd5961f963b5cc1fd7ec2b22ec4851e019cbf0a40829 WatchSource:0}: Error finding container 922e3979eafb102a4ccafd5961f963b5cc1fd7ec2b22ec4851e019cbf0a40829: Status 404 returned error can't find the container with id 922e3979eafb102a4ccafd5961f963b5cc1fd7ec2b22ec4851e019cbf0a40829 Oct 30 17:26:45.339283 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:45.339260 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fea415b0-z26b7"] Oct 30 17:26:46.147240 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:46.147210 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fc7054e-4f5c-4fdf-8b7a-d6561d562ae5" path="/var/lib/kubelet/pods/1fc7054e-4f5c-4fdf-8b7a-d6561d562ae5/volumes" Oct 30 17:26:46.147776 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:46.147761 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40f8b984-d0d8-4938-859e-268ae92cf8f2" path="/var/lib/kubelet/pods/40f8b984-d0d8-4938-859e-268ae92cf8f2/volumes" Oct 30 17:26:46.202089 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:46.202058 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b87f3c-85b7f76ff9-mm6sf"] Oct 30 17:26:46.202791 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:46.202775 4522 cpu_manager.go:414] "containerMap: removing container" podUID="40f8b984-d0d8-4938-859e-268ae92cf8f2" containerName="istio-proxy" Oct 30 17:26:46.202844 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:46.202792 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40f8b984-d0d8-4938-859e-268ae92cf8f2" containerName="istio-proxy" Oct 30 17:26:46.206629 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:46.206609 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b87f3c-85b7f76ff9-mm6sf" Oct 30 17:26:46.225423 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:46.225392 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b87f3c-85b7f76ff9-mm6sf"] Oct 30 17:26:46.233361 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:46.233332 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-b87f3c\"" Oct 30 17:26:46.233361 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:46.233344 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-b87f3c-dockercfg-zfwg9\"" Oct 30 17:26:46.308155 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:46.308121 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-b87f3c-f97f9ffc4-f5qb8"] Oct 30 17:26:46.328691 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:46.328658 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b87f3c-f97f9ffc4-f5qb8" Oct 30 17:26:46.331621 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:46.331590 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c9jb\" (UniqueName: \"kubernetes.io/projected/d718a4ec-8b4d-4d81-bd11-bc1e6c704b1f-kube-api-access-6c9jb\") pod \"payload-pre-processing-e2e-shared-a-w6-b87f3c-85b7f76ff9-mm6sf\" (UID: \"d718a4ec-8b4d-4d81-bd11-bc1e6c704b1f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b87f3c-85b7f76ff9-mm6sf" Oct 30 17:26:46.331965 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:46.331946 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d718a4ec-8b4d-4d81-bd11-bc1e6c704b1f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-b87f3c-85b7f76ff9-mm6sf\" (UID: \"d718a4ec-8b4d-4d81-bd11-bc1e6c704b1f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b87f3c-85b7f76ff9-mm6sf" Oct 30 17:26:46.374368 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:46.374293 4522 generic.go:356] "Generic (PLEG): container finished" podID="5e3b947c-7748-4cf7-a326-e3187e2b2b1d" containerID="8d0cb6649488b13c2b1c766f7c9823719429746ad887aaebd0efdeeb4b9a8bd8" exitCode=0 Oct 30 17:26:46.374881 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:46.374692 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea415b0-z26b7" event={"ID":"5e3b947c-7748-4cf7-a326-e3187e2b2b1d","Type":"ContainerDied","Data":"8d0cb6649488b13c2b1c766f7c9823719429746ad887aaebd0efdeeb4b9a8bd8"} Oct 30 17:26:46.374881 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:46.374752 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea415b0-z26b7" event={"ID":"5e3b947c-7748-4cf7-a326-e3187e2b2b1d","Type":"ContainerStarted","Data":"922e3979eafb102a4ccafd5961f963b5cc1fd7ec2b22ec4851e019cbf0a40829"} Oct 30 17:26:46.385259 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:46.380707 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-b87f3c-f97f9ffc4-f5qb8"] Oct 30 17:26:46.434879 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:46.433974 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c9jb\" (UniqueName: \"kubernetes.io/projected/d718a4ec-8b4d-4d81-bd11-bc1e6c704b1f-kube-api-access-6c9jb\") pod \"payload-pre-processing-e2e-shared-a-w6-b87f3c-85b7f76ff9-mm6sf\" (UID: \"d718a4ec-8b4d-4d81-bd11-bc1e6c704b1f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b87f3c-85b7f76ff9-mm6sf" Oct 30 17:26:46.434879 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:46.434103 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d718a4ec-8b4d-4d81-bd11-bc1e6c704b1f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-b87f3c-85b7f76ff9-mm6sf\" (UID: \"d718a4ec-8b4d-4d81-bd11-bc1e6c704b1f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b87f3c-85b7f76ff9-mm6sf" Oct 30 17:26:46.434879 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:46.434281 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6szwn\" (UniqueName: \"kubernetes.io/projected/f26c6749-26c9-44cc-81bf-0e49d20ddc1a-kube-api-access-6szwn\") pod \"payload-processing-e2e-shared-a-w6-b87f3c-f97f9ffc4-f5qb8\" (UID: \"f26c6749-26c9-44cc-81bf-0e49d20ddc1a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b87f3c-f97f9ffc4-f5qb8" Oct 30 17:26:46.434879 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:46.434368 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f26c6749-26c9-44cc-81bf-0e49d20ddc1a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-b87f3c-f97f9ffc4-f5qb8\" (UID: \"f26c6749-26c9-44cc-81bf-0e49d20ddc1a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b87f3c-f97f9ffc4-f5qb8" Oct 30 17:26:46.435268 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:46.435220 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d718a4ec-8b4d-4d81-bd11-bc1e6c704b1f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-b87f3c-85b7f76ff9-mm6sf\" (UID: \"d718a4ec-8b4d-4d81-bd11-bc1e6c704b1f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b87f3c-85b7f76ff9-mm6sf" Oct 30 17:26:46.459463 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:46.459426 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c9jb\" (UniqueName: \"kubernetes.io/projected/d718a4ec-8b4d-4d81-bd11-bc1e6c704b1f-kube-api-access-6c9jb\") pod \"payload-pre-processing-e2e-shared-a-w6-b87f3c-85b7f76ff9-mm6sf\" (UID: \"d718a4ec-8b4d-4d81-bd11-bc1e6c704b1f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b87f3c-85b7f76ff9-mm6sf" Oct 30 17:26:46.536164 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:46.536129 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6szwn\" (UniqueName: \"kubernetes.io/projected/f26c6749-26c9-44cc-81bf-0e49d20ddc1a-kube-api-access-6szwn\") pod \"payload-processing-e2e-shared-a-w6-b87f3c-f97f9ffc4-f5qb8\" (UID: \"f26c6749-26c9-44cc-81bf-0e49d20ddc1a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b87f3c-f97f9ffc4-f5qb8" Oct 30 17:26:46.536402 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:46.536188 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f26c6749-26c9-44cc-81bf-0e49d20ddc1a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-b87f3c-f97f9ffc4-f5qb8\" (UID: \"f26c6749-26c9-44cc-81bf-0e49d20ddc1a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b87f3c-f97f9ffc4-f5qb8" Oct 30 17:26:46.536966 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:46.536939 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f26c6749-26c9-44cc-81bf-0e49d20ddc1a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-b87f3c-f97f9ffc4-f5qb8\" (UID: \"f26c6749-26c9-44cc-81bf-0e49d20ddc1a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b87f3c-f97f9ffc4-f5qb8" Oct 30 17:26:46.561341 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:46.561293 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6szwn\" (UniqueName: \"kubernetes.io/projected/f26c6749-26c9-44cc-81bf-0e49d20ddc1a-kube-api-access-6szwn\") pod \"payload-processing-e2e-shared-a-w6-b87f3c-f97f9ffc4-f5qb8\" (UID: \"f26c6749-26c9-44cc-81bf-0e49d20ddc1a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b87f3c-f97f9ffc4-f5qb8" Oct 30 17:26:46.590731 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:46.590698 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b87f3c-85b7f76ff9-mm6sf" Oct 30 17:26:46.674859 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:46.674817 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b87f3c-f97f9ffc4-f5qb8" Oct 30 17:26:46.902872 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:26:46.902838 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd718a4ec_8b4d_4d81_bd11_bc1e6c704b1f.slice/crio-3410c19db18aca985a0fc4f3a0f14474144cc6c3d55597998eec28a53d93bdff WatchSource:0}: Error finding container 3410c19db18aca985a0fc4f3a0f14474144cc6c3d55597998eec28a53d93bdff: Status 404 returned error can't find the container with id 3410c19db18aca985a0fc4f3a0f14474144cc6c3d55597998eec28a53d93bdff Oct 30 17:26:46.905212 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:46.905187 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b87f3c-85b7f76ff9-mm6sf"] Oct 30 17:26:47.005818 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:26:47.005791 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf26c6749_26c9_44cc_81bf_0e49d20ddc1a.slice/crio-d2ea113d79d94255e77d323cac5ef71d57703dd28c6804a21770d45247ee84c8 WatchSource:0}: Error finding container d2ea113d79d94255e77d323cac5ef71d57703dd28c6804a21770d45247ee84c8: Status 404 returned error can't find the container with id d2ea113d79d94255e77d323cac5ef71d57703dd28c6804a21770d45247ee84c8 Oct 30 17:26:47.008289 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:47.008265 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-b87f3c-f97f9ffc4-f5qb8"] Oct 30 17:26:47.384275 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:47.384235 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b87f3c-f97f9ffc4-f5qb8" event={"ID":"f26c6749-26c9-44cc-81bf-0e49d20ddc1a","Type":"ContainerStarted","Data":"592d1ad5a47798de16b10641da42a340a8f5671970c0e1a316331b4b1703b744"} Oct 30 17:26:47.384704 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:47.384299 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b87f3c-f97f9ffc4-f5qb8" event={"ID":"f26c6749-26c9-44cc-81bf-0e49d20ddc1a","Type":"ContainerStarted","Data":"d2ea113d79d94255e77d323cac5ef71d57703dd28c6804a21770d45247ee84c8"} Oct 30 17:26:47.384704 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:47.384341 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b87f3c-f97f9ffc4-f5qb8" Oct 30 17:26:47.386029 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:47.385996 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b87f3c-85b7f76ff9-mm6sf" event={"ID":"d718a4ec-8b4d-4d81-bd11-bc1e6c704b1f","Type":"ContainerStarted","Data":"1df72c2f845671c180833129ae22615d1b2ca295ebe8ba7050e658fa56a0b3a2"} Oct 30 17:26:47.386143 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:47.386054 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b87f3c-85b7f76ff9-mm6sf" event={"ID":"d718a4ec-8b4d-4d81-bd11-bc1e6c704b1f","Type":"ContainerStarted","Data":"3410c19db18aca985a0fc4f3a0f14474144cc6c3d55597998eec28a53d93bdff"} Oct 30 17:26:47.386189 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:47.386148 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b87f3c-85b7f76ff9-mm6sf" Oct 30 17:26:47.429384 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:47.429326 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b87f3c-f97f9ffc4-f5qb8" podStartSLOduration=1.4292992660000001 podStartE2EDuration="1.429299266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:26:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:26:47.428033293 +0000 UTC m=+2381.428788194" watchObservedRunningTime="2026-10-30 17:26:47.429299266 +0000 UTC m=+2381.430054167" Oct 30 17:26:47.538569 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:47.538539 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea415b0-z26b7" Oct 30 17:26:47.583131 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:47.583069 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b87f3c-85b7f76ff9-mm6sf" podStartSLOduration=1.58304883 podStartE2EDuration="1.58304883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:26:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:26:47.483263853 +0000 UTC m=+2381.484018754" watchObservedRunningTime="2026-10-30 17:26:47.58304883 +0000 UTC m=+2381.583803732" Oct 30 17:26:47.656091 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:47.656000 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e3b947c-7748-4cf7-a326-e3187e2b2b1d-kube-api-access-jxrm2\" (UniqueName: \"kubernetes.io/projected/5e3b947c-7748-4cf7-a326-e3187e2b2b1d-kube-api-access-jxrm2\") pod \"5e3b947c-7748-4cf7-a326-e3187e2b2b1d\" (UID: \"5e3b947c-7748-4cf7-a326-e3187e2b2b1d\") " Oct 30 17:26:47.656091 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:47.656047 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e3b947c-7748-4cf7-a326-e3187e2b2b1d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e3b947c-7748-4cf7-a326-e3187e2b2b1d-maas-api-openshift-ca\") pod \"5e3b947c-7748-4cf7-a326-e3187e2b2b1d\" (UID: \"5e3b947c-7748-4cf7-a326-e3187e2b2b1d\") " Oct 30 17:26:47.656382 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:47.656259 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e3b947c-7748-4cf7-a326-e3187e2b2b1d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e3b947c-7748-4cf7-a326-e3187e2b2b1d-maas-api-xks-ca\") pod \"5e3b947c-7748-4cf7-a326-e3187e2b2b1d\" (UID: \"5e3b947c-7748-4cf7-a326-e3187e2b2b1d\") " Oct 30 17:26:47.658656 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:47.658626 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e3b947c-7748-4cf7-a326-e3187e2b2b1d-kube-api-access-jxrm2" pod "5e3b947c-7748-4cf7-a326-e3187e2b2b1d" (UID: "5e3b947c-7748-4cf7-a326-e3187e2b2b1d"). InnerVolumeSpecName "kube-api-access-jxrm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:26:47.659009 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:47.658734 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e3b947c-7748-4cf7-a326-e3187e2b2b1d-maas-api-openshift-ca" pod "5e3b947c-7748-4cf7-a326-e3187e2b2b1d" (UID: "5e3b947c-7748-4cf7-a326-e3187e2b2b1d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:26:47.660868 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:47.660842 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e3b947c-7748-4cf7-a326-e3187e2b2b1d-maas-api-xks-ca" pod "5e3b947c-7748-4cf7-a326-e3187e2b2b1d" (UID: "5e3b947c-7748-4cf7-a326-e3187e2b2b1d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:26:47.759191 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:47.759155 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxrm2\" (UniqueName: \"kubernetes.io/projected/5e3b947c-7748-4cf7-a326-e3187e2b2b1d-kube-api-access-jxrm2\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:26:47.759191 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:47.759186 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e3b947c-7748-4cf7-a326-e3187e2b2b1d-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:26:47.759191 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:47.759198 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e3b947c-7748-4cf7-a326-e3187e2b2b1d-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:26:48.396059 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:48.396029 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea415b0-z26b7" Oct 30 17:26:48.396617 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:48.396111 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea415b0-z26b7" event={"ID":"5e3b947c-7748-4cf7-a326-e3187e2b2b1d","Type":"ContainerDied","Data":"922e3979eafb102a4ccafd5961f963b5cc1fd7ec2b22ec4851e019cbf0a40829"} Oct 30 17:26:48.396617 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:48.396163 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="922e3979eafb102a4ccafd5961f963b5cc1fd7ec2b22ec4851e019cbf0a40829" Oct 30 17:26:55.081176 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:55.081084 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d678803a-22zr8"] Oct 30 17:26:55.081888 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:55.081872 4522 cpu_manager.go:414] "containerMap: removing container" podUID="5e3b947c-7748-4cf7-a326-e3187e2b2b1d" containerName="revoke-keys" Oct 30 17:26:55.081940 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:55.081890 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e3b947c-7748-4cf7-a326-e3187e2b2b1d" containerName="revoke-keys" Oct 30 17:26:55.082057 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:55.082047 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e3b947c-7748-4cf7-a326-e3187e2b2b1d" containerName="revoke-keys" Oct 30 17:26:55.086893 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:55.086874 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d678803a-22zr8" Oct 30 17:26:55.098170 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:55.098149 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f89e1d2-dcf7c"] Oct 30 17:26:55.112981 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:55.112948 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f89e1d2-dcf7c"] Oct 30 17:26:55.116005 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:55.115979 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:26:55.116191 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:55.116072 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:26:55.120510 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:55.120484 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d678803a-22zr8"] Oct 30 17:26:55.181486 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:55.181453 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgsqt\" (UniqueName: \"kubernetes.io/projected/b122ae6b-31a7-4085-821e-6f3d69f2076e-kube-api-access-jgsqt\") pod \"maas-api-revoke-sub-d678803a-22zr8\" (UID: \"b122ae6b-31a7-4085-821e-6f3d69f2076e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d678803a-22zr8" Oct 30 17:26:55.181687 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:55.181668 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b122ae6b-31a7-4085-821e-6f3d69f2076e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d678803a-22zr8\" (UID: \"b122ae6b-31a7-4085-821e-6f3d69f2076e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d678803a-22zr8" Oct 30 17:26:55.182244 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:55.182220 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b122ae6b-31a7-4085-821e-6f3d69f2076e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d678803a-22zr8\" (UID: \"b122ae6b-31a7-4085-821e-6f3d69f2076e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d678803a-22zr8" Oct 30 17:26:55.283857 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:55.283817 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgsqt\" (UniqueName: \"kubernetes.io/projected/b122ae6b-31a7-4085-821e-6f3d69f2076e-kube-api-access-jgsqt\") pod \"maas-api-revoke-sub-d678803a-22zr8\" (UID: \"b122ae6b-31a7-4085-821e-6f3d69f2076e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d678803a-22zr8" Oct 30 17:26:55.284043 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:55.283978 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b122ae6b-31a7-4085-821e-6f3d69f2076e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d678803a-22zr8\" (UID: \"b122ae6b-31a7-4085-821e-6f3d69f2076e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d678803a-22zr8" Oct 30 17:26:55.284138 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:55.284117 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b122ae6b-31a7-4085-821e-6f3d69f2076e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d678803a-22zr8\" (UID: \"b122ae6b-31a7-4085-821e-6f3d69f2076e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d678803a-22zr8" Oct 30 17:26:55.284952 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:55.284922 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b122ae6b-31a7-4085-821e-6f3d69f2076e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d678803a-22zr8\" (UID: \"b122ae6b-31a7-4085-821e-6f3d69f2076e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d678803a-22zr8" Oct 30 17:26:55.288826 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:55.288799 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b122ae6b-31a7-4085-821e-6f3d69f2076e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d678803a-22zr8\" (UID: \"b122ae6b-31a7-4085-821e-6f3d69f2076e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d678803a-22zr8" Oct 30 17:26:55.309064 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:55.309033 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgsqt\" (UniqueName: \"kubernetes.io/projected/b122ae6b-31a7-4085-821e-6f3d69f2076e-kube-api-access-jgsqt\") pod \"maas-api-revoke-sub-d678803a-22zr8\" (UID: \"b122ae6b-31a7-4085-821e-6f3d69f2076e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d678803a-22zr8" Oct 30 17:26:55.402593 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:55.402515 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d678803a-22zr8" Oct 30 17:26:55.753376 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:55.746523 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d678803a-22zr8"] Oct 30 17:26:55.774603 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:26:55.774570 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb122ae6b_31a7_4085_821e_6f3d69f2076e.slice/crio-e35046a0c8e063290a1408241febe27704e5483c9b94311d964baa5328e19dff WatchSource:0}: Error finding container e35046a0c8e063290a1408241febe27704e5483c9b94311d964baa5328e19dff: Status 404 returned error can't find the container with id e35046a0c8e063290a1408241febe27704e5483c9b94311d964baa5328e19dff Oct 30 17:26:56.147807 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:56.147775 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc35c2ee-ed6b-4cf3-8bd9-0e13ed0ede13" path="/var/lib/kubelet/pods/cc35c2ee-ed6b-4cf3-8bd9-0e13ed0ede13/volumes" Oct 30 17:26:56.480275 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:56.480241 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d678803a-22zr8" event={"ID":"b122ae6b-31a7-4085-821e-6f3d69f2076e","Type":"ContainerStarted","Data":"d2440bf73ef85a9d632c435af42acdaa93b2f53f9bd837325d0d496ec059acc7"} Oct 30 17:26:56.480501 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:56.480291 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d678803a-22zr8" event={"ID":"b122ae6b-31a7-4085-821e-6f3d69f2076e","Type":"ContainerStarted","Data":"e35046a0c8e063290a1408241febe27704e5483c9b94311d964baa5328e19dff"} Oct 30 17:26:56.528209 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:56.528155 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d678803a-22zr8" podStartSLOduration=1.528142045 podStartE2EDuration="1.528142045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:26:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:26:56.523623353 +0000 UTC m=+2390.524378276" watchObservedRunningTime="2026-10-30 17:26:56.528142045 +0000 UTC m=+2390.528896946" Oct 30 17:26:57.491873 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:57.491838 4522 generic.go:356] "Generic (PLEG): container finished" podID="b122ae6b-31a7-4085-821e-6f3d69f2076e" containerID="d2440bf73ef85a9d632c435af42acdaa93b2f53f9bd837325d0d496ec059acc7" exitCode=0 Oct 30 17:26:57.492276 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:57.491926 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d678803a-22zr8" event={"ID":"b122ae6b-31a7-4085-821e-6f3d69f2076e","Type":"ContainerDied","Data":"d2440bf73ef85a9d632c435af42acdaa93b2f53f9bd837325d0d496ec059acc7"} Oct 30 17:26:58.399676 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:58.399649 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b87f3c-f97f9ffc4-f5qb8" Oct 30 17:26:58.399835 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:58.399718 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b87f3c-85b7f76ff9-mm6sf" Oct 30 17:26:58.627274 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:58.627240 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-11f480/sub-test-model-11f480-kserve-6946856dd5-j4hmg"] Oct 30 17:26:58.719246 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:58.719221 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d678803a-22zr8" Oct 30 17:26:58.866292 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:58.866259 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b122ae6b-31a7-4085-821e-6f3d69f2076e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b122ae6b-31a7-4085-821e-6f3d69f2076e-maas-api-xks-ca\") pod \"b122ae6b-31a7-4085-821e-6f3d69f2076e\" (UID: \"b122ae6b-31a7-4085-821e-6f3d69f2076e\") " Oct 30 17:26:58.866525 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:58.866380 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b122ae6b-31a7-4085-821e-6f3d69f2076e-kube-api-access-jgsqt\" (UniqueName: \"kubernetes.io/projected/b122ae6b-31a7-4085-821e-6f3d69f2076e-kube-api-access-jgsqt\") pod \"b122ae6b-31a7-4085-821e-6f3d69f2076e\" (UID: \"b122ae6b-31a7-4085-821e-6f3d69f2076e\") " Oct 30 17:26:58.866525 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:58.866487 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b122ae6b-31a7-4085-821e-6f3d69f2076e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b122ae6b-31a7-4085-821e-6f3d69f2076e-maas-api-openshift-ca\") pod \"b122ae6b-31a7-4085-821e-6f3d69f2076e\" (UID: \"b122ae6b-31a7-4085-821e-6f3d69f2076e\") " Oct 30 17:26:58.866916 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:58.866884 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b122ae6b-31a7-4085-821e-6f3d69f2076e-maas-api-openshift-ca" pod "b122ae6b-31a7-4085-821e-6f3d69f2076e" (UID: "b122ae6b-31a7-4085-821e-6f3d69f2076e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:26:58.867357 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:58.867291 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b122ae6b-31a7-4085-821e-6f3d69f2076e-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:26:58.868601 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:58.868573 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b122ae6b-31a7-4085-821e-6f3d69f2076e-maas-api-xks-ca" pod "b122ae6b-31a7-4085-821e-6f3d69f2076e" (UID: "b122ae6b-31a7-4085-821e-6f3d69f2076e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:26:58.868956 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:58.868941 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b122ae6b-31a7-4085-821e-6f3d69f2076e-kube-api-access-jgsqt" pod "b122ae6b-31a7-4085-821e-6f3d69f2076e" (UID: "b122ae6b-31a7-4085-821e-6f3d69f2076e"). InnerVolumeSpecName "kube-api-access-jgsqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:26:58.968958 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:58.968920 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b122ae6b-31a7-4085-821e-6f3d69f2076e-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:26:58.968958 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:58.968953 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgsqt\" (UniqueName: \"kubernetes.io/projected/b122ae6b-31a7-4085-821e-6f3d69f2076e-kube-api-access-jgsqt\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:26:59.511743 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:59.511713 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d678803a-22zr8" Oct 30 17:26:59.511743 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:59.511734 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d678803a-22zr8" event={"ID":"b122ae6b-31a7-4085-821e-6f3d69f2076e","Type":"ContainerDied","Data":"e35046a0c8e063290a1408241febe27704e5483c9b94311d964baa5328e19dff"} Oct 30 17:26:59.511958 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:26:59.511781 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e35046a0c8e063290a1408241febe27704e5483c9b94311d964baa5328e19dff" Oct 30 17:27:01.412558 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:01.412524 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75586d-79556b99-k6d8b"] Oct 30 17:27:01.413298 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:01.413281 4522 cpu_manager.go:414] "containerMap: removing container" podUID="b122ae6b-31a7-4085-821e-6f3d69f2076e" containerName="revoke-keys" Oct 30 17:27:01.413298 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:01.413299 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b122ae6b-31a7-4085-821e-6f3d69f2076e" containerName="revoke-keys" Oct 30 17:27:01.413471 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:01.413461 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="b122ae6b-31a7-4085-821e-6f3d69f2076e" containerName="revoke-keys" Oct 30 17:27:01.417489 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:01.417472 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75586d-79556b99-k6d8b" Oct 30 17:27:01.440599 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:01.440559 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75586d-79556b99-k6d8b"] Oct 30 17:27:01.446840 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:01.446816 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-75586d-dockercfg-bknc6\"" Oct 30 17:27:01.490330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:01.482463 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-75586d\"" Oct 30 17:27:01.521327 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:01.521192 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvplq\" (UniqueName: \"kubernetes.io/projected/e25622db-17ca-433c-bd0a-94589349d821-kube-api-access-vvplq\") pod \"payload-pre-processing-e2e-shared-b-w0-75586d-79556b99-k6d8b\" (UID: \"e25622db-17ca-433c-bd0a-94589349d821\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75586d-79556b99-k6d8b" Oct 30 17:27:01.521544 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:01.521366 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e25622db-17ca-433c-bd0a-94589349d821-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-75586d-79556b99-k6d8b\" (UID: \"e25622db-17ca-433c-bd0a-94589349d821\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75586d-79556b99-k6d8b" Oct 30 17:27:01.631397 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:01.631355 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e25622db-17ca-433c-bd0a-94589349d821-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-75586d-79556b99-k6d8b\" (UID: \"e25622db-17ca-433c-bd0a-94589349d821\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75586d-79556b99-k6d8b" Oct 30 17:27:01.631658 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:01.631641 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvplq\" (UniqueName: \"kubernetes.io/projected/e25622db-17ca-433c-bd0a-94589349d821-kube-api-access-vvplq\") pod \"payload-pre-processing-e2e-shared-b-w0-75586d-79556b99-k6d8b\" (UID: \"e25622db-17ca-433c-bd0a-94589349d821\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75586d-79556b99-k6d8b" Oct 30 17:27:01.632096 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:01.632076 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e25622db-17ca-433c-bd0a-94589349d821-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-75586d-79556b99-k6d8b\" (UID: \"e25622db-17ca-433c-bd0a-94589349d821\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75586d-79556b99-k6d8b" Oct 30 17:27:01.657292 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:01.657259 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvplq\" (UniqueName: \"kubernetes.io/projected/e25622db-17ca-433c-bd0a-94589349d821-kube-api-access-vvplq\") pod \"payload-pre-processing-e2e-shared-b-w0-75586d-79556b99-k6d8b\" (UID: \"e25622db-17ca-433c-bd0a-94589349d821\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75586d-79556b99-k6d8b" Oct 30 17:27:01.733633 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:01.733585 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75586d-79556b99-k6d8b" Oct 30 17:27:02.045336 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:27:02.045285 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode25622db_17ca_433c_bd0a_94589349d821.slice/crio-17975eede459bff93b5d17a5f63ed78719e302a208602962186b3ddfc7fb122a WatchSource:0}: Error finding container 17975eede459bff93b5d17a5f63ed78719e302a208602962186b3ddfc7fb122a: Status 404 returned error can't find the container with id 17975eede459bff93b5d17a5f63ed78719e302a208602962186b3ddfc7fb122a Oct 30 17:27:02.047885 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:02.047858 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75586d-79556b99-k6d8b"] Oct 30 17:27:02.592583 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:02.592545 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75586d-79556b99-k6d8b" event={"ID":"e25622db-17ca-433c-bd0a-94589349d821","Type":"ContainerStarted","Data":"96874ed144797dc90b7ec6c43d856e9a91e7c3ccbef6c0a44f93f92dddf13dec"} Oct 30 17:27:02.592997 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:02.592600 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75586d-79556b99-k6d8b" event={"ID":"e25622db-17ca-433c-bd0a-94589349d821","Type":"ContainerStarted","Data":"17975eede459bff93b5d17a5f63ed78719e302a208602962186b3ddfc7fb122a"} Oct 30 17:27:02.592997 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:02.592621 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75586d-79556b99-k6d8b" Oct 30 17:27:02.641824 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:02.641768 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75586d-79556b99-k6d8b" podStartSLOduration=1.6417511519999999 podStartE2EDuration="1.641751152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:27:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:27:02.63782152 +0000 UTC m=+2396.638576422" watchObservedRunningTime="2026-10-30 17:27:02.641751152 +0000 UTC m=+2396.642506052" Oct 30 17:27:06.023361 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:06.023298 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac8bb032-kq9dm"] Oct 30 17:27:06.029204 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:06.029171 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac8bb032-kq9dm" Oct 30 17:27:06.056745 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:06.056699 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac8bb032-kq9dm"] Oct 30 17:27:06.100325 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:06.099659 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:27:06.100325 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:06.099735 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:27:06.124330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:06.114060 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e6bdbf9-3f04-49c9-b38f-9b7d20e14743-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac8bb032-kq9dm\" (UID: \"3e6bdbf9-3f04-49c9-b38f-9b7d20e14743\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac8bb032-kq9dm" Oct 30 17:27:06.124330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:06.114144 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e6bdbf9-3f04-49c9-b38f-9b7d20e14743-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac8bb032-kq9dm\" (UID: \"3e6bdbf9-3f04-49c9-b38f-9b7d20e14743\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac8bb032-kq9dm" Oct 30 17:27:06.124330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:06.114272 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-668l8\" (UniqueName: \"kubernetes.io/projected/3e6bdbf9-3f04-49c9-b38f-9b7d20e14743-kube-api-access-668l8\") pod \"maas-api-revoke-sub-ac8bb032-kq9dm\" (UID: \"3e6bdbf9-3f04-49c9-b38f-9b7d20e14743\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac8bb032-kq9dm" Oct 30 17:27:06.220329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:06.219536 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-668l8\" (UniqueName: \"kubernetes.io/projected/3e6bdbf9-3f04-49c9-b38f-9b7d20e14743-kube-api-access-668l8\") pod \"maas-api-revoke-sub-ac8bb032-kq9dm\" (UID: \"3e6bdbf9-3f04-49c9-b38f-9b7d20e14743\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac8bb032-kq9dm" Oct 30 17:27:06.220329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:06.219794 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e6bdbf9-3f04-49c9-b38f-9b7d20e14743-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac8bb032-kq9dm\" (UID: \"3e6bdbf9-3f04-49c9-b38f-9b7d20e14743\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac8bb032-kq9dm" Oct 30 17:27:06.220329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:06.219891 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e6bdbf9-3f04-49c9-b38f-9b7d20e14743-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac8bb032-kq9dm\" (UID: \"3e6bdbf9-3f04-49c9-b38f-9b7d20e14743\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac8bb032-kq9dm" Oct 30 17:27:06.221880 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:06.221839 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e6bdbf9-3f04-49c9-b38f-9b7d20e14743-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac8bb032-kq9dm\" (UID: \"3e6bdbf9-3f04-49c9-b38f-9b7d20e14743\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac8bb032-kq9dm" Oct 30 17:27:06.235875 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:06.235848 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:27:06.243372 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:06.243340 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e6bdbf9-3f04-49c9-b38f-9b7d20e14743-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac8bb032-kq9dm\" (UID: \"3e6bdbf9-3f04-49c9-b38f-9b7d20e14743\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac8bb032-kq9dm" Oct 30 17:27:06.245581 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:06.245547 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-668l8\" (UniqueName: \"kubernetes.io/projected/3e6bdbf9-3f04-49c9-b38f-9b7d20e14743-kube-api-access-668l8\") pod \"maas-api-revoke-sub-ac8bb032-kq9dm\" (UID: \"3e6bdbf9-3f04-49c9-b38f-9b7d20e14743\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac8bb032-kq9dm" Oct 30 17:27:06.453783 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:06.453732 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:27:06.458752 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:06.458510 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac8bb032-kq9dm" Oct 30 17:27:06.856331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:06.843664 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-6d4f99/sub-test-model-6d4f99-kserve-9ddf765b5-r59ft"] Oct 30 17:27:07.154033 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:07.114891 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d44f8491-5k49b"] Oct 30 17:27:07.233469 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:07.233441 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d44f8491-5k49b"] Oct 30 17:27:07.683714 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:07.683676 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac8bb032-kq9dm"] Oct 30 17:27:07.738720 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:07.738687 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac8bb032-kq9dm" event={"ID":"3e6bdbf9-3f04-49c9-b38f-9b7d20e14743","Type":"ContainerStarted","Data":"1b90336db71c5458ba97d4d00d6e702fe67ce72e149ecf9d9f4903832bb1b26f"} Oct 30 17:27:08.146983 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:08.146950 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fada1708-60a3-43d6-b7cc-8884c8e11341" path="/var/lib/kubelet/pods/fada1708-60a3-43d6-b7cc-8884c8e11341/volumes" Oct 30 17:27:08.748574 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:08.748543 4522 generic.go:356] "Generic (PLEG): container finished" podID="3e6bdbf9-3f04-49c9-b38f-9b7d20e14743" containerID="9fb0e8c52b35a00a4ba1e4c5044bba645048229b0f5a156cee5f775f2a14ea81" exitCode=0 Oct 30 17:27:08.748956 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:08.748634 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac8bb032-kq9dm" event={"ID":"3e6bdbf9-3f04-49c9-b38f-9b7d20e14743","Type":"ContainerDied","Data":"9fb0e8c52b35a00a4ba1e4c5044bba645048229b0f5a156cee5f775f2a14ea81"} Oct 30 17:27:09.382396 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:09.382364 4522 scope.go:122] "RemoveContainer" containerID="e33fa64c57006dc50543fc442c4d19f5e7c3e3124b65c0b28eccc70ce507355d" Oct 30 17:27:09.393262 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:09.393241 4522 scope.go:122] "RemoveContainer" containerID="efa661720c8b048df143196128e076e3d0f4b94c92ff966bfd6932ad5cb94d63" Oct 30 17:27:09.403219 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:09.403192 4522 scope.go:122] "RemoveContainer" containerID="5199b0bb903fd1ced149feb230811eaaa4d92c725f7c96c5899f710ebc50408e" Oct 30 17:27:09.923614 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:09.923586 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac8bb032-kq9dm" Oct 30 17:27:10.016533 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:10.016448 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e6bdbf9-3f04-49c9-b38f-9b7d20e14743-kube-api-access-668l8\" (UniqueName: \"kubernetes.io/projected/3e6bdbf9-3f04-49c9-b38f-9b7d20e14743-kube-api-access-668l8\") pod \"3e6bdbf9-3f04-49c9-b38f-9b7d20e14743\" (UID: \"3e6bdbf9-3f04-49c9-b38f-9b7d20e14743\") " Oct 30 17:27:10.016676 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:10.016538 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e6bdbf9-3f04-49c9-b38f-9b7d20e14743-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e6bdbf9-3f04-49c9-b38f-9b7d20e14743-maas-api-openshift-ca\") pod \"3e6bdbf9-3f04-49c9-b38f-9b7d20e14743\" (UID: \"3e6bdbf9-3f04-49c9-b38f-9b7d20e14743\") " Oct 30 17:27:10.016783 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:10.016763 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e6bdbf9-3f04-49c9-b38f-9b7d20e14743-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e6bdbf9-3f04-49c9-b38f-9b7d20e14743-maas-api-xks-ca\") pod \"3e6bdbf9-3f04-49c9-b38f-9b7d20e14743\" (UID: \"3e6bdbf9-3f04-49c9-b38f-9b7d20e14743\") " Oct 30 17:27:10.016902 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:10.016880 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e6bdbf9-3f04-49c9-b38f-9b7d20e14743-maas-api-openshift-ca" pod "3e6bdbf9-3f04-49c9-b38f-9b7d20e14743" (UID: "3e6bdbf9-3f04-49c9-b38f-9b7d20e14743"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:27:10.017443 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:10.017420 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e6bdbf9-3f04-49c9-b38f-9b7d20e14743-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:27:10.018790 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:10.018762 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e6bdbf9-3f04-49c9-b38f-9b7d20e14743-maas-api-xks-ca" pod "3e6bdbf9-3f04-49c9-b38f-9b7d20e14743" (UID: "3e6bdbf9-3f04-49c9-b38f-9b7d20e14743"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:27:10.018876 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:10.018805 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e6bdbf9-3f04-49c9-b38f-9b7d20e14743-kube-api-access-668l8" pod "3e6bdbf9-3f04-49c9-b38f-9b7d20e14743" (UID: "3e6bdbf9-3f04-49c9-b38f-9b7d20e14743"). InnerVolumeSpecName "kube-api-access-668l8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:27:10.118952 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:10.118920 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e6bdbf9-3f04-49c9-b38f-9b7d20e14743-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:27:10.118952 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:10.118948 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-668l8\" (UniqueName: \"kubernetes.io/projected/3e6bdbf9-3f04-49c9-b38f-9b7d20e14743-kube-api-access-668l8\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:27:10.766564 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:10.766530 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac8bb032-kq9dm" Oct 30 17:27:10.766724 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:10.766571 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac8bb032-kq9dm" event={"ID":"3e6bdbf9-3f04-49c9-b38f-9b7d20e14743","Type":"ContainerDied","Data":"1b90336db71c5458ba97d4d00d6e702fe67ce72e149ecf9d9f4903832bb1b26f"} Oct 30 17:27:10.766724 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:10.766620 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b90336db71c5458ba97d4d00d6e702fe67ce72e149ecf9d9f4903832bb1b26f" Oct 30 17:27:12.959359 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:27:12.957935 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:27:13.604005 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:13.603974 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75586d-79556b99-k6d8b" Oct 30 17:27:13.659244 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:13.659173 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-11f480/sub-test-model-11f480-kserve-6946856dd5-j4hmg" podUID="d98b9965-90fe-44d6-b5de-9f861148d07a" containerName="main" containerID="cri-o://ea9b773c9a84fb0fcbc56b6f6884693e98bb1b3c75f1ae3d83b70cd7f663e3bb" gracePeriod=45 Oct 30 17:27:13.801515 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:13.801481 4522 generic.go:356] "Generic (PLEG): container finished" podID="d98b9965-90fe-44d6-b5de-9f861148d07a" containerID="ea9b773c9a84fb0fcbc56b6f6884693e98bb1b3c75f1ae3d83b70cd7f663e3bb" exitCode=0 Oct 30 17:27:13.801710 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:13.801558 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-11f480/sub-test-model-11f480-kserve-6946856dd5-j4hmg" event={"ID":"d98b9965-90fe-44d6-b5de-9f861148d07a","Type":"ContainerDied","Data":"ea9b773c9a84fb0fcbc56b6f6884693e98bb1b3c75f1ae3d83b70cd7f663e3bb"} Oct 30 17:27:13.947825 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:13.947794 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-11f480/sub-test-model-11f480-kserve-6946856dd5-j4hmg" Oct 30 17:27:14.071160 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.071126 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-424ecfc9-ds2z7"] Oct 30 17:27:14.072003 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.071984 4522 cpu_manager.go:414] "containerMap: removing container" podUID="d98b9965-90fe-44d6-b5de-9f861148d07a" containerName="main" Oct 30 17:27:14.072003 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.072003 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d98b9965-90fe-44d6-b5de-9f861148d07a" containerName="main" Oct 30 17:27:14.072127 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.072044 4522 cpu_manager.go:414] "containerMap: removing container" podUID="3e6bdbf9-3f04-49c9-b38f-9b7d20e14743" containerName="revoke-keys" Oct 30 17:27:14.072127 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.072050 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e6bdbf9-3f04-49c9-b38f-9b7d20e14743" containerName="revoke-keys" Oct 30 17:27:14.072196 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.072179 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="d98b9965-90fe-44d6-b5de-9f861148d07a" containerName="main" Oct 30 17:27:14.072196 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.072191 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e6bdbf9-3f04-49c9-b38f-9b7d20e14743" containerName="revoke-keys" Oct 30 17:27:14.076216 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.076187 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-424ecfc9-ds2z7" Oct 30 17:27:14.081523 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.081498 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d98b9965-90fe-44d6-b5de-9f861148d07a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d98b9965-90fe-44d6-b5de-9f861148d07a-dshm\") pod \"d98b9965-90fe-44d6-b5de-9f861148d07a\" (UID: \"d98b9965-90fe-44d6-b5de-9f861148d07a\") " Oct 30 17:27:14.081736 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.081715 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d98b9965-90fe-44d6-b5de-9f861148d07a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d98b9965-90fe-44d6-b5de-9f861148d07a-model-cache\") pod \"d98b9965-90fe-44d6-b5de-9f861148d07a\" (UID: \"d98b9965-90fe-44d6-b5de-9f861148d07a\") " Oct 30 17:27:14.081891 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.081876 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d98b9965-90fe-44d6-b5de-9f861148d07a-tls-certs\" (UniqueName: \"kubernetes.io/secret/d98b9965-90fe-44d6-b5de-9f861148d07a-tls-certs\") pod \"d98b9965-90fe-44d6-b5de-9f861148d07a\" (UID: \"d98b9965-90fe-44d6-b5de-9f861148d07a\") " Oct 30 17:27:14.082038 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.082014 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d98b9965-90fe-44d6-b5de-9f861148d07a-model-cache" pod "d98b9965-90fe-44d6-b5de-9f861148d07a" (UID: "d98b9965-90fe-44d6-b5de-9f861148d07a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:27:14.082146 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.082131 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d98b9965-90fe-44d6-b5de-9f861148d07a-home\" (UniqueName: \"kubernetes.io/empty-dir/d98b9965-90fe-44d6-b5de-9f861148d07a-home\") pod \"d98b9965-90fe-44d6-b5de-9f861148d07a\" (UID: \"d98b9965-90fe-44d6-b5de-9f861148d07a\") " Oct 30 17:27:14.082269 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.082253 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d98b9965-90fe-44d6-b5de-9f861148d07a-kube-api-access-bmdms\" (UniqueName: \"kubernetes.io/projected/d98b9965-90fe-44d6-b5de-9f861148d07a-kube-api-access-bmdms\") pod \"d98b9965-90fe-44d6-b5de-9f861148d07a\" (UID: \"d98b9965-90fe-44d6-b5de-9f861148d07a\") " Oct 30 17:27:14.082399 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.082384 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d98b9965-90fe-44d6-b5de-9f861148d07a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d98b9965-90fe-44d6-b5de-9f861148d07a-tmp-dir\") pod \"d98b9965-90fe-44d6-b5de-9f861148d07a\" (UID: \"d98b9965-90fe-44d6-b5de-9f861148d07a\") " Oct 30 17:27:14.082666 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.082640 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d98b9965-90fe-44d6-b5de-9f861148d07a-home" pod "d98b9965-90fe-44d6-b5de-9f861148d07a" (UID: "d98b9965-90fe-44d6-b5de-9f861148d07a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:27:14.083971 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.083783 4522 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d98b9965-90fe-44d6-b5de-9f861148d07a-model-cache\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:27:14.083971 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.083808 4522 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d98b9965-90fe-44d6-b5de-9f861148d07a-home\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:27:14.084394 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.084370 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d98b9965-90fe-44d6-b5de-9f861148d07a-tmp-dir" pod "d98b9965-90fe-44d6-b5de-9f861148d07a" (UID: "d98b9965-90fe-44d6-b5de-9f861148d07a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:27:14.084565 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.084544 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d98b9965-90fe-44d6-b5de-9f861148d07a-dshm" pod "d98b9965-90fe-44d6-b5de-9f861148d07a" (UID: "d98b9965-90fe-44d6-b5de-9f861148d07a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:27:14.086349 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.086322 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d98b9965-90fe-44d6-b5de-9f861148d07a-tls-certs" pod "d98b9965-90fe-44d6-b5de-9f861148d07a" (UID: "d98b9965-90fe-44d6-b5de-9f861148d07a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:27:14.086445 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.086402 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d98b9965-90fe-44d6-b5de-9f861148d07a-kube-api-access-bmdms" pod "d98b9965-90fe-44d6-b5de-9f861148d07a" (UID: "d98b9965-90fe-44d6-b5de-9f861148d07a"). InnerVolumeSpecName "kube-api-access-bmdms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:27:14.092481 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.092456 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:27:14.103849 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.103824 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-424ecfc9-ds2z7"] Oct 30 17:27:14.173198 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.173138 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:27:14.185221 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.185193 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b54956ec-9536-4d68-af21-67b316b5b169-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-424ecfc9-ds2z7\" (UID: \"b54956ec-9536-4d68-af21-67b316b5b169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-424ecfc9-ds2z7" Oct 30 17:27:14.185351 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.185234 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b54956ec-9536-4d68-af21-67b316b5b169-maas-api-xks-ca\") pod \"maas-api-revoke-sub-424ecfc9-ds2z7\" (UID: \"b54956ec-9536-4d68-af21-67b316b5b169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-424ecfc9-ds2z7" Oct 30 17:27:14.185505 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.185492 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj55s\" (UniqueName: \"kubernetes.io/projected/b54956ec-9536-4d68-af21-67b316b5b169-kube-api-access-pj55s\") pod \"maas-api-revoke-sub-424ecfc9-ds2z7\" (UID: \"b54956ec-9536-4d68-af21-67b316b5b169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-424ecfc9-ds2z7" Oct 30 17:27:14.185622 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.185611 4522 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d98b9965-90fe-44d6-b5de-9f861148d07a-dshm\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:27:14.185664 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.185625 4522 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d98b9965-90fe-44d6-b5de-9f861148d07a-tls-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:27:14.185664 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.185635 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmdms\" (UniqueName: \"kubernetes.io/projected/d98b9965-90fe-44d6-b5de-9f861148d07a-kube-api-access-bmdms\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:27:14.185664 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.185644 4522 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d98b9965-90fe-44d6-b5de-9f861148d07a-tmp-dir\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:27:14.223060 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.223030 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-41cb25-db9f8d799-t27kf"] Oct 30 17:27:14.228188 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.228167 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41cb25-db9f8d799-t27kf" Oct 30 17:27:14.245373 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.245339 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-41cb25-dockercfg-tllvr\"" Oct 30 17:27:14.245486 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.245426 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-41cb25\"" Oct 30 17:27:14.248758 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.248733 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-41cb25-db9f8d799-t27kf"] Oct 30 17:27:14.289154 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.287032 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj55s\" (UniqueName: \"kubernetes.io/projected/b54956ec-9536-4d68-af21-67b316b5b169-kube-api-access-pj55s\") pod \"maas-api-revoke-sub-424ecfc9-ds2z7\" (UID: \"b54956ec-9536-4d68-af21-67b316b5b169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-424ecfc9-ds2z7" Oct 30 17:27:14.289154 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.287181 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b54956ec-9536-4d68-af21-67b316b5b169-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-424ecfc9-ds2z7\" (UID: \"b54956ec-9536-4d68-af21-67b316b5b169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-424ecfc9-ds2z7" Oct 30 17:27:14.289154 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.287230 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b54956ec-9536-4d68-af21-67b316b5b169-maas-api-xks-ca\") pod \"maas-api-revoke-sub-424ecfc9-ds2z7\" (UID: \"b54956ec-9536-4d68-af21-67b316b5b169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-424ecfc9-ds2z7" Oct 30 17:27:14.289154 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.288795 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b54956ec-9536-4d68-af21-67b316b5b169-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-424ecfc9-ds2z7\" (UID: \"b54956ec-9536-4d68-af21-67b316b5b169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-424ecfc9-ds2z7" Oct 30 17:27:14.290773 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.290747 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b54956ec-9536-4d68-af21-67b316b5b169-maas-api-xks-ca\") pod \"maas-api-revoke-sub-424ecfc9-ds2z7\" (UID: \"b54956ec-9536-4d68-af21-67b316b5b169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-424ecfc9-ds2z7" Oct 30 17:27:14.381841 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.376220 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-41cb25-56b855cbbb-x7mzc"] Oct 30 17:27:14.422361 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.400008 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dce37be9-f255-4c2d-8203-93b5058d86bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-41cb25-db9f8d799-t27kf\" (UID: \"dce37be9-f255-4c2d-8203-93b5058d86bf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41cb25-db9f8d799-t27kf" Oct 30 17:27:14.422361 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.400090 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnkdd\" (UniqueName: \"kubernetes.io/projected/dce37be9-f255-4c2d-8203-93b5058d86bf-kube-api-access-rnkdd\") pod \"payload-pre-processing-e2e-ipp-a-41cb25-db9f8d799-t27kf\" (UID: \"dce37be9-f255-4c2d-8203-93b5058d86bf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41cb25-db9f8d799-t27kf" Oct 30 17:27:14.422361 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.402146 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj55s\" (UniqueName: \"kubernetes.io/projected/b54956ec-9536-4d68-af21-67b316b5b169-kube-api-access-pj55s\") pod \"maas-api-revoke-sub-424ecfc9-ds2z7\" (UID: \"b54956ec-9536-4d68-af21-67b316b5b169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-424ecfc9-ds2z7" Oct 30 17:27:14.450462 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.440864 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-424ecfc9-ds2z7" Oct 30 17:27:14.450462 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.447153 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-41cb25-56b855cbbb-x7mzc" Oct 30 17:27:14.474251 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.473225 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-41cb25-56b855cbbb-x7mzc"] Oct 30 17:27:14.501588 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.501546 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttj2k\" (UniqueName: \"kubernetes.io/projected/7e7ab767-5219-47a3-988f-fb4240a88652-kube-api-access-ttj2k\") pod \"payload-processing-e2e-ipp-a-41cb25-56b855cbbb-x7mzc\" (UID: \"7e7ab767-5219-47a3-988f-fb4240a88652\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-41cb25-56b855cbbb-x7mzc" Oct 30 17:27:14.501776 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.501630 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e7ab767-5219-47a3-988f-fb4240a88652-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-41cb25-56b855cbbb-x7mzc\" (UID: \"7e7ab767-5219-47a3-988f-fb4240a88652\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-41cb25-56b855cbbb-x7mzc" Oct 30 17:27:14.501776 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.501702 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dce37be9-f255-4c2d-8203-93b5058d86bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-41cb25-db9f8d799-t27kf\" (UID: \"dce37be9-f255-4c2d-8203-93b5058d86bf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41cb25-db9f8d799-t27kf" Oct 30 17:27:14.501776 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.501746 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnkdd\" (UniqueName: \"kubernetes.io/projected/dce37be9-f255-4c2d-8203-93b5058d86bf-kube-api-access-rnkdd\") pod \"payload-pre-processing-e2e-ipp-a-41cb25-db9f8d799-t27kf\" (UID: \"dce37be9-f255-4c2d-8203-93b5058d86bf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41cb25-db9f8d799-t27kf" Oct 30 17:27:14.502759 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.502731 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dce37be9-f255-4c2d-8203-93b5058d86bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-41cb25-db9f8d799-t27kf\" (UID: \"dce37be9-f255-4c2d-8203-93b5058d86bf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41cb25-db9f8d799-t27kf" Oct 30 17:27:14.528227 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.528197 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnkdd\" (UniqueName: \"kubernetes.io/projected/dce37be9-f255-4c2d-8203-93b5058d86bf-kube-api-access-rnkdd\") pod \"payload-pre-processing-e2e-ipp-a-41cb25-db9f8d799-t27kf\" (UID: \"dce37be9-f255-4c2d-8203-93b5058d86bf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41cb25-db9f8d799-t27kf" Oct 30 17:27:14.543620 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.543591 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41cb25-db9f8d799-t27kf" Oct 30 17:27:14.603807 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.603757 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttj2k\" (UniqueName: \"kubernetes.io/projected/7e7ab767-5219-47a3-988f-fb4240a88652-kube-api-access-ttj2k\") pod \"payload-processing-e2e-ipp-a-41cb25-56b855cbbb-x7mzc\" (UID: \"7e7ab767-5219-47a3-988f-fb4240a88652\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-41cb25-56b855cbbb-x7mzc" Oct 30 17:27:14.603967 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.603901 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e7ab767-5219-47a3-988f-fb4240a88652-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-41cb25-56b855cbbb-x7mzc\" (UID: \"7e7ab767-5219-47a3-988f-fb4240a88652\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-41cb25-56b855cbbb-x7mzc" Oct 30 17:27:14.605129 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.605099 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e7ab767-5219-47a3-988f-fb4240a88652-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-41cb25-56b855cbbb-x7mzc\" (UID: \"7e7ab767-5219-47a3-988f-fb4240a88652\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-41cb25-56b855cbbb-x7mzc" Oct 30 17:27:14.631929 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.631882 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttj2k\" (UniqueName: \"kubernetes.io/projected/7e7ab767-5219-47a3-988f-fb4240a88652-kube-api-access-ttj2k\") pod \"payload-processing-e2e-ipp-a-41cb25-56b855cbbb-x7mzc\" (UID: \"7e7ab767-5219-47a3-988f-fb4240a88652\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-41cb25-56b855cbbb-x7mzc" Oct 30 17:27:14.787010 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.786856 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-41cb25-56b855cbbb-x7mzc" Oct 30 17:27:14.791672 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.791572 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-424ecfc9-ds2z7"] Oct 30 17:27:14.792509 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:27:14.792484 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb54956ec_9536_4d68_af21_67b316b5b169.slice/crio-e89f40a33bb179fd5630bedd73e3657ac3eb402bd4b7ba204902ee3efced2e5e WatchSource:0}: Error finding container e89f40a33bb179fd5630bedd73e3657ac3eb402bd4b7ba204902ee3efced2e5e: Status 404 returned error can't find the container with id e89f40a33bb179fd5630bedd73e3657ac3eb402bd4b7ba204902ee3efced2e5e Oct 30 17:27:14.813772 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.813749 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-11f480/sub-test-model-11f480-kserve-6946856dd5-j4hmg" Oct 30 17:27:14.814554 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.814478 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-11f480/sub-test-model-11f480-kserve-6946856dd5-j4hmg" event={"ID":"d98b9965-90fe-44d6-b5de-9f861148d07a","Type":"ContainerDied","Data":"478dc94e718fc83460e5c32f60df443606a213c73df6a159bbab574d5a7faf8e"} Oct 30 17:27:14.814554 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.814527 4522 scope.go:122] "RemoveContainer" containerID="ea9b773c9a84fb0fcbc56b6f6884693e98bb1b3c75f1ae3d83b70cd7f663e3bb" Oct 30 17:27:14.816456 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.816428 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-424ecfc9-ds2z7" event={"ID":"b54956ec-9536-4d68-af21-67b316b5b169","Type":"ContainerStarted","Data":"e89f40a33bb179fd5630bedd73e3657ac3eb402bd4b7ba204902ee3efced2e5e"} Oct 30 17:27:14.883135 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.883106 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-41cb25-db9f8d799-t27kf"] Oct 30 17:27:14.888285 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:27:14.887296 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce37be9_f255_4c2d_8203_93b5058d86bf.slice/crio-89564b597971ebb500523d1b8bbd1189eefd1093f674e262bf29cff5c7b367bb WatchSource:0}: Error finding container 89564b597971ebb500523d1b8bbd1189eefd1093f674e262bf29cff5c7b367bb: Status 404 returned error can't find the container with id 89564b597971ebb500523d1b8bbd1189eefd1093f674e262bf29cff5c7b367bb Oct 30 17:27:14.903591 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.903565 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-11f480/sub-test-model-11f480-kserve-6946856dd5-j4hmg"] Oct 30 17:27:14.910648 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:14.910622 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-11f480/sub-test-model-11f480-kserve-6946856dd5-j4hmg"] Oct 30 17:27:15.119266 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:27:15.119239 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e7ab767_5219_47a3_988f_fb4240a88652.slice/crio-4da1ec0d2f858d69433d2e5a62ac5efa374dbf6121462bbcc366429eb583650b WatchSource:0}: Error finding container 4da1ec0d2f858d69433d2e5a62ac5efa374dbf6121462bbcc366429eb583650b: Status 404 returned error can't find the container with id 4da1ec0d2f858d69433d2e5a62ac5efa374dbf6121462bbcc366429eb583650b Oct 30 17:27:15.122756 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:15.122736 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-41cb25-56b855cbbb-x7mzc"] Oct 30 17:27:15.839451 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:15.839405 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41cb25-db9f8d799-t27kf" event={"ID":"dce37be9-f255-4c2d-8203-93b5058d86bf","Type":"ContainerStarted","Data":"d7a1be5fc1bea3d37a890fb9341706d8b5987c6b3fae634f18228407b7315d69"} Oct 30 17:27:15.839658 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:15.839476 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41cb25-db9f8d799-t27kf" event={"ID":"dce37be9-f255-4c2d-8203-93b5058d86bf","Type":"ContainerStarted","Data":"89564b597971ebb500523d1b8bbd1189eefd1093f674e262bf29cff5c7b367bb"} Oct 30 17:27:15.839759 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:15.839739 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41cb25-db9f8d799-t27kf" Oct 30 17:27:15.844435 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:15.844403 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-424ecfc9-ds2z7" event={"ID":"b54956ec-9536-4d68-af21-67b316b5b169","Type":"ContainerStarted","Data":"23d6552ae62ccd4b24db33537b2f72820d0335076fd9b1a8ba821dab10a91ee0"} Oct 30 17:27:15.846598 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:15.846559 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-41cb25-56b855cbbb-x7mzc" event={"ID":"7e7ab767-5219-47a3-988f-fb4240a88652","Type":"ContainerStarted","Data":"bd4aade4914249fb22638a2d7ac8557f7d3a3d5a1c17187a63ff949abecc3f5a"} Oct 30 17:27:15.846719 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:15.846622 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-41cb25-56b855cbbb-x7mzc" event={"ID":"7e7ab767-5219-47a3-988f-fb4240a88652","Type":"ContainerStarted","Data":"4da1ec0d2f858d69433d2e5a62ac5efa374dbf6121462bbcc366429eb583650b"} Oct 30 17:27:15.846719 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:15.846662 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-41cb25-56b855cbbb-x7mzc" Oct 30 17:27:15.890523 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:15.890466 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41cb25-db9f8d799-t27kf" podStartSLOduration=1.890451688 podStartE2EDuration="1.890451688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:27:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:27:15.88517058 +0000 UTC m=+2409.885925482" watchObservedRunningTime="2026-10-30 17:27:15.890451688 +0000 UTC m=+2409.891206586" Oct 30 17:27:15.939061 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:15.938992 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-abe243-c6b579d8-t25w4"] Oct 30 17:27:15.949627 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:15.948004 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-abe243-c6b579d8-t25w4" Oct 30 17:27:15.953119 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:15.952109 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-424ecfc9-ds2z7" podStartSLOduration=1.952091166 podStartE2EDuration="1.952091166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:27:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:27:15.931981425 +0000 UTC m=+2409.932736324" watchObservedRunningTime="2026-10-30 17:27:15.952091166 +0000 UTC m=+2409.952846074" Oct 30 17:27:15.964889 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:15.964857 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-abe243\"" Oct 30 17:27:15.967496 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:15.966528 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-abe243-dockercfg-zmf9h\"" Oct 30 17:27:15.967496 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:15.967128 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-abe243-c6b579d8-t25w4"] Oct 30 17:27:15.986268 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:15.983352 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-41cb25-56b855cbbb-x7mzc" podStartSLOduration=1.9833359590000001 podStartE2EDuration="1.983335959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:27:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:27:15.980196217 +0000 UTC m=+2409.980951139" watchObservedRunningTime="2026-10-30 17:27:15.983335959 +0000 UTC m=+2409.984090857" Oct 30 17:27:16.032919 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:16.032889 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ct5t\" (UniqueName: \"kubernetes.io/projected/3ea67f7d-62df-49ff-a3c2-6395420ae0b9-kube-api-access-8ct5t\") pod \"payload-processing-e2e-ipp-a-abe243-c6b579d8-t25w4\" (UID: \"3ea67f7d-62df-49ff-a3c2-6395420ae0b9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-abe243-c6b579d8-t25w4" Oct 30 17:27:16.033096 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:16.033023 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ea67f7d-62df-49ff-a3c2-6395420ae0b9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-abe243-c6b579d8-t25w4\" (UID: \"3ea67f7d-62df-49ff-a3c2-6395420ae0b9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-abe243-c6b579d8-t25w4" Oct 30 17:27:16.134587 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:16.134497 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ct5t\" (UniqueName: \"kubernetes.io/projected/3ea67f7d-62df-49ff-a3c2-6395420ae0b9-kube-api-access-8ct5t\") pod \"payload-processing-e2e-ipp-a-abe243-c6b579d8-t25w4\" (UID: \"3ea67f7d-62df-49ff-a3c2-6395420ae0b9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-abe243-c6b579d8-t25w4" Oct 30 17:27:16.135064 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:16.134638 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ea67f7d-62df-49ff-a3c2-6395420ae0b9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-abe243-c6b579d8-t25w4\" (UID: \"3ea67f7d-62df-49ff-a3c2-6395420ae0b9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-abe243-c6b579d8-t25w4" Oct 30 17:27:16.135498 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:16.135471 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ea67f7d-62df-49ff-a3c2-6395420ae0b9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-abe243-c6b579d8-t25w4\" (UID: \"3ea67f7d-62df-49ff-a3c2-6395420ae0b9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-abe243-c6b579d8-t25w4" Oct 30 17:27:16.150134 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:16.150098 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d98b9965-90fe-44d6-b5de-9f861148d07a" path="/var/lib/kubelet/pods/d98b9965-90fe-44d6-b5de-9f861148d07a/volumes" Oct 30 17:27:16.159799 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:16.159774 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ct5t\" (UniqueName: \"kubernetes.io/projected/3ea67f7d-62df-49ff-a3c2-6395420ae0b9-kube-api-access-8ct5t\") pod \"payload-processing-e2e-ipp-a-abe243-c6b579d8-t25w4\" (UID: \"3ea67f7d-62df-49ff-a3c2-6395420ae0b9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-abe243-c6b579d8-t25w4" Oct 30 17:27:16.319886 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:16.319849 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-abe243-c6b579d8-t25w4" Oct 30 17:27:16.597018 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:16.596989 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d2d985-bdd4b595-zmgvf"] Oct 30 17:27:16.602967 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:16.602945 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d2d985-bdd4b595-zmgvf" Oct 30 17:27:16.638547 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:16.638471 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-d2d985-serving-cert\"" Oct 30 17:27:16.646550 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:16.642716 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6197c2d3-3465-4091-857f-9d8ccdcc2eb3-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d2d985-bdd4b595-zmgvf\" (UID: \"6197c2d3-3465-4091-857f-9d8ccdcc2eb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d2d985-bdd4b595-zmgvf" Oct 30 17:27:16.646550 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:16.642811 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6197c2d3-3465-4091-857f-9d8ccdcc2eb3-metrics-certs\") pod \"maas-api-e2e-ipp-a-d2d985-bdd4b595-zmgvf\" (UID: \"6197c2d3-3465-4091-857f-9d8ccdcc2eb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d2d985-bdd4b595-zmgvf" Oct 30 17:27:16.646550 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:16.642860 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-755cj\" (UniqueName: \"kubernetes.io/projected/6197c2d3-3465-4091-857f-9d8ccdcc2eb3-kube-api-access-755cj\") pod \"maas-api-e2e-ipp-a-d2d985-bdd4b595-zmgvf\" (UID: \"6197c2d3-3465-4091-857f-9d8ccdcc2eb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d2d985-bdd4b595-zmgvf" Oct 30 17:27:16.646550 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:16.642888 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6197c2d3-3465-4091-857f-9d8ccdcc2eb3-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d2d985-bdd4b595-zmgvf\" (UID: \"6197c2d3-3465-4091-857f-9d8ccdcc2eb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d2d985-bdd4b595-zmgvf" Oct 30 17:27:16.646550 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:16.643075 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d2d985-bdd4b595-zmgvf"] Oct 30 17:27:16.648153 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:16.648127 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-abe243-c6b579d8-t25w4"] Oct 30 17:27:16.681333 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:27:16.670190 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ea67f7d_62df_49ff_a3c2_6395420ae0b9.slice/crio-aa9dc01910bb9a9b5ac17e7b756a4c1185331d47b0cde8a0bfec6259aaa6dafb WatchSource:0}: Error finding container aa9dc01910bb9a9b5ac17e7b756a4c1185331d47b0cde8a0bfec6259aaa6dafb: Status 404 returned error can't find the container with id aa9dc01910bb9a9b5ac17e7b756a4c1185331d47b0cde8a0bfec6259aaa6dafb Oct 30 17:27:16.681333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:16.672990 4522 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:27:16.717857 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:16.717829 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a20b19b-7krsq"] Oct 30 17:27:16.722997 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:16.722978 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a20b19b-7krsq" Oct 30 17:27:16.744540 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:16.744514 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a20b19b-7krsq"] Oct 30 17:27:16.745098 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:16.745069 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5a593f9-c4b1-45a5-aef4-f88b8bc76e5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a20b19b-7krsq\" (UID: \"e5a593f9-c4b1-45a5-aef4-f88b8bc76e5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a20b19b-7krsq" Oct 30 17:27:16.745245 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:16.745119 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6197c2d3-3465-4091-857f-9d8ccdcc2eb3-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d2d985-bdd4b595-zmgvf\" (UID: \"6197c2d3-3465-4091-857f-9d8ccdcc2eb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d2d985-bdd4b595-zmgvf" Oct 30 17:27:16.745245 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:16.745237 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6197c2d3-3465-4091-857f-9d8ccdcc2eb3-metrics-certs\") pod \"maas-api-e2e-ipp-a-d2d985-bdd4b595-zmgvf\" (UID: \"6197c2d3-3465-4091-857f-9d8ccdcc2eb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d2d985-bdd4b595-zmgvf" Oct 30 17:27:16.745509 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:16.745294 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-755cj\" (UniqueName: \"kubernetes.io/projected/6197c2d3-3465-4091-857f-9d8ccdcc2eb3-kube-api-access-755cj\") pod \"maas-api-e2e-ipp-a-d2d985-bdd4b595-zmgvf\" (UID: \"6197c2d3-3465-4091-857f-9d8ccdcc2eb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d2d985-bdd4b595-zmgvf" Oct 30 17:27:16.745509 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:16.745363 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6197c2d3-3465-4091-857f-9d8ccdcc2eb3-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d2d985-bdd4b595-zmgvf\" (UID: \"6197c2d3-3465-4091-857f-9d8ccdcc2eb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d2d985-bdd4b595-zmgvf" Oct 30 17:27:16.745751 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:16.745676 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5a593f9-c4b1-45a5-aef4-f88b8bc76e5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a20b19b-7krsq\" (UID: \"e5a593f9-c4b1-45a5-aef4-f88b8bc76e5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a20b19b-7krsq" Oct 30 17:27:16.745751 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:16.745719 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzkp9\" (UniqueName: \"kubernetes.io/projected/e5a593f9-c4b1-45a5-aef4-f88b8bc76e5f-kube-api-access-zzkp9\") pod \"maas-api-revoke-sub-5a20b19b-7krsq\" (UID: \"e5a593f9-c4b1-45a5-aef4-f88b8bc76e5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a20b19b-7krsq" Oct 30 17:27:16.747364 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:16.746185 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6197c2d3-3465-4091-857f-9d8ccdcc2eb3-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d2d985-bdd4b595-zmgvf\" (UID: \"6197c2d3-3465-4091-857f-9d8ccdcc2eb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d2d985-bdd4b595-zmgvf" Oct 30 17:27:16.747364 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:27:16.746345 4522 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-d2d985-serving-cert: secret "maas-api-e2e-ipp-a-d2d985-serving-cert" not found Oct 30 17:27:16.747364 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:27:16.746422 4522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6197c2d3-3465-4091-857f-9d8ccdcc2eb3-maas-api-tls podName:6197c2d3-3465-4091-857f-9d8ccdcc2eb3 nodeName:}" failed. No retries permitted until 2026-10-30 17:27:17.246402657 +0000 UTC m=+2411.247157536 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6197c2d3-3465-4091-857f-9d8ccdcc2eb3-maas-api-tls") pod "maas-api-e2e-ipp-a-d2d985-bdd4b595-zmgvf" (UID: "6197c2d3-3465-4091-857f-9d8ccdcc2eb3") : secret "maas-api-e2e-ipp-a-d2d985-serving-cert" not found Oct 30 17:27:16.749348 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:16.748899 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6197c2d3-3465-4091-857f-9d8ccdcc2eb3-metrics-certs\") pod \"maas-api-e2e-ipp-a-d2d985-bdd4b595-zmgvf\" (UID: \"6197c2d3-3465-4091-857f-9d8ccdcc2eb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d2d985-bdd4b595-zmgvf" Oct 30 17:27:16.780251 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:16.780213 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-755cj\" (UniqueName: \"kubernetes.io/projected/6197c2d3-3465-4091-857f-9d8ccdcc2eb3-kube-api-access-755cj\") pod \"maas-api-e2e-ipp-a-d2d985-bdd4b595-zmgvf\" (UID: \"6197c2d3-3465-4091-857f-9d8ccdcc2eb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d2d985-bdd4b595-zmgvf" Oct 30 17:27:16.847635 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:16.847603 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5a593f9-c4b1-45a5-aef4-f88b8bc76e5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a20b19b-7krsq\" (UID: \"e5a593f9-c4b1-45a5-aef4-f88b8bc76e5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a20b19b-7krsq" Oct 30 17:27:16.847876 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:16.847862 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5a593f9-c4b1-45a5-aef4-f88b8bc76e5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a20b19b-7krsq\" (UID: \"e5a593f9-c4b1-45a5-aef4-f88b8bc76e5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a20b19b-7krsq" Oct 30 17:27:16.847947 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:16.847888 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzkp9\" (UniqueName: \"kubernetes.io/projected/e5a593f9-c4b1-45a5-aef4-f88b8bc76e5f-kube-api-access-zzkp9\") pod \"maas-api-revoke-sub-5a20b19b-7krsq\" (UID: \"e5a593f9-c4b1-45a5-aef4-f88b8bc76e5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a20b19b-7krsq" Oct 30 17:27:16.848338 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:16.848279 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5a593f9-c4b1-45a5-aef4-f88b8bc76e5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a20b19b-7krsq\" (UID: \"e5a593f9-c4b1-45a5-aef4-f88b8bc76e5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a20b19b-7krsq" Oct 30 17:27:16.850527 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:16.850491 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5a593f9-c4b1-45a5-aef4-f88b8bc76e5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a20b19b-7krsq\" (UID: \"e5a593f9-c4b1-45a5-aef4-f88b8bc76e5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a20b19b-7krsq" Oct 30 17:27:16.884027 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:16.883994 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzkp9\" (UniqueName: \"kubernetes.io/projected/e5a593f9-c4b1-45a5-aef4-f88b8bc76e5f-kube-api-access-zzkp9\") pod \"maas-api-revoke-sub-5a20b19b-7krsq\" (UID: \"e5a593f9-c4b1-45a5-aef4-f88b8bc76e5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a20b19b-7krsq" Oct 30 17:27:16.887171 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:16.887139 4522 generic.go:356] "Generic (PLEG): container finished" podID="b54956ec-9536-4d68-af21-67b316b5b169" containerID="23d6552ae62ccd4b24db33537b2f72820d0335076fd9b1a8ba821dab10a91ee0" exitCode=0 Oct 30 17:27:16.887341 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:16.887223 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-424ecfc9-ds2z7" event={"ID":"b54956ec-9536-4d68-af21-67b316b5b169","Type":"ContainerDied","Data":"23d6552ae62ccd4b24db33537b2f72820d0335076fd9b1a8ba821dab10a91ee0"} Oct 30 17:27:16.888747 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:16.888676 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-abe243-c6b579d8-t25w4" event={"ID":"3ea67f7d-62df-49ff-a3c2-6395420ae0b9","Type":"ContainerStarted","Data":"aa9dc01910bb9a9b5ac17e7b756a4c1185331d47b0cde8a0bfec6259aaa6dafb"} Oct 30 17:27:17.038478 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:17.038445 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a20b19b-7krsq" Oct 30 17:27:17.253577 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:17.253544 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6197c2d3-3465-4091-857f-9d8ccdcc2eb3-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d2d985-bdd4b595-zmgvf\" (UID: \"6197c2d3-3465-4091-857f-9d8ccdcc2eb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d2d985-bdd4b595-zmgvf" Oct 30 17:27:17.256258 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:17.256232 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6197c2d3-3465-4091-857f-9d8ccdcc2eb3-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d2d985-bdd4b595-zmgvf\" (UID: \"6197c2d3-3465-4091-857f-9d8ccdcc2eb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d2d985-bdd4b595-zmgvf" Oct 30 17:27:17.355215 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:17.355182 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a20b19b-7krsq"] Oct 30 17:27:17.355661 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:27:17.355627 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5a593f9_c4b1_45a5_aef4_f88b8bc76e5f.slice/crio-b2a1fb3aaf6e1a7daeac09cf98acfa734e45bc06d794a432115fa1be9b5ca9c5 WatchSource:0}: Error finding container b2a1fb3aaf6e1a7daeac09cf98acfa734e45bc06d794a432115fa1be9b5ca9c5: Status 404 returned error can't find the container with id b2a1fb3aaf6e1a7daeac09cf98acfa734e45bc06d794a432115fa1be9b5ca9c5 Oct 30 17:27:17.519334 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:17.519242 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d2d985-bdd4b595-zmgvf" Oct 30 17:27:17.922136 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:17.921574 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d2d985-bdd4b595-zmgvf"] Oct 30 17:27:17.939337 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:17.937292 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a20b19b-7krsq" event={"ID":"e5a593f9-c4b1-45a5-aef4-f88b8bc76e5f","Type":"ContainerStarted","Data":"e8208bb4a261b3b389bc8f0348a9bac057893075eddcc66bb1adc0ffe7c5d767"} Oct 30 17:27:17.939337 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:17.937371 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a20b19b-7krsq" event={"ID":"e5a593f9-c4b1-45a5-aef4-f88b8bc76e5f","Type":"ContainerStarted","Data":"b2a1fb3aaf6e1a7daeac09cf98acfa734e45bc06d794a432115fa1be9b5ca9c5"} Oct 30 17:27:18.205192 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:18.205154 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-424ecfc9-ds2z7" Oct 30 17:27:18.252918 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:18.252859 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a20b19b-7krsq" podStartSLOduration=2.252843408 podStartE2EDuration="2.252843408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:27:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:27:18.013212008 +0000 UTC m=+2412.013966910" watchObservedRunningTime="2026-10-30 17:27:18.252843408 +0000 UTC m=+2412.253598310" Oct 30 17:27:18.296855 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:18.294266 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b54956ec-9536-4d68-af21-67b316b5b169-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b54956ec-9536-4d68-af21-67b316b5b169-maas-api-openshift-ca\") pod \"b54956ec-9536-4d68-af21-67b316b5b169\" (UID: \"b54956ec-9536-4d68-af21-67b316b5b169\") " Oct 30 17:27:18.296855 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:18.294482 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b54956ec-9536-4d68-af21-67b316b5b169-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b54956ec-9536-4d68-af21-67b316b5b169-maas-api-xks-ca\") pod \"b54956ec-9536-4d68-af21-67b316b5b169\" (UID: \"b54956ec-9536-4d68-af21-67b316b5b169\") " Oct 30 17:27:18.296855 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:18.294680 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b54956ec-9536-4d68-af21-67b316b5b169-kube-api-access-pj55s\" (UniqueName: \"kubernetes.io/projected/b54956ec-9536-4d68-af21-67b316b5b169-kube-api-access-pj55s\") pod \"b54956ec-9536-4d68-af21-67b316b5b169\" (UID: \"b54956ec-9536-4d68-af21-67b316b5b169\") " Oct 30 17:27:18.296855 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:18.294720 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b54956ec-9536-4d68-af21-67b316b5b169-maas-api-openshift-ca" pod "b54956ec-9536-4d68-af21-67b316b5b169" (UID: "b54956ec-9536-4d68-af21-67b316b5b169"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:27:18.296855 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:18.295367 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b54956ec-9536-4d68-af21-67b316b5b169-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:27:18.303275 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:18.300891 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b54956ec-9536-4d68-af21-67b316b5b169-maas-api-xks-ca" pod "b54956ec-9536-4d68-af21-67b316b5b169" (UID: "b54956ec-9536-4d68-af21-67b316b5b169"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:27:18.303275 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:18.301007 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b54956ec-9536-4d68-af21-67b316b5b169-kube-api-access-pj55s" pod "b54956ec-9536-4d68-af21-67b316b5b169" (UID: "b54956ec-9536-4d68-af21-67b316b5b169"). InnerVolumeSpecName "kube-api-access-pj55s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:27:18.396915 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:18.396884 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pj55s\" (UniqueName: \"kubernetes.io/projected/b54956ec-9536-4d68-af21-67b316b5b169-kube-api-access-pj55s\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:27:18.396915 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:18.396918 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b54956ec-9536-4d68-af21-67b316b5b169-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:27:18.937513 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:18.937430 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d2d985-bdd4b595-zmgvf" event={"ID":"6197c2d3-3465-4091-857f-9d8ccdcc2eb3","Type":"ContainerStarted","Data":"7271aee885831857cb9bb7b58ce053d46f2dfc5df5723400b9a634ffeff702e4"} Oct 30 17:27:18.937513 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:18.937512 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d2d985-bdd4b595-zmgvf" event={"ID":"6197c2d3-3465-4091-857f-9d8ccdcc2eb3","Type":"ContainerStarted","Data":"5fb0dd07039c231002535a2c72a5bf2e31cc13e51bb66c115630338364b05e8e"} Oct 30 17:27:18.937730 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:18.937568 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d2d985-bdd4b595-zmgvf" Oct 30 17:27:18.939261 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:18.939235 4522 generic.go:356] "Generic (PLEG): container finished" podID="e5a593f9-c4b1-45a5-aef4-f88b8bc76e5f" containerID="e8208bb4a261b3b389bc8f0348a9bac057893075eddcc66bb1adc0ffe7c5d767" exitCode=0 Oct 30 17:27:18.939398 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:18.939315 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a20b19b-7krsq" event={"ID":"e5a593f9-c4b1-45a5-aef4-f88b8bc76e5f","Type":"ContainerDied","Data":"e8208bb4a261b3b389bc8f0348a9bac057893075eddcc66bb1adc0ffe7c5d767"} Oct 30 17:27:18.940850 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:18.940833 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-424ecfc9-ds2z7" Oct 30 17:27:18.940956 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:18.940835 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-424ecfc9-ds2z7" event={"ID":"b54956ec-9536-4d68-af21-67b316b5b169","Type":"ContainerDied","Data":"e89f40a33bb179fd5630bedd73e3657ac3eb402bd4b7ba204902ee3efced2e5e"} Oct 30 17:27:18.940956 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:18.940942 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e89f40a33bb179fd5630bedd73e3657ac3eb402bd4b7ba204902ee3efced2e5e" Oct 30 17:27:18.942553 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:18.942532 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-abe243-c6b579d8-t25w4" event={"ID":"3ea67f7d-62df-49ff-a3c2-6395420ae0b9","Type":"ContainerStarted","Data":"5b00d19009eea15a0537dff45f879ab01140af5b2afeed01419530e4dd50b595"} Oct 30 17:27:18.942668 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:18.942604 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-abe243-c6b579d8-t25w4" Oct 30 17:27:18.982870 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:18.982832 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d2d985-bdd4b595-zmgvf" podStartSLOduration=2.632105469 podStartE2EDuration="2.982820843s" totalImagesPullingTime="350.715374ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:27:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:27:18.981758717 +0000 UTC m=+2412.982513655" watchObservedRunningTime="2026-10-30 17:27:18.982820843 +0000 UTC m=+2412.983575803" Oct 30 17:27:19.091362 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:19.091293 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-abe243-c6b579d8-t25w4" podStartSLOduration=2.0688513139999998 podStartE2EDuration="4.091278937s" totalImagesPullingTime="2.022427623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:27:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:27:19.089703181 +0000 UTC m=+2413.090458083" watchObservedRunningTime="2026-10-30 17:27:19.091278937 +0000 UTC m=+2413.092033837" Oct 30 17:27:20.133870 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:20.133846 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a20b19b-7krsq" Oct 30 17:27:20.227238 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:20.227149 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5a593f9-c4b1-45a5-aef4-f88b8bc76e5f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5a593f9-c4b1-45a5-aef4-f88b8bc76e5f-maas-api-xks-ca\") pod \"e5a593f9-c4b1-45a5-aef4-f88b8bc76e5f\" (UID: \"e5a593f9-c4b1-45a5-aef4-f88b8bc76e5f\") " Oct 30 17:27:20.227442 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:20.227344 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5a593f9-c4b1-45a5-aef4-f88b8bc76e5f-kube-api-access-zzkp9\" (UniqueName: \"kubernetes.io/projected/e5a593f9-c4b1-45a5-aef4-f88b8bc76e5f-kube-api-access-zzkp9\") pod \"e5a593f9-c4b1-45a5-aef4-f88b8bc76e5f\" (UID: \"e5a593f9-c4b1-45a5-aef4-f88b8bc76e5f\") " Oct 30 17:27:20.227524 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:20.227450 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5a593f9-c4b1-45a5-aef4-f88b8bc76e5f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5a593f9-c4b1-45a5-aef4-f88b8bc76e5f-maas-api-openshift-ca\") pod \"e5a593f9-c4b1-45a5-aef4-f88b8bc76e5f\" (UID: \"e5a593f9-c4b1-45a5-aef4-f88b8bc76e5f\") " Oct 30 17:27:20.228032 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:20.228002 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5a593f9-c4b1-45a5-aef4-f88b8bc76e5f-maas-api-openshift-ca" pod "e5a593f9-c4b1-45a5-aef4-f88b8bc76e5f" (UID: "e5a593f9-c4b1-45a5-aef4-f88b8bc76e5f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:27:20.228371 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:20.228349 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5a593f9-c4b1-45a5-aef4-f88b8bc76e5f-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:27:20.229604 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:20.229582 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5a593f9-c4b1-45a5-aef4-f88b8bc76e5f-maas-api-xks-ca" pod "e5a593f9-c4b1-45a5-aef4-f88b8bc76e5f" (UID: "e5a593f9-c4b1-45a5-aef4-f88b8bc76e5f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:27:20.229698 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:20.229622 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5a593f9-c4b1-45a5-aef4-f88b8bc76e5f-kube-api-access-zzkp9" pod "e5a593f9-c4b1-45a5-aef4-f88b8bc76e5f" (UID: "e5a593f9-c4b1-45a5-aef4-f88b8bc76e5f"). InnerVolumeSpecName "kube-api-access-zzkp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:27:20.330010 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:20.329972 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5a593f9-c4b1-45a5-aef4-f88b8bc76e5f-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:27:20.330010 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:20.330002 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzkp9\" (UniqueName: \"kubernetes.io/projected/e5a593f9-c4b1-45a5-aef4-f88b8bc76e5f-kube-api-access-zzkp9\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:27:20.961586 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:20.961556 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a20b19b-7krsq" Oct 30 17:27:20.961773 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:20.961589 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a20b19b-7krsq" event={"ID":"e5a593f9-c4b1-45a5-aef4-f88b8bc76e5f","Type":"ContainerDied","Data":"b2a1fb3aaf6e1a7daeac09cf98acfa734e45bc06d794a432115fa1be9b5ca9c5"} Oct 30 17:27:20.961773 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:20.961635 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2a1fb3aaf6e1a7daeac09cf98acfa734e45bc06d794a432115fa1be9b5ca9c5" Oct 30 17:27:22.013675 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:22.013634 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-6d4f99/sub-test-model-6d4f99-kserve-9ddf765b5-r59ft" podUID="8f146a44-f336-4ec0-9419-ffba69c846dc" containerName="main" containerID="cri-o://383df1458a8cdda85ea2bff3085ca8f5b480248912730335b4f2d0fd20a1348b" gracePeriod=45 Oct 30 17:27:22.322729 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:22.322699 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-6d4f99/sub-test-model-6d4f99-kserve-9ddf765b5-r59ft" Oct 30 17:27:22.360532 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:22.360496 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f146a44-f336-4ec0-9419-ffba69c846dc-home\" (UniqueName: \"kubernetes.io/empty-dir/8f146a44-f336-4ec0-9419-ffba69c846dc-home\") pod \"8f146a44-f336-4ec0-9419-ffba69c846dc\" (UID: \"8f146a44-f336-4ec0-9419-ffba69c846dc\") " Oct 30 17:27:22.360747 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:22.360545 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f146a44-f336-4ec0-9419-ffba69c846dc-tls-certs\" (UniqueName: \"kubernetes.io/secret/8f146a44-f336-4ec0-9419-ffba69c846dc-tls-certs\") pod \"8f146a44-f336-4ec0-9419-ffba69c846dc\" (UID: \"8f146a44-f336-4ec0-9419-ffba69c846dc\") " Oct 30 17:27:22.360747 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:22.360655 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f146a44-f336-4ec0-9419-ffba69c846dc-kube-api-access-l6m6c\" (UniqueName: \"kubernetes.io/projected/8f146a44-f336-4ec0-9419-ffba69c846dc-kube-api-access-l6m6c\") pod \"8f146a44-f336-4ec0-9419-ffba69c846dc\" (UID: \"8f146a44-f336-4ec0-9419-ffba69c846dc\") " Oct 30 17:27:22.360889 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:22.360759 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f146a44-f336-4ec0-9419-ffba69c846dc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f146a44-f336-4ec0-9419-ffba69c846dc-dshm\") pod \"8f146a44-f336-4ec0-9419-ffba69c846dc\" (UID: \"8f146a44-f336-4ec0-9419-ffba69c846dc\") " Oct 30 17:27:22.360889 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:22.360793 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f146a44-f336-4ec0-9419-ffba69c846dc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f146a44-f336-4ec0-9419-ffba69c846dc-model-cache\") pod \"8f146a44-f336-4ec0-9419-ffba69c846dc\" (UID: \"8f146a44-f336-4ec0-9419-ffba69c846dc\") " Oct 30 17:27:22.361242 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:22.360972 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f146a44-f336-4ec0-9419-ffba69c846dc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f146a44-f336-4ec0-9419-ffba69c846dc-tmp-dir\") pod \"8f146a44-f336-4ec0-9419-ffba69c846dc\" (UID: \"8f146a44-f336-4ec0-9419-ffba69c846dc\") " Oct 30 17:27:22.361242 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:22.361008 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f146a44-f336-4ec0-9419-ffba69c846dc-home" pod "8f146a44-f336-4ec0-9419-ffba69c846dc" (UID: "8f146a44-f336-4ec0-9419-ffba69c846dc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:27:22.361242 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:22.361227 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f146a44-f336-4ec0-9419-ffba69c846dc-model-cache" pod "8f146a44-f336-4ec0-9419-ffba69c846dc" (UID: "8f146a44-f336-4ec0-9419-ffba69c846dc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:27:22.361457 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:22.361365 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f146a44-f336-4ec0-9419-ffba69c846dc-tmp-dir" pod "8f146a44-f336-4ec0-9419-ffba69c846dc" (UID: "8f146a44-f336-4ec0-9419-ffba69c846dc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:27:22.362221 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:22.362152 4522 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f146a44-f336-4ec0-9419-ffba69c846dc-model-cache\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:27:22.362221 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:22.362183 4522 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f146a44-f336-4ec0-9419-ffba69c846dc-tmp-dir\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:27:22.362221 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:22.362197 4522 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f146a44-f336-4ec0-9419-ffba69c846dc-home\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:27:22.364002 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:22.363979 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f146a44-f336-4ec0-9419-ffba69c846dc-dshm" pod "8f146a44-f336-4ec0-9419-ffba69c846dc" (UID: "8f146a44-f336-4ec0-9419-ffba69c846dc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:27:22.364221 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:22.364194 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f146a44-f336-4ec0-9419-ffba69c846dc-kube-api-access-l6m6c" pod "8f146a44-f336-4ec0-9419-ffba69c846dc" (UID: "8f146a44-f336-4ec0-9419-ffba69c846dc"). InnerVolumeSpecName "kube-api-access-l6m6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:27:22.364396 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:22.364301 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f146a44-f336-4ec0-9419-ffba69c846dc-tls-certs" pod "8f146a44-f336-4ec0-9419-ffba69c846dc" (UID: "8f146a44-f336-4ec0-9419-ffba69c846dc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:27:22.464030 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:22.463994 4522 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f146a44-f336-4ec0-9419-ffba69c846dc-dshm\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:27:22.464030 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:22.464024 4522 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f146a44-f336-4ec0-9419-ffba69c846dc-tls-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:27:22.464030 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:22.464035 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6m6c\" (UniqueName: \"kubernetes.io/projected/8f146a44-f336-4ec0-9419-ffba69c846dc-kube-api-access-l6m6c\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:27:22.980815 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:22.980777 4522 generic.go:356] "Generic (PLEG): container finished" podID="8f146a44-f336-4ec0-9419-ffba69c846dc" containerID="383df1458a8cdda85ea2bff3085ca8f5b480248912730335b4f2d0fd20a1348b" exitCode=0 Oct 30 17:27:22.980976 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:22.980826 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-6d4f99/sub-test-model-6d4f99-kserve-9ddf765b5-r59ft" Oct 30 17:27:22.980976 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:22.980860 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-6d4f99/sub-test-model-6d4f99-kserve-9ddf765b5-r59ft" event={"ID":"8f146a44-f336-4ec0-9419-ffba69c846dc","Type":"ContainerDied","Data":"383df1458a8cdda85ea2bff3085ca8f5b480248912730335b4f2d0fd20a1348b"} Oct 30 17:27:22.980976 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:22.980908 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-6d4f99/sub-test-model-6d4f99-kserve-9ddf765b5-r59ft" event={"ID":"8f146a44-f336-4ec0-9419-ffba69c846dc","Type":"ContainerDied","Data":"ec2d0d1805a7b9a9dbcbe5019169aab10323d8ac4d5fe5a557939fc9df13189c"} Oct 30 17:27:22.980976 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:22.980929 4522 scope.go:122] "RemoveContainer" containerID="383df1458a8cdda85ea2bff3085ca8f5b480248912730335b4f2d0fd20a1348b" Oct 30 17:27:22.993499 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:22.993475 4522 scope.go:122] "RemoveContainer" containerID="383df1458a8cdda85ea2bff3085ca8f5b480248912730335b4f2d0fd20a1348b" Oct 30 17:27:22.993823 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:27:22.993803 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"383df1458a8cdda85ea2bff3085ca8f5b480248912730335b4f2d0fd20a1348b\": container with ID starting with 383df1458a8cdda85ea2bff3085ca8f5b480248912730335b4f2d0fd20a1348b not found: ID does not exist" containerID="383df1458a8cdda85ea2bff3085ca8f5b480248912730335b4f2d0fd20a1348b" Oct 30 17:27:22.993871 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:22.993831 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"383df1458a8cdda85ea2bff3085ca8f5b480248912730335b4f2d0fd20a1348b"} err="failed to get container status \"383df1458a8cdda85ea2bff3085ca8f5b480248912730335b4f2d0fd20a1348b\": rpc error: code = NotFound desc = could not find container \"383df1458a8cdda85ea2bff3085ca8f5b480248912730335b4f2d0fd20a1348b\": container with ID starting with 383df1458a8cdda85ea2bff3085ca8f5b480248912730335b4f2d0fd20a1348b not found: ID does not exist" Oct 30 17:27:23.044624 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:23.044590 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-6d4f99/sub-test-model-6d4f99-kserve-9ddf765b5-r59ft"] Oct 30 17:27:23.049269 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:23.049243 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-6d4f99/sub-test-model-6d4f99-kserve-9ddf765b5-r59ft"] Oct 30 17:27:23.257336 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:23.257218 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-6d4f99/sub-test-model-6d4f99-kserve-9ddf765b5-r59ft" podUID="8f146a44-f336-4ec0-9419-ffba69c846dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.247:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:27:24.146801 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:24.146769 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f146a44-f336-4ec0-9419-ffba69c846dc" path="/var/lib/kubelet/pods/8f146a44-f336-4ec0-9419-ffba69c846dc/volumes" Oct 30 17:27:24.380967 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:24.380934 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02ca91a1-njlr4"] Oct 30 17:27:24.381753 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:24.381737 4522 cpu_manager.go:414] "containerMap: removing container" podUID="8f146a44-f336-4ec0-9419-ffba69c846dc" containerName="main" Oct 30 17:27:24.381800 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:24.381756 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f146a44-f336-4ec0-9419-ffba69c846dc" containerName="main" Oct 30 17:27:24.381800 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:24.381770 4522 cpu_manager.go:414] "containerMap: removing container" podUID="b54956ec-9536-4d68-af21-67b316b5b169" containerName="revoke-keys" Oct 30 17:27:24.381800 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:24.381775 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b54956ec-9536-4d68-af21-67b316b5b169" containerName="revoke-keys" Oct 30 17:27:24.381800 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:24.381793 4522 cpu_manager.go:414] "containerMap: removing container" podUID="e5a593f9-c4b1-45a5-aef4-f88b8bc76e5f" containerName="revoke-keys" Oct 30 17:27:24.381800 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:24.381799 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5a593f9-c4b1-45a5-aef4-f88b8bc76e5f" containerName="revoke-keys" Oct 30 17:27:24.381975 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:24.381933 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f146a44-f336-4ec0-9419-ffba69c846dc" containerName="main" Oct 30 17:27:24.381975 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:24.381941 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="b54956ec-9536-4d68-af21-67b316b5b169" containerName="revoke-keys" Oct 30 17:27:24.381975 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:24.381948 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5a593f9-c4b1-45a5-aef4-f88b8bc76e5f" containerName="revoke-keys" Oct 30 17:27:24.387432 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:24.387408 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02ca91a1-njlr4" Oct 30 17:27:24.404423 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:24.404358 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02ca91a1-njlr4"] Oct 30 17:27:24.411756 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:24.411726 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:27:24.411966 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:24.411944 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:27:24.494255 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:24.494217 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16c3823f-3221-46bc-bbf6-bf3c00559c72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02ca91a1-njlr4\" (UID: \"16c3823f-3221-46bc-bbf6-bf3c00559c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02ca91a1-njlr4" Oct 30 17:27:24.494431 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:24.494265 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5lls\" (UniqueName: \"kubernetes.io/projected/16c3823f-3221-46bc-bbf6-bf3c00559c72-kube-api-access-z5lls\") pod \"maas-api-revoke-sub-02ca91a1-njlr4\" (UID: \"16c3823f-3221-46bc-bbf6-bf3c00559c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02ca91a1-njlr4" Oct 30 17:27:24.494496 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:24.494437 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16c3823f-3221-46bc-bbf6-bf3c00559c72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02ca91a1-njlr4\" (UID: \"16c3823f-3221-46bc-bbf6-bf3c00559c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02ca91a1-njlr4" Oct 30 17:27:24.596494 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:24.596452 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16c3823f-3221-46bc-bbf6-bf3c00559c72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02ca91a1-njlr4\" (UID: \"16c3823f-3221-46bc-bbf6-bf3c00559c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02ca91a1-njlr4" Oct 30 17:27:24.596494 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:24.596497 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5lls\" (UniqueName: \"kubernetes.io/projected/16c3823f-3221-46bc-bbf6-bf3c00559c72-kube-api-access-z5lls\") pod \"maas-api-revoke-sub-02ca91a1-njlr4\" (UID: \"16c3823f-3221-46bc-bbf6-bf3c00559c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02ca91a1-njlr4" Oct 30 17:27:24.596749 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:24.596555 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16c3823f-3221-46bc-bbf6-bf3c00559c72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02ca91a1-njlr4\" (UID: \"16c3823f-3221-46bc-bbf6-bf3c00559c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02ca91a1-njlr4" Oct 30 17:27:24.597175 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:24.597147 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16c3823f-3221-46bc-bbf6-bf3c00559c72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02ca91a1-njlr4\" (UID: \"16c3823f-3221-46bc-bbf6-bf3c00559c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02ca91a1-njlr4" Oct 30 17:27:24.598914 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:24.598893 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16c3823f-3221-46bc-bbf6-bf3c00559c72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02ca91a1-njlr4\" (UID: \"16c3823f-3221-46bc-bbf6-bf3c00559c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02ca91a1-njlr4" Oct 30 17:27:24.621894 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:24.621867 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5lls\" (UniqueName: \"kubernetes.io/projected/16c3823f-3221-46bc-bbf6-bf3c00559c72-kube-api-access-z5lls\") pod \"maas-api-revoke-sub-02ca91a1-njlr4\" (UID: \"16c3823f-3221-46bc-bbf6-bf3c00559c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02ca91a1-njlr4" Oct 30 17:27:24.709726 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:24.709695 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02ca91a1-njlr4" Oct 30 17:27:24.957912 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:24.957881 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d2d985-bdd4b595-zmgvf" Oct 30 17:27:25.025463 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:27:25.025431 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16c3823f_3221_46bc_bbf6_bf3c00559c72.slice/crio-ff666d369706995808916274c9dcaf6fd47f5318b833f105ddd3537ae21d9402 WatchSource:0}: Error finding container ff666d369706995808916274c9dcaf6fd47f5318b833f105ddd3537ae21d9402: Status 404 returned error can't find the container with id ff666d369706995808916274c9dcaf6fd47f5318b833f105ddd3537ae21d9402 Oct 30 17:27:25.029425 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:25.029396 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02ca91a1-njlr4"] Oct 30 17:27:26.011263 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:26.011226 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02ca91a1-njlr4" event={"ID":"16c3823f-3221-46bc-bbf6-bf3c00559c72","Type":"ContainerStarted","Data":"d6642267e3bea08551ca1d203d17f3d74686a4d3a72d988c521c7bb33b3479af"} Oct 30 17:27:26.011712 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:26.011286 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02ca91a1-njlr4" event={"ID":"16c3823f-3221-46bc-bbf6-bf3c00559c72","Type":"ContainerStarted","Data":"ff666d369706995808916274c9dcaf6fd47f5318b833f105ddd3537ae21d9402"} Oct 30 17:27:26.048145 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:26.048113 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca1e9ef0-rk7j8"] Oct 30 17:27:26.052970 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:26.052942 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca1e9ef0-rk7j8"] Oct 30 17:27:26.071739 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:26.071688 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02ca91a1-njlr4" podStartSLOduration=2.071675351 podStartE2EDuration="2.071675351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:27:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:27:26.070476188 +0000 UTC m=+2420.071231089" watchObservedRunningTime="2026-10-30 17:27:26.071675351 +0000 UTC m=+2420.072430308" Oct 30 17:27:26.148009 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:26.147980 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8113928a-b589-43fe-ab47-c6e4b4c4fd87" path="/var/lib/kubelet/pods/8113928a-b589-43fe-ab47-c6e4b4c4fd87/volumes" Oct 30 17:27:26.891739 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:26.891704 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41cb25-db9f8d799-t27kf" Oct 30 17:27:26.891912 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:26.891785 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-41cb25-56b855cbbb-x7mzc" Oct 30 17:27:27.023148 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:27.023112 4522 generic.go:356] "Generic (PLEG): container finished" podID="16c3823f-3221-46bc-bbf6-bf3c00559c72" containerID="d6642267e3bea08551ca1d203d17f3d74686a4d3a72d988c521c7bb33b3479af" exitCode=0 Oct 30 17:27:27.023548 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:27.023175 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02ca91a1-njlr4" event={"ID":"16c3823f-3221-46bc-bbf6-bf3c00559c72","Type":"ContainerDied","Data":"d6642267e3bea08551ca1d203d17f3d74686a4d3a72d988c521c7bb33b3479af"} Oct 30 17:27:28.203127 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:28.203097 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02ca91a1-njlr4" Oct 30 17:27:28.374070 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:28.373976 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16c3823f-3221-46bc-bbf6-bf3c00559c72-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16c3823f-3221-46bc-bbf6-bf3c00559c72-maas-api-xks-ca\") pod \"16c3823f-3221-46bc-bbf6-bf3c00559c72\" (UID: \"16c3823f-3221-46bc-bbf6-bf3c00559c72\") " Oct 30 17:27:28.374291 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:28.374277 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16c3823f-3221-46bc-bbf6-bf3c00559c72-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16c3823f-3221-46bc-bbf6-bf3c00559c72-maas-api-openshift-ca\") pod \"16c3823f-3221-46bc-bbf6-bf3c00559c72\" (UID: \"16c3823f-3221-46bc-bbf6-bf3c00559c72\") " Oct 30 17:27:28.374448 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:28.374430 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16c3823f-3221-46bc-bbf6-bf3c00559c72-kube-api-access-z5lls\" (UniqueName: \"kubernetes.io/projected/16c3823f-3221-46bc-bbf6-bf3c00559c72-kube-api-access-z5lls\") pod \"16c3823f-3221-46bc-bbf6-bf3c00559c72\" (UID: \"16c3823f-3221-46bc-bbf6-bf3c00559c72\") " Oct 30 17:27:28.374674 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:28.374645 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16c3823f-3221-46bc-bbf6-bf3c00559c72-maas-api-openshift-ca" pod "16c3823f-3221-46bc-bbf6-bf3c00559c72" (UID: "16c3823f-3221-46bc-bbf6-bf3c00559c72"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:27:28.375089 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:28.375068 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16c3823f-3221-46bc-bbf6-bf3c00559c72-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:27:28.376289 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:28.376266 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16c3823f-3221-46bc-bbf6-bf3c00559c72-maas-api-xks-ca" pod "16c3823f-3221-46bc-bbf6-bf3c00559c72" (UID: "16c3823f-3221-46bc-bbf6-bf3c00559c72"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:27:28.376619 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:28.376596 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16c3823f-3221-46bc-bbf6-bf3c00559c72-kube-api-access-z5lls" pod "16c3823f-3221-46bc-bbf6-bf3c00559c72" (UID: "16c3823f-3221-46bc-bbf6-bf3c00559c72"). InnerVolumeSpecName "kube-api-access-z5lls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:27:28.477068 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:28.477033 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5lls\" (UniqueName: \"kubernetes.io/projected/16c3823f-3221-46bc-bbf6-bf3c00559c72-kube-api-access-z5lls\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:27:28.477068 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:28.477063 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16c3823f-3221-46bc-bbf6-bf3c00559c72-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:27:29.042045 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:29.042016 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02ca91a1-njlr4" Oct 30 17:27:29.042233 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:29.042048 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02ca91a1-njlr4" event={"ID":"16c3823f-3221-46bc-bbf6-bf3c00559c72","Type":"ContainerDied","Data":"ff666d369706995808916274c9dcaf6fd47f5318b833f105ddd3537ae21d9402"} Oct 30 17:27:29.042233 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:29.042093 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff666d369706995808916274c9dcaf6fd47f5318b833f105ddd3537ae21d9402" Oct 30 17:27:29.954052 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:29.954023 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-abe243-c6b579d8-t25w4" Oct 30 17:27:31.334901 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:31.330784 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8681b5/sub-test-model-8681b5-kserve-6dcbf55b89-hsdx9"] Oct 30 17:27:32.421916 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:32.421877 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph"] Oct 30 17:27:32.422757 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:32.422740 4522 cpu_manager.go:414] "containerMap: removing container" podUID="16c3823f-3221-46bc-bbf6-bf3c00559c72" containerName="revoke-keys" Oct 30 17:27:32.422757 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:32.422757 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16c3823f-3221-46bc-bbf6-bf3c00559c72" containerName="revoke-keys" Oct 30 17:27:32.422911 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:32.422897 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="16c3823f-3221-46bc-bbf6-bf3c00559c72" containerName="revoke-keys" Oct 30 17:27:32.427065 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:32.427044 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph" Oct 30 17:27:32.448805 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:32.448751 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ca9280e-ce64-48e6-9a1e-6a104853002e-workload-socket\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph\" (UID: \"8ca9280e-ce64-48e6-9a1e-6a104853002e\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph" Oct 30 17:27:32.450010 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:32.449976 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ca9280e-ce64-48e6-9a1e-6a104853002e-istiod-ca-cert\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph\" (UID: \"8ca9280e-ce64-48e6-9a1e-6a104853002e\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph" Oct 30 17:27:32.450132 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:32.450037 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdrzz\" (UniqueName: \"kubernetes.io/projected/8ca9280e-ce64-48e6-9a1e-6a104853002e-kube-api-access-sdrzz\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph\" (UID: \"8ca9280e-ce64-48e6-9a1e-6a104853002e\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph" Oct 30 17:27:32.450132 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:32.450080 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ca9280e-ce64-48e6-9a1e-6a104853002e-istio-envoy\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph\" (UID: \"8ca9280e-ce64-48e6-9a1e-6a104853002e\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph" Oct 30 17:27:32.450132 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:32.450120 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ca9280e-ce64-48e6-9a1e-6a104853002e-credential-socket\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph\" (UID: \"8ca9280e-ce64-48e6-9a1e-6a104853002e\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph" Oct 30 17:27:32.450333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:32.450224 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ca9280e-ce64-48e6-9a1e-6a104853002e-workload-certs\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph\" (UID: \"8ca9280e-ce64-48e6-9a1e-6a104853002e\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph" Oct 30 17:27:32.450333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:32.450286 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ca9280e-ce64-48e6-9a1e-6a104853002e-istio-token\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph\" (UID: \"8ca9280e-ce64-48e6-9a1e-6a104853002e\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph" Oct 30 17:27:32.450445 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:32.450340 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ca9280e-ce64-48e6-9a1e-6a104853002e-istio-data\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph\" (UID: \"8ca9280e-ce64-48e6-9a1e-6a104853002e\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph" Oct 30 17:27:32.450445 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:32.450375 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ca9280e-ce64-48e6-9a1e-6a104853002e-istio-podinfo\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph\" (UID: \"8ca9280e-ce64-48e6-9a1e-6a104853002e\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph" Oct 30 17:27:32.454689 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:32.454663 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph"] Oct 30 17:27:32.551258 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:32.551224 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ca9280e-ce64-48e6-9a1e-6a104853002e-workload-certs\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph\" (UID: \"8ca9280e-ce64-48e6-9a1e-6a104853002e\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph" Oct 30 17:27:32.551491 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:32.551281 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ca9280e-ce64-48e6-9a1e-6a104853002e-istio-token\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph\" (UID: \"8ca9280e-ce64-48e6-9a1e-6a104853002e\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph" Oct 30 17:27:32.551491 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:32.551323 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ca9280e-ce64-48e6-9a1e-6a104853002e-istio-data\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph\" (UID: \"8ca9280e-ce64-48e6-9a1e-6a104853002e\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph" Oct 30 17:27:32.551491 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:32.551346 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ca9280e-ce64-48e6-9a1e-6a104853002e-istio-podinfo\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph\" (UID: \"8ca9280e-ce64-48e6-9a1e-6a104853002e\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph" Oct 30 17:27:32.551657 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:32.551497 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ca9280e-ce64-48e6-9a1e-6a104853002e-workload-socket\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph\" (UID: \"8ca9280e-ce64-48e6-9a1e-6a104853002e\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph" Oct 30 17:27:32.551742 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:32.551722 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ca9280e-ce64-48e6-9a1e-6a104853002e-istiod-ca-cert\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph\" (UID: \"8ca9280e-ce64-48e6-9a1e-6a104853002e\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph" Oct 30 17:27:32.551797 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:32.551768 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ca9280e-ce64-48e6-9a1e-6a104853002e-istio-data\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph\" (UID: \"8ca9280e-ce64-48e6-9a1e-6a104853002e\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph" Oct 30 17:27:32.551797 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:32.551724 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ca9280e-ce64-48e6-9a1e-6a104853002e-workload-certs\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph\" (UID: \"8ca9280e-ce64-48e6-9a1e-6a104853002e\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph" Oct 30 17:27:32.551911 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:32.551786 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdrzz\" (UniqueName: \"kubernetes.io/projected/8ca9280e-ce64-48e6-9a1e-6a104853002e-kube-api-access-sdrzz\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph\" (UID: \"8ca9280e-ce64-48e6-9a1e-6a104853002e\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph" Oct 30 17:27:32.551911 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:32.551868 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ca9280e-ce64-48e6-9a1e-6a104853002e-istio-envoy\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph\" (UID: \"8ca9280e-ce64-48e6-9a1e-6a104853002e\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph" Oct 30 17:27:32.551911 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:32.551870 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ca9280e-ce64-48e6-9a1e-6a104853002e-workload-socket\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph\" (UID: \"8ca9280e-ce64-48e6-9a1e-6a104853002e\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph" Oct 30 17:27:32.552069 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:32.551926 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ca9280e-ce64-48e6-9a1e-6a104853002e-credential-socket\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph\" (UID: \"8ca9280e-ce64-48e6-9a1e-6a104853002e\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph" Oct 30 17:27:32.552266 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:32.552240 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ca9280e-ce64-48e6-9a1e-6a104853002e-credential-socket\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph\" (UID: \"8ca9280e-ce64-48e6-9a1e-6a104853002e\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph" Oct 30 17:27:32.552431 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:32.552369 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ca9280e-ce64-48e6-9a1e-6a104853002e-istiod-ca-cert\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph\" (UID: \"8ca9280e-ce64-48e6-9a1e-6a104853002e\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph" Oct 30 17:27:32.554025 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:32.553998 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ca9280e-ce64-48e6-9a1e-6a104853002e-istio-podinfo\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph\" (UID: \"8ca9280e-ce64-48e6-9a1e-6a104853002e\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph" Oct 30 17:27:32.554335 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:32.554291 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ca9280e-ce64-48e6-9a1e-6a104853002e-istio-envoy\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph\" (UID: \"8ca9280e-ce64-48e6-9a1e-6a104853002e\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph" Oct 30 17:27:32.576423 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:32.576389 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ca9280e-ce64-48e6-9a1e-6a104853002e-istio-token\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph\" (UID: \"8ca9280e-ce64-48e6-9a1e-6a104853002e\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph" Oct 30 17:27:32.576592 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:32.576433 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdrzz\" (UniqueName: \"kubernetes.io/projected/8ca9280e-ce64-48e6-9a1e-6a104853002e-kube-api-access-sdrzz\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph\" (UID: \"8ca9280e-ce64-48e6-9a1e-6a104853002e\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph" Oct 30 17:27:32.745817 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:32.745782 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph" Oct 30 17:27:33.071698 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:27:33.071664 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ca9280e_ce64_48e6_9a1e_6a104853002e.slice/crio-2ef5de0c6f1f7cc73f0585836425190d0f06e78404590a68763693ae72225647 WatchSource:0}: Error finding container 2ef5de0c6f1f7cc73f0585836425190d0f06e78404590a68763693ae72225647: Status 404 returned error can't find the container with id 2ef5de0c6f1f7cc73f0585836425190d0f06e78404590a68763693ae72225647 Oct 30 17:27:33.074411 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:33.074382 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph"] Oct 30 17:27:33.074758 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:33.074729 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:27:33.074804 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:33.074793 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:27:33.074837 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:33.074821 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:27:34.089049 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:34.089014 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph" event={"ID":"8ca9280e-ce64-48e6-9a1e-6a104853002e","Type":"ContainerStarted","Data":"cd4e75e748e3bf0fcdc6f167eed9079be2947f3a176a7afd88868b483680512c"} Oct 30 17:27:34.089509 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:34.089069 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph" event={"ID":"8ca9280e-ce64-48e6-9a1e-6a104853002e","Type":"ContainerStarted","Data":"2ef5de0c6f1f7cc73f0585836425190d0f06e78404590a68763693ae72225647"} Oct 30 17:27:34.153975 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:34.153924 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph" podStartSLOduration=2.153908291 podStartE2EDuration="2.153908291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:27:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:27:34.151464006 +0000 UTC m=+2428.152218906" watchObservedRunningTime="2026-10-30 17:27:34.153908291 +0000 UTC m=+2428.154663191" Oct 30 17:27:34.746768 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:34.746731 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph" Oct 30 17:27:34.748665 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:34.748639 4522 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" start-of-body= Oct 30 17:27:34.748755 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:34.748701 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph" podUID="8ca9280e-ce64-48e6-9a1e-6a104853002e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" Oct 30 17:27:34.967017 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:34.966977 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9"] Oct 30 17:27:34.972940 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:34.972912 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9" Oct 30 17:27:35.036401 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:35.036297 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9"] Oct 30 17:27:35.045591 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:35.045561 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-63e60d-openshift-default-dockercfg-z8q4j\"" Oct 30 17:27:35.126680 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:35.126641 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e904bd74-db7c-4430-863a-5277580a6342-istio-token\") pod \"e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9\" (UID: \"e904bd74-db7c-4430-863a-5277580a6342\") " pod="openshift-ingress/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9" Oct 30 17:27:35.127098 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:35.126797 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e904bd74-db7c-4430-863a-5277580a6342-istiod-ca-cert\") pod \"e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9\" (UID: \"e904bd74-db7c-4430-863a-5277580a6342\") " pod="openshift-ingress/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9" Oct 30 17:27:35.127098 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:35.126856 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e904bd74-db7c-4430-863a-5277580a6342-credential-socket\") pod \"e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9\" (UID: \"e904bd74-db7c-4430-863a-5277580a6342\") " pod="openshift-ingress/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9" Oct 30 17:27:35.127098 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:35.126951 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e904bd74-db7c-4430-863a-5277580a6342-istio-envoy\") pod \"e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9\" (UID: \"e904bd74-db7c-4430-863a-5277580a6342\") " pod="openshift-ingress/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9" Oct 30 17:27:35.127098 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:35.127081 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e904bd74-db7c-4430-863a-5277580a6342-istio-podinfo\") pod \"e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9\" (UID: \"e904bd74-db7c-4430-863a-5277580a6342\") " pod="openshift-ingress/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9" Oct 30 17:27:35.127275 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:35.127149 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e904bd74-db7c-4430-863a-5277580a6342-workload-socket\") pod \"e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9\" (UID: \"e904bd74-db7c-4430-863a-5277580a6342\") " pod="openshift-ingress/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9" Oct 30 17:27:35.127275 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:35.127185 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e904bd74-db7c-4430-863a-5277580a6342-workload-certs\") pod \"e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9\" (UID: \"e904bd74-db7c-4430-863a-5277580a6342\") " pod="openshift-ingress/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9" Oct 30 17:27:35.127275 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:35.127201 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2rrn\" (UniqueName: \"kubernetes.io/projected/e904bd74-db7c-4430-863a-5277580a6342-kube-api-access-v2rrn\") pod \"e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9\" (UID: \"e904bd74-db7c-4430-863a-5277580a6342\") " pod="openshift-ingress/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9" Oct 30 17:27:35.127404 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:35.127290 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e904bd74-db7c-4430-863a-5277580a6342-istio-data\") pod \"e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9\" (UID: \"e904bd74-db7c-4430-863a-5277580a6342\") " pod="openshift-ingress/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9" Oct 30 17:27:35.236331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:35.234369 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e904bd74-db7c-4430-863a-5277580a6342-istio-podinfo\") pod \"e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9\" (UID: \"e904bd74-db7c-4430-863a-5277580a6342\") " pod="openshift-ingress/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9" Oct 30 17:27:35.236331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:35.234533 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e904bd74-db7c-4430-863a-5277580a6342-workload-socket\") pod \"e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9\" (UID: \"e904bd74-db7c-4430-863a-5277580a6342\") " pod="openshift-ingress/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9" Oct 30 17:27:35.236331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:35.234579 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e904bd74-db7c-4430-863a-5277580a6342-workload-certs\") pod \"e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9\" (UID: \"e904bd74-db7c-4430-863a-5277580a6342\") " pod="openshift-ingress/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9" Oct 30 17:27:35.236331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:35.234603 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2rrn\" (UniqueName: \"kubernetes.io/projected/e904bd74-db7c-4430-863a-5277580a6342-kube-api-access-v2rrn\") pod \"e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9\" (UID: \"e904bd74-db7c-4430-863a-5277580a6342\") " pod="openshift-ingress/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9" Oct 30 17:27:35.236331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:35.234648 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e904bd74-db7c-4430-863a-5277580a6342-istio-data\") pod \"e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9\" (UID: \"e904bd74-db7c-4430-863a-5277580a6342\") " pod="openshift-ingress/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9" Oct 30 17:27:35.236331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:35.234676 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e904bd74-db7c-4430-863a-5277580a6342-istio-token\") pod \"e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9\" (UID: \"e904bd74-db7c-4430-863a-5277580a6342\") " pod="openshift-ingress/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9" Oct 30 17:27:35.236331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:35.234809 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e904bd74-db7c-4430-863a-5277580a6342-istiod-ca-cert\") pod \"e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9\" (UID: \"e904bd74-db7c-4430-863a-5277580a6342\") " pod="openshift-ingress/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9" Oct 30 17:27:35.236331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:35.234866 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e904bd74-db7c-4430-863a-5277580a6342-credential-socket\") pod \"e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9\" (UID: \"e904bd74-db7c-4430-863a-5277580a6342\") " pod="openshift-ingress/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9" Oct 30 17:27:35.236331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:35.234917 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e904bd74-db7c-4430-863a-5277580a6342-istio-envoy\") pod \"e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9\" (UID: \"e904bd74-db7c-4430-863a-5277580a6342\") " pod="openshift-ingress/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9" Oct 30 17:27:35.256327 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:35.252637 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e904bd74-db7c-4430-863a-5277580a6342-istio-data\") pod \"e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9\" (UID: \"e904bd74-db7c-4430-863a-5277580a6342\") " pod="openshift-ingress/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9" Oct 30 17:27:35.256327 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:35.253137 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e904bd74-db7c-4430-863a-5277580a6342-workload-socket\") pod \"e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9\" (UID: \"e904bd74-db7c-4430-863a-5277580a6342\") " pod="openshift-ingress/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9" Oct 30 17:27:35.256327 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:35.253922 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e904bd74-db7c-4430-863a-5277580a6342-credential-socket\") pod \"e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9\" (UID: \"e904bd74-db7c-4430-863a-5277580a6342\") " pod="openshift-ingress/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9" Oct 30 17:27:35.256327 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:35.254711 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e904bd74-db7c-4430-863a-5277580a6342-istiod-ca-cert\") pod \"e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9\" (UID: \"e904bd74-db7c-4430-863a-5277580a6342\") " pod="openshift-ingress/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9" Oct 30 17:27:35.263504 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:35.260485 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e904bd74-db7c-4430-863a-5277580a6342-workload-certs\") pod \"e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9\" (UID: \"e904bd74-db7c-4430-863a-5277580a6342\") " pod="openshift-ingress/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9" Oct 30 17:27:35.263971 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:35.263945 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e904bd74-db7c-4430-863a-5277580a6342-istio-envoy\") pod \"e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9\" (UID: \"e904bd74-db7c-4430-863a-5277580a6342\") " pod="openshift-ingress/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9" Oct 30 17:27:35.288117 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:35.287851 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e904bd74-db7c-4430-863a-5277580a6342-istio-podinfo\") pod \"e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9\" (UID: \"e904bd74-db7c-4430-863a-5277580a6342\") " pod="openshift-ingress/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9" Oct 30 17:27:35.294991 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:35.294957 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e904bd74-db7c-4430-863a-5277580a6342-istio-token\") pod \"e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9\" (UID: \"e904bd74-db7c-4430-863a-5277580a6342\") " pod="openshift-ingress/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9" Oct 30 17:27:35.309067 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:35.309032 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2rrn\" (UniqueName: \"kubernetes.io/projected/e904bd74-db7c-4430-863a-5277580a6342-kube-api-access-v2rrn\") pod \"e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9\" (UID: \"e904bd74-db7c-4430-863a-5277580a6342\") " pod="openshift-ingress/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9" Oct 30 17:27:35.341867 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:35.341832 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9" Oct 30 17:27:35.802239 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:35.801112 4522 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" start-of-body= Oct 30 17:27:35.802239 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:35.801169 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph" podUID="8ca9280e-ce64-48e6-9a1e-6a104853002e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" Oct 30 17:27:36.058520 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:27:36.058488 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode904bd74_db7c_4430_863a_5277580a6342.slice/crio-83625eef45b74b1fc6cca417339c3499b7e771df0702e252a4a6e62436462fb4 WatchSource:0}: Error finding container 83625eef45b74b1fc6cca417339c3499b7e771df0702e252a4a6e62436462fb4: Status 404 returned error can't find the container with id 83625eef45b74b1fc6cca417339c3499b7e771df0702e252a4a6e62436462fb4 Oct 30 17:27:36.060780 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:36.060756 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9"] Oct 30 17:27:36.061568 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:36.061540 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:27:36.061615 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:36.061601 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:27:36.061659 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:36.061648 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:27:36.136231 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:36.136198 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9" event={"ID":"e904bd74-db7c-4430-863a-5277580a6342","Type":"ContainerStarted","Data":"83625eef45b74b1fc6cca417339c3499b7e771df0702e252a4a6e62436462fb4"} Oct 30 17:27:36.750511 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:36.750471 4522 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" start-of-body= Oct 30 17:27:36.750768 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:36.750541 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph" podUID="8ca9280e-ce64-48e6-9a1e-6a104853002e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" Oct 30 17:27:37.146637 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:37.146546 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9" event={"ID":"e904bd74-db7c-4430-863a-5277580a6342","Type":"ContainerStarted","Data":"9d6632e288587cddb15621dad4ff6694cd4720783e0a7dfa7b7d37923bdce115"} Oct 30 17:27:37.231221 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:37.231169 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9" podStartSLOduration=3.231153099 podStartE2EDuration="3.231153099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:27:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:27:37.213226177 +0000 UTC m=+2431.213981110" watchObservedRunningTime="2026-10-30 17:27:37.231153099 +0000 UTC m=+2431.231907999" Oct 30 17:27:37.343072 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:37.343035 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9" Oct 30 17:27:37.359899 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:37.359871 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9" Oct 30 17:27:37.746072 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:37.746037 4522 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" start-of-body= Oct 30 17:27:37.746260 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:37.746085 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph" podUID="8ca9280e-ce64-48e6-9a1e-6a104853002e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" Oct 30 17:27:38.154958 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:38.154866 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9" Oct 30 17:27:38.156037 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:38.156017 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9" Oct 30 17:27:38.746614 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:38.746580 4522 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" start-of-body= Oct 30 17:27:38.746823 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:38.746632 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph" podUID="8ca9280e-ce64-48e6-9a1e-6a104853002e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" Oct 30 17:27:39.746585 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:39.746548 4522 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" start-of-body= Oct 30 17:27:39.747015 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:39.746619 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph" podUID="8ca9280e-ce64-48e6-9a1e-6a104853002e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" Oct 30 17:27:40.751971 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:40.751937 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph" Oct 30 17:27:40.752423 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:40.752414 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph" Oct 30 17:27:40.753219 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:40.753198 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph" Oct 30 17:27:41.528384 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:41.528348 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-63e60d-5f6f7b8cc5-7wsxk"] Oct 30 17:27:41.532917 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:41.532899 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e60d-5f6f7b8cc5-7wsxk" Oct 30 17:27:41.552282 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:41.552255 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-63e60d-dockercfg-fgr4z\"" Oct 30 17:27:41.552822 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:41.552802 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-63e60d\"" Oct 30 17:27:41.559636 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:41.559614 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-63e60d-5f6f7b8cc5-7wsxk"] Oct 30 17:27:41.649403 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:41.649368 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-63e60d-6c68cfcd78-7wlrg"] Oct 30 17:27:41.688741 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:41.687890 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-63e60d-6c68cfcd78-7wlrg" Oct 30 17:27:41.746179 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:41.746145 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-63e60d-6c68cfcd78-7wlrg"] Oct 30 17:27:41.770299 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:41.770265 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj245\" (UniqueName: \"kubernetes.io/projected/5c473c8d-2321-447a-8059-eb3c1e6e105e-kube-api-access-bj245\") pod \"payload-pre-processing-e2e-ipp-b-63e60d-5f6f7b8cc5-7wsxk\" (UID: \"5c473c8d-2321-447a-8059-eb3c1e6e105e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e60d-5f6f7b8cc5-7wsxk" Oct 30 17:27:41.770706 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:41.770353 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c419fcd5-9b32-4075-ba32-1614e365b132-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-63e60d-6c68cfcd78-7wlrg\" (UID: \"c419fcd5-9b32-4075-ba32-1614e365b132\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-63e60d-6c68cfcd78-7wlrg" Oct 30 17:27:41.770706 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:41.770431 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2cfs\" (UniqueName: \"kubernetes.io/projected/c419fcd5-9b32-4075-ba32-1614e365b132-kube-api-access-m2cfs\") pod \"payload-processing-e2e-ipp-b-63e60d-6c68cfcd78-7wlrg\" (UID: \"c419fcd5-9b32-4075-ba32-1614e365b132\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-63e60d-6c68cfcd78-7wlrg" Oct 30 17:27:41.770706 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:41.770468 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c473c8d-2321-447a-8059-eb3c1e6e105e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-63e60d-5f6f7b8cc5-7wsxk\" (UID: \"5c473c8d-2321-447a-8059-eb3c1e6e105e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e60d-5f6f7b8cc5-7wsxk" Oct 30 17:27:41.872802 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:41.872713 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj245\" (UniqueName: \"kubernetes.io/projected/5c473c8d-2321-447a-8059-eb3c1e6e105e-kube-api-access-bj245\") pod \"payload-pre-processing-e2e-ipp-b-63e60d-5f6f7b8cc5-7wsxk\" (UID: \"5c473c8d-2321-447a-8059-eb3c1e6e105e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e60d-5f6f7b8cc5-7wsxk" Oct 30 17:27:41.872966 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:41.872808 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c419fcd5-9b32-4075-ba32-1614e365b132-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-63e60d-6c68cfcd78-7wlrg\" (UID: \"c419fcd5-9b32-4075-ba32-1614e365b132\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-63e60d-6c68cfcd78-7wlrg" Oct 30 17:27:41.872966 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:41.872862 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2cfs\" (UniqueName: \"kubernetes.io/projected/c419fcd5-9b32-4075-ba32-1614e365b132-kube-api-access-m2cfs\") pod \"payload-processing-e2e-ipp-b-63e60d-6c68cfcd78-7wlrg\" (UID: \"c419fcd5-9b32-4075-ba32-1614e365b132\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-63e60d-6c68cfcd78-7wlrg" Oct 30 17:27:41.872966 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:41.872920 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c473c8d-2321-447a-8059-eb3c1e6e105e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-63e60d-5f6f7b8cc5-7wsxk\" (UID: \"5c473c8d-2321-447a-8059-eb3c1e6e105e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e60d-5f6f7b8cc5-7wsxk" Oct 30 17:27:41.873639 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:41.873617 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c419fcd5-9b32-4075-ba32-1614e365b132-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-63e60d-6c68cfcd78-7wlrg\" (UID: \"c419fcd5-9b32-4075-ba32-1614e365b132\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-63e60d-6c68cfcd78-7wlrg" Oct 30 17:27:41.873703 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:41.873654 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c473c8d-2321-447a-8059-eb3c1e6e105e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-63e60d-5f6f7b8cc5-7wsxk\" (UID: \"5c473c8d-2321-447a-8059-eb3c1e6e105e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e60d-5f6f7b8cc5-7wsxk" Oct 30 17:27:41.898061 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:41.898028 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2cfs\" (UniqueName: \"kubernetes.io/projected/c419fcd5-9b32-4075-ba32-1614e365b132-kube-api-access-m2cfs\") pod \"payload-processing-e2e-ipp-b-63e60d-6c68cfcd78-7wlrg\" (UID: \"c419fcd5-9b32-4075-ba32-1614e365b132\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-63e60d-6c68cfcd78-7wlrg" Oct 30 17:27:41.898216 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:41.898164 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj245\" (UniqueName: \"kubernetes.io/projected/5c473c8d-2321-447a-8059-eb3c1e6e105e-kube-api-access-bj245\") pod \"payload-pre-processing-e2e-ipp-b-63e60d-5f6f7b8cc5-7wsxk\" (UID: \"5c473c8d-2321-447a-8059-eb3c1e6e105e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e60d-5f6f7b8cc5-7wsxk" Oct 30 17:27:42.074344 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:42.074275 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-63e60d-6c68cfcd78-7wlrg" Oct 30 17:27:42.159353 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:42.156963 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e60d-5f6f7b8cc5-7wsxk" Oct 30 17:27:42.386724 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:27:42.386693 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc419fcd5_9b32_4075_ba32_1614e365b132.slice/crio-6f267f43de6b33476207aaa1fa9f5449c656cb4ff5139f6ce12a39818cc239eb WatchSource:0}: Error finding container 6f267f43de6b33476207aaa1fa9f5449c656cb4ff5139f6ce12a39818cc239eb: Status 404 returned error can't find the container with id 6f267f43de6b33476207aaa1fa9f5449c656cb4ff5139f6ce12a39818cc239eb Oct 30 17:27:42.393550 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:42.393528 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-63e60d-6c68cfcd78-7wlrg"] Oct 30 17:27:42.494075 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:27:42.494031 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c473c8d_2321_447a_8059_eb3c1e6e105e.slice/crio-9d938e021da747859fe01b128fe68cab5e8975c27f925c4c2127690fb745bc94 WatchSource:0}: Error finding container 9d938e021da747859fe01b128fe68cab5e8975c27f925c4c2127690fb745bc94: Status 404 returned error can't find the container with id 9d938e021da747859fe01b128fe68cab5e8975c27f925c4c2127690fb745bc94 Oct 30 17:27:42.501891 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:42.501864 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-63e60d-5f6f7b8cc5-7wsxk"] Oct 30 17:27:42.948966 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:27:42.948933 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:27:43.216404 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:43.216354 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e60d-5f6f7b8cc5-7wsxk" event={"ID":"5c473c8d-2321-447a-8059-eb3c1e6e105e","Type":"ContainerStarted","Data":"f3aa3bc01158ecdb4328da87ef784120096ff65829186f0a85fc4efe3f8d8c2f"} Oct 30 17:27:43.216594 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:43.216422 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e60d-5f6f7b8cc5-7wsxk" event={"ID":"5c473c8d-2321-447a-8059-eb3c1e6e105e","Type":"ContainerStarted","Data":"9d938e021da747859fe01b128fe68cab5e8975c27f925c4c2127690fb745bc94"} Oct 30 17:27:43.216594 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:43.216455 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e60d-5f6f7b8cc5-7wsxk" Oct 30 17:27:43.218442 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:43.218414 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-63e60d-6c68cfcd78-7wlrg" event={"ID":"c419fcd5-9b32-4075-ba32-1614e365b132","Type":"ContainerStarted","Data":"3c4fa462a6505e77311a74c27e808999eb9dba02f003e357583a9fd1d4252055"} Oct 30 17:27:43.218561 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:43.218459 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-63e60d-6c68cfcd78-7wlrg" event={"ID":"c419fcd5-9b32-4075-ba32-1614e365b132","Type":"ContainerStarted","Data":"6f267f43de6b33476207aaa1fa9f5449c656cb4ff5139f6ce12a39818cc239eb"} Oct 30 17:27:43.218561 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:43.218482 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-63e60d-6c68cfcd78-7wlrg" Oct 30 17:27:43.317388 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:43.317294 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e60d-5f6f7b8cc5-7wsxk" podStartSLOduration=2.317279684 podStartE2EDuration="2.317279684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:27:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:27:43.269626008 +0000 UTC m=+2437.270380925" watchObservedRunningTime="2026-10-30 17:27:43.317279684 +0000 UTC m=+2437.318034585" Oct 30 17:27:43.319596 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:43.319558 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-63e60d-6c68cfcd78-7wlrg" podStartSLOduration=2.31954592 podStartE2EDuration="2.31954592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:27:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:27:43.314241349 +0000 UTC m=+2437.314996250" watchObservedRunningTime="2026-10-30 17:27:43.31954592 +0000 UTC m=+2437.320300821" Oct 30 17:27:45.461379 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:45.461339 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-db8417/sub-test-model-db8417-kserve-fb7fdb549-6xvgx"] Oct 30 17:27:46.363276 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:46.363236 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-8681b5/sub-test-model-8681b5-kserve-6dcbf55b89-hsdx9" podUID="3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b" containerName="main" containerID="cri-o://928c0baa704782b9abf12e214a469d7a1ac9b97c602be85e7f5d8e1eb3530d08" gracePeriod=45 Oct 30 17:27:46.744777 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:46.744747 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd"] Oct 30 17:27:46.760787 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:46.756971 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd" Oct 30 17:27:46.772190 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:46.772142 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd"] Oct 30 17:27:46.778673 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:46.778643 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-c17e8c-openshift-default-dockercfg-fgnp8\"" Oct 30 17:27:46.866800 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:46.866761 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173-istio-token\") pod \"e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd\" (UID: \"1e03b6f6-d0a5-45ac-af7f-3f1c50faf173\") " pod="openshift-ingress/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd" Oct 30 17:27:46.867148 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:46.867125 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173-istiod-ca-cert\") pod \"e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd\" (UID: \"1e03b6f6-d0a5-45ac-af7f-3f1c50faf173\") " pod="openshift-ingress/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd" Oct 30 17:27:46.867345 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:46.867326 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173-credential-socket\") pod \"e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd\" (UID: \"1e03b6f6-d0a5-45ac-af7f-3f1c50faf173\") " pod="openshift-ingress/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd" Oct 30 17:27:46.867485 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:46.867467 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p46z\" (UniqueName: \"kubernetes.io/projected/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173-kube-api-access-4p46z\") pod \"e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd\" (UID: \"1e03b6f6-d0a5-45ac-af7f-3f1c50faf173\") " pod="openshift-ingress/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd" Oct 30 17:27:46.870384 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:46.869978 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173-istio-envoy\") pod \"e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd\" (UID: \"1e03b6f6-d0a5-45ac-af7f-3f1c50faf173\") " pod="openshift-ingress/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd" Oct 30 17:27:46.870384 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:46.870089 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173-istio-podinfo\") pod \"e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd\" (UID: \"1e03b6f6-d0a5-45ac-af7f-3f1c50faf173\") " pod="openshift-ingress/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd" Oct 30 17:27:46.870384 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:46.870151 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173-istio-data\") pod \"e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd\" (UID: \"1e03b6f6-d0a5-45ac-af7f-3f1c50faf173\") " pod="openshift-ingress/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd" Oct 30 17:27:46.870384 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:46.870220 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173-workload-certs\") pod \"e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd\" (UID: \"1e03b6f6-d0a5-45ac-af7f-3f1c50faf173\") " pod="openshift-ingress/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd" Oct 30 17:27:46.870701 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:46.870546 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173-workload-socket\") pod \"e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd\" (UID: \"1e03b6f6-d0a5-45ac-af7f-3f1c50faf173\") " pod="openshift-ingress/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd" Oct 30 17:27:46.875745 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:46.875732 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-8681b5/sub-test-model-8681b5-kserve-6dcbf55b89-hsdx9" Oct 30 17:27:46.971866 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:46.971832 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b-dshm\") pod \"3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b\" (UID: \"3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b\") " Oct 30 17:27:46.972069 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:46.971891 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b-home\" (UniqueName: \"kubernetes.io/empty-dir/3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b-home\") pod \"3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b\" (UID: \"3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b\") " Oct 30 17:27:46.972069 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:46.971954 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b-model-cache\") pod \"3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b\" (UID: \"3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b\") " Oct 30 17:27:46.972069 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:46.972051 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b-tls-certs\" (UniqueName: \"kubernetes.io/secret/3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b-tls-certs\") pod \"3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b\" (UID: \"3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b\") " Oct 30 17:27:46.972266 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:46.972111 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b-tmp-dir\") pod \"3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b\" (UID: \"3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b\") " Oct 30 17:27:46.972266 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:46.972150 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b-kube-api-access-r84kk\" (UniqueName: \"kubernetes.io/projected/3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b-kube-api-access-r84kk\") pod \"3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b\" (UID: \"3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b\") " Oct 30 17:27:46.972266 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:46.972251 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b-home" pod "3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b" (UID: "3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:27:46.972822 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:46.972376 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b-model-cache" pod "3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b" (UID: "3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:27:46.972822 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:46.972552 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173-workload-certs\") pod \"e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd\" (UID: \"1e03b6f6-d0a5-45ac-af7f-3f1c50faf173\") " pod="openshift-ingress/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd" Oct 30 17:27:46.972822 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:46.972728 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b-tmp-dir" pod "3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b" (UID: "3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:27:46.972822 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:46.972806 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173-workload-socket\") pod \"e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd\" (UID: \"1e03b6f6-d0a5-45ac-af7f-3f1c50faf173\") " pod="openshift-ingress/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd" Oct 30 17:27:46.973129 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:46.972951 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173-istio-token\") pod \"e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd\" (UID: \"1e03b6f6-d0a5-45ac-af7f-3f1c50faf173\") " pod="openshift-ingress/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd" Oct 30 17:27:46.973129 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:46.973007 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173-istiod-ca-cert\") pod \"e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd\" (UID: \"1e03b6f6-d0a5-45ac-af7f-3f1c50faf173\") " pod="openshift-ingress/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd" Oct 30 17:27:46.973129 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:46.973038 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173-credential-socket\") pod \"e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd\" (UID: \"1e03b6f6-d0a5-45ac-af7f-3f1c50faf173\") " pod="openshift-ingress/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd" Oct 30 17:27:46.973129 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:46.973065 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p46z\" (UniqueName: \"kubernetes.io/projected/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173-kube-api-access-4p46z\") pod \"e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd\" (UID: \"1e03b6f6-d0a5-45ac-af7f-3f1c50faf173\") " pod="openshift-ingress/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd" Oct 30 17:27:46.973129 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:46.973124 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173-istio-envoy\") pod \"e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd\" (UID: \"1e03b6f6-d0a5-45ac-af7f-3f1c50faf173\") " pod="openshift-ingress/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd" Oct 30 17:27:46.973448 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:46.973178 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173-istio-podinfo\") pod \"e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd\" (UID: \"1e03b6f6-d0a5-45ac-af7f-3f1c50faf173\") " pod="openshift-ingress/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd" Oct 30 17:27:46.973448 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:46.973215 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173-istio-data\") pod \"e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd\" (UID: \"1e03b6f6-d0a5-45ac-af7f-3f1c50faf173\") " pod="openshift-ingress/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd" Oct 30 17:27:46.973448 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:46.973241 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173-workload-certs\") pod \"e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd\" (UID: \"1e03b6f6-d0a5-45ac-af7f-3f1c50faf173\") " pod="openshift-ingress/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd" Oct 30 17:27:46.973942 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:46.973735 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173-workload-socket\") pod \"e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd\" (UID: \"1e03b6f6-d0a5-45ac-af7f-3f1c50faf173\") " pod="openshift-ingress/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd" Oct 30 17:27:46.974057 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:46.974001 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173-credential-socket\") pod \"e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd\" (UID: \"1e03b6f6-d0a5-45ac-af7f-3f1c50faf173\") " pod="openshift-ingress/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd" Oct 30 17:27:46.974123 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:46.974067 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173-istiod-ca-cert\") pod \"e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd\" (UID: \"1e03b6f6-d0a5-45ac-af7f-3f1c50faf173\") " pod="openshift-ingress/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd" Oct 30 17:27:46.974204 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:46.974185 4522 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b-home\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:27:46.974364 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:46.974344 4522 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b-model-cache\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:27:46.974443 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:46.974371 4522 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b-tmp-dir\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:27:46.974635 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:46.974615 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173-istio-data\") pod \"e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd\" (UID: \"1e03b6f6-d0a5-45ac-af7f-3f1c50faf173\") " pod="openshift-ingress/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd" Oct 30 17:27:46.975051 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:46.975027 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b-dshm" pod "3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b" (UID: "3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:27:46.975943 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:46.975914 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b-kube-api-access-r84kk" pod "3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b" (UID: "3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b"). InnerVolumeSpecName "kube-api-access-r84kk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:27:46.976739 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:46.976693 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173-istio-envoy\") pod \"e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd\" (UID: \"1e03b6f6-d0a5-45ac-af7f-3f1c50faf173\") " pod="openshift-ingress/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd" Oct 30 17:27:46.977080 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:46.977059 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b-tls-certs" pod "3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b" (UID: "3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:27:46.977519 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:46.977498 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173-istio-podinfo\") pod \"e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd\" (UID: \"1e03b6f6-d0a5-45ac-af7f-3f1c50faf173\") " pod="openshift-ingress/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd" Oct 30 17:27:47.005185 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:47.005126 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173-istio-token\") pod \"e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd\" (UID: \"1e03b6f6-d0a5-45ac-af7f-3f1c50faf173\") " pod="openshift-ingress/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd" Oct 30 17:27:47.005471 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:47.005446 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p46z\" (UniqueName: \"kubernetes.io/projected/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173-kube-api-access-4p46z\") pod \"e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd\" (UID: \"1e03b6f6-d0a5-45ac-af7f-3f1c50faf173\") " pod="openshift-ingress/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd" Oct 30 17:27:47.076930 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:47.076898 4522 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b-tls-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:27:47.076930 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:47.076928 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r84kk\" (UniqueName: \"kubernetes.io/projected/3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b-kube-api-access-r84kk\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:27:47.076930 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:47.076939 4522 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b-dshm\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:27:47.114235 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:47.114202 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd" Oct 30 17:27:47.268094 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:47.268007 4522 generic.go:356] "Generic (PLEG): container finished" podID="3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b" containerID="928c0baa704782b9abf12e214a469d7a1ac9b97c602be85e7f5d8e1eb3530d08" exitCode=0 Oct 30 17:27:47.268249 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:47.268172 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-8681b5/sub-test-model-8681b5-kserve-6dcbf55b89-hsdx9" event={"ID":"3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b","Type":"ContainerDied","Data":"928c0baa704782b9abf12e214a469d7a1ac9b97c602be85e7f5d8e1eb3530d08"} Oct 30 17:27:47.268249 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:47.268204 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-8681b5/sub-test-model-8681b5-kserve-6dcbf55b89-hsdx9" event={"ID":"3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b","Type":"ContainerDied","Data":"c136c0d1d54e4c9bd2608eae7e715d80a0d70bae5f341b21b38d9f0693d926fe"} Oct 30 17:27:47.268249 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:47.268222 4522 scope.go:122] "RemoveContainer" containerID="928c0baa704782b9abf12e214a469d7a1ac9b97c602be85e7f5d8e1eb3530d08" Oct 30 17:27:47.268461 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:47.268380 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-8681b5/sub-test-model-8681b5-kserve-6dcbf55b89-hsdx9" Oct 30 17:27:47.295143 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:47.294548 4522 scope.go:122] "RemoveContainer" containerID="928c0baa704782b9abf12e214a469d7a1ac9b97c602be85e7f5d8e1eb3530d08" Oct 30 17:27:47.295143 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:27:47.295007 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"928c0baa704782b9abf12e214a469d7a1ac9b97c602be85e7f5d8e1eb3530d08\": container with ID starting with 928c0baa704782b9abf12e214a469d7a1ac9b97c602be85e7f5d8e1eb3530d08 not found: ID does not exist" containerID="928c0baa704782b9abf12e214a469d7a1ac9b97c602be85e7f5d8e1eb3530d08" Oct 30 17:27:47.295143 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:47.295051 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"928c0baa704782b9abf12e214a469d7a1ac9b97c602be85e7f5d8e1eb3530d08"} err="failed to get container status \"928c0baa704782b9abf12e214a469d7a1ac9b97c602be85e7f5d8e1eb3530d08\": rpc error: code = NotFound desc = could not find container \"928c0baa704782b9abf12e214a469d7a1ac9b97c602be85e7f5d8e1eb3530d08\": container with ID starting with 928c0baa704782b9abf12e214a469d7a1ac9b97c602be85e7f5d8e1eb3530d08 not found: ID does not exist" Oct 30 17:27:47.349152 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:47.349121 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8681b5/sub-test-model-8681b5-kserve-6dcbf55b89-hsdx9"] Oct 30 17:27:47.353895 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:47.353869 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8681b5/sub-test-model-8681b5-kserve-6dcbf55b89-hsdx9"] Oct 30 17:27:47.466957 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:27:47.466926 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e03b6f6_d0a5_45ac_af7f_3f1c50faf173.slice/crio-d2cbc10460aab23ba64c799e8ab243e026c48f30acb4f25b78cf621a347a01b1 WatchSource:0}: Error finding container d2cbc10460aab23ba64c799e8ab243e026c48f30acb4f25b78cf621a347a01b1: Status 404 returned error can't find the container with id d2cbc10460aab23ba64c799e8ab243e026c48f30acb4f25b78cf621a347a01b1 Oct 30 17:27:47.470675 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:47.470640 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:27:47.470799 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:47.470724 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:27:47.470799 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:47.470772 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:27:47.472673 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:47.472654 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd"] Oct 30 17:27:47.860931 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:47.860842 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-8681b5/sub-test-model-8681b5-kserve-6dcbf55b89-hsdx9" podUID="3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.244:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:27:48.150226 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:48.150149 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b" path="/var/lib/kubelet/pods/3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b/volumes" Oct 30 17:27:48.287148 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:48.287111 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd" event={"ID":"1e03b6f6-d0a5-45ac-af7f-3f1c50faf173","Type":"ContainerStarted","Data":"2c5da7495b1854a538ccac343f5ce39d06254be0b95fa5557e4f0a1bdb5c6ec5"} Oct 30 17:27:48.287355 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:48.287173 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd" event={"ID":"1e03b6f6-d0a5-45ac-af7f-3f1c50faf173","Type":"ContainerStarted","Data":"d2cbc10460aab23ba64c799e8ab243e026c48f30acb4f25b78cf621a347a01b1"} Oct 30 17:27:48.337560 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:48.337509 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd" podStartSLOduration=2.337496614 podStartE2EDuration="2.337496614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:27:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:27:48.335071655 +0000 UTC m=+2442.335826576" watchObservedRunningTime="2026-10-30 17:27:48.337496614 +0000 UTC m=+2442.338251514" Oct 30 17:27:49.115466 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:49.115427 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd" Oct 30 17:27:49.121696 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:49.121670 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd" Oct 30 17:27:49.299170 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:49.299142 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd" Oct 30 17:27:49.300009 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:49.299989 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd" Oct 30 17:27:52.703683 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:52.703637 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c17e8c-7cd6d6db99-wffp5"] Oct 30 17:27:52.705775 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:52.705744 4522 cpu_manager.go:414] "containerMap: removing container" podUID="3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b" containerName="main" Oct 30 17:27:52.705775 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:52.705775 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b" containerName="main" Oct 30 17:27:52.706288 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:52.706265 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="3977a0e9-bb6b-45e3-8b8c-ee5d4733a37b" containerName="main" Oct 30 17:27:52.714721 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:52.714697 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c17e8c-7cd6d6db99-wffp5" Oct 30 17:27:52.724803 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:52.724776 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c17e8c-7cd6d6db99-wffp5"] Oct 30 17:27:52.746524 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:52.746501 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-c17e8c-dockercfg-hkj62\"" Oct 30 17:27:52.746524 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:52.746501 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c17e8c\"" Oct 30 17:27:52.809732 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:52.809695 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c17e8c-54fd5c99d-m7v82"] Oct 30 17:27:52.868420 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:52.867519 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c17e8c-54fd5c99d-m7v82" Oct 30 17:27:52.886076 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:52.883380 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/645a0822-b642-405c-8b44-1bde28fa322e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c17e8c-7cd6d6db99-wffp5\" (UID: \"645a0822-b642-405c-8b44-1bde28fa322e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c17e8c-7cd6d6db99-wffp5" Oct 30 17:27:52.886076 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:52.883667 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7gb6\" (UniqueName: \"kubernetes.io/projected/645a0822-b642-405c-8b44-1bde28fa322e-kube-api-access-l7gb6\") pod \"payload-pre-processing-e2e-ipp-a-c17e8c-7cd6d6db99-wffp5\" (UID: \"645a0822-b642-405c-8b44-1bde28fa322e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c17e8c-7cd6d6db99-wffp5" Oct 30 17:27:52.891338 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:52.891059 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c17e8c-54fd5c99d-m7v82"] Oct 30 17:27:52.985752 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:52.985664 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7gb6\" (UniqueName: \"kubernetes.io/projected/645a0822-b642-405c-8b44-1bde28fa322e-kube-api-access-l7gb6\") pod \"payload-pre-processing-e2e-ipp-a-c17e8c-7cd6d6db99-wffp5\" (UID: \"645a0822-b642-405c-8b44-1bde28fa322e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c17e8c-7cd6d6db99-wffp5" Oct 30 17:27:52.986077 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:52.986058 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae18756-7182-45d8-af84-4b2b41920db0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c17e8c-54fd5c99d-m7v82\" (UID: \"1ae18756-7182-45d8-af84-4b2b41920db0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c17e8c-54fd5c99d-m7v82" Oct 30 17:27:52.986226 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:52.986122 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/645a0822-b642-405c-8b44-1bde28fa322e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c17e8c-7cd6d6db99-wffp5\" (UID: \"645a0822-b642-405c-8b44-1bde28fa322e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c17e8c-7cd6d6db99-wffp5" Oct 30 17:27:52.986226 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:52.986184 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttpdg\" (UniqueName: \"kubernetes.io/projected/1ae18756-7182-45d8-af84-4b2b41920db0-kube-api-access-ttpdg\") pod \"payload-processing-e2e-ipp-a-c17e8c-54fd5c99d-m7v82\" (UID: \"1ae18756-7182-45d8-af84-4b2b41920db0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c17e8c-54fd5c99d-m7v82" Oct 30 17:27:52.987531 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:52.987446 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/645a0822-b642-405c-8b44-1bde28fa322e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c17e8c-7cd6d6db99-wffp5\" (UID: \"645a0822-b642-405c-8b44-1bde28fa322e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c17e8c-7cd6d6db99-wffp5" Oct 30 17:27:53.010808 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:53.010778 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7gb6\" (UniqueName: \"kubernetes.io/projected/645a0822-b642-405c-8b44-1bde28fa322e-kube-api-access-l7gb6\") pod \"payload-pre-processing-e2e-ipp-a-c17e8c-7cd6d6db99-wffp5\" (UID: \"645a0822-b642-405c-8b44-1bde28fa322e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c17e8c-7cd6d6db99-wffp5" Oct 30 17:27:53.034376 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:53.034348 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c17e8c-7cd6d6db99-wffp5" Oct 30 17:27:53.087887 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:53.087848 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae18756-7182-45d8-af84-4b2b41920db0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c17e8c-54fd5c99d-m7v82\" (UID: \"1ae18756-7182-45d8-af84-4b2b41920db0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c17e8c-54fd5c99d-m7v82" Oct 30 17:27:53.088096 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:53.087942 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttpdg\" (UniqueName: \"kubernetes.io/projected/1ae18756-7182-45d8-af84-4b2b41920db0-kube-api-access-ttpdg\") pod \"payload-processing-e2e-ipp-a-c17e8c-54fd5c99d-m7v82\" (UID: \"1ae18756-7182-45d8-af84-4b2b41920db0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c17e8c-54fd5c99d-m7v82" Oct 30 17:27:53.088619 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:53.088595 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae18756-7182-45d8-af84-4b2b41920db0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c17e8c-54fd5c99d-m7v82\" (UID: \"1ae18756-7182-45d8-af84-4b2b41920db0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c17e8c-54fd5c99d-m7v82" Oct 30 17:27:53.117548 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:53.115945 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttpdg\" (UniqueName: \"kubernetes.io/projected/1ae18756-7182-45d8-af84-4b2b41920db0-kube-api-access-ttpdg\") pod \"payload-processing-e2e-ipp-a-c17e8c-54fd5c99d-m7v82\" (UID: \"1ae18756-7182-45d8-af84-4b2b41920db0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c17e8c-54fd5c99d-m7v82" Oct 30 17:27:53.198560 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:53.198525 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c17e8c-54fd5c99d-m7v82" Oct 30 17:27:53.353050 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:53.353021 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c17e8c-7cd6d6db99-wffp5"] Oct 30 17:27:53.521122 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:27:53.521085 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ae18756_7182_45d8_af84_4b2b41920db0.slice/crio-88e8d644dedff428620a75de88262b902bde63fe83b5aaec1cfef57531a48f8f WatchSource:0}: Error finding container 88e8d644dedff428620a75de88262b902bde63fe83b5aaec1cfef57531a48f8f: Status 404 returned error can't find the container with id 88e8d644dedff428620a75de88262b902bde63fe83b5aaec1cfef57531a48f8f Oct 30 17:27:53.524341 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:53.524295 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c17e8c-54fd5c99d-m7v82"] Oct 30 17:27:54.231415 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:54.231381 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e60d-5f6f7b8cc5-7wsxk" Oct 30 17:27:54.231885 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:54.231783 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-63e60d-6c68cfcd78-7wlrg" Oct 30 17:27:54.356565 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:54.356503 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c17e8c-7cd6d6db99-wffp5" event={"ID":"645a0822-b642-405c-8b44-1bde28fa322e","Type":"ContainerStarted","Data":"a7ebf865e21dd6f31a4e0444a3d1483180d4a2ddfa938a8b20def40db7bab146"} Oct 30 17:27:54.356776 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:54.356678 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c17e8c-7cd6d6db99-wffp5" Oct 30 17:27:54.356776 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:54.356715 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c17e8c-7cd6d6db99-wffp5" event={"ID":"645a0822-b642-405c-8b44-1bde28fa322e","Type":"ContainerStarted","Data":"529e0dce97c5bcfd25b56e6d9dbf51ce8917dbb41de5c7f1f3f90c8a6b6ce086"} Oct 30 17:27:54.358990 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:54.358963 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c17e8c-54fd5c99d-m7v82" event={"ID":"1ae18756-7182-45d8-af84-4b2b41920db0","Type":"ContainerStarted","Data":"dd7c62cc30e69bf75354572b53e016cd30943d6da12cdd2e39cb05f6a58171ce"} Oct 30 17:27:54.359145 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:54.359009 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c17e8c-54fd5c99d-m7v82" event={"ID":"1ae18756-7182-45d8-af84-4b2b41920db0","Type":"ContainerStarted","Data":"88e8d644dedff428620a75de88262b902bde63fe83b5aaec1cfef57531a48f8f"} Oct 30 17:27:54.359213 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:54.359149 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c17e8c-54fd5c99d-m7v82" Oct 30 17:27:54.407342 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:54.407277 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c17e8c-7cd6d6db99-wffp5" podStartSLOduration=2.407260037 podStartE2EDuration="2.407260037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:27:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:27:54.406668839 +0000 UTC m=+2448.407423741" watchObservedRunningTime="2026-10-30 17:27:54.407260037 +0000 UTC m=+2448.408014938" Oct 30 17:27:54.452411 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:54.452358 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-c17e8c-54fd5c99d-m7v82" podStartSLOduration=2.452345802 podStartE2EDuration="2.452345802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:27:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:27:54.45123465 +0000 UTC m=+2448.451989551" watchObservedRunningTime="2026-10-30 17:27:54.452345802 +0000 UTC m=+2448.453100703" Oct 30 17:27:55.577964 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:55.577928 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-41cb25-54df857bfd-mljxr"] Oct 30 17:27:55.583342 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:55.583296 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41cb25-54df857bfd-mljxr" Oct 30 17:27:55.605209 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:55.605183 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-41cb25-54df857bfd-mljxr"] Oct 30 17:27:55.609961 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:55.608088 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-41cb25-serving-cert\"" Oct 30 17:27:55.632674 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:55.632639 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02a70782-464d-476c-916d-7f539cc59183-metrics-certs\") pod \"maas-api-e2e-ipp-a-41cb25-54df857bfd-mljxr\" (UID: \"02a70782-464d-476c-916d-7f539cc59183\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41cb25-54df857bfd-mljxr" Oct 30 17:27:55.632888 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:55.632763 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02a70782-464d-476c-916d-7f539cc59183-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-41cb25-54df857bfd-mljxr\" (UID: \"02a70782-464d-476c-916d-7f539cc59183\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41cb25-54df857bfd-mljxr" Oct 30 17:27:55.632888 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:55.632841 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9f52\" (UniqueName: \"kubernetes.io/projected/02a70782-464d-476c-916d-7f539cc59183-kube-api-access-z9f52\") pod \"maas-api-e2e-ipp-a-41cb25-54df857bfd-mljxr\" (UID: \"02a70782-464d-476c-916d-7f539cc59183\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41cb25-54df857bfd-mljxr" Oct 30 17:27:55.633040 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:55.632896 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02a70782-464d-476c-916d-7f539cc59183-maas-api-tls\") pod \"maas-api-e2e-ipp-a-41cb25-54df857bfd-mljxr\" (UID: \"02a70782-464d-476c-916d-7f539cc59183\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41cb25-54df857bfd-mljxr" Oct 30 17:27:55.734423 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:55.734383 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02a70782-464d-476c-916d-7f539cc59183-maas-api-tls\") pod \"maas-api-e2e-ipp-a-41cb25-54df857bfd-mljxr\" (UID: \"02a70782-464d-476c-916d-7f539cc59183\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41cb25-54df857bfd-mljxr" Oct 30 17:27:55.734679 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:55.734660 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02a70782-464d-476c-916d-7f539cc59183-metrics-certs\") pod \"maas-api-e2e-ipp-a-41cb25-54df857bfd-mljxr\" (UID: \"02a70782-464d-476c-916d-7f539cc59183\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41cb25-54df857bfd-mljxr" Oct 30 17:27:55.734793 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:55.734776 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02a70782-464d-476c-916d-7f539cc59183-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-41cb25-54df857bfd-mljxr\" (UID: \"02a70782-464d-476c-916d-7f539cc59183\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41cb25-54df857bfd-mljxr" Oct 30 17:27:55.734873 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:55.734859 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9f52\" (UniqueName: \"kubernetes.io/projected/02a70782-464d-476c-916d-7f539cc59183-kube-api-access-z9f52\") pod \"maas-api-e2e-ipp-a-41cb25-54df857bfd-mljxr\" (UID: \"02a70782-464d-476c-916d-7f539cc59183\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41cb25-54df857bfd-mljxr" Oct 30 17:27:55.735411 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:55.735386 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02a70782-464d-476c-916d-7f539cc59183-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-41cb25-54df857bfd-mljxr\" (UID: \"02a70782-464d-476c-916d-7f539cc59183\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41cb25-54df857bfd-mljxr" Oct 30 17:27:55.737105 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:55.737083 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02a70782-464d-476c-916d-7f539cc59183-metrics-certs\") pod \"maas-api-e2e-ipp-a-41cb25-54df857bfd-mljxr\" (UID: \"02a70782-464d-476c-916d-7f539cc59183\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41cb25-54df857bfd-mljxr" Oct 30 17:27:55.737199 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:55.737113 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02a70782-464d-476c-916d-7f539cc59183-maas-api-tls\") pod \"maas-api-e2e-ipp-a-41cb25-54df857bfd-mljxr\" (UID: \"02a70782-464d-476c-916d-7f539cc59183\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41cb25-54df857bfd-mljxr" Oct 30 17:27:55.760293 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:55.760261 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9f52\" (UniqueName: \"kubernetes.io/projected/02a70782-464d-476c-916d-7f539cc59183-kube-api-access-z9f52\") pod \"maas-api-e2e-ipp-a-41cb25-54df857bfd-mljxr\" (UID: \"02a70782-464d-476c-916d-7f539cc59183\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41cb25-54df857bfd-mljxr" Oct 30 17:27:55.904166 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:55.904091 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41cb25-54df857bfd-mljxr" Oct 30 17:27:56.224013 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:56.223978 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-41cb25-54df857bfd-mljxr"] Oct 30 17:27:56.226759 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:27:56.226729 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02a70782_464d_476c_916d_7f539cc59183.slice/crio-83a94b901aaefb699af1b5e59e4c73393f99287fc3bb7cefb7774160619e2da9 WatchSource:0}: Error finding container 83a94b901aaefb699af1b5e59e4c73393f99287fc3bb7cefb7774160619e2da9: Status 404 returned error can't find the container with id 83a94b901aaefb699af1b5e59e4c73393f99287fc3bb7cefb7774160619e2da9 Oct 30 17:27:56.378783 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:56.378749 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41cb25-54df857bfd-mljxr" event={"ID":"02a70782-464d-476c-916d-7f539cc59183","Type":"ContainerStarted","Data":"83a94b901aaefb699af1b5e59e4c73393f99287fc3bb7cefb7774160619e2da9"} Oct 30 17:27:57.389527 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:57.389487 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41cb25-54df857bfd-mljxr" event={"ID":"02a70782-464d-476c-916d-7f539cc59183","Type":"ContainerStarted","Data":"b352e0bfa19421d1536baa494d03b6d7abd33f1ca2480e13d9bcd0c9c8312d09"} Oct 30 17:27:57.389945 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:57.389570 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41cb25-54df857bfd-mljxr" Oct 30 17:27:57.440037 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:57.439982 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41cb25-54df857bfd-mljxr" podStartSLOduration=2.107042347 podStartE2EDuration="2.439968064s" totalImagesPullingTime="332.925717ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:27:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:27:57.434882335 +0000 UTC m=+2451.435637255" watchObservedRunningTime="2026-10-30 17:27:57.439968064 +0000 UTC m=+2451.440722965" Oct 30 17:27:58.103989 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:58.103948 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-78c9c2/rate-test-model-78c9c2-kserve-6fbd556d4f-6bbfp"] Oct 30 17:27:59.036542 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:59.036502 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71437729-w8n4k"] Oct 30 17:27:59.049706 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:27:59.049674 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71437729-w8n4k"] Oct 30 17:28:00.149498 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:00.149421 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad6583a4-0e86-4e83-8d27-5ec3f17060bd" path="/var/lib/kubelet/pods/ad6583a4-0e86-4e83-8d27-5ec3f17060bd/volumes" Oct 30 17:28:00.495019 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:00.494970 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-db8417/sub-test-model-db8417-kserve-fb7fdb549-6xvgx" podUID="f922e81e-fa9c-46ad-af32-5e0266311e85" containerName="main" containerID="cri-o://8428cadbae9a3eea3d9c784cda8d128aa1daa7cacb5447b3ec7719581847cfcd" gracePeriod=45 Oct 30 17:28:00.773122 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:00.773094 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-db8417/sub-test-model-db8417-kserve-fb7fdb549-6xvgx" Oct 30 17:28:00.821146 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:00.820378 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f922e81e-fa9c-46ad-af32-5e0266311e85-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f922e81e-fa9c-46ad-af32-5e0266311e85-dshm\") pod \"f922e81e-fa9c-46ad-af32-5e0266311e85\" (UID: \"f922e81e-fa9c-46ad-af32-5e0266311e85\") " Oct 30 17:28:00.821146 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:00.820487 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f922e81e-fa9c-46ad-af32-5e0266311e85-kube-api-access-zl227\" (UniqueName: \"kubernetes.io/projected/f922e81e-fa9c-46ad-af32-5e0266311e85-kube-api-access-zl227\") pod \"f922e81e-fa9c-46ad-af32-5e0266311e85\" (UID: \"f922e81e-fa9c-46ad-af32-5e0266311e85\") " Oct 30 17:28:00.821146 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:00.820527 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f922e81e-fa9c-46ad-af32-5e0266311e85-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f922e81e-fa9c-46ad-af32-5e0266311e85-tmp-dir\") pod \"f922e81e-fa9c-46ad-af32-5e0266311e85\" (UID: \"f922e81e-fa9c-46ad-af32-5e0266311e85\") " Oct 30 17:28:00.821146 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:00.820780 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f922e81e-fa9c-46ad-af32-5e0266311e85-home\" (UniqueName: \"kubernetes.io/empty-dir/f922e81e-fa9c-46ad-af32-5e0266311e85-home\") pod \"f922e81e-fa9c-46ad-af32-5e0266311e85\" (UID: \"f922e81e-fa9c-46ad-af32-5e0266311e85\") " Oct 30 17:28:00.821146 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:00.820806 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f922e81e-fa9c-46ad-af32-5e0266311e85-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f922e81e-fa9c-46ad-af32-5e0266311e85-model-cache\") pod \"f922e81e-fa9c-46ad-af32-5e0266311e85\" (UID: \"f922e81e-fa9c-46ad-af32-5e0266311e85\") " Oct 30 17:28:00.821146 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:00.820841 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f922e81e-fa9c-46ad-af32-5e0266311e85-tls-certs\" (UniqueName: \"kubernetes.io/secret/f922e81e-fa9c-46ad-af32-5e0266311e85-tls-certs\") pod \"f922e81e-fa9c-46ad-af32-5e0266311e85\" (UID: \"f922e81e-fa9c-46ad-af32-5e0266311e85\") " Oct 30 17:28:00.821636 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:00.821477 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f922e81e-fa9c-46ad-af32-5e0266311e85-tmp-dir" pod "f922e81e-fa9c-46ad-af32-5e0266311e85" (UID: "f922e81e-fa9c-46ad-af32-5e0266311e85"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:28:00.821722 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:00.821692 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f922e81e-fa9c-46ad-af32-5e0266311e85-home" pod "f922e81e-fa9c-46ad-af32-5e0266311e85" (UID: "f922e81e-fa9c-46ad-af32-5e0266311e85"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:28:00.821887 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:00.821843 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f922e81e-fa9c-46ad-af32-5e0266311e85-model-cache" pod "f922e81e-fa9c-46ad-af32-5e0266311e85" (UID: "f922e81e-fa9c-46ad-af32-5e0266311e85"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:28:00.822407 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:00.822376 4522 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f922e81e-fa9c-46ad-af32-5e0266311e85-home\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:28:00.822407 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:00.822403 4522 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f922e81e-fa9c-46ad-af32-5e0266311e85-model-cache\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:28:00.822593 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:00.822416 4522 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f922e81e-fa9c-46ad-af32-5e0266311e85-tmp-dir\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:28:00.823721 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:00.823690 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f922e81e-fa9c-46ad-af32-5e0266311e85-dshm" pod "f922e81e-fa9c-46ad-af32-5e0266311e85" (UID: "f922e81e-fa9c-46ad-af32-5e0266311e85"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:28:00.824585 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:00.824551 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f922e81e-fa9c-46ad-af32-5e0266311e85-kube-api-access-zl227" pod "f922e81e-fa9c-46ad-af32-5e0266311e85" (UID: "f922e81e-fa9c-46ad-af32-5e0266311e85"). InnerVolumeSpecName "kube-api-access-zl227". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:28:00.824931 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:00.824910 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f922e81e-fa9c-46ad-af32-5e0266311e85-tls-certs" pod "f922e81e-fa9c-46ad-af32-5e0266311e85" (UID: "f922e81e-fa9c-46ad-af32-5e0266311e85"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:28:00.923602 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:00.923561 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl227\" (UniqueName: \"kubernetes.io/projected/f922e81e-fa9c-46ad-af32-5e0266311e85-kube-api-access-zl227\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:28:00.923602 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:00.923600 4522 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f922e81e-fa9c-46ad-af32-5e0266311e85-tls-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:28:00.923602 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:00.923613 4522 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f922e81e-fa9c-46ad-af32-5e0266311e85-dshm\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:28:01.439175 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:01.439141 4522 generic.go:356] "Generic (PLEG): container finished" podID="f922e81e-fa9c-46ad-af32-5e0266311e85" containerID="8428cadbae9a3eea3d9c784cda8d128aa1daa7cacb5447b3ec7719581847cfcd" exitCode=0 Oct 30 17:28:01.439632 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:01.439185 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-db8417/sub-test-model-db8417-kserve-fb7fdb549-6xvgx" Oct 30 17:28:01.439632 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:01.439181 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-db8417/sub-test-model-db8417-kserve-fb7fdb549-6xvgx" event={"ID":"f922e81e-fa9c-46ad-af32-5e0266311e85","Type":"ContainerDied","Data":"8428cadbae9a3eea3d9c784cda8d128aa1daa7cacb5447b3ec7719581847cfcd"} Oct 30 17:28:01.439632 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:01.439296 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-db8417/sub-test-model-db8417-kserve-fb7fdb549-6xvgx" event={"ID":"f922e81e-fa9c-46ad-af32-5e0266311e85","Type":"ContainerDied","Data":"1a8889ae68bfcfedd3890085018cef5be2dcc6246f59e8f6bd5397c37b8382c3"} Oct 30 17:28:01.439632 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:01.439326 4522 scope.go:122] "RemoveContainer" containerID="8428cadbae9a3eea3d9c784cda8d128aa1daa7cacb5447b3ec7719581847cfcd" Oct 30 17:28:01.452651 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:01.452630 4522 scope.go:122] "RemoveContainer" containerID="8428cadbae9a3eea3d9c784cda8d128aa1daa7cacb5447b3ec7719581847cfcd" Oct 30 17:28:01.452978 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:28:01.452953 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8428cadbae9a3eea3d9c784cda8d128aa1daa7cacb5447b3ec7719581847cfcd\": container with ID starting with 8428cadbae9a3eea3d9c784cda8d128aa1daa7cacb5447b3ec7719581847cfcd not found: ID does not exist" containerID="8428cadbae9a3eea3d9c784cda8d128aa1daa7cacb5447b3ec7719581847cfcd" Oct 30 17:28:01.453073 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:01.452989 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8428cadbae9a3eea3d9c784cda8d128aa1daa7cacb5447b3ec7719581847cfcd"} err="failed to get container status \"8428cadbae9a3eea3d9c784cda8d128aa1daa7cacb5447b3ec7719581847cfcd\": rpc error: code = NotFound desc = could not find container \"8428cadbae9a3eea3d9c784cda8d128aa1daa7cacb5447b3ec7719581847cfcd\": container with ID starting with 8428cadbae9a3eea3d9c784cda8d128aa1daa7cacb5447b3ec7719581847cfcd not found: ID does not exist" Oct 30 17:28:01.499266 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:01.499233 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-db8417/sub-test-model-db8417-kserve-fb7fdb549-6xvgx"] Oct 30 17:28:01.511828 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:01.511801 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-db8417/sub-test-model-db8417-kserve-fb7fdb549-6xvgx"] Oct 30 17:28:02.149231 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:02.149199 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f922e81e-fa9c-46ad-af32-5e0266311e85" path="/var/lib/kubelet/pods/f922e81e-fa9c-46ad-af32-5e0266311e85/volumes" Oct 30 17:28:03.417359 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:03.417324 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41cb25-54df857bfd-mljxr" Oct 30 17:28:05.372140 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:05.372113 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c17e8c-54fd5c99d-m7v82" Oct 30 17:28:05.372624 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:05.372180 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c17e8c-7cd6d6db99-wffp5" Oct 30 17:28:09.459145 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:09.459113 4522 scope.go:122] "RemoveContainer" containerID="55e15fe15e61e17f1d74b22dd655c95b7253f1d198b15a4e84b472670e05a342" Oct 30 17:28:09.471131 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:09.471106 4522 scope.go:122] "RemoveContainer" containerID="93568b5b3ab2a334c1e0a062fa4ff50394d0d0a00a8493afe23d54a57f71263e" Oct 30 17:28:09.481729 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:09.481704 4522 scope.go:122] "RemoveContainer" containerID="f54fb7d618e31f7718806d99270edc505f1aabe25e80009f1ae698eaef73f815" Oct 30 17:28:09.496953 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:09.496926 4522 scope.go:122] "RemoveContainer" containerID="c76350e808b0571974a268287f88334bc42c818495753b7ff746a66aa92b8a36" Oct 30 17:28:09.507779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:09.507757 4522 scope.go:122] "RemoveContainer" containerID="deee53d28ea23c79e2d1215ebc7faefca10a653dc8ceebb4854a5a7687b71009" Oct 30 17:28:12.952829 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:28:12.952794 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:28:13.128853 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:13.128815 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w5-78c9c2/rate-test-model-78c9c2-kserve-6fbd556d4f-6bbfp" podUID="47d89bea-97aa-48c6-ab76-760d42f18344" containerName="main" containerID="cri-o://bb74e67eae7a465f1b54227cfc19084827ba34acbdbf796c73e4f834219bae71" gracePeriod=45 Oct 30 17:28:13.565823 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:13.565739 4522 generic.go:356] "Generic (PLEG): container finished" podID="47d89bea-97aa-48c6-ab76-760d42f18344" containerID="bb74e67eae7a465f1b54227cfc19084827ba34acbdbf796c73e4f834219bae71" exitCode=0 Oct 30 17:28:13.565983 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:13.565944 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-78c9c2/rate-test-model-78c9c2-kserve-6fbd556d4f-6bbfp" event={"ID":"47d89bea-97aa-48c6-ab76-760d42f18344","Type":"ContainerDied","Data":"bb74e67eae7a465f1b54227cfc19084827ba34acbdbf796c73e4f834219bae71"} Oct 30 17:28:13.605965 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:13.605940 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-78c9c2/rate-test-model-78c9c2-kserve-6fbd556d4f-6bbfp" Oct 30 17:28:13.664344 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:13.664296 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47d89bea-97aa-48c6-ab76-760d42f18344-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47d89bea-97aa-48c6-ab76-760d42f18344-model-cache\") pod \"47d89bea-97aa-48c6-ab76-760d42f18344\" (UID: \"47d89bea-97aa-48c6-ab76-760d42f18344\") " Oct 30 17:28:13.664546 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:13.664439 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47d89bea-97aa-48c6-ab76-760d42f18344-dshm\" (UniqueName: \"kubernetes.io/empty-dir/47d89bea-97aa-48c6-ab76-760d42f18344-dshm\") pod \"47d89bea-97aa-48c6-ab76-760d42f18344\" (UID: \"47d89bea-97aa-48c6-ab76-760d42f18344\") " Oct 30 17:28:13.664546 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:13.664499 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47d89bea-97aa-48c6-ab76-760d42f18344-kube-api-access-59n7b\" (UniqueName: \"kubernetes.io/projected/47d89bea-97aa-48c6-ab76-760d42f18344-kube-api-access-59n7b\") pod \"47d89bea-97aa-48c6-ab76-760d42f18344\" (UID: \"47d89bea-97aa-48c6-ab76-760d42f18344\") " Oct 30 17:28:13.664679 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:13.664605 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47d89bea-97aa-48c6-ab76-760d42f18344-model-cache" pod "47d89bea-97aa-48c6-ab76-760d42f18344" (UID: "47d89bea-97aa-48c6-ab76-760d42f18344"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:28:13.664679 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:13.664627 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47d89bea-97aa-48c6-ab76-760d42f18344-home\" (UniqueName: \"kubernetes.io/empty-dir/47d89bea-97aa-48c6-ab76-760d42f18344-home\") pod \"47d89bea-97aa-48c6-ab76-760d42f18344\" (UID: \"47d89bea-97aa-48c6-ab76-760d42f18344\") " Oct 30 17:28:13.664790 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:13.664692 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47d89bea-97aa-48c6-ab76-760d42f18344-tls-certs\" (UniqueName: \"kubernetes.io/secret/47d89bea-97aa-48c6-ab76-760d42f18344-tls-certs\") pod \"47d89bea-97aa-48c6-ab76-760d42f18344\" (UID: \"47d89bea-97aa-48c6-ab76-760d42f18344\") " Oct 30 17:28:13.664843 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:13.664825 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47d89bea-97aa-48c6-ab76-760d42f18344-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47d89bea-97aa-48c6-ab76-760d42f18344-tmp-dir\") pod \"47d89bea-97aa-48c6-ab76-760d42f18344\" (UID: \"47d89bea-97aa-48c6-ab76-760d42f18344\") " Oct 30 17:28:13.664948 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:13.664903 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47d89bea-97aa-48c6-ab76-760d42f18344-home" pod "47d89bea-97aa-48c6-ab76-760d42f18344" (UID: "47d89bea-97aa-48c6-ab76-760d42f18344"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:28:13.665287 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:13.665253 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47d89bea-97aa-48c6-ab76-760d42f18344-tmp-dir" pod "47d89bea-97aa-48c6-ab76-760d42f18344" (UID: "47d89bea-97aa-48c6-ab76-760d42f18344"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:28:13.665720 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:13.665698 4522 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47d89bea-97aa-48c6-ab76-760d42f18344-home\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:28:13.665783 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:13.665751 4522 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47d89bea-97aa-48c6-ab76-760d42f18344-tmp-dir\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:28:13.665783 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:13.665768 4522 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47d89bea-97aa-48c6-ab76-760d42f18344-model-cache\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:28:13.667076 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:13.667048 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47d89bea-97aa-48c6-ab76-760d42f18344-dshm" pod "47d89bea-97aa-48c6-ab76-760d42f18344" (UID: "47d89bea-97aa-48c6-ab76-760d42f18344"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:28:13.667188 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:13.667168 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47d89bea-97aa-48c6-ab76-760d42f18344-tls-certs" pod "47d89bea-97aa-48c6-ab76-760d42f18344" (UID: "47d89bea-97aa-48c6-ab76-760d42f18344"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:28:13.667270 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:13.667249 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47d89bea-97aa-48c6-ab76-760d42f18344-kube-api-access-59n7b" pod "47d89bea-97aa-48c6-ab76-760d42f18344" (UID: "47d89bea-97aa-48c6-ab76-760d42f18344"). InnerVolumeSpecName "kube-api-access-59n7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:28:13.767322 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:13.767275 4522 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47d89bea-97aa-48c6-ab76-760d42f18344-dshm\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:28:13.767322 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:13.767317 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59n7b\" (UniqueName: \"kubernetes.io/projected/47d89bea-97aa-48c6-ab76-760d42f18344-kube-api-access-59n7b\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:28:13.767538 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:13.767334 4522 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47d89bea-97aa-48c6-ab76-760d42f18344-tls-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:28:14.457757 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:14.457726 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w5-78c9c2/rate-test-model-78c9c2-kserve-6fbd556d4f-6bbfp" podUID="47d89bea-97aa-48c6-ab76-760d42f18344" containerName="main" probeResult="failure" output="Get \"https://10.134.0.241:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:28:14.576258 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:14.576225 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-78c9c2/rate-test-model-78c9c2-kserve-6fbd556d4f-6bbfp" Oct 30 17:28:14.576431 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:14.576264 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-78c9c2/rate-test-model-78c9c2-kserve-6fbd556d4f-6bbfp" event={"ID":"47d89bea-97aa-48c6-ab76-760d42f18344","Type":"ContainerDied","Data":"3622b41d655d9479bd461b27201c0019fce29c6ea7ee123eefa6c7bb277166e7"} Oct 30 17:28:14.576431 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:14.576354 4522 scope.go:122] "RemoveContainer" containerID="bb74e67eae7a465f1b54227cfc19084827ba34acbdbf796c73e4f834219bae71" Oct 30 17:28:14.638910 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:14.638881 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-78c9c2/rate-test-model-78c9c2-kserve-6fbd556d4f-6bbfp"] Oct 30 17:28:14.651433 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:14.651403 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w5-78c9c2/rate-test-model-78c9c2-kserve-6fbd556d4f-6bbfp"] Oct 30 17:28:15.927345 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:15.927294 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-82243b/rate-test-model-82243b-kserve-75d85889df-fd25g"] Oct 30 17:28:16.148235 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:16.148206 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47d89bea-97aa-48c6-ab76-760d42f18344" path="/var/lib/kubelet/pods/47d89bea-97aa-48c6-ab76-760d42f18344/volumes" Oct 30 17:28:17.847771 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:17.847739 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb"] Oct 30 17:28:17.848977 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:17.848952 4522 cpu_manager.go:414] "containerMap: removing container" podUID="f922e81e-fa9c-46ad-af32-5e0266311e85" containerName="main" Oct 30 17:28:17.848977 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:17.848976 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f922e81e-fa9c-46ad-af32-5e0266311e85" containerName="main" Oct 30 17:28:17.849167 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:17.849021 4522 cpu_manager.go:414] "containerMap: removing container" podUID="47d89bea-97aa-48c6-ab76-760d42f18344" containerName="main" Oct 30 17:28:17.849167 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:17.849029 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47d89bea-97aa-48c6-ab76-760d42f18344" containerName="main" Oct 30 17:28:17.849277 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:17.849260 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="47d89bea-97aa-48c6-ab76-760d42f18344" containerName="main" Oct 30 17:28:17.849385 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:17.849293 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="f922e81e-fa9c-46ad-af32-5e0266311e85" containerName="main" Oct 30 17:28:17.869234 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:17.869203 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb" Oct 30 17:28:17.894555 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:17.894520 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb"] Oct 30 17:28:17.898763 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:17.898739 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-e13a51-openshift-default-dockercfg-h59r9\"" Oct 30 17:28:17.957519 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:17.957476 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d64632e-0973-4589-9c43-b7b3b7bb95b3-istio-envoy\") pod \"e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb\" (UID: \"2d64632e-0973-4589-9c43-b7b3b7bb95b3\") " pod="openshift-ingress/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb" Oct 30 17:28:17.957714 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:17.957536 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4lvq\" (UniqueName: \"kubernetes.io/projected/2d64632e-0973-4589-9c43-b7b3b7bb95b3-kube-api-access-m4lvq\") pod \"e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb\" (UID: \"2d64632e-0973-4589-9c43-b7b3b7bb95b3\") " pod="openshift-ingress/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb" Oct 30 17:28:17.957852 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:17.957821 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d64632e-0973-4589-9c43-b7b3b7bb95b3-credential-socket\") pod \"e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb\" (UID: \"2d64632e-0973-4589-9c43-b7b3b7bb95b3\") " pod="openshift-ingress/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb" Oct 30 17:28:17.957911 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:17.957878 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d64632e-0973-4589-9c43-b7b3b7bb95b3-istiod-ca-cert\") pod \"e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb\" (UID: \"2d64632e-0973-4589-9c43-b7b3b7bb95b3\") " pod="openshift-ingress/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb" Oct 30 17:28:17.958003 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:17.957986 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d64632e-0973-4589-9c43-b7b3b7bb95b3-istio-podinfo\") pod \"e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb\" (UID: \"2d64632e-0973-4589-9c43-b7b3b7bb95b3\") " pod="openshift-ingress/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb" Oct 30 17:28:17.958056 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:17.958043 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d64632e-0973-4589-9c43-b7b3b7bb95b3-istio-token\") pod \"e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb\" (UID: \"2d64632e-0973-4589-9c43-b7b3b7bb95b3\") " pod="openshift-ingress/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb" Oct 30 17:28:17.958182 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:17.958163 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d64632e-0973-4589-9c43-b7b3b7bb95b3-workload-certs\") pod \"e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb\" (UID: \"2d64632e-0973-4589-9c43-b7b3b7bb95b3\") " pod="openshift-ingress/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb" Oct 30 17:28:17.958251 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:17.958200 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d64632e-0973-4589-9c43-b7b3b7bb95b3-istio-data\") pod \"e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb\" (UID: \"2d64632e-0973-4589-9c43-b7b3b7bb95b3\") " pod="openshift-ingress/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb" Oct 30 17:28:17.958394 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:17.958374 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d64632e-0973-4589-9c43-b7b3b7bb95b3-workload-socket\") pod \"e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb\" (UID: \"2d64632e-0973-4589-9c43-b7b3b7bb95b3\") " pod="openshift-ingress/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb" Oct 30 17:28:18.060360 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:18.060298 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d64632e-0973-4589-9c43-b7b3b7bb95b3-workload-socket\") pod \"e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb\" (UID: \"2d64632e-0973-4589-9c43-b7b3b7bb95b3\") " pod="openshift-ingress/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb" Oct 30 17:28:18.060551 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:18.060451 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d64632e-0973-4589-9c43-b7b3b7bb95b3-istio-envoy\") pod \"e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb\" (UID: \"2d64632e-0973-4589-9c43-b7b3b7bb95b3\") " pod="openshift-ingress/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb" Oct 30 17:28:18.060551 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:18.060501 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4lvq\" (UniqueName: \"kubernetes.io/projected/2d64632e-0973-4589-9c43-b7b3b7bb95b3-kube-api-access-m4lvq\") pod \"e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb\" (UID: \"2d64632e-0973-4589-9c43-b7b3b7bb95b3\") " pod="openshift-ingress/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb" Oct 30 17:28:18.060714 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:18.060684 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d64632e-0973-4589-9c43-b7b3b7bb95b3-credential-socket\") pod \"e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb\" (UID: \"2d64632e-0973-4589-9c43-b7b3b7bb95b3\") " pod="openshift-ingress/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb" Oct 30 17:28:18.060802 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:18.060722 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d64632e-0973-4589-9c43-b7b3b7bb95b3-istiod-ca-cert\") pod \"e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb\" (UID: \"2d64632e-0973-4589-9c43-b7b3b7bb95b3\") " pod="openshift-ingress/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb" Oct 30 17:28:18.060802 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:18.060762 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d64632e-0973-4589-9c43-b7b3b7bb95b3-istio-podinfo\") pod \"e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb\" (UID: \"2d64632e-0973-4589-9c43-b7b3b7bb95b3\") " pod="openshift-ingress/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb" Oct 30 17:28:18.060802 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:18.060720 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d64632e-0973-4589-9c43-b7b3b7bb95b3-workload-socket\") pod \"e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb\" (UID: \"2d64632e-0973-4589-9c43-b7b3b7bb95b3\") " pod="openshift-ingress/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb" Oct 30 17:28:18.060982 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:18.060899 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d64632e-0973-4589-9c43-b7b3b7bb95b3-istio-token\") pod \"e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb\" (UID: \"2d64632e-0973-4589-9c43-b7b3b7bb95b3\") " pod="openshift-ingress/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb" Oct 30 17:28:18.060982 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:18.060973 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d64632e-0973-4589-9c43-b7b3b7bb95b3-workload-certs\") pod \"e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb\" (UID: \"2d64632e-0973-4589-9c43-b7b3b7bb95b3\") " pod="openshift-ingress/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb" Oct 30 17:28:18.061095 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:18.061005 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d64632e-0973-4589-9c43-b7b3b7bb95b3-istio-data\") pod \"e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb\" (UID: \"2d64632e-0973-4589-9c43-b7b3b7bb95b3\") " pod="openshift-ingress/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb" Oct 30 17:28:18.061197 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:18.060975 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d64632e-0973-4589-9c43-b7b3b7bb95b3-credential-socket\") pod \"e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb\" (UID: \"2d64632e-0973-4589-9c43-b7b3b7bb95b3\") " pod="openshift-ingress/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb" Oct 30 17:28:18.061289 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:18.061187 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d64632e-0973-4589-9c43-b7b3b7bb95b3-workload-certs\") pod \"e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb\" (UID: \"2d64632e-0973-4589-9c43-b7b3b7bb95b3\") " pod="openshift-ingress/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb" Oct 30 17:28:18.061493 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:18.061470 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d64632e-0973-4589-9c43-b7b3b7bb95b3-istio-data\") pod \"e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb\" (UID: \"2d64632e-0973-4589-9c43-b7b3b7bb95b3\") " pod="openshift-ingress/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb" Oct 30 17:28:18.061596 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:18.061572 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d64632e-0973-4589-9c43-b7b3b7bb95b3-istiod-ca-cert\") pod \"e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb\" (UID: \"2d64632e-0973-4589-9c43-b7b3b7bb95b3\") " pod="openshift-ingress/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb" Oct 30 17:28:18.063190 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:18.063162 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d64632e-0973-4589-9c43-b7b3b7bb95b3-istio-envoy\") pod \"e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb\" (UID: \"2d64632e-0973-4589-9c43-b7b3b7bb95b3\") " pod="openshift-ingress/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb" Oct 30 17:28:18.063487 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:18.063457 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d64632e-0973-4589-9c43-b7b3b7bb95b3-istio-podinfo\") pod \"e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb\" (UID: \"2d64632e-0973-4589-9c43-b7b3b7bb95b3\") " pod="openshift-ingress/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb" Oct 30 17:28:18.088299 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:18.088270 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4lvq\" (UniqueName: \"kubernetes.io/projected/2d64632e-0973-4589-9c43-b7b3b7bb95b3-kube-api-access-m4lvq\") pod \"e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb\" (UID: \"2d64632e-0973-4589-9c43-b7b3b7bb95b3\") " pod="openshift-ingress/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb" Oct 30 17:28:18.088539 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:18.088518 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d64632e-0973-4589-9c43-b7b3b7bb95b3-istio-token\") pod \"e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb\" (UID: \"2d64632e-0973-4589-9c43-b7b3b7bb95b3\") " pod="openshift-ingress/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb" Oct 30 17:28:18.222084 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:18.222053 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb" Oct 30 17:28:18.576534 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:28:18.576503 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d64632e_0973_4589_9c43_b7b3b7bb95b3.slice/crio-d6c26fc2dc39bc21b85df0e2d00830a69043d800db559721e7cd13a50457211a WatchSource:0}: Error finding container d6c26fc2dc39bc21b85df0e2d00830a69043d800db559721e7cd13a50457211a: Status 404 returned error can't find the container with id d6c26fc2dc39bc21b85df0e2d00830a69043d800db559721e7cd13a50457211a Oct 30 17:28:18.578005 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:18.577977 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb"] Oct 30 17:28:18.579961 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:18.579931 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:28:18.580035 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:18.579994 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:28:18.580035 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:18.580026 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:28:18.766907 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:18.766864 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb" event={"ID":"2d64632e-0973-4589-9c43-b7b3b7bb95b3","Type":"ContainerStarted","Data":"af69effeda096f045d40224956e652f26bcd057525521bd203c2fbe7fe259de0"} Oct 30 17:28:18.767193 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:18.767173 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb" event={"ID":"2d64632e-0973-4589-9c43-b7b3b7bb95b3","Type":"ContainerStarted","Data":"d6c26fc2dc39bc21b85df0e2d00830a69043d800db559721e7cd13a50457211a"} Oct 30 17:28:18.829037 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:18.828906 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb" podStartSLOduration=1.8288870560000001 podStartE2EDuration="1.828887056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:28:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:28:18.826976682 +0000 UTC m=+2472.827731585" watchObservedRunningTime="2026-10-30 17:28:18.828887056 +0000 UTC m=+2472.829641961" Oct 30 17:28:19.227332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:19.224336 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb" Oct 30 17:28:19.227332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:19.225090 4522 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" start-of-body= Oct 30 17:28:19.227332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:19.225142 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb" podUID="2d64632e-0973-4589-9c43-b7b3b7bb95b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" Oct 30 17:28:20.228529 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:20.228497 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb" Oct 30 17:28:20.798190 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:20.798153 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb" Oct 30 17:28:20.799248 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:20.799230 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb" Oct 30 17:28:23.928851 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:23.928813 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e13a51-7f74f8bfcf-5wstg"] Oct 30 17:28:23.934211 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:23.934192 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e13a51-7f74f8bfcf-5wstg" Oct 30 17:28:23.953757 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:23.953728 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e13a51-7f74f8bfcf-5wstg"] Oct 30 17:28:23.961195 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:23.961164 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-e13a51\"" Oct 30 17:28:23.961402 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:23.961235 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-e13a51-dockercfg-n7jrv\"" Oct 30 17:28:24.002462 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:24.002428 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-e13a51-b5f84f694-7tz8j"] Oct 30 17:28:24.007418 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:24.007399 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e13a51-b5f84f694-7tz8j" Oct 30 17:28:24.015867 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:24.015822 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eef74243-b455-4c59-baf2-d4ae87e3e5c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-e13a51-7f74f8bfcf-5wstg\" (UID: \"eef74243-b455-4c59-baf2-d4ae87e3e5c4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e13a51-7f74f8bfcf-5wstg" Oct 30 17:28:24.016509 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:24.016483 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrtzf\" (UniqueName: \"kubernetes.io/projected/eef74243-b455-4c59-baf2-d4ae87e3e5c4-kube-api-access-zrtzf\") pod \"payload-pre-processing-e2e-shared-b-w6-e13a51-7f74f8bfcf-5wstg\" (UID: \"eef74243-b455-4c59-baf2-d4ae87e3e5c4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e13a51-7f74f8bfcf-5wstg" Oct 30 17:28:24.083355 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:24.071135 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-e13a51-b5f84f694-7tz8j"] Oct 30 17:28:24.126328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:24.124652 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrtzf\" (UniqueName: \"kubernetes.io/projected/eef74243-b455-4c59-baf2-d4ae87e3e5c4-kube-api-access-zrtzf\") pod \"payload-pre-processing-e2e-shared-b-w6-e13a51-7f74f8bfcf-5wstg\" (UID: \"eef74243-b455-4c59-baf2-d4ae87e3e5c4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e13a51-7f74f8bfcf-5wstg" Oct 30 17:28:24.126328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:24.124798 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eef74243-b455-4c59-baf2-d4ae87e3e5c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-e13a51-7f74f8bfcf-5wstg\" (UID: \"eef74243-b455-4c59-baf2-d4ae87e3e5c4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e13a51-7f74f8bfcf-5wstg" Oct 30 17:28:24.141370 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:24.140100 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eef74243-b455-4c59-baf2-d4ae87e3e5c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-e13a51-7f74f8bfcf-5wstg\" (UID: \"eef74243-b455-4c59-baf2-d4ae87e3e5c4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e13a51-7f74f8bfcf-5wstg" Oct 30 17:28:24.231325 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:24.228667 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4135bd88-8ef0-44c0-8679-9f6719ef3a18-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-e13a51-b5f84f694-7tz8j\" (UID: \"4135bd88-8ef0-44c0-8679-9f6719ef3a18\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e13a51-b5f84f694-7tz8j" Oct 30 17:28:24.231325 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:24.229036 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6hjs\" (UniqueName: \"kubernetes.io/projected/4135bd88-8ef0-44c0-8679-9f6719ef3a18-kube-api-access-m6hjs\") pod \"payload-processing-e2e-shared-b-w6-e13a51-b5f84f694-7tz8j\" (UID: \"4135bd88-8ef0-44c0-8679-9f6719ef3a18\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e13a51-b5f84f694-7tz8j" Oct 30 17:28:24.236902 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:24.236823 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrtzf\" (UniqueName: \"kubernetes.io/projected/eef74243-b455-4c59-baf2-d4ae87e3e5c4-kube-api-access-zrtzf\") pod \"payload-pre-processing-e2e-shared-b-w6-e13a51-7f74f8bfcf-5wstg\" (UID: \"eef74243-b455-4c59-baf2-d4ae87e3e5c4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e13a51-7f74f8bfcf-5wstg" Oct 30 17:28:24.253128 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:24.253099 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e13a51-7f74f8bfcf-5wstg" Oct 30 17:28:24.331532 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:24.331496 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6hjs\" (UniqueName: \"kubernetes.io/projected/4135bd88-8ef0-44c0-8679-9f6719ef3a18-kube-api-access-m6hjs\") pod \"payload-processing-e2e-shared-b-w6-e13a51-b5f84f694-7tz8j\" (UID: \"4135bd88-8ef0-44c0-8679-9f6719ef3a18\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e13a51-b5f84f694-7tz8j" Oct 30 17:28:24.331883 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:24.331860 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4135bd88-8ef0-44c0-8679-9f6719ef3a18-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-e13a51-b5f84f694-7tz8j\" (UID: \"4135bd88-8ef0-44c0-8679-9f6719ef3a18\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e13a51-b5f84f694-7tz8j" Oct 30 17:28:24.335948 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:24.335913 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4135bd88-8ef0-44c0-8679-9f6719ef3a18-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-e13a51-b5f84f694-7tz8j\" (UID: \"4135bd88-8ef0-44c0-8679-9f6719ef3a18\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e13a51-b5f84f694-7tz8j" Oct 30 17:28:24.374385 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:24.371342 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6hjs\" (UniqueName: \"kubernetes.io/projected/4135bd88-8ef0-44c0-8679-9f6719ef3a18-kube-api-access-m6hjs\") pod \"payload-processing-e2e-shared-b-w6-e13a51-b5f84f694-7tz8j\" (UID: \"4135bd88-8ef0-44c0-8679-9f6719ef3a18\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e13a51-b5f84f694-7tz8j" Oct 30 17:28:24.586516 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:28:24.586480 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeef74243_b455_4c59_baf2_d4ae87e3e5c4.slice/crio-900dda9a2f4c724b299142f6ce42c3c236341ac5514f25b8c3185d916e76683a WatchSource:0}: Error finding container 900dda9a2f4c724b299142f6ce42c3c236341ac5514f25b8c3185d916e76683a: Status 404 returned error can't find the container with id 900dda9a2f4c724b299142f6ce42c3c236341ac5514f25b8c3185d916e76683a Oct 30 17:28:24.589228 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:24.589201 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e13a51-7f74f8bfcf-5wstg"] Oct 30 17:28:24.663246 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:24.663216 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e13a51-b5f84f694-7tz8j" Oct 30 17:28:24.839897 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:24.839769 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e13a51-7f74f8bfcf-5wstg" event={"ID":"eef74243-b455-4c59-baf2-d4ae87e3e5c4","Type":"ContainerStarted","Data":"9df31bc48b79b7872a08ab9617b8719fe068abc5cfa06a45c27eddb926eeb5e7"} Oct 30 17:28:24.839897 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:24.839832 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e13a51-7f74f8bfcf-5wstg" event={"ID":"eef74243-b455-4c59-baf2-d4ae87e3e5c4","Type":"ContainerStarted","Data":"900dda9a2f4c724b299142f6ce42c3c236341ac5514f25b8c3185d916e76683a"} Oct 30 17:28:24.839897 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:24.839860 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e13a51-7f74f8bfcf-5wstg" Oct 30 17:28:24.891107 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:24.891050 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e13a51-7f74f8bfcf-5wstg" podStartSLOduration=1.891035123 podStartE2EDuration="1.891035123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:28:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:28:24.886291712 +0000 UTC m=+2478.887046612" watchObservedRunningTime="2026-10-30 17:28:24.891035123 +0000 UTC m=+2478.891790044" Oct 30 17:28:24.974735 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:28:24.974704 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4135bd88_8ef0_44c0_8679_9f6719ef3a18.slice/crio-bc66c6dedeebd3acdafdbcbf1d439099805e46f9debea09f2732cfbf997b0c46 WatchSource:0}: Error finding container bc66c6dedeebd3acdafdbcbf1d439099805e46f9debea09f2732cfbf997b0c46: Status 404 returned error can't find the container with id bc66c6dedeebd3acdafdbcbf1d439099805e46f9debea09f2732cfbf997b0c46 Oct 30 17:28:24.977473 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:24.977451 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-e13a51-b5f84f694-7tz8j"] Oct 30 17:28:25.620098 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:25.620054 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8a7e4a-d55dd8b4d-hd4qr"] Oct 30 17:28:25.625459 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:25.625436 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8a7e4a-d55dd8b4d-hd4qr" Oct 30 17:28:25.644710 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:25.644402 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-8a7e4a-dockercfg-m25tr\"" Oct 30 17:28:25.645087 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:25.645067 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-8a7e4a\"" Oct 30 17:28:25.651373 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:25.651344 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8a7e4a-d55dd8b4d-hd4qr"] Oct 30 17:28:25.764277 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:25.763200 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/901bc210-b26c-41df-b40f-e94a12c578c5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8a7e4a-d55dd8b4d-hd4qr\" (UID: \"901bc210-b26c-41df-b40f-e94a12c578c5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8a7e4a-d55dd8b4d-hd4qr" Oct 30 17:28:25.764277 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:25.763615 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j5f4\" (UniqueName: \"kubernetes.io/projected/901bc210-b26c-41df-b40f-e94a12c578c5-kube-api-access-6j5f4\") pod \"payload-processing-e2e-ipp-a-8a7e4a-d55dd8b4d-hd4qr\" (UID: \"901bc210-b26c-41df-b40f-e94a12c578c5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8a7e4a-d55dd8b4d-hd4qr" Oct 30 17:28:25.869382 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:25.868715 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j5f4\" (UniqueName: \"kubernetes.io/projected/901bc210-b26c-41df-b40f-e94a12c578c5-kube-api-access-6j5f4\") pod \"payload-processing-e2e-ipp-a-8a7e4a-d55dd8b4d-hd4qr\" (UID: \"901bc210-b26c-41df-b40f-e94a12c578c5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8a7e4a-d55dd8b4d-hd4qr" Oct 30 17:28:25.869382 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:25.868824 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/901bc210-b26c-41df-b40f-e94a12c578c5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8a7e4a-d55dd8b4d-hd4qr\" (UID: \"901bc210-b26c-41df-b40f-e94a12c578c5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8a7e4a-d55dd8b4d-hd4qr" Oct 30 17:28:25.899750 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:25.899674 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/901bc210-b26c-41df-b40f-e94a12c578c5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8a7e4a-d55dd8b4d-hd4qr\" (UID: \"901bc210-b26c-41df-b40f-e94a12c578c5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8a7e4a-d55dd8b4d-hd4qr" Oct 30 17:28:25.923682 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:25.922940 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e13a51-b5f84f694-7tz8j" event={"ID":"4135bd88-8ef0-44c0-8679-9f6719ef3a18","Type":"ContainerStarted","Data":"9152f62938dfa61b26630e9b83ed63c9536a27d663e9ad339b936567ab797007"} Oct 30 17:28:25.923682 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:25.922992 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e13a51-b5f84f694-7tz8j" event={"ID":"4135bd88-8ef0-44c0-8679-9f6719ef3a18","Type":"ContainerStarted","Data":"bc66c6dedeebd3acdafdbcbf1d439099805e46f9debea09f2732cfbf997b0c46"} Oct 30 17:28:25.926052 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:25.926021 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j5f4\" (UniqueName: \"kubernetes.io/projected/901bc210-b26c-41df-b40f-e94a12c578c5-kube-api-access-6j5f4\") pod \"payload-processing-e2e-ipp-a-8a7e4a-d55dd8b4d-hd4qr\" (UID: \"901bc210-b26c-41df-b40f-e94a12c578c5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8a7e4a-d55dd8b4d-hd4qr" Oct 30 17:28:25.944228 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:25.944190 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8a7e4a-d55dd8b4d-hd4qr" Oct 30 17:28:25.977776 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:25.977591 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e13a51-b5f84f694-7tz8j" podStartSLOduration=2.977572438 podStartE2EDuration="2.977572438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:28:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:28:25.970514647 +0000 UTC m=+2479.971269549" watchObservedRunningTime="2026-10-30 17:28:25.977572438 +0000 UTC m=+2479.978327340" Oct 30 17:28:26.265287 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:28:26.264876 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod901bc210_b26c_41df_b40f_e94a12c578c5.slice/crio-305e40cee1369994de2fb89da5ad57b093d1a5fd5e4c16bae46ff47defe022ab WatchSource:0}: Error finding container 305e40cee1369994de2fb89da5ad57b093d1a5fd5e4c16bae46ff47defe022ab: Status 404 returned error can't find the container with id 305e40cee1369994de2fb89da5ad57b093d1a5fd5e4c16bae46ff47defe022ab Oct 30 17:28:26.267384 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:26.267337 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8a7e4a-d55dd8b4d-hd4qr"] Oct 30 17:28:26.933821 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:26.933787 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8a7e4a-d55dd8b4d-hd4qr" event={"ID":"901bc210-b26c-41df-b40f-e94a12c578c5","Type":"ContainerStarted","Data":"86f928ca250d042a74c0658e77b9760e7fa6d5fef063db548206bcc856d499b7"} Oct 30 17:28:26.934004 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:26.933839 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8a7e4a-d55dd8b4d-hd4qr" event={"ID":"901bc210-b26c-41df-b40f-e94a12c578c5","Type":"ContainerStarted","Data":"305e40cee1369994de2fb89da5ad57b093d1a5fd5e4c16bae46ff47defe022ab"} Oct 30 17:28:26.934004 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:26.933934 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e13a51-b5f84f694-7tz8j" Oct 30 17:28:26.981037 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:26.980987 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-8a7e4a-d55dd8b4d-hd4qr" podStartSLOduration=1.980972848 podStartE2EDuration="1.980972848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:28:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:28:26.980328109 +0000 UTC m=+2480.981083011" watchObservedRunningTime="2026-10-30 17:28:26.980972848 +0000 UTC m=+2480.981727785" Oct 30 17:28:27.943211 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:27.943180 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8a7e4a-d55dd8b4d-hd4qr" Oct 30 17:28:30.949489 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:30.949451 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w5-82243b/rate-test-model-82243b-kserve-75d85889df-fd25g" podUID="882b8f38-848d-4fe0-aec9-bd81b63ace42" containerName="main" containerID="cri-o://35788251ba76cbd2479bd7a25f117bfb2c16954584f5982894c6356339ec77bb" gracePeriod=45 Oct 30 17:28:31.243982 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:31.243955 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-82243b/rate-test-model-82243b-kserve-75d85889df-fd25g" Oct 30 17:28:31.368516 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:31.368467 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/882b8f38-848d-4fe0-aec9-bd81b63ace42-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/882b8f38-848d-4fe0-aec9-bd81b63ace42-tmp-dir\") pod \"882b8f38-848d-4fe0-aec9-bd81b63ace42\" (UID: \"882b8f38-848d-4fe0-aec9-bd81b63ace42\") " Oct 30 17:28:31.368721 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:31.368645 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/882b8f38-848d-4fe0-aec9-bd81b63ace42-tls-certs\" (UniqueName: \"kubernetes.io/secret/882b8f38-848d-4fe0-aec9-bd81b63ace42-tls-certs\") pod \"882b8f38-848d-4fe0-aec9-bd81b63ace42\" (UID: \"882b8f38-848d-4fe0-aec9-bd81b63ace42\") " Oct 30 17:28:31.368721 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:31.368684 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/882b8f38-848d-4fe0-aec9-bd81b63ace42-home\" (UniqueName: \"kubernetes.io/empty-dir/882b8f38-848d-4fe0-aec9-bd81b63ace42-home\") pod \"882b8f38-848d-4fe0-aec9-bd81b63ace42\" (UID: \"882b8f38-848d-4fe0-aec9-bd81b63ace42\") " Oct 30 17:28:31.368859 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:31.368737 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/882b8f38-848d-4fe0-aec9-bd81b63ace42-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/882b8f38-848d-4fe0-aec9-bd81b63ace42-model-cache\") pod \"882b8f38-848d-4fe0-aec9-bd81b63ace42\" (UID: \"882b8f38-848d-4fe0-aec9-bd81b63ace42\") " Oct 30 17:28:31.368859 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:31.368788 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/882b8f38-848d-4fe0-aec9-bd81b63ace42-tmp-dir" pod "882b8f38-848d-4fe0-aec9-bd81b63ace42" (UID: "882b8f38-848d-4fe0-aec9-bd81b63ace42"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:28:31.368859 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:31.368799 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/882b8f38-848d-4fe0-aec9-bd81b63ace42-dshm\" (UniqueName: \"kubernetes.io/empty-dir/882b8f38-848d-4fe0-aec9-bd81b63ace42-dshm\") pod \"882b8f38-848d-4fe0-aec9-bd81b63ace42\" (UID: \"882b8f38-848d-4fe0-aec9-bd81b63ace42\") " Oct 30 17:28:31.369038 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:31.368949 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/882b8f38-848d-4fe0-aec9-bd81b63ace42-home" pod "882b8f38-848d-4fe0-aec9-bd81b63ace42" (UID: "882b8f38-848d-4fe0-aec9-bd81b63ace42"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:28:31.369111 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:31.369060 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/882b8f38-848d-4fe0-aec9-bd81b63ace42-model-cache" pod "882b8f38-848d-4fe0-aec9-bd81b63ace42" (UID: "882b8f38-848d-4fe0-aec9-bd81b63ace42"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:28:31.369229 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:31.369117 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/882b8f38-848d-4fe0-aec9-bd81b63ace42-kube-api-access-4hqzt\" (UniqueName: \"kubernetes.io/projected/882b8f38-848d-4fe0-aec9-bd81b63ace42-kube-api-access-4hqzt\") pod \"882b8f38-848d-4fe0-aec9-bd81b63ace42\" (UID: \"882b8f38-848d-4fe0-aec9-bd81b63ace42\") " Oct 30 17:28:31.370022 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:31.369994 4522 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/882b8f38-848d-4fe0-aec9-bd81b63ace42-tmp-dir\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:28:31.370022 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:31.370019 4522 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/882b8f38-848d-4fe0-aec9-bd81b63ace42-home\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:28:31.370022 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:31.370027 4522 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/882b8f38-848d-4fe0-aec9-bd81b63ace42-model-cache\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:28:31.371157 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:31.371132 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/882b8f38-848d-4fe0-aec9-bd81b63ace42-tls-certs" pod "882b8f38-848d-4fe0-aec9-bd81b63ace42" (UID: "882b8f38-848d-4fe0-aec9-bd81b63ace42"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:28:31.371257 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:31.371181 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/882b8f38-848d-4fe0-aec9-bd81b63ace42-dshm" pod "882b8f38-848d-4fe0-aec9-bd81b63ace42" (UID: "882b8f38-848d-4fe0-aec9-bd81b63ace42"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:28:31.371257 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:31.371226 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/882b8f38-848d-4fe0-aec9-bd81b63ace42-kube-api-access-4hqzt" pod "882b8f38-848d-4fe0-aec9-bd81b63ace42" (UID: "882b8f38-848d-4fe0-aec9-bd81b63ace42"). InnerVolumeSpecName "kube-api-access-4hqzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:28:31.472008 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:31.471931 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hqzt\" (UniqueName: \"kubernetes.io/projected/882b8f38-848d-4fe0-aec9-bd81b63ace42-kube-api-access-4hqzt\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:28:31.472008 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:31.471959 4522 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/882b8f38-848d-4fe0-aec9-bd81b63ace42-tls-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:28:31.472008 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:31.471969 4522 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/882b8f38-848d-4fe0-aec9-bd81b63ace42-dshm\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:28:31.981466 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:31.981434 4522 generic.go:356] "Generic (PLEG): container finished" podID="882b8f38-848d-4fe0-aec9-bd81b63ace42" containerID="35788251ba76cbd2479bd7a25f117bfb2c16954584f5982894c6356339ec77bb" exitCode=0 Oct 30 17:28:31.981993 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:31.981500 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-82243b/rate-test-model-82243b-kserve-75d85889df-fd25g" event={"ID":"882b8f38-848d-4fe0-aec9-bd81b63ace42","Type":"ContainerDied","Data":"35788251ba76cbd2479bd7a25f117bfb2c16954584f5982894c6356339ec77bb"} Oct 30 17:28:31.981993 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:31.981529 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-82243b/rate-test-model-82243b-kserve-75d85889df-fd25g" event={"ID":"882b8f38-848d-4fe0-aec9-bd81b63ace42","Type":"ContainerDied","Data":"ef09d2261b7db2529d669f2513bdc5a474a2ed64a4a37c4153cc1dcbefd3b7af"} Oct 30 17:28:31.981993 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:31.981545 4522 scope.go:122] "RemoveContainer" containerID="35788251ba76cbd2479bd7a25f117bfb2c16954584f5982894c6356339ec77bb" Oct 30 17:28:31.981993 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:31.981501 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-82243b/rate-test-model-82243b-kserve-75d85889df-fd25g" Oct 30 17:28:31.995134 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:31.995114 4522 scope.go:122] "RemoveContainer" containerID="35788251ba76cbd2479bd7a25f117bfb2c16954584f5982894c6356339ec77bb" Oct 30 17:28:31.995438 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:28:31.995418 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35788251ba76cbd2479bd7a25f117bfb2c16954584f5982894c6356339ec77bb\": container with ID starting with 35788251ba76cbd2479bd7a25f117bfb2c16954584f5982894c6356339ec77bb not found: ID does not exist" containerID="35788251ba76cbd2479bd7a25f117bfb2c16954584f5982894c6356339ec77bb" Oct 30 17:28:31.995513 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:31.995448 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35788251ba76cbd2479bd7a25f117bfb2c16954584f5982894c6356339ec77bb"} err="failed to get container status \"35788251ba76cbd2479bd7a25f117bfb2c16954584f5982894c6356339ec77bb\": rpc error: code = NotFound desc = could not find container \"35788251ba76cbd2479bd7a25f117bfb2c16954584f5982894c6356339ec77bb\": container with ID starting with 35788251ba76cbd2479bd7a25f117bfb2c16954584f5982894c6356339ec77bb not found: ID does not exist" Oct 30 17:28:32.042806 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:32.042777 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-82243b/rate-test-model-82243b-kserve-75d85889df-fd25g"] Oct 30 17:28:32.055457 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:32.055431 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w5-82243b/rate-test-model-82243b-kserve-75d85889df-fd25g"] Oct 30 17:28:32.147867 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:32.147836 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="882b8f38-848d-4fe0-aec9-bd81b63ace42" path="/var/lib/kubelet/pods/882b8f38-848d-4fe0-aec9-bd81b63ace42/volumes" Oct 30 17:28:34.035162 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:34.035127 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e08b0a0b-6xr9w"] Oct 30 17:28:34.052206 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:34.052165 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e08b0a0b-6xr9w"] Oct 30 17:28:34.171214 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:34.171178 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5528580-d2f5-4201-859e-f9f2c1e6c588" path="/var/lib/kubelet/pods/f5528580-d2f5-4201-859e-f9f2c1e6c588/volumes" Oct 30 17:28:35.927275 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:35.927245 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e13a51-7f74f8bfcf-5wstg" Oct 30 17:28:37.945696 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:37.945668 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e13a51-b5f84f694-7tz8j" Oct 30 17:28:38.955373 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:38.955345 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8a7e4a-d55dd8b4d-hd4qr" Oct 30 17:28:42.949549 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:28:42.949514 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:28:54.865570 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:54.865528 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-41cb25-56b855cbbb-x7mzc"] Oct 30 17:28:54.900396 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:54.900366 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-41cb25-db9f8d799-t27kf"] Oct 30 17:28:57.380330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:57.377399 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d2d985-688fcd588c-45k5x"] Oct 30 17:28:57.380330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:57.378919 4522 cpu_manager.go:414] "containerMap: removing container" podUID="882b8f38-848d-4fe0-aec9-bd81b63ace42" containerName="main" Oct 30 17:28:57.380330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:57.378944 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="882b8f38-848d-4fe0-aec9-bd81b63ace42" containerName="main" Oct 30 17:28:57.380330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:57.379196 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="882b8f38-848d-4fe0-aec9-bd81b63ace42" containerName="main" Oct 30 17:28:57.393448 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:57.393416 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d2d985-688fcd588c-45k5x" Oct 30 17:28:57.395212 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:57.395188 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d2d985-688fcd588c-45k5x"] Oct 30 17:28:57.445542 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:57.435336 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgxft\" (UniqueName: \"kubernetes.io/projected/93279e37-0139-4f89-a5a0-55372047919c-kube-api-access-xgxft\") pod \"payload-pre-processing-e2e-ipp-a-d2d985-688fcd588c-45k5x\" (UID: \"93279e37-0139-4f89-a5a0-55372047919c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d2d985-688fcd588c-45k5x" Oct 30 17:28:57.445542 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:57.435413 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93279e37-0139-4f89-a5a0-55372047919c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d2d985-688fcd588c-45k5x\" (UID: \"93279e37-0139-4f89-a5a0-55372047919c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d2d985-688fcd588c-45k5x" Oct 30 17:28:57.445542 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:57.438338 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-d2d985\"" Oct 30 17:28:57.445542 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:57.438410 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-d2d985-dockercfg-q8r5n\"" Oct 30 17:28:57.531829 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:57.531798 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d2d985-698fbdbbdb-z4wjk"] Oct 30 17:28:57.536997 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:57.536963 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgxft\" (UniqueName: \"kubernetes.io/projected/93279e37-0139-4f89-a5a0-55372047919c-kube-api-access-xgxft\") pod \"payload-pre-processing-e2e-ipp-a-d2d985-688fcd588c-45k5x\" (UID: \"93279e37-0139-4f89-a5a0-55372047919c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d2d985-688fcd588c-45k5x" Oct 30 17:28:57.537189 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:57.537022 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93279e37-0139-4f89-a5a0-55372047919c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d2d985-688fcd588c-45k5x\" (UID: \"93279e37-0139-4f89-a5a0-55372047919c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d2d985-688fcd588c-45k5x" Oct 30 17:28:57.537423 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:57.537403 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d2d985-698fbdbbdb-z4wjk" Oct 30 17:28:57.537975 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:57.537955 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93279e37-0139-4f89-a5a0-55372047919c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d2d985-688fcd588c-45k5x\" (UID: \"93279e37-0139-4f89-a5a0-55372047919c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d2d985-688fcd588c-45k5x" Oct 30 17:28:57.557998 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:57.557965 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d2d985-698fbdbbdb-z4wjk"] Oct 30 17:28:57.582521 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:57.581465 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgxft\" (UniqueName: \"kubernetes.io/projected/93279e37-0139-4f89-a5a0-55372047919c-kube-api-access-xgxft\") pod \"payload-pre-processing-e2e-ipp-a-d2d985-688fcd588c-45k5x\" (UID: \"93279e37-0139-4f89-a5a0-55372047919c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d2d985-688fcd588c-45k5x" Oct 30 17:28:57.638663 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:57.638566 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9db299c4-22b3-4237-8d6c-b8dc9c20b117-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d2d985-698fbdbbdb-z4wjk\" (UID: \"9db299c4-22b3-4237-8d6c-b8dc9c20b117\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d2d985-698fbdbbdb-z4wjk" Oct 30 17:28:57.638846 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:57.638716 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf5tw\" (UniqueName: \"kubernetes.io/projected/9db299c4-22b3-4237-8d6c-b8dc9c20b117-kube-api-access-rf5tw\") pod \"payload-processing-e2e-ipp-a-d2d985-698fbdbbdb-z4wjk\" (UID: \"9db299c4-22b3-4237-8d6c-b8dc9c20b117\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d2d985-698fbdbbdb-z4wjk" Oct 30 17:28:57.740839 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:57.740806 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9db299c4-22b3-4237-8d6c-b8dc9c20b117-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d2d985-698fbdbbdb-z4wjk\" (UID: \"9db299c4-22b3-4237-8d6c-b8dc9c20b117\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d2d985-698fbdbbdb-z4wjk" Oct 30 17:28:57.741042 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:57.740891 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf5tw\" (UniqueName: \"kubernetes.io/projected/9db299c4-22b3-4237-8d6c-b8dc9c20b117-kube-api-access-rf5tw\") pod \"payload-processing-e2e-ipp-a-d2d985-698fbdbbdb-z4wjk\" (UID: \"9db299c4-22b3-4237-8d6c-b8dc9c20b117\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d2d985-698fbdbbdb-z4wjk" Oct 30 17:28:57.741641 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:57.741617 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9db299c4-22b3-4237-8d6c-b8dc9c20b117-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d2d985-698fbdbbdb-z4wjk\" (UID: \"9db299c4-22b3-4237-8d6c-b8dc9c20b117\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d2d985-698fbdbbdb-z4wjk" Oct 30 17:28:57.766051 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:57.766025 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf5tw\" (UniqueName: \"kubernetes.io/projected/9db299c4-22b3-4237-8d6c-b8dc9c20b117-kube-api-access-rf5tw\") pod \"payload-processing-e2e-ipp-a-d2d985-698fbdbbdb-z4wjk\" (UID: \"9db299c4-22b3-4237-8d6c-b8dc9c20b117\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d2d985-698fbdbbdb-z4wjk" Oct 30 17:28:57.769924 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:57.769905 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d2d985-688fcd588c-45k5x" Oct 30 17:28:57.859864 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:57.859833 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d2d985-698fbdbbdb-z4wjk" Oct 30 17:28:58.081994 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:28:58.081960 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93279e37_0139_4f89_a5a0_55372047919c.slice/crio-55bc2394915daa66f731801ca13c18ab878c42d4132a09d11d1720b06b10a748 WatchSource:0}: Error finding container 55bc2394915daa66f731801ca13c18ab878c42d4132a09d11d1720b06b10a748: Status 404 returned error can't find the container with id 55bc2394915daa66f731801ca13c18ab878c42d4132a09d11d1720b06b10a748 Oct 30 17:28:58.084429 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:58.084405 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d2d985-688fcd588c-45k5x"] Oct 30 17:28:58.189820 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:28:58.189785 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9db299c4_22b3_4237_8d6c_b8dc9c20b117.slice/crio-96e73121dc4d9d8e1fbb9681d66b95c4dfba7dddb7d09bf4a58ba06f0561a184 WatchSource:0}: Error finding container 96e73121dc4d9d8e1fbb9681d66b95c4dfba7dddb7d09bf4a58ba06f0561a184: Status 404 returned error can't find the container with id 96e73121dc4d9d8e1fbb9681d66b95c4dfba7dddb7d09bf4a58ba06f0561a184 Oct 30 17:28:58.193079 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:58.193052 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d2d985-698fbdbbdb-z4wjk"] Oct 30 17:28:58.252253 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:58.252222 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d2d985-688fcd588c-45k5x" event={"ID":"93279e37-0139-4f89-a5a0-55372047919c","Type":"ContainerStarted","Data":"3ac0a2388198a40a87a2708d8ef6a474f824c69ddcc2122af06b51ed7ec4d12b"} Oct 30 17:28:58.252430 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:58.252283 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d2d985-688fcd588c-45k5x" event={"ID":"93279e37-0139-4f89-a5a0-55372047919c","Type":"ContainerStarted","Data":"55bc2394915daa66f731801ca13c18ab878c42d4132a09d11d1720b06b10a748"} Oct 30 17:28:58.252518 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:58.252499 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d2d985-688fcd588c-45k5x" Oct 30 17:28:58.253976 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:58.253948 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d2d985-698fbdbbdb-z4wjk" event={"ID":"9db299c4-22b3-4237-8d6c-b8dc9c20b117","Type":"ContainerStarted","Data":"96e73121dc4d9d8e1fbb9681d66b95c4dfba7dddb7d09bf4a58ba06f0561a184"} Oct 30 17:28:58.298237 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:58.298186 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d2d985-688fcd588c-45k5x" podStartSLOduration=1.298172817 podStartE2EDuration="1.298172817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:28:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:28:58.29719095 +0000 UTC m=+2512.297945852" watchObservedRunningTime="2026-10-30 17:28:58.298172817 +0000 UTC m=+2512.298927718" Oct 30 17:28:59.035666 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:59.035628 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-094cd551-s65fn"] Oct 30 17:28:59.048735 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:59.048701 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-094cd551-s65fn"] Oct 30 17:28:59.267273 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:59.267229 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d2d985-698fbdbbdb-z4wjk" event={"ID":"9db299c4-22b3-4237-8d6c-b8dc9c20b117","Type":"ContainerStarted","Data":"e0ba3abdae1db74c4f59777844c113213ae0618eb8c35130c23f35e4c726ca65"} Oct 30 17:28:59.267800 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:59.267778 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d2d985-698fbdbbdb-z4wjk" Oct 30 17:28:59.314486 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:59.314391 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-d2d985-698fbdbbdb-z4wjk" podStartSLOduration=2.314378161 podStartE2EDuration="2.314378161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:28:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:28:59.311758332 +0000 UTC m=+2513.312513263" watchObservedRunningTime="2026-10-30 17:28:59.314378161 +0000 UTC m=+2513.315133062" Oct 30 17:28:59.865895 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:59.865856 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-41cb25-56b855cbbb-x7mzc" podUID="7e7ab767-5219-47a3-988f-fb4240a88652" containerName="payload-processing" containerID="cri-o://bd4aade4914249fb22638a2d7ac8557f7d3a3d5a1c17187a63ff949abecc3f5a" gracePeriod=25 Oct 30 17:28:59.900689 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:28:59.900653 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41cb25-db9f8d799-t27kf" podUID="dce37be9-f255-4c2d-8203-93b5058d86bf" containerName="payload-pre-processing" containerID="cri-o://d7a1be5fc1bea3d37a890fb9341706d8b5987c6b3fae634f18228407b7315d69" gracePeriod=25 Oct 30 17:29:00.161326 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:00.159294 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6caa6499-3ae1-4a95-9983-6c597d6e1436" path="/var/lib/kubelet/pods/6caa6499-3ae1-4a95-9983-6c597d6e1436/volumes" Oct 30 17:29:00.281997 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:00.281962 4522 generic.go:356] "Generic (PLEG): container finished" podID="dce37be9-f255-4c2d-8203-93b5058d86bf" containerID="d7a1be5fc1bea3d37a890fb9341706d8b5987c6b3fae634f18228407b7315d69" exitCode=0 Oct 30 17:29:00.282169 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:00.282113 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41cb25-db9f8d799-t27kf" event={"ID":"dce37be9-f255-4c2d-8203-93b5058d86bf","Type":"ContainerDied","Data":"d7a1be5fc1bea3d37a890fb9341706d8b5987c6b3fae634f18228407b7315d69"} Oct 30 17:29:00.282169 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:00.282162 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41cb25-db9f8d799-t27kf" event={"ID":"dce37be9-f255-4c2d-8203-93b5058d86bf","Type":"ContainerDied","Data":"89564b597971ebb500523d1b8bbd1189eefd1093f674e262bf29cff5c7b367bb"} Oct 30 17:29:00.282289 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:00.282187 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89564b597971ebb500523d1b8bbd1189eefd1093f674e262bf29cff5c7b367bb" Oct 30 17:29:00.285072 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:00.285044 4522 generic.go:356] "Generic (PLEG): container finished" podID="7e7ab767-5219-47a3-988f-fb4240a88652" containerID="bd4aade4914249fb22638a2d7ac8557f7d3a3d5a1c17187a63ff949abecc3f5a" exitCode=0 Oct 30 17:29:00.285560 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:00.285538 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-41cb25-56b855cbbb-x7mzc" event={"ID":"7e7ab767-5219-47a3-988f-fb4240a88652","Type":"ContainerDied","Data":"bd4aade4914249fb22638a2d7ac8557f7d3a3d5a1c17187a63ff949abecc3f5a"} Oct 30 17:29:00.285634 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:00.285586 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-41cb25-56b855cbbb-x7mzc" event={"ID":"7e7ab767-5219-47a3-988f-fb4240a88652","Type":"ContainerDied","Data":"4da1ec0d2f858d69433d2e5a62ac5efa374dbf6121462bbcc366429eb583650b"} Oct 30 17:29:00.285634 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:00.285607 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4da1ec0d2f858d69433d2e5a62ac5efa374dbf6121462bbcc366429eb583650b" Oct 30 17:29:00.296667 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:00.296445 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-41cb25-56b855cbbb-x7mzc" Oct 30 17:29:00.303242 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:00.303222 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41cb25-db9f8d799-t27kf" Oct 30 17:29:00.388065 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:00.387677 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e7ab767-5219-47a3-988f-fb4240a88652-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e7ab767-5219-47a3-988f-fb4240a88652-plugins-config-volume\") pod \"7e7ab767-5219-47a3-988f-fb4240a88652\" (UID: \"7e7ab767-5219-47a3-988f-fb4240a88652\") " Oct 30 17:29:00.388065 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:00.387749 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e7ab767-5219-47a3-988f-fb4240a88652-kube-api-access-ttj2k\" (UniqueName: \"kubernetes.io/projected/7e7ab767-5219-47a3-988f-fb4240a88652-kube-api-access-ttj2k\") pod \"7e7ab767-5219-47a3-988f-fb4240a88652\" (UID: \"7e7ab767-5219-47a3-988f-fb4240a88652\") " Oct 30 17:29:00.388065 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:00.387827 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dce37be9-f255-4c2d-8203-93b5058d86bf-kube-api-access-rnkdd\" (UniqueName: \"kubernetes.io/projected/dce37be9-f255-4c2d-8203-93b5058d86bf-kube-api-access-rnkdd\") pod \"dce37be9-f255-4c2d-8203-93b5058d86bf\" (UID: \"dce37be9-f255-4c2d-8203-93b5058d86bf\") " Oct 30 17:29:00.388065 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:00.387935 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dce37be9-f255-4c2d-8203-93b5058d86bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dce37be9-f255-4c2d-8203-93b5058d86bf-plugins-config-volume\") pod \"dce37be9-f255-4c2d-8203-93b5058d86bf\" (UID: \"dce37be9-f255-4c2d-8203-93b5058d86bf\") " Oct 30 17:29:00.389585 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:00.388864 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e7ab767-5219-47a3-988f-fb4240a88652-plugins-config-volume" pod "7e7ab767-5219-47a3-988f-fb4240a88652" (UID: "7e7ab767-5219-47a3-988f-fb4240a88652"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:29:00.389585 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:00.389381 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dce37be9-f255-4c2d-8203-93b5058d86bf-plugins-config-volume" pod "dce37be9-f255-4c2d-8203-93b5058d86bf" (UID: "dce37be9-f255-4c2d-8203-93b5058d86bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:29:00.392548 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:00.392520 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e7ab767-5219-47a3-988f-fb4240a88652-kube-api-access-ttj2k" pod "7e7ab767-5219-47a3-988f-fb4240a88652" (UID: "7e7ab767-5219-47a3-988f-fb4240a88652"). InnerVolumeSpecName "kube-api-access-ttj2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:29:00.398406 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:00.394344 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dce37be9-f255-4c2d-8203-93b5058d86bf-kube-api-access-rnkdd" pod "dce37be9-f255-4c2d-8203-93b5058d86bf" (UID: "dce37be9-f255-4c2d-8203-93b5058d86bf"). InnerVolumeSpecName "kube-api-access-rnkdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:29:00.490688 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:00.490658 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e7ab767-5219-47a3-988f-fb4240a88652-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:29:00.490688 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:00.490686 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttj2k\" (UniqueName: \"kubernetes.io/projected/7e7ab767-5219-47a3-988f-fb4240a88652-kube-api-access-ttj2k\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:29:00.490887 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:00.490698 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnkdd\" (UniqueName: \"kubernetes.io/projected/dce37be9-f255-4c2d-8203-93b5058d86bf-kube-api-access-rnkdd\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:29:00.490887 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:00.490708 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dce37be9-f255-4c2d-8203-93b5058d86bf-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:29:01.294687 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:01.294658 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-41cb25-56b855cbbb-x7mzc" Oct 30 17:29:01.295144 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:01.294659 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41cb25-db9f8d799-t27kf" Oct 30 17:29:01.370817 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:01.370786 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-41cb25-56b855cbbb-x7mzc"] Oct 30 17:29:01.384110 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:01.384079 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-41cb25-56b855cbbb-x7mzc"] Oct 30 17:29:01.413155 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:01.413114 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-41cb25-db9f8d799-t27kf"] Oct 30 17:29:01.418164 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:01.418141 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-41cb25-db9f8d799-t27kf"] Oct 30 17:29:02.147443 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:02.147408 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e7ab767-5219-47a3-988f-fb4240a88652" path="/var/lib/kubelet/pods/7e7ab767-5219-47a3-988f-fb4240a88652/volumes" Oct 30 17:29:02.147912 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:02.147898 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dce37be9-f255-4c2d-8203-93b5058d86bf" path="/var/lib/kubelet/pods/dce37be9-f255-4c2d-8203-93b5058d86bf/volumes" Oct 30 17:29:06.977998 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:06.977962 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7mdgd"] Oct 30 17:29:06.978522 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:06.978226 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7mdgd" podUID="86c0131b-dd1d-426b-ae9d-124a6e45e010" containerName="istio-proxy" containerID="cri-o://7266d81c21073083f051b3596b54023d612e854ca4a6055bee056fc9cfe6bed1" gracePeriod=30 Oct 30 17:29:09.270222 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:09.270183 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d2d985-688fcd588c-45k5x" Oct 30 17:29:09.588832 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:09.588747 4522 scope.go:122] "RemoveContainer" containerID="b2794999c1c383a7c35424f4744f6eed996cbb2c07e29ce961af4b5c2a72b10d" Oct 30 17:29:09.648363 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:09.648329 4522 scope.go:122] "RemoveContainer" containerID="eeb6730f9a6bb710f6c16e9cd118989127194b6b0be57a2883ef2ada20333088" Oct 30 17:29:09.660022 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:09.660000 4522 scope.go:122] "RemoveContainer" containerID="f243f76b3e4047e0e08014a2b2cb0d3d9536f0ea24036a6cfa61cd19858a972e" Oct 30 17:29:09.670771 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:09.670750 4522 scope.go:122] "RemoveContainer" containerID="e878ddfaa5b78346ad5a6f3177c3eb364d5b025d9c0edc5b5d80c13291443fc9" Oct 30 17:29:10.288234 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:10.288203 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d2d985-698fbdbbdb-z4wjk" Oct 30 17:29:11.347994 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:11.347926 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-75586d/auth-test-model-75586d-kserve-76fdc47ddf-7j94j"] Oct 30 17:29:11.349443 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:11.349416 4522 cpu_manager.go:414] "containerMap: removing container" podUID="7e7ab767-5219-47a3-988f-fb4240a88652" containerName="payload-processing" Oct 30 17:29:11.349961 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:11.349938 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e7ab767-5219-47a3-988f-fb4240a88652" containerName="payload-processing" Oct 30 17:29:11.350156 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:11.350144 4522 cpu_manager.go:414] "containerMap: removing container" podUID="dce37be9-f255-4c2d-8203-93b5058d86bf" containerName="payload-pre-processing" Oct 30 17:29:11.350252 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:11.350239 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dce37be9-f255-4c2d-8203-93b5058d86bf" containerName="payload-pre-processing" Oct 30 17:29:11.350764 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:11.350743 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e7ab767-5219-47a3-988f-fb4240a88652" containerName="payload-processing" Oct 30 17:29:11.350908 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:11.350897 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="dce37be9-f255-4c2d-8203-93b5058d86bf" containerName="payload-pre-processing" Oct 30 17:29:11.357745 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:11.357727 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-75586d/auth-test-model-75586d-kserve-76fdc47ddf-7j94j"] Oct 30 17:29:11.357904 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:11.357852 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-75586d/auth-test-model-75586d-kserve-76fdc47ddf-7j94j" Oct 30 17:29:11.387532 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:11.387507 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-75586d\"/\"default-dockercfg-d67br\"" Oct 30 17:29:11.387749 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:11.387729 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-75586d\"/\"kube-root-ca.crt\"" Oct 30 17:29:11.387847 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:11.387749 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-75586d\"/\"openshift-service-ca.crt\"" Oct 30 17:29:11.387847 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:11.387737 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-75586d\"/\"auth-test-model-75586d-kserve-self-signed-certs\"" Oct 30 17:29:11.426338 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:11.426022 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/920b5d6f-bdb8-47a6-b573-627e25beb8c5-home\") pod \"auth-test-model-75586d-kserve-76fdc47ddf-7j94j\" (UID: \"920b5d6f-bdb8-47a6-b573-627e25beb8c5\") " pod="ai-tenant-e2e-shared-b-w0-75586d/auth-test-model-75586d-kserve-76fdc47ddf-7j94j" Oct 30 17:29:11.426338 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:11.426183 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/920b5d6f-bdb8-47a6-b573-627e25beb8c5-tls-certs\") pod \"auth-test-model-75586d-kserve-76fdc47ddf-7j94j\" (UID: \"920b5d6f-bdb8-47a6-b573-627e25beb8c5\") " pod="ai-tenant-e2e-shared-b-w0-75586d/auth-test-model-75586d-kserve-76fdc47ddf-7j94j" Oct 30 17:29:11.426338 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:11.426283 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klb2q\" (UniqueName: \"kubernetes.io/projected/920b5d6f-bdb8-47a6-b573-627e25beb8c5-kube-api-access-klb2q\") pod \"auth-test-model-75586d-kserve-76fdc47ddf-7j94j\" (UID: \"920b5d6f-bdb8-47a6-b573-627e25beb8c5\") " pod="ai-tenant-e2e-shared-b-w0-75586d/auth-test-model-75586d-kserve-76fdc47ddf-7j94j" Oct 30 17:29:11.426665 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:11.426360 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/920b5d6f-bdb8-47a6-b573-627e25beb8c5-model-cache\") pod \"auth-test-model-75586d-kserve-76fdc47ddf-7j94j\" (UID: \"920b5d6f-bdb8-47a6-b573-627e25beb8c5\") " pod="ai-tenant-e2e-shared-b-w0-75586d/auth-test-model-75586d-kserve-76fdc47ddf-7j94j" Oct 30 17:29:11.426665 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:11.426488 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/920b5d6f-bdb8-47a6-b573-627e25beb8c5-tmp-dir\") pod \"auth-test-model-75586d-kserve-76fdc47ddf-7j94j\" (UID: \"920b5d6f-bdb8-47a6-b573-627e25beb8c5\") " pod="ai-tenant-e2e-shared-b-w0-75586d/auth-test-model-75586d-kserve-76fdc47ddf-7j94j" Oct 30 17:29:11.426665 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:11.426568 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/920b5d6f-bdb8-47a6-b573-627e25beb8c5-dshm\") pod \"auth-test-model-75586d-kserve-76fdc47ddf-7j94j\" (UID: \"920b5d6f-bdb8-47a6-b573-627e25beb8c5\") " pod="ai-tenant-e2e-shared-b-w0-75586d/auth-test-model-75586d-kserve-76fdc47ddf-7j94j" Oct 30 17:29:11.542466 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:11.539520 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/920b5d6f-bdb8-47a6-b573-627e25beb8c5-home\") pod \"auth-test-model-75586d-kserve-76fdc47ddf-7j94j\" (UID: \"920b5d6f-bdb8-47a6-b573-627e25beb8c5\") " pod="ai-tenant-e2e-shared-b-w0-75586d/auth-test-model-75586d-kserve-76fdc47ddf-7j94j" Oct 30 17:29:11.542864 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:11.542838 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/920b5d6f-bdb8-47a6-b573-627e25beb8c5-tls-certs\") pod \"auth-test-model-75586d-kserve-76fdc47ddf-7j94j\" (UID: \"920b5d6f-bdb8-47a6-b573-627e25beb8c5\") " pod="ai-tenant-e2e-shared-b-w0-75586d/auth-test-model-75586d-kserve-76fdc47ddf-7j94j" Oct 30 17:29:11.543050 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:11.543033 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klb2q\" (UniqueName: \"kubernetes.io/projected/920b5d6f-bdb8-47a6-b573-627e25beb8c5-kube-api-access-klb2q\") pod \"auth-test-model-75586d-kserve-76fdc47ddf-7j94j\" (UID: \"920b5d6f-bdb8-47a6-b573-627e25beb8c5\") " pod="ai-tenant-e2e-shared-b-w0-75586d/auth-test-model-75586d-kserve-76fdc47ddf-7j94j" Oct 30 17:29:11.543208 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:11.543191 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/920b5d6f-bdb8-47a6-b573-627e25beb8c5-model-cache\") pod \"auth-test-model-75586d-kserve-76fdc47ddf-7j94j\" (UID: \"920b5d6f-bdb8-47a6-b573-627e25beb8c5\") " pod="ai-tenant-e2e-shared-b-w0-75586d/auth-test-model-75586d-kserve-76fdc47ddf-7j94j" Oct 30 17:29:11.543385 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:11.543368 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/920b5d6f-bdb8-47a6-b573-627e25beb8c5-tmp-dir\") pod \"auth-test-model-75586d-kserve-76fdc47ddf-7j94j\" (UID: \"920b5d6f-bdb8-47a6-b573-627e25beb8c5\") " pod="ai-tenant-e2e-shared-b-w0-75586d/auth-test-model-75586d-kserve-76fdc47ddf-7j94j" Oct 30 17:29:11.543960 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:11.543036 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/920b5d6f-bdb8-47a6-b573-627e25beb8c5-home\") pod \"auth-test-model-75586d-kserve-76fdc47ddf-7j94j\" (UID: \"920b5d6f-bdb8-47a6-b573-627e25beb8c5\") " pod="ai-tenant-e2e-shared-b-w0-75586d/auth-test-model-75586d-kserve-76fdc47ddf-7j94j" Oct 30 17:29:11.543960 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:11.543743 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/920b5d6f-bdb8-47a6-b573-627e25beb8c5-tmp-dir\") pod \"auth-test-model-75586d-kserve-76fdc47ddf-7j94j\" (UID: \"920b5d6f-bdb8-47a6-b573-627e25beb8c5\") " pod="ai-tenant-e2e-shared-b-w0-75586d/auth-test-model-75586d-kserve-76fdc47ddf-7j94j" Oct 30 17:29:11.543960 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:11.543787 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/920b5d6f-bdb8-47a6-b573-627e25beb8c5-model-cache\") pod \"auth-test-model-75586d-kserve-76fdc47ddf-7j94j\" (UID: \"920b5d6f-bdb8-47a6-b573-627e25beb8c5\") " pod="ai-tenant-e2e-shared-b-w0-75586d/auth-test-model-75586d-kserve-76fdc47ddf-7j94j" Oct 30 17:29:11.544803 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:11.544785 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/920b5d6f-bdb8-47a6-b573-627e25beb8c5-dshm\") pod \"auth-test-model-75586d-kserve-76fdc47ddf-7j94j\" (UID: \"920b5d6f-bdb8-47a6-b573-627e25beb8c5\") " pod="ai-tenant-e2e-shared-b-w0-75586d/auth-test-model-75586d-kserve-76fdc47ddf-7j94j" Oct 30 17:29:11.547480 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:11.547458 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/920b5d6f-bdb8-47a6-b573-627e25beb8c5-tls-certs\") pod \"auth-test-model-75586d-kserve-76fdc47ddf-7j94j\" (UID: \"920b5d6f-bdb8-47a6-b573-627e25beb8c5\") " pod="ai-tenant-e2e-shared-b-w0-75586d/auth-test-model-75586d-kserve-76fdc47ddf-7j94j" Oct 30 17:29:11.548681 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:11.548637 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/920b5d6f-bdb8-47a6-b573-627e25beb8c5-dshm\") pod \"auth-test-model-75586d-kserve-76fdc47ddf-7j94j\" (UID: \"920b5d6f-bdb8-47a6-b573-627e25beb8c5\") " pod="ai-tenant-e2e-shared-b-w0-75586d/auth-test-model-75586d-kserve-76fdc47ddf-7j94j" Oct 30 17:29:11.569128 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:11.569095 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klb2q\" (UniqueName: \"kubernetes.io/projected/920b5d6f-bdb8-47a6-b573-627e25beb8c5-kube-api-access-klb2q\") pod \"auth-test-model-75586d-kserve-76fdc47ddf-7j94j\" (UID: \"920b5d6f-bdb8-47a6-b573-627e25beb8c5\") " pod="ai-tenant-e2e-shared-b-w0-75586d/auth-test-model-75586d-kserve-76fdc47ddf-7j94j" Oct 30 17:29:11.682171 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:11.677725 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-75586d/auth-test-model-75586d-kserve-76fdc47ddf-7j94j" Oct 30 17:29:12.147635 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:12.140032 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-75586d/auth-test-model-75586d-kserve-76fdc47ddf-7j94j"] Oct 30 17:29:12.152116 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:29:12.152078 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod920b5d6f_bdb8_47a6_b573_627e25beb8c5.slice/crio-be67b01f30353185439074b56e8fef6ad66bfd17e4bc238e620796b833f62813 WatchSource:0}: Error finding container be67b01f30353185439074b56e8fef6ad66bfd17e4bc238e620796b833f62813: Status 404 returned error can't find the container with id be67b01f30353185439074b56e8fef6ad66bfd17e4bc238e620796b833f62813 Oct 30 17:29:12.444347 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:12.441542 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7mdgd" Oct 30 17:29:12.485223 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:12.485189 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86c0131b-dd1d-426b-ae9d-124a6e45e010-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86c0131b-dd1d-426b-ae9d-124a6e45e010-istio-data\") pod \"86c0131b-dd1d-426b-ae9d-124a6e45e010\" (UID: \"86c0131b-dd1d-426b-ae9d-124a6e45e010\") " Oct 30 17:29:12.485412 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:12.485248 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86c0131b-dd1d-426b-ae9d-124a6e45e010-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86c0131b-dd1d-426b-ae9d-124a6e45e010-istiod-ca-cert\") pod \"86c0131b-dd1d-426b-ae9d-124a6e45e010\" (UID: \"86c0131b-dd1d-426b-ae9d-124a6e45e010\") " Oct 30 17:29:12.485412 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:12.485337 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86c0131b-dd1d-426b-ae9d-124a6e45e010-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86c0131b-dd1d-426b-ae9d-124a6e45e010-istio-envoy\") pod \"86c0131b-dd1d-426b-ae9d-124a6e45e010\" (UID: \"86c0131b-dd1d-426b-ae9d-124a6e45e010\") " Oct 30 17:29:12.485412 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:12.485383 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86c0131b-dd1d-426b-ae9d-124a6e45e010-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c0131b-dd1d-426b-ae9d-124a6e45e010-workload-socket\") pod \"86c0131b-dd1d-426b-ae9d-124a6e45e010\" (UID: \"86c0131b-dd1d-426b-ae9d-124a6e45e010\") " Oct 30 17:29:12.485550 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:12.485466 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86c0131b-dd1d-426b-ae9d-124a6e45e010-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86c0131b-dd1d-426b-ae9d-124a6e45e010-workload-certs\") pod \"86c0131b-dd1d-426b-ae9d-124a6e45e010\" (UID: \"86c0131b-dd1d-426b-ae9d-124a6e45e010\") " Oct 30 17:29:12.485550 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:12.485508 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86c0131b-dd1d-426b-ae9d-124a6e45e010-kube-api-access-hm7wz\" (UniqueName: \"kubernetes.io/projected/86c0131b-dd1d-426b-ae9d-124a6e45e010-kube-api-access-hm7wz\") pod \"86c0131b-dd1d-426b-ae9d-124a6e45e010\" (UID: \"86c0131b-dd1d-426b-ae9d-124a6e45e010\") " Oct 30 17:29:12.485631 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:12.485574 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86c0131b-dd1d-426b-ae9d-124a6e45e010-istio-token\" (UniqueName: \"kubernetes.io/projected/86c0131b-dd1d-426b-ae9d-124a6e45e010-istio-token\") pod \"86c0131b-dd1d-426b-ae9d-124a6e45e010\" (UID: \"86c0131b-dd1d-426b-ae9d-124a6e45e010\") " Oct 30 17:29:12.485670 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:12.485650 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86c0131b-dd1d-426b-ae9d-124a6e45e010-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86c0131b-dd1d-426b-ae9d-124a6e45e010-istio-podinfo\") pod \"86c0131b-dd1d-426b-ae9d-124a6e45e010\" (UID: \"86c0131b-dd1d-426b-ae9d-124a6e45e010\") " Oct 30 17:29:12.485734 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:12.485710 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86c0131b-dd1d-426b-ae9d-124a6e45e010-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c0131b-dd1d-426b-ae9d-124a6e45e010-credential-socket\") pod \"86c0131b-dd1d-426b-ae9d-124a6e45e010\" (UID: \"86c0131b-dd1d-426b-ae9d-124a6e45e010\") " Oct 30 17:29:12.486177 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:12.485979 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86c0131b-dd1d-426b-ae9d-124a6e45e010-workload-socket" pod "86c0131b-dd1d-426b-ae9d-124a6e45e010" (UID: "86c0131b-dd1d-426b-ae9d-124a6e45e010"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:29:12.486867 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:12.486536 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86c0131b-dd1d-426b-ae9d-124a6e45e010-workload-certs" pod "86c0131b-dd1d-426b-ae9d-124a6e45e010" (UID: "86c0131b-dd1d-426b-ae9d-124a6e45e010"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:29:12.486867 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:12.486614 4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c0131b-dd1d-426b-ae9d-124a6e45e010-workload-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:29:12.486867 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:12.486674 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86c0131b-dd1d-426b-ae9d-124a6e45e010-istiod-ca-cert" pod "86c0131b-dd1d-426b-ae9d-124a6e45e010" (UID: "86c0131b-dd1d-426b-ae9d-124a6e45e010"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:29:12.487124 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:12.486950 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86c0131b-dd1d-426b-ae9d-124a6e45e010-istio-data" pod "86c0131b-dd1d-426b-ae9d-124a6e45e010" (UID: "86c0131b-dd1d-426b-ae9d-124a6e45e010"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:29:12.487124 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:12.487030 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86c0131b-dd1d-426b-ae9d-124a6e45e010-credential-socket" pod "86c0131b-dd1d-426b-ae9d-124a6e45e010" (UID: "86c0131b-dd1d-426b-ae9d-124a6e45e010"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:29:12.489644 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:12.489558 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86c0131b-dd1d-426b-ae9d-124a6e45e010-istio-envoy" pod "86c0131b-dd1d-426b-ae9d-124a6e45e010" (UID: "86c0131b-dd1d-426b-ae9d-124a6e45e010"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:29:12.489644 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:12.489605 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86c0131b-dd1d-426b-ae9d-124a6e45e010-istio-podinfo" pod "86c0131b-dd1d-426b-ae9d-124a6e45e010" (UID: "86c0131b-dd1d-426b-ae9d-124a6e45e010"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:29:12.557383 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:12.498733 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-75586d/auth-test-model-75586d-kserve-76fdc47ddf-7j94j" event={"ID":"920b5d6f-bdb8-47a6-b573-627e25beb8c5","Type":"ContainerStarted","Data":"d8f528ea5fd3a4efa3eb95b8d7e8b302db91ec092ff4a0b410da52b0087ce662"} Oct 30 17:29:12.557383 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:12.498786 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-75586d/auth-test-model-75586d-kserve-76fdc47ddf-7j94j" event={"ID":"920b5d6f-bdb8-47a6-b573-627e25beb8c5","Type":"ContainerStarted","Data":"be67b01f30353185439074b56e8fef6ad66bfd17e4bc238e620796b833f62813"} Oct 30 17:29:12.557383 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:12.499625 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86c0131b-dd1d-426b-ae9d-124a6e45e010-kube-api-access-hm7wz" pod "86c0131b-dd1d-426b-ae9d-124a6e45e010" (UID: "86c0131b-dd1d-426b-ae9d-124a6e45e010"). InnerVolumeSpecName "kube-api-access-hm7wz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:29:12.557383 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:12.555984 4522 generic.go:356] "Generic (PLEG): container finished" podID="86c0131b-dd1d-426b-ae9d-124a6e45e010" containerID="7266d81c21073083f051b3596b54023d612e854ca4a6055bee056fc9cfe6bed1" exitCode=0 Oct 30 17:29:12.557383 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:12.556097 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7mdgd" event={"ID":"86c0131b-dd1d-426b-ae9d-124a6e45e010","Type":"ContainerDied","Data":"7266d81c21073083f051b3596b54023d612e854ca4a6055bee056fc9cfe6bed1"} Oct 30 17:29:12.557383 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:12.556147 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7mdgd" event={"ID":"86c0131b-dd1d-426b-ae9d-124a6e45e010","Type":"ContainerDied","Data":"2b3d6725d5c6303ea289868259a41300b5c4fd41e0d60776b9fc638883bacd7a"} Oct 30 17:29:12.557383 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:12.556179 4522 scope.go:122] "RemoveContainer" containerID="7266d81c21073083f051b3596b54023d612e854ca4a6055bee056fc9cfe6bed1" Oct 30 17:29:12.557383 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:12.556379 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7mdgd" Oct 30 17:29:12.563541 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:12.563430 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86c0131b-dd1d-426b-ae9d-124a6e45e010-istio-token" pod "86c0131b-dd1d-426b-ae9d-124a6e45e010" (UID: "86c0131b-dd1d-426b-ae9d-124a6e45e010"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:29:12.582532 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:12.582484 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-75586d/auth-test-model-75586d-kserve-76fdc47ddf-7j94j" podStartSLOduration=1.582470987 podStartE2EDuration="1.582470987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:29:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:29:12.579038592 +0000 UTC m=+2526.579793510" watchObservedRunningTime="2026-10-30 17:29:12.582470987 +0000 UTC m=+2526.583225887" Oct 30 17:29:12.589598 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:12.589567 4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86c0131b-dd1d-426b-ae9d-124a6e45e010-istio-data\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:29:12.589769 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:12.589757 4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86c0131b-dd1d-426b-ae9d-124a6e45e010-istiod-ca-cert\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:29:12.589872 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:12.589860 4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86c0131b-dd1d-426b-ae9d-124a6e45e010-istio-envoy\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:29:12.589969 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:12.589957 4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86c0131b-dd1d-426b-ae9d-124a6e45e010-workload-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:29:12.590071 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:12.590058 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hm7wz\" (UniqueName: \"kubernetes.io/projected/86c0131b-dd1d-426b-ae9d-124a6e45e010-kube-api-access-hm7wz\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:29:12.590152 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:12.590140 4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86c0131b-dd1d-426b-ae9d-124a6e45e010-istio-token\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:29:12.590230 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:12.590221 4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86c0131b-dd1d-426b-ae9d-124a6e45e010-istio-podinfo\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:29:12.590394 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:12.590381 4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c0131b-dd1d-426b-ae9d-124a6e45e010-credential-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:29:12.664732 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:12.664703 4522 scope.go:122] "RemoveContainer" containerID="7266d81c21073083f051b3596b54023d612e854ca4a6055bee056fc9cfe6bed1" Oct 30 17:29:12.665114 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:29:12.665086 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7266d81c21073083f051b3596b54023d612e854ca4a6055bee056fc9cfe6bed1\": container with ID starting with 7266d81c21073083f051b3596b54023d612e854ca4a6055bee056fc9cfe6bed1 not found: ID does not exist" containerID="7266d81c21073083f051b3596b54023d612e854ca4a6055bee056fc9cfe6bed1" Oct 30 17:29:12.665262 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:12.665238 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7266d81c21073083f051b3596b54023d612e854ca4a6055bee056fc9cfe6bed1"} err="failed to get container status \"7266d81c21073083f051b3596b54023d612e854ca4a6055bee056fc9cfe6bed1\": rpc error: code = NotFound desc = could not find container \"7266d81c21073083f051b3596b54023d612e854ca4a6055bee056fc9cfe6bed1\": container with ID starting with 7266d81c21073083f051b3596b54023d612e854ca4a6055bee056fc9cfe6bed1 not found: ID does not exist" Oct 30 17:29:12.679535 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:12.678801 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-75586d/auth-test-model-75586d-kserve-76fdc47ddf-7j94j" Oct 30 17:29:12.950423 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:12.950383 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7mdgd"] Oct 30 17:29:12.957500 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:12.957472 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7mdgd"] Oct 30 17:29:12.980105 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:29:12.980067 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:29:14.154692 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:14.154657 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86c0131b-dd1d-426b-ae9d-124a6e45e010" path="/var/lib/kubelet/pods/86c0131b-dd1d-426b-ae9d-124a6e45e010/volumes" Oct 30 17:29:14.287875 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:14.285862 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e13a51-75c9c5946-dcfnn"] Oct 30 17:29:14.287875 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:14.286761 4522 cpu_manager.go:414] "containerMap: removing container" podUID="86c0131b-dd1d-426b-ae9d-124a6e45e010" containerName="istio-proxy" Oct 30 17:29:14.287875 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:14.286778 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86c0131b-dd1d-426b-ae9d-124a6e45e010" containerName="istio-proxy" Oct 30 17:29:14.287875 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:14.286960 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="86c0131b-dd1d-426b-ae9d-124a6e45e010" containerName="istio-proxy" Oct 30 17:29:14.314331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:14.310593 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e13a51-75c9c5946-dcfnn" Oct 30 17:29:14.361586 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:14.361515 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w6-e13a51-serving-cert\"" Oct 30 17:29:14.364952 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:14.364924 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e13a51-75c9c5946-dcfnn"] Oct 30 17:29:14.376408 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:14.376378 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5451bfad-41b1-4c30-bdbb-6821a3ca6727-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-e13a51-75c9c5946-dcfnn\" (UID: \"5451bfad-41b1-4c30-bdbb-6821a3ca6727\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e13a51-75c9c5946-dcfnn" Oct 30 17:29:14.376538 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:14.376428 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5451bfad-41b1-4c30-bdbb-6821a3ca6727-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-e13a51-75c9c5946-dcfnn\" (UID: \"5451bfad-41b1-4c30-bdbb-6821a3ca6727\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e13a51-75c9c5946-dcfnn" Oct 30 17:29:14.376643 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:14.376626 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn6xg\" (UniqueName: \"kubernetes.io/projected/5451bfad-41b1-4c30-bdbb-6821a3ca6727-kube-api-access-mn6xg\") pod \"maas-api-e2e-shared-b-w6-e13a51-75c9c5946-dcfnn\" (UID: \"5451bfad-41b1-4c30-bdbb-6821a3ca6727\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e13a51-75c9c5946-dcfnn" Oct 30 17:29:14.376742 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:14.376727 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5451bfad-41b1-4c30-bdbb-6821a3ca6727-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-e13a51-75c9c5946-dcfnn\" (UID: \"5451bfad-41b1-4c30-bdbb-6821a3ca6727\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e13a51-75c9c5946-dcfnn" Oct 30 17:29:14.515946 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:14.515909 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn6xg\" (UniqueName: \"kubernetes.io/projected/5451bfad-41b1-4c30-bdbb-6821a3ca6727-kube-api-access-mn6xg\") pod \"maas-api-e2e-shared-b-w6-e13a51-75c9c5946-dcfnn\" (UID: \"5451bfad-41b1-4c30-bdbb-6821a3ca6727\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e13a51-75c9c5946-dcfnn" Oct 30 17:29:14.516149 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:14.516068 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5451bfad-41b1-4c30-bdbb-6821a3ca6727-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-e13a51-75c9c5946-dcfnn\" (UID: \"5451bfad-41b1-4c30-bdbb-6821a3ca6727\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e13a51-75c9c5946-dcfnn" Oct 30 17:29:14.516256 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:14.516220 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5451bfad-41b1-4c30-bdbb-6821a3ca6727-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-e13a51-75c9c5946-dcfnn\" (UID: \"5451bfad-41b1-4c30-bdbb-6821a3ca6727\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e13a51-75c9c5946-dcfnn" Oct 30 17:29:14.516340 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:14.516261 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5451bfad-41b1-4c30-bdbb-6821a3ca6727-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-e13a51-75c9c5946-dcfnn\" (UID: \"5451bfad-41b1-4c30-bdbb-6821a3ca6727\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e13a51-75c9c5946-dcfnn" Oct 30 17:29:14.516889 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:29:14.516569 4522 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e13a51-serving-cert: secret "maas-api-e2e-shared-b-w6-e13a51-serving-cert" not found Oct 30 17:29:14.516889 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:29:14.516640 4522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5451bfad-41b1-4c30-bdbb-6821a3ca6727-maas-api-tls podName:5451bfad-41b1-4c30-bdbb-6821a3ca6727 nodeName:}" failed. No retries permitted until 2026-10-30 17:29:15.016619069 +0000 UTC m=+2529.017373952 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5451bfad-41b1-4c30-bdbb-6821a3ca6727-maas-api-tls") pod "maas-api-e2e-shared-b-w6-e13a51-75c9c5946-dcfnn" (UID: "5451bfad-41b1-4c30-bdbb-6821a3ca6727") : secret "maas-api-e2e-shared-b-w6-e13a51-serving-cert" not found Oct 30 17:29:14.517651 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:14.517620 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5451bfad-41b1-4c30-bdbb-6821a3ca6727-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-e13a51-75c9c5946-dcfnn\" (UID: \"5451bfad-41b1-4c30-bdbb-6821a3ca6727\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e13a51-75c9c5946-dcfnn" Oct 30 17:29:14.574216 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:14.574180 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn6xg\" (UniqueName: \"kubernetes.io/projected/5451bfad-41b1-4c30-bdbb-6821a3ca6727-kube-api-access-mn6xg\") pod \"maas-api-e2e-shared-b-w6-e13a51-75c9c5946-dcfnn\" (UID: \"5451bfad-41b1-4c30-bdbb-6821a3ca6727\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e13a51-75c9c5946-dcfnn" Oct 30 17:29:14.576161 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:14.576105 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5451bfad-41b1-4c30-bdbb-6821a3ca6727-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-e13a51-75c9c5946-dcfnn\" (UID: \"5451bfad-41b1-4c30-bdbb-6821a3ca6727\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e13a51-75c9c5946-dcfnn" Oct 30 17:29:15.038112 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:15.038075 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5451bfad-41b1-4c30-bdbb-6821a3ca6727-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-e13a51-75c9c5946-dcfnn\" (UID: \"5451bfad-41b1-4c30-bdbb-6821a3ca6727\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e13a51-75c9c5946-dcfnn" Oct 30 17:29:15.040829 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:15.040803 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5451bfad-41b1-4c30-bdbb-6821a3ca6727-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-e13a51-75c9c5946-dcfnn\" (UID: \"5451bfad-41b1-4c30-bdbb-6821a3ca6727\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e13a51-75c9c5946-dcfnn" Oct 30 17:29:15.282884 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:15.282849 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e13a51-75c9c5946-dcfnn" Oct 30 17:29:15.878285 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:29:15.877781 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5451bfad_41b1_4c30_bdbb_6821a3ca6727.slice/crio-c571eff37b4c10208dd6e116a47dfd01350a232a013727a66fa0aab28e97a3b1 WatchSource:0}: Error finding container c571eff37b4c10208dd6e116a47dfd01350a232a013727a66fa0aab28e97a3b1: Status 404 returned error can't find the container with id c571eff37b4c10208dd6e116a47dfd01350a232a013727a66fa0aab28e97a3b1 Oct 30 17:29:15.881999 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:15.878751 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e13a51-75c9c5946-dcfnn"] Oct 30 17:29:16.456956 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:16.456905 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a7e4a-77547b685c-pm9ms"] Oct 30 17:29:16.463350 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:16.463321 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a7e4a-77547b685c-pm9ms" Oct 30 17:29:16.484372 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:16.482512 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-8a7e4a-serving-cert\"" Oct 30 17:29:16.486504 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:16.484947 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c633f97f-ac84-44f1-a029-75ead2b39058-metrics-certs\") pod \"maas-api-e2e-ipp-a-8a7e4a-77547b685c-pm9ms\" (UID: \"c633f97f-ac84-44f1-a029-75ead2b39058\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a7e4a-77547b685c-pm9ms" Oct 30 17:29:16.486504 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:16.485054 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf78z\" (UniqueName: \"kubernetes.io/projected/c633f97f-ac84-44f1-a029-75ead2b39058-kube-api-access-cf78z\") pod \"maas-api-e2e-ipp-a-8a7e4a-77547b685c-pm9ms\" (UID: \"c633f97f-ac84-44f1-a029-75ead2b39058\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a7e4a-77547b685c-pm9ms" Oct 30 17:29:16.486504 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:16.485181 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c633f97f-ac84-44f1-a029-75ead2b39058-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-8a7e4a-77547b685c-pm9ms\" (UID: \"c633f97f-ac84-44f1-a029-75ead2b39058\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a7e4a-77547b685c-pm9ms" Oct 30 17:29:16.486504 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:16.485292 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c633f97f-ac84-44f1-a029-75ead2b39058-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8a7e4a-77547b685c-pm9ms\" (UID: \"c633f97f-ac84-44f1-a029-75ead2b39058\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a7e4a-77547b685c-pm9ms" Oct 30 17:29:16.499375 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:16.499349 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a7e4a-77547b685c-pm9ms"] Oct 30 17:29:16.586500 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:16.586463 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf78z\" (UniqueName: \"kubernetes.io/projected/c633f97f-ac84-44f1-a029-75ead2b39058-kube-api-access-cf78z\") pod \"maas-api-e2e-ipp-a-8a7e4a-77547b685c-pm9ms\" (UID: \"c633f97f-ac84-44f1-a029-75ead2b39058\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a7e4a-77547b685c-pm9ms" Oct 30 17:29:16.586664 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:16.586604 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c633f97f-ac84-44f1-a029-75ead2b39058-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-8a7e4a-77547b685c-pm9ms\" (UID: \"c633f97f-ac84-44f1-a029-75ead2b39058\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a7e4a-77547b685c-pm9ms" Oct 30 17:29:16.586731 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:16.586714 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c633f97f-ac84-44f1-a029-75ead2b39058-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8a7e4a-77547b685c-pm9ms\" (UID: \"c633f97f-ac84-44f1-a029-75ead2b39058\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a7e4a-77547b685c-pm9ms" Oct 30 17:29:16.586846 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:16.586827 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c633f97f-ac84-44f1-a029-75ead2b39058-metrics-certs\") pod \"maas-api-e2e-ipp-a-8a7e4a-77547b685c-pm9ms\" (UID: \"c633f97f-ac84-44f1-a029-75ead2b39058\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a7e4a-77547b685c-pm9ms" Oct 30 17:29:16.587457 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:16.587432 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c633f97f-ac84-44f1-a029-75ead2b39058-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-8a7e4a-77547b685c-pm9ms\" (UID: \"c633f97f-ac84-44f1-a029-75ead2b39058\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a7e4a-77547b685c-pm9ms" Oct 30 17:29:16.589650 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:16.589624 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c633f97f-ac84-44f1-a029-75ead2b39058-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8a7e4a-77547b685c-pm9ms\" (UID: \"c633f97f-ac84-44f1-a029-75ead2b39058\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a7e4a-77547b685c-pm9ms" Oct 30 17:29:16.590014 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:16.589994 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c633f97f-ac84-44f1-a029-75ead2b39058-metrics-certs\") pod \"maas-api-e2e-ipp-a-8a7e4a-77547b685c-pm9ms\" (UID: \"c633f97f-ac84-44f1-a029-75ead2b39058\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a7e4a-77547b685c-pm9ms" Oct 30 17:29:16.617218 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:16.617176 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf78z\" (UniqueName: \"kubernetes.io/projected/c633f97f-ac84-44f1-a029-75ead2b39058-kube-api-access-cf78z\") pod \"maas-api-e2e-ipp-a-8a7e4a-77547b685c-pm9ms\" (UID: \"c633f97f-ac84-44f1-a029-75ead2b39058\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a7e4a-77547b685c-pm9ms" Oct 30 17:29:16.673700 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:16.666766 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e13a51-75c9c5946-dcfnn" event={"ID":"5451bfad-41b1-4c30-bdbb-6821a3ca6727","Type":"ContainerStarted","Data":"72b50ec3728559dc6e41c6af2cb0646490d1a929d0b6648ed71bb6bfca65db51"} Oct 30 17:29:16.673700 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:16.667041 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e13a51-75c9c5946-dcfnn" Oct 30 17:29:16.673700 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:16.667082 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e13a51-75c9c5946-dcfnn" event={"ID":"5451bfad-41b1-4c30-bdbb-6821a3ca6727","Type":"ContainerStarted","Data":"c571eff37b4c10208dd6e116a47dfd01350a232a013727a66fa0aab28e97a3b1"} Oct 30 17:29:16.730284 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:16.728919 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e13a51-75c9c5946-dcfnn" podStartSLOduration=2.371907567 podStartE2EDuration="2.728894908s" totalImagesPullingTime="356.987341ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:29:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:29:16.726063208 +0000 UTC m=+2530.726818111" watchObservedRunningTime="2026-10-30 17:29:16.728894908 +0000 UTC m=+2530.729649814" Oct 30 17:29:16.820364 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:16.820236 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a7e4a-77547b685c-pm9ms" Oct 30 17:29:17.197342 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:29:17.183920 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc633f97f_ac84_44f1_a029_75ead2b39058.slice/crio-cb914cf7752111d263cd5d47d0c9cbf9f53c38039edfadbb7680e582fc47a984 WatchSource:0}: Error finding container cb914cf7752111d263cd5d47d0c9cbf9f53c38039edfadbb7680e582fc47a984: Status 404 returned error can't find the container with id cb914cf7752111d263cd5d47d0c9cbf9f53c38039edfadbb7680e582fc47a984 Oct 30 17:29:17.197342 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:17.186196 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a7e4a-77547b685c-pm9ms"] Oct 30 17:29:17.681161 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:17.681122 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a7e4a-77547b685c-pm9ms" event={"ID":"c633f97f-ac84-44f1-a029-75ead2b39058","Type":"ContainerStarted","Data":"cb914cf7752111d263cd5d47d0c9cbf9f53c38039edfadbb7680e582fc47a984"} Oct 30 17:29:18.697431 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:18.697299 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a7e4a-77547b685c-pm9ms" event={"ID":"c633f97f-ac84-44f1-a029-75ead2b39058","Type":"ContainerStarted","Data":"7ec8a8f4fc63cebc2ceee27c90565f761a7550d0d1f63f0a8e8dfcfac278044a"} Oct 30 17:29:18.697906 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:18.697588 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a7e4a-77547b685c-pm9ms" Oct 30 17:29:18.746977 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:18.746920 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a7e4a-77547b685c-pm9ms" podStartSLOduration=1.68912499 podStartE2EDuration="2.746908289s" totalImagesPullingTime="1.057783299s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:29:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:29:18.746185177 +0000 UTC m=+2532.746940089" watchObservedRunningTime="2026-10-30 17:29:18.746908289 +0000 UTC m=+2532.747663189" Oct 30 17:29:19.655907 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:19.655675 4522 patch_prober.go:29] interesting pod/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.30:15021/healthz/ready\": read tcp 10.134.0.2:38468->10.134.1.30:15021: read: connection reset by peer" start-of-body= Oct 30 17:29:19.655907 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:19.655751 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd" podUID="1e03b6f6-d0a5-45ac-af7f-3f1c50faf173" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.30:15021/healthz/ready\": read tcp 10.134.0.2:38468->10.134.1.30:15021: read: connection reset by peer" Oct 30 17:29:19.825073 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:19.708515 4522 generic.go:356] "Generic (PLEG): container finished" podID="1e03b6f6-d0a5-45ac-af7f-3f1c50faf173" containerID="2c5da7495b1854a538ccac343f5ce39d06254be0b95fa5557e4f0a1bdb5c6ec5" exitCode=0 Oct 30 17:29:19.825073 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:19.708590 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd" event={"ID":"1e03b6f6-d0a5-45ac-af7f-3f1c50faf173","Type":"ContainerDied","Data":"2c5da7495b1854a538ccac343f5ce39d06254be0b95fa5557e4f0a1bdb5c6ec5"} Oct 30 17:29:19.825073 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:19.709084 4522 scope.go:122] "RemoveContainer" containerID="2c5da7495b1854a538ccac343f5ce39d06254be0b95fa5557e4f0a1bdb5c6ec5" Oct 30 17:29:19.825073 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:19.710898 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:29:19.825073 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:19.710961 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:29:19.825073 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:19.710994 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:29:20.728639 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:20.728599 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd" event={"ID":"1e03b6f6-d0a5-45ac-af7f-3f1c50faf173","Type":"ContainerStarted","Data":"ada327d8a66a878d51ad3cfb2ee3c115b90383176a43c6db6f271bb03f62220d"} Oct 30 17:29:20.730372 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:20.730346 4522 generic.go:356] "Generic (PLEG): container finished" podID="e904bd74-db7c-4430-863a-5277580a6342" containerID="9d6632e288587cddb15621dad4ff6694cd4720783e0a7dfa7b7d37923bdce115" exitCode=0 Oct 30 17:29:20.730533 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:20.730418 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9" event={"ID":"e904bd74-db7c-4430-863a-5277580a6342","Type":"ContainerDied","Data":"9d6632e288587cddb15621dad4ff6694cd4720783e0a7dfa7b7d37923bdce115"} Oct 30 17:29:20.730898 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:20.730880 4522 scope.go:122] "RemoveContainer" containerID="9d6632e288587cddb15621dad4ff6694cd4720783e0a7dfa7b7d37923bdce115" Oct 30 17:29:20.732801 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:20.732774 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:29:20.732873 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:20.732843 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:29:20.732912 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:20.732877 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:29:21.121388 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:21.120740 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd" Oct 30 17:29:21.121388 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:21.120874 4522 patch_prober.go:29] interesting pod/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.30:15021/healthz/ready\": dial tcp 10.134.1.30:15021: connect: connection refused" start-of-body= Oct 30 17:29:21.121388 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:21.120920 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd" podUID="1e03b6f6-d0a5-45ac-af7f-3f1c50faf173" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.30:15021/healthz/ready\": dial tcp 10.134.1.30:15021: connect: connection refused" Oct 30 17:29:21.538782 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:21.538751 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-93e020-6c9d97748-pnfp5"] Oct 30 17:29:21.552287 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:21.551292 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-93e020-6c9d97748-pnfp5"] Oct 30 17:29:21.552287 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:21.551473 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-93e020-6c9d97748-pnfp5" Oct 30 17:29:21.568718 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:21.568688 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-93e020-dockercfg-p99rl\"" Oct 30 17:29:21.568905 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:21.568779 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-93e020\"" Oct 30 17:29:21.667669 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:21.667402 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af795373-0e85-4f37-80ab-2caa6e2dd689-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-93e020-6c9d97748-pnfp5\" (UID: \"af795373-0e85-4f37-80ab-2caa6e2dd689\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-93e020-6c9d97748-pnfp5" Oct 30 17:29:21.670595 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:21.668562 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5l2k\" (UniqueName: \"kubernetes.io/projected/af795373-0e85-4f37-80ab-2caa6e2dd689-kube-api-access-n5l2k\") pod \"payload-processing-e2e-ipp-b-93e020-6c9d97748-pnfp5\" (UID: \"af795373-0e85-4f37-80ab-2caa6e2dd689\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-93e020-6c9d97748-pnfp5" Oct 30 17:29:21.686499 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:21.686467 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-75586d/auth-test-model-75586d-kserve-76fdc47ddf-7j94j" Oct 30 17:29:21.744331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:21.743221 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-75586d/auth-test-model-75586d-kserve-76fdc47ddf-7j94j" Oct 30 17:29:21.752329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:21.750827 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9" event={"ID":"e904bd74-db7c-4430-863a-5277580a6342","Type":"ContainerStarted","Data":"67401d16a05f49d8810f2279c1776d3596a4fd4967912d5a1abec21b8bac73c8"} Oct 30 17:29:21.766112 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:21.766074 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-75586d/auth-test-model-75586d-kserve-76fdc47ddf-7j94j" Oct 30 17:29:21.770173 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:21.770142 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5l2k\" (UniqueName: \"kubernetes.io/projected/af795373-0e85-4f37-80ab-2caa6e2dd689-kube-api-access-n5l2k\") pod \"payload-processing-e2e-ipp-b-93e020-6c9d97748-pnfp5\" (UID: \"af795373-0e85-4f37-80ab-2caa6e2dd689\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-93e020-6c9d97748-pnfp5" Oct 30 17:29:21.770350 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:21.770286 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af795373-0e85-4f37-80ab-2caa6e2dd689-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-93e020-6c9d97748-pnfp5\" (UID: \"af795373-0e85-4f37-80ab-2caa6e2dd689\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-93e020-6c9d97748-pnfp5" Oct 30 17:29:21.771147 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:21.771120 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af795373-0e85-4f37-80ab-2caa6e2dd689-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-93e020-6c9d97748-pnfp5\" (UID: \"af795373-0e85-4f37-80ab-2caa6e2dd689\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-93e020-6c9d97748-pnfp5" Oct 30 17:29:21.822393 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:21.818358 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5l2k\" (UniqueName: \"kubernetes.io/projected/af795373-0e85-4f37-80ab-2caa6e2dd689-kube-api-access-n5l2k\") pod \"payload-processing-e2e-ipp-b-93e020-6c9d97748-pnfp5\" (UID: \"af795373-0e85-4f37-80ab-2caa6e2dd689\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-93e020-6c9d97748-pnfp5" Oct 30 17:29:21.874224 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:21.874189 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-93e020-6c9d97748-pnfp5" Oct 30 17:29:22.146398 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:22.119780 4522 patch_prober.go:29] interesting pod/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.30:15021/healthz/ready\": dial tcp 10.134.1.30:15021: connect: connection refused" start-of-body= Oct 30 17:29:22.146398 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:22.119841 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd" podUID="1e03b6f6-d0a5-45ac-af7f-3f1c50faf173" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.30:15021/healthz/ready\": dial tcp 10.134.1.30:15021: connect: connection refused" Oct 30 17:29:22.347525 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:22.345504 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9" Oct 30 17:29:22.349086 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:22.348574 4522 patch_prober.go:29] interesting pod/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused" start-of-body= Oct 30 17:29:22.349086 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:22.348622 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9" podUID="e904bd74-db7c-4430-863a-5277580a6342" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused" Oct 30 17:29:22.352406 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:22.352296 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-93e020-6c9d97748-pnfp5"] Oct 30 17:29:22.686974 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:22.686939 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e13a51-75c9c5946-dcfnn" Oct 30 17:29:22.765376 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:22.762660 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-93e020-6c9d97748-pnfp5" event={"ID":"af795373-0e85-4f37-80ab-2caa6e2dd689","Type":"ContainerStarted","Data":"2053bff1310a76a1aafcae8f1ccbcb25bc9843ee6d47361eca6bcd6e2393929e"} Oct 30 17:29:22.765376 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:22.762765 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-93e020-6c9d97748-pnfp5" event={"ID":"af795373-0e85-4f37-80ab-2caa6e2dd689","Type":"ContainerStarted","Data":"1bd6ee6ccf3869f2481d598274672e20399b371751d95448d0d340d671a63e05"} Oct 30 17:29:22.765376 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:22.762796 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-93e020-6c9d97748-pnfp5" Oct 30 17:29:22.847546 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:22.847499 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-93e020-6c9d97748-pnfp5" podStartSLOduration=1.8474851129999998 podStartE2EDuration="1.847485113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:29:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:29:22.847060046 +0000 UTC m=+2536.847814949" watchObservedRunningTime="2026-10-30 17:29:22.847485113 +0000 UTC m=+2536.848240013" Oct 30 17:29:23.123389 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:23.116722 4522 patch_prober.go:29] interesting pod/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.30:15021/healthz/ready\": dial tcp 10.134.1.30:15021: connect: connection refused" start-of-body= Oct 30 17:29:23.123389 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:23.116786 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd" podUID="1e03b6f6-d0a5-45ac-af7f-3f1c50faf173" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.30:15021/healthz/ready\": dial tcp 10.134.1.30:15021: connect: connection refused" Oct 30 17:29:23.155270 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:23.155232 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9" Oct 30 17:29:23.343329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:23.342618 4522 patch_prober.go:29] interesting pod/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused" start-of-body= Oct 30 17:29:23.343329 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:23.342665 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9" podUID="e904bd74-db7c-4430-863a-5277580a6342" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused" Oct 30 17:29:24.118338 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:24.117771 4522 patch_prober.go:29] interesting pod/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.30:15021/healthz/ready\": dial tcp 10.134.1.30:15021: connect: connection refused" start-of-body= Oct 30 17:29:24.118338 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:24.117853 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd" podUID="1e03b6f6-d0a5-45ac-af7f-3f1c50faf173" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.30:15021/healthz/ready\": dial tcp 10.134.1.30:15021: connect: connection refused" Oct 30 17:29:24.352334 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:24.351673 4522 patch_prober.go:29] interesting pod/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused" start-of-body= Oct 30 17:29:24.352334 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:24.351737 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9" podUID="e904bd74-db7c-4430-863a-5277580a6342" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused" Oct 30 17:29:24.742337 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:24.742201 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a7e4a-77547b685c-pm9ms" Oct 30 17:29:25.118388 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:25.117734 4522 patch_prober.go:29] interesting pod/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.30:15021/healthz/ready\": dial tcp 10.134.1.30:15021: connect: connection refused" start-of-body= Oct 30 17:29:25.118388 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:25.117801 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd" podUID="1e03b6f6-d0a5-45ac-af7f-3f1c50faf173" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.30:15021/healthz/ready\": dial tcp 10.134.1.30:15021: connect: connection refused" Oct 30 17:29:25.343332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:25.343250 4522 patch_prober.go:29] interesting pod/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused" start-of-body= Oct 30 17:29:25.343518 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:25.343347 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9" podUID="e904bd74-db7c-4430-863a-5277580a6342" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused" Oct 30 17:29:26.289370 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:26.280106 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd" Oct 30 17:29:26.289370 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:26.281255 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd" Oct 30 17:29:26.289370 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:26.283346 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd" Oct 30 17:29:26.343371 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:26.343289 4522 patch_prober.go:29] interesting pod/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused" start-of-body= Oct 30 17:29:26.343576 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:26.343387 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9" podUID="e904bd74-db7c-4430-863a-5277580a6342" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused" Oct 30 17:29:27.297553 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:27.297519 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-63e60d-6c68cfcd78-7wlrg"] Oct 30 17:29:27.322507 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:27.322476 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-63e60d-5f6f7b8cc5-7wsxk"] Oct 30 17:29:27.342729 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:27.342657 4522 patch_prober.go:29] interesting pod/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused" start-of-body= Oct 30 17:29:27.342918 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:27.342761 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9" podUID="e904bd74-db7c-4430-863a-5277580a6342" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused" Oct 30 17:29:28.364951 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:28.364910 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9" Oct 30 17:29:28.366188 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:28.366165 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9" Oct 30 17:29:32.036739 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:32.036705 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9af8e32c-642v5"] Oct 30 17:29:32.050302 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:32.050267 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9af8e32c-642v5"] Oct 30 17:29:32.148317 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:32.148275 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d25a3aba-e336-4d85-9b43-edb6aa07b38d" path="/var/lib/kubelet/pods/d25a3aba-e336-4d85-9b43-edb6aa07b38d/volumes" Oct 30 17:29:32.298021 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:32.297936 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-63e60d-6c68cfcd78-7wlrg" podUID="c419fcd5-9b32-4075-ba32-1614e365b132" containerName="payload-processing" containerID="cri-o://3c4fa462a6505e77311a74c27e808999eb9dba02f003e357583a9fd1d4252055" gracePeriod=25 Oct 30 17:29:32.323113 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:32.323081 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e60d-5f6f7b8cc5-7wsxk" podUID="5c473c8d-2321-447a-8059-eb3c1e6e105e" containerName="payload-pre-processing" containerID="cri-o://f3aa3bc01158ecdb4328da87ef784120096ff65829186f0a85fc4efe3f8d8c2f" gracePeriod=25 Oct 30 17:29:32.653801 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:32.653775 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-63e60d-6c68cfcd78-7wlrg" Oct 30 17:29:32.665272 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:32.665244 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e60d-5f6f7b8cc5-7wsxk" Oct 30 17:29:32.799006 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:32.798959 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c473c8d-2321-447a-8059-eb3c1e6e105e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c473c8d-2321-447a-8059-eb3c1e6e105e-plugins-config-volume\") pod \"5c473c8d-2321-447a-8059-eb3c1e6e105e\" (UID: \"5c473c8d-2321-447a-8059-eb3c1e6e105e\") " Oct 30 17:29:32.799188 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:32.799136 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c473c8d-2321-447a-8059-eb3c1e6e105e-kube-api-access-bj245\" (UniqueName: \"kubernetes.io/projected/5c473c8d-2321-447a-8059-eb3c1e6e105e-kube-api-access-bj245\") pod \"5c473c8d-2321-447a-8059-eb3c1e6e105e\" (UID: \"5c473c8d-2321-447a-8059-eb3c1e6e105e\") " Oct 30 17:29:32.799372 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:32.799348 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c419fcd5-9b32-4075-ba32-1614e365b132-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c419fcd5-9b32-4075-ba32-1614e365b132-plugins-config-volume\") pod \"c419fcd5-9b32-4075-ba32-1614e365b132\" (UID: \"c419fcd5-9b32-4075-ba32-1614e365b132\") " Oct 30 17:29:32.799550 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:32.799523 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c473c8d-2321-447a-8059-eb3c1e6e105e-plugins-config-volume" pod "5c473c8d-2321-447a-8059-eb3c1e6e105e" (UID: "5c473c8d-2321-447a-8059-eb3c1e6e105e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:29:32.799640 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:32.799564 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c419fcd5-9b32-4075-ba32-1614e365b132-kube-api-access-m2cfs\" (UniqueName: \"kubernetes.io/projected/c419fcd5-9b32-4075-ba32-1614e365b132-kube-api-access-m2cfs\") pod \"c419fcd5-9b32-4075-ba32-1614e365b132\" (UID: \"c419fcd5-9b32-4075-ba32-1614e365b132\") " Oct 30 17:29:32.800046 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:32.799924 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c419fcd5-9b32-4075-ba32-1614e365b132-plugins-config-volume" pod "c419fcd5-9b32-4075-ba32-1614e365b132" (UID: "c419fcd5-9b32-4075-ba32-1614e365b132"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:29:32.800420 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:32.800397 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c419fcd5-9b32-4075-ba32-1614e365b132-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:29:32.800635 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:32.800423 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c473c8d-2321-447a-8059-eb3c1e6e105e-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:29:32.802351 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:32.802297 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c419fcd5-9b32-4075-ba32-1614e365b132-kube-api-access-m2cfs" pod "c419fcd5-9b32-4075-ba32-1614e365b132" (UID: "c419fcd5-9b32-4075-ba32-1614e365b132"). InnerVolumeSpecName "kube-api-access-m2cfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:29:32.802451 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:32.802381 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c473c8d-2321-447a-8059-eb3c1e6e105e-kube-api-access-bj245" pod "5c473c8d-2321-447a-8059-eb3c1e6e105e" (UID: "5c473c8d-2321-447a-8059-eb3c1e6e105e"). InnerVolumeSpecName "kube-api-access-bj245". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:29:32.902179 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:32.902096 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2cfs\" (UniqueName: \"kubernetes.io/projected/c419fcd5-9b32-4075-ba32-1614e365b132-kube-api-access-m2cfs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:29:32.902179 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:32.902129 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bj245\" (UniqueName: \"kubernetes.io/projected/5c473c8d-2321-447a-8059-eb3c1e6e105e-kube-api-access-bj245\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:29:32.975657 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:32.975617 4522 generic.go:356] "Generic (PLEG): container finished" podID="5c473c8d-2321-447a-8059-eb3c1e6e105e" containerID="f3aa3bc01158ecdb4328da87ef784120096ff65829186f0a85fc4efe3f8d8c2f" exitCode=0 Oct 30 17:29:32.975810 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:32.975665 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e60d-5f6f7b8cc5-7wsxk" Oct 30 17:29:32.975810 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:32.975695 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e60d-5f6f7b8cc5-7wsxk" event={"ID":"5c473c8d-2321-447a-8059-eb3c1e6e105e","Type":"ContainerDied","Data":"f3aa3bc01158ecdb4328da87ef784120096ff65829186f0a85fc4efe3f8d8c2f"} Oct 30 17:29:32.975810 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:32.975773 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e60d-5f6f7b8cc5-7wsxk" event={"ID":"5c473c8d-2321-447a-8059-eb3c1e6e105e","Type":"ContainerDied","Data":"9d938e021da747859fe01b128fe68cab5e8975c27f925c4c2127690fb745bc94"} Oct 30 17:29:32.975810 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:32.975796 4522 scope.go:122] "RemoveContainer" containerID="f3aa3bc01158ecdb4328da87ef784120096ff65829186f0a85fc4efe3f8d8c2f" Oct 30 17:29:32.977411 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:32.977389 4522 generic.go:356] "Generic (PLEG): container finished" podID="c419fcd5-9b32-4075-ba32-1614e365b132" containerID="3c4fa462a6505e77311a74c27e808999eb9dba02f003e357583a9fd1d4252055" exitCode=0 Oct 30 17:29:32.977522 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:32.977442 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-63e60d-6c68cfcd78-7wlrg" Oct 30 17:29:32.977522 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:32.977493 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-63e60d-6c68cfcd78-7wlrg" event={"ID":"c419fcd5-9b32-4075-ba32-1614e365b132","Type":"ContainerDied","Data":"3c4fa462a6505e77311a74c27e808999eb9dba02f003e357583a9fd1d4252055"} Oct 30 17:29:32.977620 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:32.977550 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-63e60d-6c68cfcd78-7wlrg" event={"ID":"c419fcd5-9b32-4075-ba32-1614e365b132","Type":"ContainerDied","Data":"6f267f43de6b33476207aaa1fa9f5449c656cb4ff5139f6ce12a39818cc239eb"} Oct 30 17:29:32.988945 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:32.988925 4522 scope.go:122] "RemoveContainer" containerID="f3aa3bc01158ecdb4328da87ef784120096ff65829186f0a85fc4efe3f8d8c2f" Oct 30 17:29:32.989289 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:29:32.989270 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3aa3bc01158ecdb4328da87ef784120096ff65829186f0a85fc4efe3f8d8c2f\": container with ID starting with f3aa3bc01158ecdb4328da87ef784120096ff65829186f0a85fc4efe3f8d8c2f not found: ID does not exist" containerID="f3aa3bc01158ecdb4328da87ef784120096ff65829186f0a85fc4efe3f8d8c2f" Oct 30 17:29:32.989639 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:32.989299 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3aa3bc01158ecdb4328da87ef784120096ff65829186f0a85fc4efe3f8d8c2f"} err="failed to get container status \"f3aa3bc01158ecdb4328da87ef784120096ff65829186f0a85fc4efe3f8d8c2f\": rpc error: code = NotFound desc = could not find container \"f3aa3bc01158ecdb4328da87ef784120096ff65829186f0a85fc4efe3f8d8c2f\": container with ID starting with f3aa3bc01158ecdb4328da87ef784120096ff65829186f0a85fc4efe3f8d8c2f not found: ID does not exist" Oct 30 17:29:32.989639 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:32.989352 4522 scope.go:122] "RemoveContainer" containerID="3c4fa462a6505e77311a74c27e808999eb9dba02f003e357583a9fd1d4252055" Oct 30 17:29:33.001003 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:33.000984 4522 scope.go:122] "RemoveContainer" containerID="3c4fa462a6505e77311a74c27e808999eb9dba02f003e357583a9fd1d4252055" Oct 30 17:29:33.001269 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:29:33.001252 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c4fa462a6505e77311a74c27e808999eb9dba02f003e357583a9fd1d4252055\": container with ID starting with 3c4fa462a6505e77311a74c27e808999eb9dba02f003e357583a9fd1d4252055 not found: ID does not exist" containerID="3c4fa462a6505e77311a74c27e808999eb9dba02f003e357583a9fd1d4252055" Oct 30 17:29:33.001337 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:33.001278 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c4fa462a6505e77311a74c27e808999eb9dba02f003e357583a9fd1d4252055"} err="failed to get container status \"3c4fa462a6505e77311a74c27e808999eb9dba02f003e357583a9fd1d4252055\": rpc error: code = NotFound desc = could not find container \"3c4fa462a6505e77311a74c27e808999eb9dba02f003e357583a9fd1d4252055\": container with ID starting with 3c4fa462a6505e77311a74c27e808999eb9dba02f003e357583a9fd1d4252055 not found: ID does not exist" Oct 30 17:29:33.036860 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:33.036829 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fba7356e-m8gjf"] Oct 30 17:29:33.041924 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:33.041898 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-63e60d-5f6f7b8cc5-7wsxk"] Oct 30 17:29:33.050029 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:33.050001 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-63e60d-5f6f7b8cc5-7wsxk"] Oct 30 17:29:33.054706 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:33.054683 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fba7356e-m8gjf"] Oct 30 17:29:33.091522 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:33.091491 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-63e60d-6c68cfcd78-7wlrg"] Oct 30 17:29:33.096170 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:33.096148 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-63e60d-6c68cfcd78-7wlrg"] Oct 30 17:29:33.875659 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:33.875634 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-93e020-6c9d97748-pnfp5" Oct 30 17:29:34.148262 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:34.148180 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22c82bab-c6f4-48ba-8bb9-06b49eed06be" path="/var/lib/kubelet/pods/22c82bab-c6f4-48ba-8bb9-06b49eed06be/volumes" Oct 30 17:29:34.148772 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:34.148756 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c473c8d-2321-447a-8059-eb3c1e6e105e" path="/var/lib/kubelet/pods/5c473c8d-2321-447a-8059-eb3c1e6e105e/volumes" Oct 30 17:29:34.149182 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:34.149169 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c419fcd5-9b32-4075-ba32-1614e365b132" path="/var/lib/kubelet/pods/c419fcd5-9b32-4075-ba32-1614e365b132/volumes" Oct 30 17:29:38.424077 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:38.424043 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c17e8c-54fd5c99d-m7v82"] Oct 30 17:29:38.459002 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:38.458971 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c17e8c-7cd6d6db99-wffp5"] Oct 30 17:29:42.950973 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:29:42.950939 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:29:43.036433 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:43.036398 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de890356-fdmx7"] Oct 30 17:29:43.049840 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:43.049806 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de890356-fdmx7"] Oct 30 17:29:43.427387 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:43.424418 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-c17e8c-54fd5c99d-m7v82" podUID="1ae18756-7182-45d8-af84-4b2b41920db0" containerName="payload-processing" containerID="cri-o://dd7c62cc30e69bf75354572b53e016cd30943d6da12cdd2e39cb05f6a58171ce" gracePeriod=25 Oct 30 17:29:43.459859 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:43.459820 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c17e8c-7cd6d6db99-wffp5" podUID="645a0822-b642-405c-8b44-1bde28fa322e" containerName="payload-pre-processing" containerID="cri-o://a7ebf865e21dd6f31a4e0444a3d1483180d4a2ddfa938a8b20def40db7bab146" gracePeriod=25 Oct 30 17:29:43.823080 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:43.822959 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c17e8c-54fd5c99d-m7v82" Oct 30 17:29:43.856785 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:43.856760 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c17e8c-7cd6d6db99-wffp5" Oct 30 17:29:43.946723 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:43.946688 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ae18756-7182-45d8-af84-4b2b41920db0-kube-api-access-ttpdg\" (UniqueName: \"kubernetes.io/projected/1ae18756-7182-45d8-af84-4b2b41920db0-kube-api-access-ttpdg\") pod \"1ae18756-7182-45d8-af84-4b2b41920db0\" (UID: \"1ae18756-7182-45d8-af84-4b2b41920db0\") " Oct 30 17:29:43.946911 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:43.946828 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ae18756-7182-45d8-af84-4b2b41920db0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae18756-7182-45d8-af84-4b2b41920db0-plugins-config-volume\") pod \"1ae18756-7182-45d8-af84-4b2b41920db0\" (UID: \"1ae18756-7182-45d8-af84-4b2b41920db0\") " Oct 30 17:29:43.946987 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:43.946970 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/645a0822-b642-405c-8b44-1bde28fa322e-kube-api-access-l7gb6\" (UniqueName: \"kubernetes.io/projected/645a0822-b642-405c-8b44-1bde28fa322e-kube-api-access-l7gb6\") pod \"645a0822-b642-405c-8b44-1bde28fa322e\" (UID: \"645a0822-b642-405c-8b44-1bde28fa322e\") " Oct 30 17:29:43.947239 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:43.947213 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ae18756-7182-45d8-af84-4b2b41920db0-plugins-config-volume" pod "1ae18756-7182-45d8-af84-4b2b41920db0" (UID: "1ae18756-7182-45d8-af84-4b2b41920db0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:29:43.947760 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:43.947732 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/645a0822-b642-405c-8b44-1bde28fa322e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/645a0822-b642-405c-8b44-1bde28fa322e-plugins-config-volume\") pod \"645a0822-b642-405c-8b44-1bde28fa322e\" (UID: \"645a0822-b642-405c-8b44-1bde28fa322e\") " Oct 30 17:29:43.948164 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:43.948136 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/645a0822-b642-405c-8b44-1bde28fa322e-plugins-config-volume" pod "645a0822-b642-405c-8b44-1bde28fa322e" (UID: "645a0822-b642-405c-8b44-1bde28fa322e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:29:43.949040 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:43.949012 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/645a0822-b642-405c-8b44-1bde28fa322e-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:29:43.949130 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:43.949042 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae18756-7182-45d8-af84-4b2b41920db0-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:29:43.949185 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:43.949146 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ae18756-7182-45d8-af84-4b2b41920db0-kube-api-access-ttpdg" pod "1ae18756-7182-45d8-af84-4b2b41920db0" (UID: "1ae18756-7182-45d8-af84-4b2b41920db0"). InnerVolumeSpecName "kube-api-access-ttpdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:29:43.949294 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:43.949274 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/645a0822-b642-405c-8b44-1bde28fa322e-kube-api-access-l7gb6" pod "645a0822-b642-405c-8b44-1bde28fa322e" (UID: "645a0822-b642-405c-8b44-1bde28fa322e"). InnerVolumeSpecName "kube-api-access-l7gb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:29:44.050609 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:44.050532 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7gb6\" (UniqueName: \"kubernetes.io/projected/645a0822-b642-405c-8b44-1bde28fa322e-kube-api-access-l7gb6\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:29:44.050609 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:44.050560 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttpdg\" (UniqueName: \"kubernetes.io/projected/1ae18756-7182-45d8-af84-4b2b41920db0-kube-api-access-ttpdg\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:29:44.146419 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:44.146379 4522 generic.go:356] "Generic (PLEG): container finished" podID="645a0822-b642-405c-8b44-1bde28fa322e" containerID="a7ebf865e21dd6f31a4e0444a3d1483180d4a2ddfa938a8b20def40db7bab146" exitCode=0 Oct 30 17:29:44.146698 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:44.146669 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c17e8c-7cd6d6db99-wffp5" Oct 30 17:29:44.148645 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:44.148619 4522 generic.go:356] "Generic (PLEG): container finished" podID="1ae18756-7182-45d8-af84-4b2b41920db0" containerID="dd7c62cc30e69bf75354572b53e016cd30943d6da12cdd2e39cb05f6a58171ce" exitCode=0 Oct 30 17:29:44.148770 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:44.148744 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c17e8c-54fd5c99d-m7v82" Oct 30 17:29:44.158724 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:44.158698 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e08ea37-fe40-45d2-a36d-d4904537e910" path="/var/lib/kubelet/pods/4e08ea37-fe40-45d2-a36d-d4904537e910/volumes" Oct 30 17:29:44.159634 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:44.159610 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c17e8c-7cd6d6db99-wffp5" event={"ID":"645a0822-b642-405c-8b44-1bde28fa322e","Type":"ContainerDied","Data":"a7ebf865e21dd6f31a4e0444a3d1483180d4a2ddfa938a8b20def40db7bab146"} Oct 30 17:29:44.159715 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:44.159669 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c17e8c-7cd6d6db99-wffp5" event={"ID":"645a0822-b642-405c-8b44-1bde28fa322e","Type":"ContainerDied","Data":"529e0dce97c5bcfd25b56e6d9dbf51ce8917dbb41de5c7f1f3f90c8a6b6ce086"} Oct 30 17:29:44.159715 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:44.159697 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c17e8c-54fd5c99d-m7v82" event={"ID":"1ae18756-7182-45d8-af84-4b2b41920db0","Type":"ContainerDied","Data":"dd7c62cc30e69bf75354572b53e016cd30943d6da12cdd2e39cb05f6a58171ce"} Oct 30 17:29:44.159842 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:44.159721 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c17e8c-54fd5c99d-m7v82" event={"ID":"1ae18756-7182-45d8-af84-4b2b41920db0","Type":"ContainerDied","Data":"88e8d644dedff428620a75de88262b902bde63fe83b5aaec1cfef57531a48f8f"} Oct 30 17:29:44.159842 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:44.159744 4522 scope.go:122] "RemoveContainer" containerID="a7ebf865e21dd6f31a4e0444a3d1483180d4a2ddfa938a8b20def40db7bab146" Oct 30 17:29:44.173695 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:44.173678 4522 scope.go:122] "RemoveContainer" containerID="a7ebf865e21dd6f31a4e0444a3d1483180d4a2ddfa938a8b20def40db7bab146" Oct 30 17:29:44.174011 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:29:44.173991 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7ebf865e21dd6f31a4e0444a3d1483180d4a2ddfa938a8b20def40db7bab146\": container with ID starting with a7ebf865e21dd6f31a4e0444a3d1483180d4a2ddfa938a8b20def40db7bab146 not found: ID does not exist" containerID="a7ebf865e21dd6f31a4e0444a3d1483180d4a2ddfa938a8b20def40db7bab146" Oct 30 17:29:44.174059 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:44.174020 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7ebf865e21dd6f31a4e0444a3d1483180d4a2ddfa938a8b20def40db7bab146"} err="failed to get container status \"a7ebf865e21dd6f31a4e0444a3d1483180d4a2ddfa938a8b20def40db7bab146\": rpc error: code = NotFound desc = could not find container \"a7ebf865e21dd6f31a4e0444a3d1483180d4a2ddfa938a8b20def40db7bab146\": container with ID starting with a7ebf865e21dd6f31a4e0444a3d1483180d4a2ddfa938a8b20def40db7bab146 not found: ID does not exist" Oct 30 17:29:44.174059 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:44.174038 4522 scope.go:122] "RemoveContainer" containerID="dd7c62cc30e69bf75354572b53e016cd30943d6da12cdd2e39cb05f6a58171ce" Oct 30 17:29:44.186876 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:44.186853 4522 scope.go:122] "RemoveContainer" containerID="dd7c62cc30e69bf75354572b53e016cd30943d6da12cdd2e39cb05f6a58171ce" Oct 30 17:29:44.187167 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:29:44.187148 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd7c62cc30e69bf75354572b53e016cd30943d6da12cdd2e39cb05f6a58171ce\": container with ID starting with dd7c62cc30e69bf75354572b53e016cd30943d6da12cdd2e39cb05f6a58171ce not found: ID does not exist" containerID="dd7c62cc30e69bf75354572b53e016cd30943d6da12cdd2e39cb05f6a58171ce" Oct 30 17:29:44.187245 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:44.187176 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd7c62cc30e69bf75354572b53e016cd30943d6da12cdd2e39cb05f6a58171ce"} err="failed to get container status \"dd7c62cc30e69bf75354572b53e016cd30943d6da12cdd2e39cb05f6a58171ce\": rpc error: code = NotFound desc = could not find container \"dd7c62cc30e69bf75354572b53e016cd30943d6da12cdd2e39cb05f6a58171ce\": container with ID starting with dd7c62cc30e69bf75354572b53e016cd30943d6da12cdd2e39cb05f6a58171ce not found: ID does not exist" Oct 30 17:29:44.227361 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:44.227295 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c17e8c-7cd6d6db99-wffp5"] Oct 30 17:29:44.232291 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:44.232264 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c17e8c-7cd6d6db99-wffp5"] Oct 30 17:29:44.269809 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:44.269775 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c17e8c-54fd5c99d-m7v82"] Oct 30 17:29:44.275704 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:44.275667 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c17e8c-54fd5c99d-m7v82"] Oct 30 17:29:44.990996 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:44.990959 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-41cb25-755479dd79-dj7jc"] Oct 30 17:29:44.991937 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:44.991915 4522 cpu_manager.go:414] "containerMap: removing container" podUID="c419fcd5-9b32-4075-ba32-1614e365b132" containerName="payload-processing" Oct 30 17:29:44.991937 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:44.991936 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c419fcd5-9b32-4075-ba32-1614e365b132" containerName="payload-processing" Oct 30 17:29:44.992117 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:44.991945 4522 cpu_manager.go:414] "containerMap: removing container" podUID="1ae18756-7182-45d8-af84-4b2b41920db0" containerName="payload-processing" Oct 30 17:29:44.992117 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:44.991950 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ae18756-7182-45d8-af84-4b2b41920db0" containerName="payload-processing" Oct 30 17:29:44.992117 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:44.991984 4522 cpu_manager.go:414] "containerMap: removing container" podUID="5c473c8d-2321-447a-8059-eb3c1e6e105e" containerName="payload-pre-processing" Oct 30 17:29:44.992117 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:44.991990 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c473c8d-2321-447a-8059-eb3c1e6e105e" containerName="payload-pre-processing" Oct 30 17:29:44.992117 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:44.992009 4522 cpu_manager.go:414] "containerMap: removing container" podUID="645a0822-b642-405c-8b44-1bde28fa322e" containerName="payload-pre-processing" Oct 30 17:29:44.992117 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:44.992014 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="645a0822-b642-405c-8b44-1bde28fa322e" containerName="payload-pre-processing" Oct 30 17:29:44.992484 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:44.992131 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c473c8d-2321-447a-8059-eb3c1e6e105e" containerName="payload-pre-processing" Oct 30 17:29:44.992484 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:44.992140 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="645a0822-b642-405c-8b44-1bde28fa322e" containerName="payload-pre-processing" Oct 30 17:29:44.992484 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:44.992153 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="c419fcd5-9b32-4075-ba32-1614e365b132" containerName="payload-processing" Oct 30 17:29:44.992484 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:44.992159 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ae18756-7182-45d8-af84-4b2b41920db0" containerName="payload-processing" Oct 30 17:29:44.997695 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:44.997672 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41cb25-755479dd79-dj7jc" Oct 30 17:29:45.015165 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:45.015135 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-41cb25\"" Oct 30 17:29:45.015342 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:45.015261 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-41cb25-dockercfg-686v8\"" Oct 30 17:29:45.018508 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:45.018480 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-41cb25-755479dd79-dj7jc"] Oct 30 17:29:45.073453 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:45.073403 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3913f59f-4c2f-4163-9080-b9557dbd3987-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-41cb25-755479dd79-dj7jc\" (UID: \"3913f59f-4c2f-4163-9080-b9557dbd3987\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41cb25-755479dd79-dj7jc" Oct 30 17:29:45.073977 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:45.073757 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7ftf\" (UniqueName: \"kubernetes.io/projected/3913f59f-4c2f-4163-9080-b9557dbd3987-kube-api-access-b7ftf\") pod \"payload-pre-processing-e2e-ipp-a-41cb25-755479dd79-dj7jc\" (UID: \"3913f59f-4c2f-4163-9080-b9557dbd3987\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41cb25-755479dd79-dj7jc" Oct 30 17:29:45.163153 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:45.163113 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-41cb25-6c785598f6-vcrjm"] Oct 30 17:29:45.170209 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:45.170175 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-41cb25-6c785598f6-vcrjm" Oct 30 17:29:45.175996 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:45.175388 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7ftf\" (UniqueName: \"kubernetes.io/projected/3913f59f-4c2f-4163-9080-b9557dbd3987-kube-api-access-b7ftf\") pod \"payload-pre-processing-e2e-ipp-a-41cb25-755479dd79-dj7jc\" (UID: \"3913f59f-4c2f-4163-9080-b9557dbd3987\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41cb25-755479dd79-dj7jc" Oct 30 17:29:45.175996 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:45.175578 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3913f59f-4c2f-4163-9080-b9557dbd3987-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-41cb25-755479dd79-dj7jc\" (UID: \"3913f59f-4c2f-4163-9080-b9557dbd3987\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41cb25-755479dd79-dj7jc" Oct 30 17:29:45.180902 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:45.176538 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3913f59f-4c2f-4163-9080-b9557dbd3987-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-41cb25-755479dd79-dj7jc\" (UID: \"3913f59f-4c2f-4163-9080-b9557dbd3987\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41cb25-755479dd79-dj7jc" Oct 30 17:29:45.197404 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:45.195301 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-41cb25-6c785598f6-vcrjm"] Oct 30 17:29:45.230849 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:45.230807 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7ftf\" (UniqueName: \"kubernetes.io/projected/3913f59f-4c2f-4163-9080-b9557dbd3987-kube-api-access-b7ftf\") pod \"payload-pre-processing-e2e-ipp-a-41cb25-755479dd79-dj7jc\" (UID: \"3913f59f-4c2f-4163-9080-b9557dbd3987\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41cb25-755479dd79-dj7jc" Oct 30 17:29:45.278917 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:45.278487 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf81982b-e4fd-456c-99fe-52b89db4cced-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-41cb25-6c785598f6-vcrjm\" (UID: \"cf81982b-e4fd-456c-99fe-52b89db4cced\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-41cb25-6c785598f6-vcrjm" Oct 30 17:29:45.278917 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:45.278767 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mcj7\" (UniqueName: \"kubernetes.io/projected/cf81982b-e4fd-456c-99fe-52b89db4cced-kube-api-access-8mcj7\") pod \"payload-processing-e2e-ipp-a-41cb25-6c785598f6-vcrjm\" (UID: \"cf81982b-e4fd-456c-99fe-52b89db4cced\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-41cb25-6c785598f6-vcrjm" Oct 30 17:29:45.320649 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:45.320619 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41cb25-755479dd79-dj7jc" Oct 30 17:29:45.384772 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:45.381981 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mcj7\" (UniqueName: \"kubernetes.io/projected/cf81982b-e4fd-456c-99fe-52b89db4cced-kube-api-access-8mcj7\") pod \"payload-processing-e2e-ipp-a-41cb25-6c785598f6-vcrjm\" (UID: \"cf81982b-e4fd-456c-99fe-52b89db4cced\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-41cb25-6c785598f6-vcrjm" Oct 30 17:29:45.384772 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:45.382112 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf81982b-e4fd-456c-99fe-52b89db4cced-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-41cb25-6c785598f6-vcrjm\" (UID: \"cf81982b-e4fd-456c-99fe-52b89db4cced\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-41cb25-6c785598f6-vcrjm" Oct 30 17:29:45.384772 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:45.383203 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf81982b-e4fd-456c-99fe-52b89db4cced-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-41cb25-6c785598f6-vcrjm\" (UID: \"cf81982b-e4fd-456c-99fe-52b89db4cced\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-41cb25-6c785598f6-vcrjm" Oct 30 17:29:45.425666 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:45.425558 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mcj7\" (UniqueName: \"kubernetes.io/projected/cf81982b-e4fd-456c-99fe-52b89db4cced-kube-api-access-8mcj7\") pod \"payload-processing-e2e-ipp-a-41cb25-6c785598f6-vcrjm\" (UID: \"cf81982b-e4fd-456c-99fe-52b89db4cced\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-41cb25-6c785598f6-vcrjm" Oct 30 17:29:45.496862 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:45.496825 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-41cb25-6c785598f6-vcrjm" Oct 30 17:29:45.662913 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:45.662876 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-41cb25-755479dd79-dj7jc"] Oct 30 17:29:45.665158 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:29:45.665122 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3913f59f_4c2f_4163_9080_b9557dbd3987.slice/crio-5624cdc848d0322874c16385492009426e53fd3718c639a2831820db9d02e874 WatchSource:0}: Error finding container 5624cdc848d0322874c16385492009426e53fd3718c639a2831820db9d02e874: Status 404 returned error can't find the container with id 5624cdc848d0322874c16385492009426e53fd3718c639a2831820db9d02e874 Oct 30 17:29:45.838100 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:45.838073 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-41cb25-6c785598f6-vcrjm"] Oct 30 17:29:46.150877 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:46.150798 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ae18756-7182-45d8-af84-4b2b41920db0" path="/var/lib/kubelet/pods/1ae18756-7182-45d8-af84-4b2b41920db0/volumes" Oct 30 17:29:46.151457 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:46.151439 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="645a0822-b642-405c-8b44-1bde28fa322e" path="/var/lib/kubelet/pods/645a0822-b642-405c-8b44-1bde28fa322e/volumes" Oct 30 17:29:46.186716 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:46.186666 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41cb25-755479dd79-dj7jc" event={"ID":"3913f59f-4c2f-4163-9080-b9557dbd3987","Type":"ContainerStarted","Data":"fd99fd82304258eac2fd4074b58787637aff9988c0d362aba89a79c107e83682"} Oct 30 17:29:46.186922 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:46.186735 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41cb25-755479dd79-dj7jc" event={"ID":"3913f59f-4c2f-4163-9080-b9557dbd3987","Type":"ContainerStarted","Data":"5624cdc848d0322874c16385492009426e53fd3718c639a2831820db9d02e874"} Oct 30 17:29:46.186922 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:46.186771 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41cb25-755479dd79-dj7jc" Oct 30 17:29:46.189073 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:46.189048 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-41cb25-6c785598f6-vcrjm" event={"ID":"cf81982b-e4fd-456c-99fe-52b89db4cced","Type":"ContainerStarted","Data":"a70da96341342c94ddb60330fc268c520019d2244772f71bc6ef6831d2ea6650"} Oct 30 17:29:46.189196 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:46.189097 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-41cb25-6c785598f6-vcrjm" event={"ID":"cf81982b-e4fd-456c-99fe-52b89db4cced","Type":"ContainerStarted","Data":"fced5024e53ca9977989381216da14edcf7f945f7c2cd9285fba18a40d02e645"} Oct 30 17:29:46.189196 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:46.189145 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-41cb25-6c785598f6-vcrjm" Oct 30 17:29:46.238450 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:46.238397 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41cb25-755479dd79-dj7jc" podStartSLOduration=2.238383169 podStartE2EDuration="2.238383169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:29:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:29:46.2330891 +0000 UTC m=+2560.233844000" watchObservedRunningTime="2026-10-30 17:29:46.238383169 +0000 UTC m=+2560.239138069" Oct 30 17:29:46.285326 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:46.284833 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-41cb25-6c785598f6-vcrjm" podStartSLOduration=1.284814619 podStartE2EDuration="1.284814619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:29:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:29:46.281567645 +0000 UTC m=+2560.282322564" watchObservedRunningTime="2026-10-30 17:29:46.284814619 +0000 UTC m=+2560.285569523" Oct 30 17:29:48.209192 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:48.209162 4522 generic.go:356] "Generic (PLEG): container finished" podID="2d64632e-0973-4589-9c43-b7b3b7bb95b3" containerID="af69effeda096f045d40224956e652f26bcd057525521bd203c2fbe7fe259de0" exitCode=0 Oct 30 17:29:48.209679 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:48.209237 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb" event={"ID":"2d64632e-0973-4589-9c43-b7b3b7bb95b3","Type":"ContainerDied","Data":"af69effeda096f045d40224956e652f26bcd057525521bd203c2fbe7fe259de0"} Oct 30 17:29:48.209789 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:48.209772 4522 scope.go:122] "RemoveContainer" containerID="af69effeda096f045d40224956e652f26bcd057525521bd203c2fbe7fe259de0" Oct 30 17:29:48.212196 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:48.212159 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:29:48.212300 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:48.212253 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:29:48.212361 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:48.212299 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:29:49.230502 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:49.230426 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb" event={"ID":"2d64632e-0973-4589-9c43-b7b3b7bb95b3","Type":"ContainerStarted","Data":"29ce2393b5b40d235814699534f9ba13a83b59f1162f52c9a9951d845d970924"} Oct 30 17:29:50.222675 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:50.222628 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb" Oct 30 17:29:50.223905 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:50.223875 4522 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" start-of-body= Oct 30 17:29:50.224083 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:50.223940 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb" podUID="2d64632e-0973-4589-9c43-b7b3b7bb95b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" Oct 30 17:29:50.799446 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:50.799403 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb" Oct 30 17:29:51.222534 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:51.222496 4522 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" start-of-body= Oct 30 17:29:51.222730 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:51.222548 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb" podUID="2d64632e-0973-4589-9c43-b7b3b7bb95b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" Oct 30 17:29:52.222483 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:52.222441 4522 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" start-of-body= Oct 30 17:29:52.222965 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:52.222505 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb" podUID="2d64632e-0973-4589-9c43-b7b3b7bb95b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" Oct 30 17:29:53.222853 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:53.222801 4522 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" start-of-body= Oct 30 17:29:53.223301 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:53.222885 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb" podUID="2d64632e-0973-4589-9c43-b7b3b7bb95b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" Oct 30 17:29:54.222836 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:54.222801 4522 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" start-of-body= Oct 30 17:29:54.223029 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:54.222864 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb" podUID="2d64632e-0973-4589-9c43-b7b3b7bb95b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" Oct 30 17:29:55.228520 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:55.228491 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb" Oct 30 17:29:55.229569 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:55.229550 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb" Oct 30 17:29:57.200901 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:57.200870 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41cb25-755479dd79-dj7jc" Oct 30 17:29:57.201400 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:29:57.200939 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-41cb25-6c785598f6-vcrjm" Oct 30 17:30:00.157095 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.157015 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-987dfc-29889690-vpglf"] Oct 30 17:30:00.162435 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.162410 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-987dfc-29889690-vpglf" Oct 30 17:30:00.162435 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.162421 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-82243b-29889690-dmgd9"] Oct 30 17:30:00.167031 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.167010 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-82243b-29889690-dmgd9" Oct 30 17:30:00.168043 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.168019 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-469ad7-29889690-cfrlx"] Oct 30 17:30:00.172543 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.172523 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-469ad7-29889690-cfrlx" Oct 30 17:30:00.183363 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.183342 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:30:00.184960 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.184936 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889690-qb6xz"] Oct 30 17:30:00.192531 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.192507 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889690-qb6xz" Oct 30 17:30:00.201739 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.201713 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889690-xxcd6"] Oct 30 17:30:00.258679 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.258646 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-987dfc-29889690-vpglf"] Oct 30 17:30:00.258932 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.258918 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-82243b-29889690-dmgd9"] Oct 30 17:30:00.259056 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.259044 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-469ad7-29889690-cfrlx"] Oct 30 17:30:00.259716 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.259433 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889690-ksf2d"] Oct 30 17:30:00.259952 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.258772 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889690-xxcd6" Oct 30 17:30:00.295848 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.292636 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-925th\" (UniqueName: \"kubernetes.io/projected/f6b0d3f6-2109-40b3-b86f-b895a4616558-kube-api-access-925th\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-82243b-29889690-dmgd9\" (UID: \"f6b0d3f6-2109-40b3-b86f-b895a4616558\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-82243b-29889690-dmgd9" Oct 30 17:30:00.295848 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.292790 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c977d640-8303-4e86-8952-149525c0708c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-469ad7-29889690-cfrlx\" (UID: \"c977d640-8303-4e86-8952-149525c0708c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-469ad7-29889690-cfrlx" Oct 30 17:30:00.295848 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.292913 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8643b717-c5ec-4194-92bb-e54de07fef00-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-987dfc-29889690-vpglf\" (UID: \"8643b717-c5ec-4194-92bb-e54de07fef00\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-987dfc-29889690-vpglf" Oct 30 17:30:00.295848 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.292947 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kvtc\" (UniqueName: \"kubernetes.io/projected/8643b717-c5ec-4194-92bb-e54de07fef00-kube-api-access-9kvtc\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-987dfc-29889690-vpglf\" (UID: \"8643b717-c5ec-4194-92bb-e54de07fef00\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-987dfc-29889690-vpglf" Oct 30 17:30:00.295848 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.292998 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mksvh\" (UniqueName: \"kubernetes.io/projected/c977d640-8303-4e86-8952-149525c0708c-kube-api-access-mksvh\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-469ad7-29889690-cfrlx\" (UID: \"c977d640-8303-4e86-8952-149525c0708c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-469ad7-29889690-cfrlx" Oct 30 17:30:00.295848 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.293957 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6b0d3f6-2109-40b3-b86f-b895a4616558-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-82243b-29889690-dmgd9\" (UID: \"f6b0d3f6-2109-40b3-b86f-b895a4616558\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-82243b-29889690-dmgd9" Oct 30 17:30:00.343579 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.342621 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889690-qb6xz"] Oct 30 17:30:00.343579 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.342668 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889690-xxcd6"] Oct 30 17:30:00.343579 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.342701 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889690-d76sf"] Oct 30 17:30:00.364127 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.362451 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889690-ksf2d" Oct 30 17:30:00.369418 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.369389 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d2d985-29889690-vlq47"] Oct 30 17:30:00.369940 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.369914 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889690-d76sf" Oct 30 17:30:00.381924 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.381119 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889690-7kjml"] Oct 30 17:30:00.382398 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.382377 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d2d985-29889690-vlq47" Oct 30 17:30:00.388610 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.388537 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889690-s8ggr"] Oct 30 17:30:00.388610 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.388599 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889690-7kjml" Oct 30 17:30:00.401793 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.395166 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889690-ksf2d"] Oct 30 17:30:00.401793 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.395260 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889690-s8ggr" Oct 30 17:30:00.401793 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.395369 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-001a96-29889690-j9qwv"] Oct 30 17:30:00.401793 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.396169 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzdnr\" (UniqueName: \"kubernetes.io/projected/58bcea19-790f-486c-8994-21b1b4c20dc9-kube-api-access-nzdnr\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889690-qb6xz\" (UID: \"58bcea19-790f-486c-8994-21b1b4c20dc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889690-qb6xz" Oct 30 17:30:00.401793 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.396226 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nl59\" (UniqueName: \"kubernetes.io/projected/e1d79713-dd75-4823-a8f8-0f252befa273-kube-api-access-9nl59\") pod \"maas-api-key-cleanup-e2e-worker-w6-d384dc-29889690-xxcd6\" (UID: \"e1d79713-dd75-4823-a8f8-0f252befa273\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889690-xxcd6" Oct 30 17:30:00.401793 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.396292 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6b0d3f6-2109-40b3-b86f-b895a4616558-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-82243b-29889690-dmgd9\" (UID: \"f6b0d3f6-2109-40b3-b86f-b895a4616558\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-82243b-29889690-dmgd9" Oct 30 17:30:00.401793 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.396357 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-925th\" (UniqueName: \"kubernetes.io/projected/f6b0d3f6-2109-40b3-b86f-b895a4616558-kube-api-access-925th\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-82243b-29889690-dmgd9\" (UID: \"f6b0d3f6-2109-40b3-b86f-b895a4616558\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-82243b-29889690-dmgd9" Oct 30 17:30:00.401793 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.396455 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1d79713-dd75-4823-a8f8-0f252befa273-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-d384dc-29889690-xxcd6\" (UID: \"e1d79713-dd75-4823-a8f8-0f252befa273\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889690-xxcd6" Oct 30 17:30:00.401793 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.396527 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c977d640-8303-4e86-8952-149525c0708c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-469ad7-29889690-cfrlx\" (UID: \"c977d640-8303-4e86-8952-149525c0708c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-469ad7-29889690-cfrlx" Oct 30 17:30:00.401793 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.396565 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58bcea19-790f-486c-8994-21b1b4c20dc9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889690-qb6xz\" (UID: \"58bcea19-790f-486c-8994-21b1b4c20dc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889690-qb6xz" Oct 30 17:30:00.401793 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.396680 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8643b717-c5ec-4194-92bb-e54de07fef00-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-987dfc-29889690-vpglf\" (UID: \"8643b717-c5ec-4194-92bb-e54de07fef00\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-987dfc-29889690-vpglf" Oct 30 17:30:00.401793 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.396711 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kvtc\" (UniqueName: \"kubernetes.io/projected/8643b717-c5ec-4194-92bb-e54de07fef00-kube-api-access-9kvtc\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-987dfc-29889690-vpglf\" (UID: \"8643b717-c5ec-4194-92bb-e54de07fef00\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-987dfc-29889690-vpglf" Oct 30 17:30:00.401793 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.396764 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mksvh\" (UniqueName: \"kubernetes.io/projected/c977d640-8303-4e86-8952-149525c0708c-kube-api-access-mksvh\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-469ad7-29889690-cfrlx\" (UID: \"c977d640-8303-4e86-8952-149525c0708c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-469ad7-29889690-cfrlx" Oct 30 17:30:00.401793 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.397850 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6b0d3f6-2109-40b3-b86f-b895a4616558-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-82243b-29889690-dmgd9\" (UID: \"f6b0d3f6-2109-40b3-b86f-b895a4616558\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-82243b-29889690-dmgd9" Oct 30 17:30:00.401793 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.397928 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c977d640-8303-4e86-8952-149525c0708c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-469ad7-29889690-cfrlx\" (UID: \"c977d640-8303-4e86-8952-149525c0708c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-469ad7-29889690-cfrlx" Oct 30 17:30:00.401793 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.397928 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8643b717-c5ec-4194-92bb-e54de07fef00-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-987dfc-29889690-vpglf\" (UID: \"8643b717-c5ec-4194-92bb-e54de07fef00\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-987dfc-29889690-vpglf" Oct 30 17:30:00.406246 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.406218 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d2d985-29889690-vlq47"] Oct 30 17:30:00.406413 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.406269 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889690-d76sf"] Oct 30 17:30:00.406413 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.406291 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-001a96-29889690-j9qwv" Oct 30 17:30:00.409074 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.409010 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889690-7kjml"] Oct 30 17:30:00.417585 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.417560 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c17e8c-29889690-nvd7z"] Oct 30 17:30:00.427348 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.422520 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mksvh\" (UniqueName: \"kubernetes.io/projected/c977d640-8303-4e86-8952-149525c0708c-kube-api-access-mksvh\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-469ad7-29889690-cfrlx\" (UID: \"c977d640-8303-4e86-8952-149525c0708c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-469ad7-29889690-cfrlx" Oct 30 17:30:00.427348 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.424605 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-925th\" (UniqueName: \"kubernetes.io/projected/f6b0d3f6-2109-40b3-b86f-b895a4616558-kube-api-access-925th\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-82243b-29889690-dmgd9\" (UID: \"f6b0d3f6-2109-40b3-b86f-b895a4616558\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-82243b-29889690-dmgd9" Oct 30 17:30:00.427348 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.426246 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kvtc\" (UniqueName: \"kubernetes.io/projected/8643b717-c5ec-4194-92bb-e54de07fef00-kube-api-access-9kvtc\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-987dfc-29889690-vpglf\" (UID: \"8643b717-c5ec-4194-92bb-e54de07fef00\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-987dfc-29889690-vpglf" Oct 30 17:30:00.429692 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.429662 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889690-s8ggr"] Oct 30 17:30:00.429826 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.429771 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c17e8c-29889690-nvd7z" Oct 30 17:30:00.437667 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.437643 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-001a96-29889690-j9qwv"] Oct 30 17:30:00.444020 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.443994 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e13a51-29889690-dg6vt"] Oct 30 17:30:00.466110 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.466074 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889690-csm2k"] Oct 30 17:30:00.466563 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.466482 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e13a51-29889690-dg6vt" Oct 30 17:30:00.473977 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.473950 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c17e8c-29889690-nvd7z"] Oct 30 17:30:00.474091 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.474025 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889690-csm2k" Oct 30 17:30:00.476718 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.476701 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889690-csm2k"] Oct 30 17:30:00.480929 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.480911 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-987dfc-29889690-vpglf" Oct 30 17:30:00.489406 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.489380 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e13a51-29889690-dg6vt"] Oct 30 17:30:00.492543 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.492519 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-82243b-29889690-dmgd9" Oct 30 17:30:00.498045 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.498020 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46fba0fa-d263-42d2-84fd-ddaefea25c63-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b80d83-29889690-s8ggr\" (UID: \"46fba0fa-d263-42d2-84fd-ddaefea25c63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889690-s8ggr" Oct 30 17:30:00.498167 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.498143 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b49aaf80-dff6-40d4-8404-d55de5b568ae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-04248e-29889690-d76sf\" (UID: \"b49aaf80-dff6-40d4-8404-d55de5b568ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889690-d76sf" Oct 30 17:30:00.498262 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.498242 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqjkd\" (UniqueName: \"kubernetes.io/projected/46fba0fa-d263-42d2-84fd-ddaefea25c63-kube-api-access-jqjkd\") pod \"maas-api-key-cleanup-e2e-worker-w3-b80d83-29889690-s8ggr\" (UID: \"46fba0fa-d263-42d2-84fd-ddaefea25c63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889690-s8ggr" Oct 30 17:30:00.498366 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.498349 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w25zl\" (UniqueName: \"kubernetes.io/projected/22ffe3de-e5e2-43a1-b276-20cf180c8b01-kube-api-access-w25zl\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889690-ksf2d\" (UID: \"22ffe3de-e5e2-43a1-b276-20cf180c8b01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889690-ksf2d" Oct 30 17:30:00.498432 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.498395 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzdnr\" (UniqueName: \"kubernetes.io/projected/58bcea19-790f-486c-8994-21b1b4c20dc9-kube-api-access-nzdnr\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889690-qb6xz\" (UID: \"58bcea19-790f-486c-8994-21b1b4c20dc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889690-qb6xz" Oct 30 17:30:00.498501 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.498479 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8fqh\" (UniqueName: \"kubernetes.io/projected/ddc0e647-bfb8-474d-b423-be4d72b31b32-kube-api-access-h8fqh\") pod \"maas-api-key-cleanup-e2e-ipp-a-d2d985-29889690-vlq47\" (UID: \"ddc0e647-bfb8-474d-b423-be4d72b31b32\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d2d985-29889690-vlq47" Oct 30 17:30:00.498566 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.498535 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nl59\" (UniqueName: \"kubernetes.io/projected/e1d79713-dd75-4823-a8f8-0f252befa273-kube-api-access-9nl59\") pod \"maas-api-key-cleanup-e2e-worker-w6-d384dc-29889690-xxcd6\" (UID: \"e1d79713-dd75-4823-a8f8-0f252befa273\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889690-xxcd6" Oct 30 17:30:00.498622 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.498590 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2ce4a71-84b5-4817-b99c-9d56d4673869-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889690-7kjml\" (UID: \"e2ce4a71-84b5-4817-b99c-9d56d4673869\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889690-7kjml" Oct 30 17:30:00.498694 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.498676 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22ffe3de-e5e2-43a1-b276-20cf180c8b01-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889690-ksf2d\" (UID: \"22ffe3de-e5e2-43a1-b276-20cf180c8b01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889690-ksf2d" Oct 30 17:30:00.498776 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.498764 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1d79713-dd75-4823-a8f8-0f252befa273-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-d384dc-29889690-xxcd6\" (UID: \"e1d79713-dd75-4823-a8f8-0f252befa273\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889690-xxcd6" Oct 30 17:30:00.498847 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.498801 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjx5q\" (UniqueName: \"kubernetes.io/projected/e2ce4a71-84b5-4817-b99c-9d56d4673869-kube-api-access-zjx5q\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889690-7kjml\" (UID: \"e2ce4a71-84b5-4817-b99c-9d56d4673869\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889690-7kjml" Oct 30 17:30:00.498847 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.498839 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vbp2\" (UniqueName: \"kubernetes.io/projected/b49aaf80-dff6-40d4-8404-d55de5b568ae-kube-api-access-5vbp2\") pod \"maas-api-key-cleanup-e2e-worker-w1-04248e-29889690-d76sf\" (UID: \"b49aaf80-dff6-40d4-8404-d55de5b568ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889690-d76sf" Oct 30 17:30:00.498959 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.498910 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58bcea19-790f-486c-8994-21b1b4c20dc9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889690-qb6xz\" (UID: \"58bcea19-790f-486c-8994-21b1b4c20dc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889690-qb6xz" Oct 30 17:30:00.499073 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.499046 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddc0e647-bfb8-474d-b423-be4d72b31b32-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-d2d985-29889690-vlq47\" (UID: \"ddc0e647-bfb8-474d-b423-be4d72b31b32\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d2d985-29889690-vlq47" Oct 30 17:30:00.499909 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.499628 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58bcea19-790f-486c-8994-21b1b4c20dc9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889690-qb6xz\" (UID: \"58bcea19-790f-486c-8994-21b1b4c20dc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889690-qb6xz" Oct 30 17:30:00.500006 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.499980 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1d79713-dd75-4823-a8f8-0f252befa273-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-d384dc-29889690-xxcd6\" (UID: \"e1d79713-dd75-4823-a8f8-0f252befa273\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889690-xxcd6" Oct 30 17:30:00.514882 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.514845 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-469ad7-29889690-cfrlx" Oct 30 17:30:00.524050 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.524021 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nl59\" (UniqueName: \"kubernetes.io/projected/e1d79713-dd75-4823-a8f8-0f252befa273-kube-api-access-9nl59\") pod \"maas-api-key-cleanup-e2e-worker-w6-d384dc-29889690-xxcd6\" (UID: \"e1d79713-dd75-4823-a8f8-0f252befa273\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889690-xxcd6" Oct 30 17:30:00.524171 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.524057 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzdnr\" (UniqueName: \"kubernetes.io/projected/58bcea19-790f-486c-8994-21b1b4c20dc9-kube-api-access-nzdnr\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889690-qb6xz\" (UID: \"58bcea19-790f-486c-8994-21b1b4c20dc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889690-qb6xz" Oct 30 17:30:00.604333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.601029 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2ce4a71-84b5-4817-b99c-9d56d4673869-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889690-7kjml\" (UID: \"e2ce4a71-84b5-4817-b99c-9d56d4673869\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889690-7kjml" Oct 30 17:30:00.604333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.601104 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3c414cb-e4f0-4fbf-9bdf-946842e1297f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-001a96-29889690-j9qwv\" (UID: \"f3c414cb-e4f0-4fbf-9bdf-946842e1297f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-001a96-29889690-j9qwv" Oct 30 17:30:00.604333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.601138 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/427507de-33c6-409a-a68c-8e5a978b24ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-c17e8c-29889690-nvd7z\" (UID: \"427507de-33c6-409a-a68c-8e5a978b24ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c17e8c-29889690-nvd7z" Oct 30 17:30:00.604333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.601183 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4febc82-5717-4157-b46b-7ad8004d7f6a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-e13a51-29889690-dg6vt\" (UID: \"e4febc82-5717-4157-b46b-7ad8004d7f6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e13a51-29889690-dg6vt" Oct 30 17:30:00.604333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.601215 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22ffe3de-e5e2-43a1-b276-20cf180c8b01-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889690-ksf2d\" (UID: \"22ffe3de-e5e2-43a1-b276-20cf180c8b01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889690-ksf2d" Oct 30 17:30:00.604333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.602165 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2ce4a71-84b5-4817-b99c-9d56d4673869-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889690-7kjml\" (UID: \"e2ce4a71-84b5-4817-b99c-9d56d4673869\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889690-7kjml" Oct 30 17:30:00.616570 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.616102 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjx5q\" (UniqueName: \"kubernetes.io/projected/e2ce4a71-84b5-4817-b99c-9d56d4673869-kube-api-access-zjx5q\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889690-7kjml\" (UID: \"e2ce4a71-84b5-4817-b99c-9d56d4673869\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889690-7kjml" Oct 30 17:30:00.616570 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.616158 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vbp2\" (UniqueName: \"kubernetes.io/projected/b49aaf80-dff6-40d4-8404-d55de5b568ae-kube-api-access-5vbp2\") pod \"maas-api-key-cleanup-e2e-worker-w1-04248e-29889690-d76sf\" (UID: \"b49aaf80-dff6-40d4-8404-d55de5b568ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889690-d76sf" Oct 30 17:30:00.616570 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.616219 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kgvk\" (UniqueName: \"kubernetes.io/projected/e4febc82-5717-4157-b46b-7ad8004d7f6a-kube-api-access-6kgvk\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-e13a51-29889690-dg6vt\" (UID: \"e4febc82-5717-4157-b46b-7ad8004d7f6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e13a51-29889690-dg6vt" Oct 30 17:30:00.616570 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.616343 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4fa40dd-b387-4513-aed9-8e11394d3454-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-096d75-29889690-csm2k\" (UID: \"d4fa40dd-b387-4513-aed9-8e11394d3454\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889690-csm2k" Oct 30 17:30:00.616570 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.616383 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddc0e647-bfb8-474d-b423-be4d72b31b32-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-d2d985-29889690-vlq47\" (UID: \"ddc0e647-bfb8-474d-b423-be4d72b31b32\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d2d985-29889690-vlq47" Oct 30 17:30:00.616570 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.616428 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46fba0fa-d263-42d2-84fd-ddaefea25c63-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b80d83-29889690-s8ggr\" (UID: \"46fba0fa-d263-42d2-84fd-ddaefea25c63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889690-s8ggr" Oct 30 17:30:00.616570 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.616506 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b49aaf80-dff6-40d4-8404-d55de5b568ae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-04248e-29889690-d76sf\" (UID: \"b49aaf80-dff6-40d4-8404-d55de5b568ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889690-d76sf" Oct 30 17:30:00.616570 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.616563 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsrxd\" (UniqueName: \"kubernetes.io/projected/d4fa40dd-b387-4513-aed9-8e11394d3454-kube-api-access-jsrxd\") pod \"maas-api-key-cleanup-e2e-worker-w0-096d75-29889690-csm2k\" (UID: \"d4fa40dd-b387-4513-aed9-8e11394d3454\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889690-csm2k" Oct 30 17:30:00.617152 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.616602 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqjkd\" (UniqueName: \"kubernetes.io/projected/46fba0fa-d263-42d2-84fd-ddaefea25c63-kube-api-access-jqjkd\") pod \"maas-api-key-cleanup-e2e-worker-w3-b80d83-29889690-s8ggr\" (UID: \"46fba0fa-d263-42d2-84fd-ddaefea25c63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889690-s8ggr" Oct 30 17:30:00.617152 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.616669 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w25zl\" (UniqueName: \"kubernetes.io/projected/22ffe3de-e5e2-43a1-b276-20cf180c8b01-kube-api-access-w25zl\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889690-ksf2d\" (UID: \"22ffe3de-e5e2-43a1-b276-20cf180c8b01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889690-ksf2d" Oct 30 17:30:00.617152 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.616710 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22ffe3de-e5e2-43a1-b276-20cf180c8b01-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889690-ksf2d\" (UID: \"22ffe3de-e5e2-43a1-b276-20cf180c8b01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889690-ksf2d" Oct 30 17:30:00.617152 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.616725 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cw4t\" (UniqueName: \"kubernetes.io/projected/f3c414cb-e4f0-4fbf-9bdf-946842e1297f-kube-api-access-8cw4t\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-001a96-29889690-j9qwv\" (UID: \"f3c414cb-e4f0-4fbf-9bdf-946842e1297f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-001a96-29889690-j9qwv" Oct 30 17:30:00.617715 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.617227 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9ln8\" (UniqueName: \"kubernetes.io/projected/427507de-33c6-409a-a68c-8e5a978b24ed-kube-api-access-n9ln8\") pod \"maas-api-key-cleanup-e2e-ipp-a-c17e8c-29889690-nvd7z\" (UID: \"427507de-33c6-409a-a68c-8e5a978b24ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c17e8c-29889690-nvd7z" Oct 30 17:30:00.617715 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.617328 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8fqh\" (UniqueName: \"kubernetes.io/projected/ddc0e647-bfb8-474d-b423-be4d72b31b32-kube-api-access-h8fqh\") pod \"maas-api-key-cleanup-e2e-ipp-a-d2d985-29889690-vlq47\" (UID: \"ddc0e647-bfb8-474d-b423-be4d72b31b32\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d2d985-29889690-vlq47" Oct 30 17:30:00.617715 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.617432 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46fba0fa-d263-42d2-84fd-ddaefea25c63-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b80d83-29889690-s8ggr\" (UID: \"46fba0fa-d263-42d2-84fd-ddaefea25c63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889690-s8ggr" Oct 30 17:30:00.623454 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.618060 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddc0e647-bfb8-474d-b423-be4d72b31b32-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-d2d985-29889690-vlq47\" (UID: \"ddc0e647-bfb8-474d-b423-be4d72b31b32\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d2d985-29889690-vlq47" Oct 30 17:30:00.623454 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.618246 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b49aaf80-dff6-40d4-8404-d55de5b568ae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-04248e-29889690-d76sf\" (UID: \"b49aaf80-dff6-40d4-8404-d55de5b568ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889690-d76sf" Oct 30 17:30:00.650566 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.650526 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8fqh\" (UniqueName: \"kubernetes.io/projected/ddc0e647-bfb8-474d-b423-be4d72b31b32-kube-api-access-h8fqh\") pod \"maas-api-key-cleanup-e2e-ipp-a-d2d985-29889690-vlq47\" (UID: \"ddc0e647-bfb8-474d-b423-be4d72b31b32\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d2d985-29889690-vlq47" Oct 30 17:30:00.652207 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.652152 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vbp2\" (UniqueName: \"kubernetes.io/projected/b49aaf80-dff6-40d4-8404-d55de5b568ae-kube-api-access-5vbp2\") pod \"maas-api-key-cleanup-e2e-worker-w1-04248e-29889690-d76sf\" (UID: \"b49aaf80-dff6-40d4-8404-d55de5b568ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889690-d76sf" Oct 30 17:30:00.652207 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.652170 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjx5q\" (UniqueName: \"kubernetes.io/projected/e2ce4a71-84b5-4817-b99c-9d56d4673869-kube-api-access-zjx5q\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889690-7kjml\" (UID: \"e2ce4a71-84b5-4817-b99c-9d56d4673869\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889690-7kjml" Oct 30 17:30:00.652796 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.652776 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqjkd\" (UniqueName: \"kubernetes.io/projected/46fba0fa-d263-42d2-84fd-ddaefea25c63-kube-api-access-jqjkd\") pod \"maas-api-key-cleanup-e2e-worker-w3-b80d83-29889690-s8ggr\" (UID: \"46fba0fa-d263-42d2-84fd-ddaefea25c63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889690-s8ggr" Oct 30 17:30:00.654039 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.654018 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w25zl\" (UniqueName: \"kubernetes.io/projected/22ffe3de-e5e2-43a1-b276-20cf180c8b01-kube-api-access-w25zl\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889690-ksf2d\" (UID: \"22ffe3de-e5e2-43a1-b276-20cf180c8b01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889690-ksf2d" Oct 30 17:30:00.697437 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.685411 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889690-qb6xz" Oct 30 17:30:00.700575 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.697942 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889690-xxcd6" Oct 30 17:30:00.701168 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.701135 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889690-ksf2d" Oct 30 17:30:00.721141 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.720054 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889690-d76sf" Oct 30 17:30:00.721832 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.721804 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d2d985-29889690-vlq47" Oct 30 17:30:00.727355 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.723737 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3c414cb-e4f0-4fbf-9bdf-946842e1297f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-001a96-29889690-j9qwv\" (UID: \"f3c414cb-e4f0-4fbf-9bdf-946842e1297f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-001a96-29889690-j9qwv" Oct 30 17:30:00.727355 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.724584 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/427507de-33c6-409a-a68c-8e5a978b24ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-c17e8c-29889690-nvd7z\" (UID: \"427507de-33c6-409a-a68c-8e5a978b24ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c17e8c-29889690-nvd7z" Oct 30 17:30:00.727355 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.724602 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3c414cb-e4f0-4fbf-9bdf-946842e1297f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-001a96-29889690-j9qwv\" (UID: \"f3c414cb-e4f0-4fbf-9bdf-946842e1297f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-001a96-29889690-j9qwv" Oct 30 17:30:00.727355 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.724651 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/427507de-33c6-409a-a68c-8e5a978b24ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-c17e8c-29889690-nvd7z\" (UID: \"427507de-33c6-409a-a68c-8e5a978b24ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c17e8c-29889690-nvd7z" Oct 30 17:30:00.727355 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.724720 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4febc82-5717-4157-b46b-7ad8004d7f6a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-e13a51-29889690-dg6vt\" (UID: \"e4febc82-5717-4157-b46b-7ad8004d7f6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e13a51-29889690-dg6vt" Oct 30 17:30:00.727355 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.725429 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4febc82-5717-4157-b46b-7ad8004d7f6a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-e13a51-29889690-dg6vt\" (UID: \"e4febc82-5717-4157-b46b-7ad8004d7f6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e13a51-29889690-dg6vt" Oct 30 17:30:00.727355 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.725597 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kgvk\" (UniqueName: \"kubernetes.io/projected/e4febc82-5717-4157-b46b-7ad8004d7f6a-kube-api-access-6kgvk\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-e13a51-29889690-dg6vt\" (UID: \"e4febc82-5717-4157-b46b-7ad8004d7f6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e13a51-29889690-dg6vt" Oct 30 17:30:00.727355 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.726000 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4fa40dd-b387-4513-aed9-8e11394d3454-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-096d75-29889690-csm2k\" (UID: \"d4fa40dd-b387-4513-aed9-8e11394d3454\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889690-csm2k" Oct 30 17:30:00.727355 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.726114 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsrxd\" (UniqueName: \"kubernetes.io/projected/d4fa40dd-b387-4513-aed9-8e11394d3454-kube-api-access-jsrxd\") pod \"maas-api-key-cleanup-e2e-worker-w0-096d75-29889690-csm2k\" (UID: \"d4fa40dd-b387-4513-aed9-8e11394d3454\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889690-csm2k" Oct 30 17:30:00.727355 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.726189 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cw4t\" (UniqueName: \"kubernetes.io/projected/f3c414cb-e4f0-4fbf-9bdf-946842e1297f-kube-api-access-8cw4t\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-001a96-29889690-j9qwv\" (UID: \"f3c414cb-e4f0-4fbf-9bdf-946842e1297f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-001a96-29889690-j9qwv" Oct 30 17:30:00.727355 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.726222 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9ln8\" (UniqueName: \"kubernetes.io/projected/427507de-33c6-409a-a68c-8e5a978b24ed-kube-api-access-n9ln8\") pod \"maas-api-key-cleanup-e2e-ipp-a-c17e8c-29889690-nvd7z\" (UID: \"427507de-33c6-409a-a68c-8e5a978b24ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c17e8c-29889690-nvd7z" Oct 30 17:30:00.727355 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.727290 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4fa40dd-b387-4513-aed9-8e11394d3454-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-096d75-29889690-csm2k\" (UID: \"d4fa40dd-b387-4513-aed9-8e11394d3454\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889690-csm2k" Oct 30 17:30:00.758063 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.758029 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kgvk\" (UniqueName: \"kubernetes.io/projected/e4febc82-5717-4157-b46b-7ad8004d7f6a-kube-api-access-6kgvk\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-e13a51-29889690-dg6vt\" (UID: \"e4febc82-5717-4157-b46b-7ad8004d7f6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e13a51-29889690-dg6vt" Oct 30 17:30:00.762005 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.761426 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9ln8\" (UniqueName: \"kubernetes.io/projected/427507de-33c6-409a-a68c-8e5a978b24ed-kube-api-access-n9ln8\") pod \"maas-api-key-cleanup-e2e-ipp-a-c17e8c-29889690-nvd7z\" (UID: \"427507de-33c6-409a-a68c-8e5a978b24ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c17e8c-29889690-nvd7z" Oct 30 17:30:00.764219 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.764200 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsrxd\" (UniqueName: \"kubernetes.io/projected/d4fa40dd-b387-4513-aed9-8e11394d3454-kube-api-access-jsrxd\") pod \"maas-api-key-cleanup-e2e-worker-w0-096d75-29889690-csm2k\" (UID: \"d4fa40dd-b387-4513-aed9-8e11394d3454\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889690-csm2k" Oct 30 17:30:00.765069 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.765053 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cw4t\" (UniqueName: \"kubernetes.io/projected/f3c414cb-e4f0-4fbf-9bdf-946842e1297f-kube-api-access-8cw4t\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-001a96-29889690-j9qwv\" (UID: \"f3c414cb-e4f0-4fbf-9bdf-946842e1297f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-001a96-29889690-j9qwv" Oct 30 17:30:00.781699 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.781676 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889690-s8ggr" Oct 30 17:30:00.783224 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.783183 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889690-7kjml" Oct 30 17:30:00.793262 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.793037 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-001a96-29889690-j9qwv" Oct 30 17:30:00.832635 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.832600 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-987dfc-29889690-vpglf"] Oct 30 17:30:00.891728 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:30:00.891650 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8643b717_c5ec_4194_92bb_e54de07fef00.slice/crio-dacb3345d05247a50f48623c812290949a69b273eff4dc01f3fe5e968e458ac8 WatchSource:0}: Error finding container dacb3345d05247a50f48623c812290949a69b273eff4dc01f3fe5e968e458ac8: Status 404 returned error can't find the container with id dacb3345d05247a50f48623c812290949a69b273eff4dc01f3fe5e968e458ac8 Oct 30 17:30:00.900935 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.900696 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c17e8c-29889690-nvd7z" Oct 30 17:30:00.915513 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.915484 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-469ad7-29889690-cfrlx"] Oct 30 17:30:00.934046 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.928684 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e13a51-29889690-dg6vt" Oct 30 17:30:00.934046 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:00.933863 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889690-csm2k" Oct 30 17:30:01.062455 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:01.061143 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-82243b-29889690-dmgd9"] Oct 30 17:30:01.336408 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:01.334795 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889690-xxcd6"] Oct 30 17:30:01.369640 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:01.369612 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889690-d76sf"] Oct 30 17:30:01.379363 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:01.379205 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889690-qb6xz"] Oct 30 17:30:01.391106 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:01.391080 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d2d985-29889690-vlq47"] Oct 30 17:30:01.407087 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:01.407032 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889690-ksf2d"] Oct 30 17:30:01.413677 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:01.411921 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889690-d76sf" event={"ID":"b49aaf80-dff6-40d4-8404-d55de5b568ae","Type":"ContainerStarted","Data":"06fe34004885f96ec33bf8bf8161a1908a1dfd07c438656534df37a46c1f57be"} Oct 30 17:30:01.414220 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:01.414195 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889690-xxcd6" event={"ID":"e1d79713-dd75-4823-a8f8-0f252befa273","Type":"ContainerStarted","Data":"236eec2f64ac673811dcb9357338c132b028d6fe418ca9c013f591269ab97fda"} Oct 30 17:30:01.417617 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:01.417592 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889690-qb6xz" event={"ID":"58bcea19-790f-486c-8994-21b1b4c20dc9","Type":"ContainerStarted","Data":"1d3b149ca37f1fc8c8b9b72987a865d6faa0aa285d71fd65c6d1dfe59fe27323"} Oct 30 17:30:01.419629 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:01.419602 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-987dfc-29889690-vpglf" event={"ID":"8643b717-c5ec-4194-92bb-e54de07fef00","Type":"ContainerStarted","Data":"dacb3345d05247a50f48623c812290949a69b273eff4dc01f3fe5e968e458ac8"} Oct 30 17:30:01.421246 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:01.421221 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-82243b-29889690-dmgd9" event={"ID":"f6b0d3f6-2109-40b3-b86f-b895a4616558","Type":"ContainerStarted","Data":"762943ceeef17a1b4e12283a92b2790fa0acc76e86b253da7d8cdf610a191d82"} Oct 30 17:30:01.423587 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:01.423566 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-469ad7-29889690-cfrlx" event={"ID":"c977d640-8303-4e86-8952-149525c0708c","Type":"ContainerStarted","Data":"eeea0a3cc1f79ae6dfef84fe9c4ecab12440609bde6030b495717e3edb0527db"} Oct 30 17:30:01.452855 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:01.452826 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889690-7kjml"] Oct 30 17:30:01.458263 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:01.458238 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-001a96-29889690-j9qwv"] Oct 30 17:30:01.462816 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:30:01.462784 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3c414cb_e4f0_4fbf_9bdf_946842e1297f.slice/crio-cdbd0cdf61fd56dc15b734c0d93323b25757767745683db2346033f13aa98f4d WatchSource:0}: Error finding container cdbd0cdf61fd56dc15b734c0d93323b25757767745683db2346033f13aa98f4d: Status 404 returned error can't find the container with id cdbd0cdf61fd56dc15b734c0d93323b25757767745683db2346033f13aa98f4d Oct 30 17:30:01.463736 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:30:01.463710 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2ce4a71_84b5_4817_b99c_9d56d4673869.slice/crio-f415498cde0e270c5754dd1509c440ddca20353ea1fe8c33c48b707f6c0a91e5 WatchSource:0}: Error finding container f415498cde0e270c5754dd1509c440ddca20353ea1fe8c33c48b707f6c0a91e5: Status 404 returned error can't find the container with id f415498cde0e270c5754dd1509c440ddca20353ea1fe8c33c48b707f6c0a91e5 Oct 30 17:30:01.477259 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:30:01.477231 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46fba0fa_d263_42d2_84fd_ddaefea25c63.slice/crio-78966c1cf61cde7444fe08b5e2292924246278b005400960d78fe460df9ffbeb WatchSource:0}: Error finding container 78966c1cf61cde7444fe08b5e2292924246278b005400960d78fe460df9ffbeb: Status 404 returned error can't find the container with id 78966c1cf61cde7444fe08b5e2292924246278b005400960d78fe460df9ffbeb Oct 30 17:30:01.478200 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:01.478174 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889690-s8ggr"] Oct 30 17:30:01.501925 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:01.501855 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889690-csm2k"] Oct 30 17:30:01.502665 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:30:01.502635 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4febc82_5717_4157_b46b_7ad8004d7f6a.slice/crio-8f82066950325ee9e821de864508d3134a8cfd09924dc28b572afc31fdfaccf2 WatchSource:0}: Error finding container 8f82066950325ee9e821de864508d3134a8cfd09924dc28b572afc31fdfaccf2: Status 404 returned error can't find the container with id 8f82066950325ee9e821de864508d3134a8cfd09924dc28b572afc31fdfaccf2 Oct 30 17:30:01.520580 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:01.508679 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e13a51-29889690-dg6vt"] Oct 30 17:30:01.539553 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:01.539518 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c17e8c-29889690-nvd7z"] Oct 30 17:30:01.568343 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:30:01.568299 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod427507de_33c6_409a_a68c_8e5a978b24ed.slice/crio-17a8fe6ae91377a1c207c9a220ea5856f0d905d30cb5a942c09c0563dfbfe966 WatchSource:0}: Error finding container 17a8fe6ae91377a1c207c9a220ea5856f0d905d30cb5a942c09c0563dfbfe966: Status 404 returned error can't find the container with id 17a8fe6ae91377a1c207c9a220ea5856f0d905d30cb5a942c09c0563dfbfe966 Oct 30 17:30:02.437541 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:02.437506 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c17e8c-29889690-nvd7z" event={"ID":"427507de-33c6-409a-a68c-8e5a978b24ed","Type":"ContainerStarted","Data":"7b298573403c175882faa580bc9397fdbc004eabdaf21e7c89c13887434f18e8"} Oct 30 17:30:02.437987 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:02.437559 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c17e8c-29889690-nvd7z" event={"ID":"427507de-33c6-409a-a68c-8e5a978b24ed","Type":"ContainerStarted","Data":"17a8fe6ae91377a1c207c9a220ea5856f0d905d30cb5a942c09c0563dfbfe966"} Oct 30 17:30:02.441076 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:02.440697 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889690-s8ggr" event={"ID":"46fba0fa-d263-42d2-84fd-ddaefea25c63","Type":"ContainerStarted","Data":"1af8fb75eac1439b3d7cba5a593d197daf85ca20212106710a0a4d4a0ad6fcdb"} Oct 30 17:30:02.441076 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:02.440752 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889690-s8ggr" event={"ID":"46fba0fa-d263-42d2-84fd-ddaefea25c63","Type":"ContainerStarted","Data":"78966c1cf61cde7444fe08b5e2292924246278b005400960d78fe460df9ffbeb"} Oct 30 17:30:02.443619 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:02.443585 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-001a96-29889690-j9qwv" event={"ID":"f3c414cb-e4f0-4fbf-9bdf-946842e1297f","Type":"ContainerStarted","Data":"316f6d04ed25e92e37a99d675fd87f92f57ade7219388abf88bd6254f43d9055"} Oct 30 17:30:02.443736 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:02.443652 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-001a96-29889690-j9qwv" event={"ID":"f3c414cb-e4f0-4fbf-9bdf-946842e1297f","Type":"ContainerStarted","Data":"cdbd0cdf61fd56dc15b734c0d93323b25757767745683db2346033f13aa98f4d"} Oct 30 17:30:02.445969 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:02.445894 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e13a51-29889690-dg6vt" event={"ID":"e4febc82-5717-4157-b46b-7ad8004d7f6a","Type":"ContainerStarted","Data":"6e8b6675d0c9ad79aca50d4cd5934e44abaceb4e414016e246c7b277004f080d"} Oct 30 17:30:02.445969 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:02.445938 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e13a51-29889690-dg6vt" event={"ID":"e4febc82-5717-4157-b46b-7ad8004d7f6a","Type":"ContainerStarted","Data":"8f82066950325ee9e821de864508d3134a8cfd09924dc28b572afc31fdfaccf2"} Oct 30 17:30:02.447694 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:02.447668 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889690-d76sf" event={"ID":"b49aaf80-dff6-40d4-8404-d55de5b568ae","Type":"ContainerStarted","Data":"4fa525456d96e94d6839849732eff973c4f74309236ca33894b1a047c23eefa8"} Oct 30 17:30:02.449356 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:02.449332 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889690-xxcd6" event={"ID":"e1d79713-dd75-4823-a8f8-0f252befa273","Type":"ContainerStarted","Data":"a468335392d05d9fb04a46494fa784b0cd298b9e52dd7e8e220b7acb9bbdcbac"} Oct 30 17:30:02.451048 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:02.451028 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889690-ksf2d" event={"ID":"22ffe3de-e5e2-43a1-b276-20cf180c8b01","Type":"ContainerStarted","Data":"55fc1d8d21e93cfc8ca0d01a75501565bfcd3361a1b2a585bb2d3521da175f41"} Oct 30 17:30:02.451146 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:02.451069 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889690-ksf2d" event={"ID":"22ffe3de-e5e2-43a1-b276-20cf180c8b01","Type":"ContainerStarted","Data":"79bfde61a27ec2c20849d366e7defc3a92b9742c62281b9eb791e619ef65482e"} Oct 30 17:30:02.452732 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:02.452708 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889690-7kjml" event={"ID":"e2ce4a71-84b5-4817-b99c-9d56d4673869","Type":"ContainerStarted","Data":"e61593a363f0c42620596d829c7dfc9d11013610443089161b58b4da4c3ba3c0"} Oct 30 17:30:02.452840 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:02.452750 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889690-7kjml" event={"ID":"e2ce4a71-84b5-4817-b99c-9d56d4673869","Type":"ContainerStarted","Data":"f415498cde0e270c5754dd1509c440ddca20353ea1fe8c33c48b707f6c0a91e5"} Oct 30 17:30:02.454494 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:02.454469 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889690-qb6xz" event={"ID":"58bcea19-790f-486c-8994-21b1b4c20dc9","Type":"ContainerStarted","Data":"6eab3817ac9443abfdd6fd8794e158c702f19f4afcf2e22dfa96342323cca2df"} Oct 30 17:30:02.456251 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:02.456231 4522 generic.go:356] "Generic (PLEG): container finished" podID="8643b717-c5ec-4194-92bb-e54de07fef00" containerID="b2dfcfb8bb0997b55eb2eeb415609568b6ea2c861607782127c48789606a64a0" exitCode=60 Oct 30 17:30:02.456360 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:02.456325 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-987dfc-29889690-vpglf" event={"ID":"8643b717-c5ec-4194-92bb-e54de07fef00","Type":"ContainerDied","Data":"b2dfcfb8bb0997b55eb2eeb415609568b6ea2c861607782127c48789606a64a0"} Oct 30 17:30:02.456510 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:02.456494 4522 scope.go:122] "RemoveContainer" containerID="b2dfcfb8bb0997b55eb2eeb415609568b6ea2c861607782127c48789606a64a0" Oct 30 17:30:02.458142 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:02.458040 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-82243b-29889690-dmgd9" event={"ID":"f6b0d3f6-2109-40b3-b86f-b895a4616558","Type":"ContainerStarted","Data":"a8ea17ae6bcee2fceafa3612466b2a8a746c78fc4fe72473bfb7889b4fcb612c"} Oct 30 17:30:02.463618 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:02.463598 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d2d985-29889690-vlq47" event={"ID":"ddc0e647-bfb8-474d-b423-be4d72b31b32","Type":"ContainerStarted","Data":"2e8016a717eaef3d9bcaeeb180d6ec91107d1a56fd0bb753e7751104ddd3b218"} Oct 30 17:30:02.463695 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:02.463634 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d2d985-29889690-vlq47" event={"ID":"ddc0e647-bfb8-474d-b423-be4d72b31b32","Type":"ContainerStarted","Data":"1afcd9d6953183f4e71d5bbfd8b522eb527cec5c9abc3819dd55871fe7c0782c"} Oct 30 17:30:02.465333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:02.465300 4522 generic.go:356] "Generic (PLEG): container finished" podID="c977d640-8303-4e86-8952-149525c0708c" containerID="21bbca36a5f3dd1d44d80928f86ee84f48e67c128c9b10964fa919bbfc6b6173" exitCode=60 Oct 30 17:30:02.465436 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:02.465378 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-469ad7-29889690-cfrlx" event={"ID":"c977d640-8303-4e86-8952-149525c0708c","Type":"ContainerDied","Data":"21bbca36a5f3dd1d44d80928f86ee84f48e67c128c9b10964fa919bbfc6b6173"} Oct 30 17:30:02.465602 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:02.465589 4522 scope.go:122] "RemoveContainer" containerID="21bbca36a5f3dd1d44d80928f86ee84f48e67c128c9b10964fa919bbfc6b6173" Oct 30 17:30:02.466929 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:02.466910 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889690-csm2k" event={"ID":"d4fa40dd-b387-4513-aed9-8e11394d3454","Type":"ContainerStarted","Data":"cb47fc72be34974076f631901d2a22e154c7e41d1093b476f59386a298916204"} Oct 30 17:30:02.467039 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:02.466950 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889690-csm2k" event={"ID":"d4fa40dd-b387-4513-aed9-8e11394d3454","Type":"ContainerStarted","Data":"36728edab92c8c97c05007f642d1959b6079ed8865c7b6fd3e7698c24bbf5ab6"} Oct 30 17:30:02.485691 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:02.485635 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c17e8c-29889690-nvd7z" podStartSLOduration=2.4856188 podStartE2EDuration="2.4856188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:02.482977629 +0000 UTC m=+2576.483732531" watchObservedRunningTime="2026-10-30 17:30:02.4856188 +0000 UTC m=+2576.486373701" Oct 30 17:30:02.528182 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:02.528140 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d2d985-29889690-vlq47" podStartSLOduration=2.52812539 podStartE2EDuration="2.52812539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:02.525992007 +0000 UTC m=+2576.526746929" watchObservedRunningTime="2026-10-30 17:30:02.52812539 +0000 UTC m=+2576.528880292" Oct 30 17:30:02.578595 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:02.578539 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889690-s8ggr" podStartSLOduration=2.578526287 podStartE2EDuration="2.578526287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:02.57297826 +0000 UTC m=+2576.573733162" watchObservedRunningTime="2026-10-30 17:30:02.578526287 +0000 UTC m=+2576.579281212" Oct 30 17:30:02.620821 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:02.620774 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889690-ksf2d" podStartSLOduration=2.620760077 podStartE2EDuration="2.620760077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:02.615282833 +0000 UTC m=+2576.616037733" watchObservedRunningTime="2026-10-30 17:30:02.620760077 +0000 UTC m=+2576.621515005" Oct 30 17:30:02.662509 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:02.662454 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889690-7kjml" podStartSLOduration=2.6624394369999997 podStartE2EDuration="2.662439437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:02.661582234 +0000 UTC m=+2576.662337156" watchObservedRunningTime="2026-10-30 17:30:02.662439437 +0000 UTC m=+2576.663194334" Oct 30 17:30:02.709197 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:02.709135 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889690-csm2k" podStartSLOduration=2.709115916 podStartE2EDuration="2.709115916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:02.707862929 +0000 UTC m=+2576.708617828" watchObservedRunningTime="2026-10-30 17:30:02.709115916 +0000 UTC m=+2576.709870820" Oct 30 17:30:02.805959 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:02.805900 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e13a51-29889690-dg6vt" podStartSLOduration=2.805880565 podStartE2EDuration="2.805880565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:02.805286928 +0000 UTC m=+2576.806041831" watchObservedRunningTime="2026-10-30 17:30:02.805880565 +0000 UTC m=+2576.806635467" Oct 30 17:30:02.851655 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:02.851602 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889690-d76sf" podStartSLOduration=2.851586762 podStartE2EDuration="2.851586762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:02.850962792 +0000 UTC m=+2576.851717692" watchObservedRunningTime="2026-10-30 17:30:02.851586762 +0000 UTC m=+2576.852341663" Oct 30 17:30:02.903955 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:02.903088 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-82243b-29889690-dmgd9" podStartSLOduration=2.903068953 podStartE2EDuration="2.903068953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:02.896185181 +0000 UTC m=+2576.896940100" watchObservedRunningTime="2026-10-30 17:30:02.903068953 +0000 UTC m=+2576.903823859" Oct 30 17:30:02.943851 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:02.943797 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889690-qb6xz" podStartSLOduration=2.9437813779999997 podStartE2EDuration="2.943781378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:02.940254247 +0000 UTC m=+2576.941009175" watchObservedRunningTime="2026-10-30 17:30:02.943781378 +0000 UTC m=+2576.944536340" Oct 30 17:30:03.052014 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:03.051957 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-001a96-29889690-j9qwv" podStartSLOduration=3.051943362 podStartE2EDuration="3.051943362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:03.047718848 +0000 UTC m=+2577.048473748" watchObservedRunningTime="2026-10-30 17:30:03.051943362 +0000 UTC m=+2577.052698273" Oct 30 17:30:03.091971 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:03.091910 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889690-xxcd6" podStartSLOduration=3.091896101 podStartE2EDuration="3.091896101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:03.09041164 +0000 UTC m=+2577.091166560" watchObservedRunningTime="2026-10-30 17:30:03.091896101 +0000 UTC m=+2577.092651003" Oct 30 17:30:03.481668 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:03.481637 4522 generic.go:356] "Generic (PLEG): container finished" podID="f6b0d3f6-2109-40b3-b86f-b895a4616558" containerID="a8ea17ae6bcee2fceafa3612466b2a8a746c78fc4fe72473bfb7889b4fcb612c" exitCode=60 Oct 30 17:30:03.482104 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:03.481720 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-82243b-29889690-dmgd9" event={"ID":"f6b0d3f6-2109-40b3-b86f-b895a4616558","Type":"ContainerDied","Data":"a8ea17ae6bcee2fceafa3612466b2a8a746c78fc4fe72473bfb7889b4fcb612c"} Oct 30 17:30:03.482104 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:03.482065 4522 scope.go:122] "RemoveContainer" containerID="a8ea17ae6bcee2fceafa3612466b2a8a746c78fc4fe72473bfb7889b4fcb612c" Oct 30 17:30:03.483766 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:03.483741 4522 generic.go:356] "Generic (PLEG): container finished" podID="ddc0e647-bfb8-474d-b423-be4d72b31b32" containerID="2e8016a717eaef3d9bcaeeb180d6ec91107d1a56fd0bb753e7751104ddd3b218" exitCode=60 Oct 30 17:30:03.483907 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:03.483830 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d2d985-29889690-vlq47" event={"ID":"ddc0e647-bfb8-474d-b423-be4d72b31b32","Type":"ContainerDied","Data":"2e8016a717eaef3d9bcaeeb180d6ec91107d1a56fd0bb753e7751104ddd3b218"} Oct 30 17:30:03.484177 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:03.484115 4522 scope.go:122] "RemoveContainer" containerID="2e8016a717eaef3d9bcaeeb180d6ec91107d1a56fd0bb753e7751104ddd3b218" Oct 30 17:30:03.486465 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:03.486445 4522 generic.go:356] "Generic (PLEG): container finished" podID="d4fa40dd-b387-4513-aed9-8e11394d3454" containerID="cb47fc72be34974076f631901d2a22e154c7e41d1093b476f59386a298916204" exitCode=60 Oct 30 17:30:03.486942 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:03.486925 4522 scope.go:122] "RemoveContainer" containerID="cb47fc72be34974076f631901d2a22e154c7e41d1093b476f59386a298916204" Oct 30 17:30:03.487014 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:03.486965 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889690-csm2k" event={"ID":"d4fa40dd-b387-4513-aed9-8e11394d3454","Type":"ContainerDied","Data":"cb47fc72be34974076f631901d2a22e154c7e41d1093b476f59386a298916204"} Oct 30 17:30:03.489165 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:03.489142 4522 generic.go:356] "Generic (PLEG): container finished" podID="427507de-33c6-409a-a68c-8e5a978b24ed" containerID="7b298573403c175882faa580bc9397fdbc004eabdaf21e7c89c13887434f18e8" exitCode=60 Oct 30 17:30:03.489278 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:03.489223 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c17e8c-29889690-nvd7z" event={"ID":"427507de-33c6-409a-a68c-8e5a978b24ed","Type":"ContainerDied","Data":"7b298573403c175882faa580bc9397fdbc004eabdaf21e7c89c13887434f18e8"} Oct 30 17:30:03.489503 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:03.489485 4522 scope.go:122] "RemoveContainer" containerID="7b298573403c175882faa580bc9397fdbc004eabdaf21e7c89c13887434f18e8" Oct 30 17:30:03.491383 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:03.491292 4522 generic.go:356] "Generic (PLEG): container finished" podID="46fba0fa-d263-42d2-84fd-ddaefea25c63" containerID="1af8fb75eac1439b3d7cba5a593d197daf85ca20212106710a0a4d4a0ad6fcdb" exitCode=60 Oct 30 17:30:03.491487 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:03.491350 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889690-s8ggr" event={"ID":"46fba0fa-d263-42d2-84fd-ddaefea25c63","Type":"ContainerDied","Data":"1af8fb75eac1439b3d7cba5a593d197daf85ca20212106710a0a4d4a0ad6fcdb"} Oct 30 17:30:03.491614 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:03.491596 4522 scope.go:122] "RemoveContainer" containerID="1af8fb75eac1439b3d7cba5a593d197daf85ca20212106710a0a4d4a0ad6fcdb" Oct 30 17:30:03.496362 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:03.496340 4522 generic.go:356] "Generic (PLEG): container finished" podID="f3c414cb-e4f0-4fbf-9bdf-946842e1297f" containerID="316f6d04ed25e92e37a99d675fd87f92f57ade7219388abf88bd6254f43d9055" exitCode=60 Oct 30 17:30:03.496601 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:03.496376 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-001a96-29889690-j9qwv" event={"ID":"f3c414cb-e4f0-4fbf-9bdf-946842e1297f","Type":"ContainerDied","Data":"316f6d04ed25e92e37a99d675fd87f92f57ade7219388abf88bd6254f43d9055"} Oct 30 17:30:03.496724 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:03.496690 4522 scope.go:122] "RemoveContainer" containerID="316f6d04ed25e92e37a99d675fd87f92f57ade7219388abf88bd6254f43d9055" Oct 30 17:30:03.498922 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:03.498903 4522 generic.go:356] "Generic (PLEG): container finished" podID="e4febc82-5717-4157-b46b-7ad8004d7f6a" containerID="6e8b6675d0c9ad79aca50d4cd5934e44abaceb4e414016e246c7b277004f080d" exitCode=60 Oct 30 17:30:03.499004 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:03.498986 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e13a51-29889690-dg6vt" event={"ID":"e4febc82-5717-4157-b46b-7ad8004d7f6a","Type":"ContainerDied","Data":"6e8b6675d0c9ad79aca50d4cd5934e44abaceb4e414016e246c7b277004f080d"} Oct 30 17:30:03.499226 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:03.499209 4522 scope.go:122] "RemoveContainer" containerID="6e8b6675d0c9ad79aca50d4cd5934e44abaceb4e414016e246c7b277004f080d" Oct 30 17:30:03.500833 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:03.500816 4522 generic.go:356] "Generic (PLEG): container finished" podID="b49aaf80-dff6-40d4-8404-d55de5b568ae" containerID="4fa525456d96e94d6839849732eff973c4f74309236ca33894b1a047c23eefa8" exitCode=60 Oct 30 17:30:03.500938 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:03.500926 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889690-d76sf" event={"ID":"b49aaf80-dff6-40d4-8404-d55de5b568ae","Type":"ContainerDied","Data":"4fa525456d96e94d6839849732eff973c4f74309236ca33894b1a047c23eefa8"} Oct 30 17:30:03.501130 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:03.501114 4522 scope.go:122] "RemoveContainer" containerID="4fa525456d96e94d6839849732eff973c4f74309236ca33894b1a047c23eefa8" Oct 30 17:30:03.502795 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:03.502772 4522 generic.go:356] "Generic (PLEG): container finished" podID="e1d79713-dd75-4823-a8f8-0f252befa273" containerID="a468335392d05d9fb04a46494fa784b0cd298b9e52dd7e8e220b7acb9bbdcbac" exitCode=60 Oct 30 17:30:03.502897 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:03.502793 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889690-xxcd6" event={"ID":"e1d79713-dd75-4823-a8f8-0f252befa273","Type":"ContainerDied","Data":"a468335392d05d9fb04a46494fa784b0cd298b9e52dd7e8e220b7acb9bbdcbac"} Oct 30 17:30:03.503081 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:03.503063 4522 scope.go:122] "RemoveContainer" containerID="a468335392d05d9fb04a46494fa784b0cd298b9e52dd7e8e220b7acb9bbdcbac" Oct 30 17:30:03.504864 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:03.504845 4522 generic.go:356] "Generic (PLEG): container finished" podID="22ffe3de-e5e2-43a1-b276-20cf180c8b01" containerID="55fc1d8d21e93cfc8ca0d01a75501565bfcd3361a1b2a585bb2d3521da175f41" exitCode=60 Oct 30 17:30:03.504991 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:03.504869 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889690-ksf2d" event={"ID":"22ffe3de-e5e2-43a1-b276-20cf180c8b01","Type":"ContainerDied","Data":"55fc1d8d21e93cfc8ca0d01a75501565bfcd3361a1b2a585bb2d3521da175f41"} Oct 30 17:30:03.505147 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:03.505130 4522 scope.go:122] "RemoveContainer" containerID="55fc1d8d21e93cfc8ca0d01a75501565bfcd3361a1b2a585bb2d3521da175f41" Oct 30 17:30:03.517442 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:03.508751 4522 generic.go:356] "Generic (PLEG): container finished" podID="e2ce4a71-84b5-4817-b99c-9d56d4673869" containerID="e61593a363f0c42620596d829c7dfc9d11013610443089161b58b4da4c3ba3c0" exitCode=60 Oct 30 17:30:03.517442 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:03.508919 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889690-7kjml" event={"ID":"e2ce4a71-84b5-4817-b99c-9d56d4673869","Type":"ContainerDied","Data":"e61593a363f0c42620596d829c7dfc9d11013610443089161b58b4da4c3ba3c0"} Oct 30 17:30:03.517442 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:03.509297 4522 scope.go:122] "RemoveContainer" containerID="e61593a363f0c42620596d829c7dfc9d11013610443089161b58b4da4c3ba3c0" Oct 30 17:30:03.517442 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:03.514712 4522 generic.go:356] "Generic (PLEG): container finished" podID="58bcea19-790f-486c-8994-21b1b4c20dc9" containerID="6eab3817ac9443abfdd6fd8794e158c702f19f4afcf2e22dfa96342323cca2df" exitCode=60 Oct 30 17:30:03.517442 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:03.514780 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889690-qb6xz" event={"ID":"58bcea19-790f-486c-8994-21b1b4c20dc9","Type":"ContainerDied","Data":"6eab3817ac9443abfdd6fd8794e158c702f19f4afcf2e22dfa96342323cca2df"} Oct 30 17:30:03.517442 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:03.515284 4522 scope.go:122] "RemoveContainer" containerID="6eab3817ac9443abfdd6fd8794e158c702f19f4afcf2e22dfa96342323cca2df" Oct 30 17:30:04.527473 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:04.527432 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889690-ksf2d" event={"ID":"22ffe3de-e5e2-43a1-b276-20cf180c8b01","Type":"ContainerStarted","Data":"a8b39d35987154a599beec11a5836636daed52c08dbcb85e9374fa3a678fc446"} Oct 30 17:30:04.529693 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:04.529659 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889690-7kjml" event={"ID":"e2ce4a71-84b5-4817-b99c-9d56d4673869","Type":"ContainerStarted","Data":"d55ced492d901c4202a262b13348021163d3262badf6ee20e85bf87f635effbc"} Oct 30 17:30:04.532198 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:04.532171 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889690-qb6xz" event={"ID":"58bcea19-790f-486c-8994-21b1b4c20dc9","Type":"ContainerStarted","Data":"d95088e00c7af151fd2607ca2473d007fd1ff4cbe931c64c641a6a35de61ad01"} Oct 30 17:30:04.534453 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:04.534428 4522 generic.go:356] "Generic (PLEG): container finished" podID="8643b717-c5ec-4194-92bb-e54de07fef00" containerID="734543e7d7f12c02b5b6dcf676c27ea7d207c0d705630b1aa03e034933faf934" exitCode=60 Oct 30 17:30:04.534578 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:04.534552 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-987dfc-29889690-vpglf" event={"ID":"8643b717-c5ec-4194-92bb-e54de07fef00","Type":"ContainerDied","Data":"734543e7d7f12c02b5b6dcf676c27ea7d207c0d705630b1aa03e034933faf934"} Oct 30 17:30:04.534623 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:04.534595 4522 scope.go:122] "RemoveContainer" containerID="b2dfcfb8bb0997b55eb2eeb415609568b6ea2c861607782127c48789606a64a0" Oct 30 17:30:04.534762 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:04.534731 4522 scope.go:122] "RemoveContainer" containerID="734543e7d7f12c02b5b6dcf676c27ea7d207c0d705630b1aa03e034933faf934" Oct 30 17:30:04.535029 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:30:04.535010 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-987dfc-29889690-vpglf_odh-ai-gateway-infra(8643b717-c5ec-4194-92bb-e54de07fef00)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-987dfc-29889690-vpglf" podUID="8643b717-c5ec-4194-92bb-e54de07fef00" Oct 30 17:30:04.537015 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:04.536994 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-82243b-29889690-dmgd9" event={"ID":"f6b0d3f6-2109-40b3-b86f-b895a4616558","Type":"ContainerStarted","Data":"b46ac2d32779d757aa0dbdfcf22ba42e8a11531ad667a0d511e37ec3f2554be1"} Oct 30 17:30:04.539104 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:04.539082 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d2d985-29889690-vlq47" event={"ID":"ddc0e647-bfb8-474d-b423-be4d72b31b32","Type":"ContainerStarted","Data":"9669c5968e097f64d7c9ea61ac6cb237243516f6758676238ab92f34e302d1b2"} Oct 30 17:30:04.541376 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:04.541353 4522 generic.go:356] "Generic (PLEG): container finished" podID="c977d640-8303-4e86-8952-149525c0708c" containerID="639c74b533eca6c9c1402f41ecd2d3621675ac9b7f6cf085df50ec136d894c5e" exitCode=60 Oct 30 17:30:04.541470 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:04.541395 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-469ad7-29889690-cfrlx" event={"ID":"c977d640-8303-4e86-8952-149525c0708c","Type":"ContainerDied","Data":"639c74b533eca6c9c1402f41ecd2d3621675ac9b7f6cf085df50ec136d894c5e"} Oct 30 17:30:04.541662 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:04.541643 4522 scope.go:122] "RemoveContainer" containerID="639c74b533eca6c9c1402f41ecd2d3621675ac9b7f6cf085df50ec136d894c5e" Oct 30 17:30:04.541941 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:30:04.541918 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-469ad7-29889690-cfrlx_odh-ai-gateway-infra(c977d640-8303-4e86-8952-149525c0708c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-469ad7-29889690-cfrlx" podUID="c977d640-8303-4e86-8952-149525c0708c" Oct 30 17:30:04.543523 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:04.543504 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889690-csm2k" event={"ID":"d4fa40dd-b387-4513-aed9-8e11394d3454","Type":"ContainerStarted","Data":"0c79a97f55e118fb1ec135a8bc0111d5de0b1e79eb6fb64a34fbae87378035de"} Oct 30 17:30:04.545597 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:04.545567 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c17e8c-29889690-nvd7z" event={"ID":"427507de-33c6-409a-a68c-8e5a978b24ed","Type":"ContainerStarted","Data":"b7879c13a8c6607368d35f8cded5358593090fb7c14c2151c205b6e0028589fb"} Oct 30 17:30:04.547991 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:04.547971 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889690-s8ggr" event={"ID":"46fba0fa-d263-42d2-84fd-ddaefea25c63","Type":"ContainerStarted","Data":"c2cae4cd665f06504d1e6b45e6b19187e916d866f395b73038e5cece7776c751"} Oct 30 17:30:04.550340 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:04.550285 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-001a96-29889690-j9qwv" event={"ID":"f3c414cb-e4f0-4fbf-9bdf-946842e1297f","Type":"ContainerStarted","Data":"d0205f49f80bb681a77c60dfb1ead99c4685ddea5bb89e926dc7888f30b2c2f1"} Oct 30 17:30:04.552537 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:04.552510 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e13a51-29889690-dg6vt" event={"ID":"e4febc82-5717-4157-b46b-7ad8004d7f6a","Type":"ContainerStarted","Data":"2ba7c2777a6beecc4d3585d7275be8aed4e341dfa09df8ce1a4d275ea1e00a20"} Oct 30 17:30:04.553080 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:04.553062 4522 scope.go:122] "RemoveContainer" containerID="21bbca36a5f3dd1d44d80928f86ee84f48e67c128c9b10964fa919bbfc6b6173" Oct 30 17:30:04.554850 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:04.554821 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889690-d76sf" event={"ID":"b49aaf80-dff6-40d4-8404-d55de5b568ae","Type":"ContainerStarted","Data":"7a757f64fb31fc0b1c5ee48e7b995d9c4bf9e3d14b101a880e406f022eb1fa35"} Oct 30 17:30:04.557046 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:04.557024 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889690-xxcd6" event={"ID":"e1d79713-dd75-4823-a8f8-0f252befa273","Type":"ContainerStarted","Data":"cfe806aa8a857e31dc2098e563c375de1f5fa3a48ab64db2708255ea77b5e03a"} Oct 30 17:30:05.569337 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:05.569207 4522 generic.go:356] "Generic (PLEG): container finished" podID="22ffe3de-e5e2-43a1-b276-20cf180c8b01" containerID="a8b39d35987154a599beec11a5836636daed52c08dbcb85e9374fa3a678fc446" exitCode=60 Oct 30 17:30:05.569337 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:05.569291 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889690-ksf2d" event={"ID":"22ffe3de-e5e2-43a1-b276-20cf180c8b01","Type":"ContainerDied","Data":"a8b39d35987154a599beec11a5836636daed52c08dbcb85e9374fa3a678fc446"} Oct 30 17:30:05.569895 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:05.569404 4522 scope.go:122] "RemoveContainer" containerID="55fc1d8d21e93cfc8ca0d01a75501565bfcd3361a1b2a585bb2d3521da175f41" Oct 30 17:30:05.569895 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:05.569766 4522 scope.go:122] "RemoveContainer" containerID="a8b39d35987154a599beec11a5836636daed52c08dbcb85e9374fa3a678fc446" Oct 30 17:30:05.570200 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:30:05.570179 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889690-ksf2d_odh-ai-gateway-infra(22ffe3de-e5e2-43a1-b276-20cf180c8b01)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889690-ksf2d" podUID="22ffe3de-e5e2-43a1-b276-20cf180c8b01" Oct 30 17:30:05.571554 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:05.571532 4522 generic.go:356] "Generic (PLEG): container finished" podID="e2ce4a71-84b5-4817-b99c-9d56d4673869" containerID="d55ced492d901c4202a262b13348021163d3262badf6ee20e85bf87f635effbc" exitCode=60 Oct 30 17:30:05.571659 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:05.571617 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889690-7kjml" event={"ID":"e2ce4a71-84b5-4817-b99c-9d56d4673869","Type":"ContainerDied","Data":"d55ced492d901c4202a262b13348021163d3262badf6ee20e85bf87f635effbc"} Oct 30 17:30:05.571908 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:05.571894 4522 scope.go:122] "RemoveContainer" containerID="d55ced492d901c4202a262b13348021163d3262badf6ee20e85bf87f635effbc" Oct 30 17:30:05.572319 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:30:05.572280 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889690-7kjml_odh-ai-gateway-infra(e2ce4a71-84b5-4817-b99c-9d56d4673869)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889690-7kjml" podUID="e2ce4a71-84b5-4817-b99c-9d56d4673869" Oct 30 17:30:05.573655 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:05.573635 4522 generic.go:356] "Generic (PLEG): container finished" podID="58bcea19-790f-486c-8994-21b1b4c20dc9" containerID="d95088e00c7af151fd2607ca2473d007fd1ff4cbe931c64c641a6a35de61ad01" exitCode=60 Oct 30 17:30:05.573743 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:05.573666 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889690-qb6xz" event={"ID":"58bcea19-790f-486c-8994-21b1b4c20dc9","Type":"ContainerDied","Data":"d95088e00c7af151fd2607ca2473d007fd1ff4cbe931c64c641a6a35de61ad01"} Oct 30 17:30:05.573954 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:05.573938 4522 scope.go:122] "RemoveContainer" containerID="d95088e00c7af151fd2607ca2473d007fd1ff4cbe931c64c641a6a35de61ad01" Oct 30 17:30:05.574237 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:30:05.574214 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889690-qb6xz_odh-ai-gateway-infra(58bcea19-790f-486c-8994-21b1b4c20dc9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889690-qb6xz" podUID="58bcea19-790f-486c-8994-21b1b4c20dc9" Oct 30 17:30:05.575753 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:05.575734 4522 scope.go:122] "RemoveContainer" containerID="734543e7d7f12c02b5b6dcf676c27ea7d207c0d705630b1aa03e034933faf934" Oct 30 17:30:05.576012 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:30:05.575990 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-987dfc-29889690-vpglf_odh-ai-gateway-infra(8643b717-c5ec-4194-92bb-e54de07fef00)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-987dfc-29889690-vpglf" podUID="8643b717-c5ec-4194-92bb-e54de07fef00" Oct 30 17:30:05.577692 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:05.577670 4522 generic.go:356] "Generic (PLEG): container finished" podID="f6b0d3f6-2109-40b3-b86f-b895a4616558" containerID="b46ac2d32779d757aa0dbdfcf22ba42e8a11531ad667a0d511e37ec3f2554be1" exitCode=60 Oct 30 17:30:05.577780 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:05.577713 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-82243b-29889690-dmgd9" event={"ID":"f6b0d3f6-2109-40b3-b86f-b895a4616558","Type":"ContainerDied","Data":"b46ac2d32779d757aa0dbdfcf22ba42e8a11531ad667a0d511e37ec3f2554be1"} Oct 30 17:30:05.578248 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:05.578009 4522 scope.go:122] "RemoveContainer" containerID="b46ac2d32779d757aa0dbdfcf22ba42e8a11531ad667a0d511e37ec3f2554be1" Oct 30 17:30:05.578358 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:30:05.578318 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-82243b-29889690-dmgd9_odh-ai-gateway-infra(f6b0d3f6-2109-40b3-b86f-b895a4616558)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-82243b-29889690-dmgd9" podUID="f6b0d3f6-2109-40b3-b86f-b895a4616558" Oct 30 17:30:05.579845 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:05.579813 4522 generic.go:356] "Generic (PLEG): container finished" podID="ddc0e647-bfb8-474d-b423-be4d72b31b32" containerID="9669c5968e097f64d7c9ea61ac6cb237243516f6758676238ab92f34e302d1b2" exitCode=60 Oct 30 17:30:05.579952 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:05.579892 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d2d985-29889690-vlq47" event={"ID":"ddc0e647-bfb8-474d-b423-be4d72b31b32","Type":"ContainerDied","Data":"9669c5968e097f64d7c9ea61ac6cb237243516f6758676238ab92f34e302d1b2"} Oct 30 17:30:05.580102 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:05.580085 4522 scope.go:122] "RemoveContainer" containerID="9669c5968e097f64d7c9ea61ac6cb237243516f6758676238ab92f34e302d1b2" Oct 30 17:30:05.580397 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:30:05.580365 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-d2d985-29889690-vlq47_odh-ai-gateway-infra(ddc0e647-bfb8-474d-b423-be4d72b31b32)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d2d985-29889690-vlq47" podUID="ddc0e647-bfb8-474d-b423-be4d72b31b32" Oct 30 17:30:05.581839 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:05.581819 4522 scope.go:122] "RemoveContainer" containerID="639c74b533eca6c9c1402f41ecd2d3621675ac9b7f6cf085df50ec136d894c5e" Oct 30 17:30:05.582102 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:30:05.582080 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-469ad7-29889690-cfrlx_odh-ai-gateway-infra(c977d640-8303-4e86-8952-149525c0708c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-469ad7-29889690-cfrlx" podUID="c977d640-8303-4e86-8952-149525c0708c" Oct 30 17:30:05.583626 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:05.583609 4522 generic.go:356] "Generic (PLEG): container finished" podID="d4fa40dd-b387-4513-aed9-8e11394d3454" containerID="0c79a97f55e118fb1ec135a8bc0111d5de0b1e79eb6fb64a34fbae87378035de" exitCode=60 Oct 30 17:30:05.583723 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:05.583688 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889690-csm2k" event={"ID":"d4fa40dd-b387-4513-aed9-8e11394d3454","Type":"ContainerDied","Data":"0c79a97f55e118fb1ec135a8bc0111d5de0b1e79eb6fb64a34fbae87378035de"} Oct 30 17:30:05.583872 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:05.583853 4522 scope.go:122] "RemoveContainer" containerID="0c79a97f55e118fb1ec135a8bc0111d5de0b1e79eb6fb64a34fbae87378035de" Oct 30 17:30:05.584097 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:30:05.584080 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-096d75-29889690-csm2k_odh-ai-gateway-infra(d4fa40dd-b387-4513-aed9-8e11394d3454)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889690-csm2k" podUID="d4fa40dd-b387-4513-aed9-8e11394d3454" Oct 30 17:30:05.591439 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:05.591418 4522 scope.go:122] "RemoveContainer" containerID="e61593a363f0c42620596d829c7dfc9d11013610443089161b58b4da4c3ba3c0" Oct 30 17:30:05.591539 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:05.591512 4522 generic.go:356] "Generic (PLEG): container finished" podID="427507de-33c6-409a-a68c-8e5a978b24ed" containerID="b7879c13a8c6607368d35f8cded5358593090fb7c14c2151c205b6e0028589fb" exitCode=60 Oct 30 17:30:05.591689 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:05.591631 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c17e8c-29889690-nvd7z" event={"ID":"427507de-33c6-409a-a68c-8e5a978b24ed","Type":"ContainerDied","Data":"b7879c13a8c6607368d35f8cded5358593090fb7c14c2151c205b6e0028589fb"} Oct 30 17:30:05.592044 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:05.591950 4522 scope.go:122] "RemoveContainer" containerID="b7879c13a8c6607368d35f8cded5358593090fb7c14c2151c205b6e0028589fb" Oct 30 17:30:05.592353 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:30:05.592325 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-c17e8c-29889690-nvd7z_odh-ai-gateway-infra(427507de-33c6-409a-a68c-8e5a978b24ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c17e8c-29889690-nvd7z" podUID="427507de-33c6-409a-a68c-8e5a978b24ed" Oct 30 17:30:05.594829 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:05.594810 4522 generic.go:356] "Generic (PLEG): container finished" podID="46fba0fa-d263-42d2-84fd-ddaefea25c63" containerID="c2cae4cd665f06504d1e6b45e6b19187e916d866f395b73038e5cece7776c751" exitCode=60 Oct 30 17:30:05.594919 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:05.594839 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889690-s8ggr" event={"ID":"46fba0fa-d263-42d2-84fd-ddaefea25c63","Type":"ContainerDied","Data":"c2cae4cd665f06504d1e6b45e6b19187e916d866f395b73038e5cece7776c751"} Oct 30 17:30:05.595179 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:05.595165 4522 scope.go:122] "RemoveContainer" containerID="c2cae4cd665f06504d1e6b45e6b19187e916d866f395b73038e5cece7776c751" Oct 30 17:30:05.595512 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:30:05.595488 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b80d83-29889690-s8ggr_odh-ai-gateway-infra(46fba0fa-d263-42d2-84fd-ddaefea25c63)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889690-s8ggr" podUID="46fba0fa-d263-42d2-84fd-ddaefea25c63" Oct 30 17:30:05.596871 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:05.596852 4522 generic.go:356] "Generic (PLEG): container finished" podID="f3c414cb-e4f0-4fbf-9bdf-946842e1297f" containerID="d0205f49f80bb681a77c60dfb1ead99c4685ddea5bb89e926dc7888f30b2c2f1" exitCode=60 Oct 30 17:30:05.596978 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:05.596927 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-001a96-29889690-j9qwv" event={"ID":"f3c414cb-e4f0-4fbf-9bdf-946842e1297f","Type":"ContainerDied","Data":"d0205f49f80bb681a77c60dfb1ead99c4685ddea5bb89e926dc7888f30b2c2f1"} Oct 30 17:30:05.597179 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:05.597161 4522 scope.go:122] "RemoveContainer" containerID="d0205f49f80bb681a77c60dfb1ead99c4685ddea5bb89e926dc7888f30b2c2f1" Oct 30 17:30:05.597472 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:30:05.597450 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-001a96-29889690-j9qwv_odh-ai-gateway-infra(f3c414cb-e4f0-4fbf-9bdf-946842e1297f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-001a96-29889690-j9qwv" podUID="f3c414cb-e4f0-4fbf-9bdf-946842e1297f" Oct 30 17:30:05.599030 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:05.599010 4522 generic.go:356] "Generic (PLEG): container finished" podID="e4febc82-5717-4157-b46b-7ad8004d7f6a" containerID="2ba7c2777a6beecc4d3585d7275be8aed4e341dfa09df8ce1a4d275ea1e00a20" exitCode=60 Oct 30 17:30:05.599125 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:05.599080 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e13a51-29889690-dg6vt" event={"ID":"e4febc82-5717-4157-b46b-7ad8004d7f6a","Type":"ContainerDied","Data":"2ba7c2777a6beecc4d3585d7275be8aed4e341dfa09df8ce1a4d275ea1e00a20"} Oct 30 17:30:05.599350 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:05.599295 4522 scope.go:122] "RemoveContainer" containerID="2ba7c2777a6beecc4d3585d7275be8aed4e341dfa09df8ce1a4d275ea1e00a20" Oct 30 17:30:05.599597 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:30:05.599572 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-e13a51-29889690-dg6vt_odh-ai-gateway-infra(e4febc82-5717-4157-b46b-7ad8004d7f6a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e13a51-29889690-dg6vt" podUID="e4febc82-5717-4157-b46b-7ad8004d7f6a" Oct 30 17:30:05.601160 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:05.601142 4522 generic.go:356] "Generic (PLEG): container finished" podID="b49aaf80-dff6-40d4-8404-d55de5b568ae" containerID="7a757f64fb31fc0b1c5ee48e7b995d9c4bf9e3d14b101a880e406f022eb1fa35" exitCode=60 Oct 30 17:30:05.601248 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:05.601173 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889690-d76sf" event={"ID":"b49aaf80-dff6-40d4-8404-d55de5b568ae","Type":"ContainerDied","Data":"7a757f64fb31fc0b1c5ee48e7b995d9c4bf9e3d14b101a880e406f022eb1fa35"} Oct 30 17:30:05.601495 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:05.601478 4522 scope.go:122] "RemoveContainer" containerID="7a757f64fb31fc0b1c5ee48e7b995d9c4bf9e3d14b101a880e406f022eb1fa35" Oct 30 17:30:05.601757 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:30:05.601738 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-04248e-29889690-d76sf_odh-ai-gateway-infra(b49aaf80-dff6-40d4-8404-d55de5b568ae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889690-d76sf" podUID="b49aaf80-dff6-40d4-8404-d55de5b568ae" Oct 30 17:30:05.603302 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:05.603285 4522 generic.go:356] "Generic (PLEG): container finished" podID="e1d79713-dd75-4823-a8f8-0f252befa273" containerID="cfe806aa8a857e31dc2098e563c375de1f5fa3a48ab64db2708255ea77b5e03a" exitCode=60 Oct 30 17:30:05.603445 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:05.603356 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889690-xxcd6" event={"ID":"e1d79713-dd75-4823-a8f8-0f252befa273","Type":"ContainerDied","Data":"cfe806aa8a857e31dc2098e563c375de1f5fa3a48ab64db2708255ea77b5e03a"} Oct 30 17:30:05.603849 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:05.603833 4522 scope.go:122] "RemoveContainer" containerID="cfe806aa8a857e31dc2098e563c375de1f5fa3a48ab64db2708255ea77b5e03a" Oct 30 17:30:05.604112 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:30:05.604093 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-d384dc-29889690-xxcd6_odh-ai-gateway-infra(e1d79713-dd75-4823-a8f8-0f252befa273)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889690-xxcd6" podUID="e1d79713-dd75-4823-a8f8-0f252befa273" Oct 30 17:30:05.610093 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:05.610072 4522 scope.go:122] "RemoveContainer" containerID="6eab3817ac9443abfdd6fd8794e158c702f19f4afcf2e22dfa96342323cca2df" Oct 30 17:30:05.626551 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:05.626532 4522 scope.go:122] "RemoveContainer" containerID="a8ea17ae6bcee2fceafa3612466b2a8a746c78fc4fe72473bfb7889b4fcb612c" Oct 30 17:30:05.643887 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:05.643867 4522 scope.go:122] "RemoveContainer" containerID="2e8016a717eaef3d9bcaeeb180d6ec91107d1a56fd0bb753e7751104ddd3b218" Oct 30 17:30:05.659282 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:05.659259 4522 scope.go:122] "RemoveContainer" containerID="cb47fc72be34974076f631901d2a22e154c7e41d1093b476f59386a298916204" Oct 30 17:30:05.677908 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:05.677886 4522 scope.go:122] "RemoveContainer" containerID="7b298573403c175882faa580bc9397fdbc004eabdaf21e7c89c13887434f18e8" Oct 30 17:30:05.696970 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:05.696947 4522 scope.go:122] "RemoveContainer" containerID="1af8fb75eac1439b3d7cba5a593d197daf85ca20212106710a0a4d4a0ad6fcdb" Oct 30 17:30:05.713073 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:05.713053 4522 scope.go:122] "RemoveContainer" containerID="316f6d04ed25e92e37a99d675fd87f92f57ade7219388abf88bd6254f43d9055" Oct 30 17:30:05.727992 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:05.727971 4522 scope.go:122] "RemoveContainer" containerID="6e8b6675d0c9ad79aca50d4cd5934e44abaceb4e414016e246c7b277004f080d" Oct 30 17:30:05.743774 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:05.743747 4522 scope.go:122] "RemoveContainer" containerID="4fa525456d96e94d6839849732eff973c4f74309236ca33894b1a047c23eefa8" Oct 30 17:30:05.760481 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:05.760451 4522 scope.go:122] "RemoveContainer" containerID="a468335392d05d9fb04a46494fa784b0cd298b9e52dd7e8e220b7acb9bbdcbac" Oct 30 17:30:06.616376 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:06.616344 4522 scope.go:122] "RemoveContainer" containerID="b46ac2d32779d757aa0dbdfcf22ba42e8a11531ad667a0d511e37ec3f2554be1" Oct 30 17:30:06.616870 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:30:06.616786 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-82243b-29889690-dmgd9_odh-ai-gateway-infra(f6b0d3f6-2109-40b3-b86f-b895a4616558)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-82243b-29889690-dmgd9" podUID="f6b0d3f6-2109-40b3-b86f-b895a4616558" Oct 30 17:30:06.618201 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:06.618169 4522 scope.go:122] "RemoveContainer" containerID="9669c5968e097f64d7c9ea61ac6cb237243516f6758676238ab92f34e302d1b2" Oct 30 17:30:06.618626 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:30:06.618597 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-d2d985-29889690-vlq47_odh-ai-gateway-infra(ddc0e647-bfb8-474d-b423-be4d72b31b32)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d2d985-29889690-vlq47" podUID="ddc0e647-bfb8-474d-b423-be4d72b31b32" Oct 30 17:30:06.619904 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:06.619885 4522 scope.go:122] "RemoveContainer" containerID="0c79a97f55e118fb1ec135a8bc0111d5de0b1e79eb6fb64a34fbae87378035de" Oct 30 17:30:06.620219 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:30:06.620197 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-096d75-29889690-csm2k_odh-ai-gateway-infra(d4fa40dd-b387-4513-aed9-8e11394d3454)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889690-csm2k" podUID="d4fa40dd-b387-4513-aed9-8e11394d3454" Oct 30 17:30:06.621587 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:06.621565 4522 scope.go:122] "RemoveContainer" containerID="b7879c13a8c6607368d35f8cded5358593090fb7c14c2151c205b6e0028589fb" Oct 30 17:30:06.621867 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:30:06.621844 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-c17e8c-29889690-nvd7z_odh-ai-gateway-infra(427507de-33c6-409a-a68c-8e5a978b24ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c17e8c-29889690-nvd7z" podUID="427507de-33c6-409a-a68c-8e5a978b24ed" Oct 30 17:30:06.623280 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:06.623256 4522 scope.go:122] "RemoveContainer" containerID="c2cae4cd665f06504d1e6b45e6b19187e916d866f395b73038e5cece7776c751" Oct 30 17:30:06.623601 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:30:06.623576 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b80d83-29889690-s8ggr_odh-ai-gateway-infra(46fba0fa-d263-42d2-84fd-ddaefea25c63)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889690-s8ggr" podUID="46fba0fa-d263-42d2-84fd-ddaefea25c63" Oct 30 17:30:06.624947 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:06.624924 4522 scope.go:122] "RemoveContainer" containerID="d0205f49f80bb681a77c60dfb1ead99c4685ddea5bb89e926dc7888f30b2c2f1" Oct 30 17:30:06.625186 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:30:06.625154 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-001a96-29889690-j9qwv_odh-ai-gateway-infra(f3c414cb-e4f0-4fbf-9bdf-946842e1297f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-001a96-29889690-j9qwv" podUID="f3c414cb-e4f0-4fbf-9bdf-946842e1297f" Oct 30 17:30:06.632283 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:06.632262 4522 scope.go:122] "RemoveContainer" containerID="2ba7c2777a6beecc4d3585d7275be8aed4e341dfa09df8ce1a4d275ea1e00a20" Oct 30 17:30:06.632585 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:30:06.632561 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-e13a51-29889690-dg6vt_odh-ai-gateway-infra(e4febc82-5717-4157-b46b-7ad8004d7f6a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e13a51-29889690-dg6vt" podUID="e4febc82-5717-4157-b46b-7ad8004d7f6a" Oct 30 17:30:06.634804 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:06.634786 4522 scope.go:122] "RemoveContainer" containerID="7a757f64fb31fc0b1c5ee48e7b995d9c4bf9e3d14b101a880e406f022eb1fa35" Oct 30 17:30:06.635027 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:30:06.635009 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-04248e-29889690-d76sf_odh-ai-gateway-infra(b49aaf80-dff6-40d4-8404-d55de5b568ae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889690-d76sf" podUID="b49aaf80-dff6-40d4-8404-d55de5b568ae" Oct 30 17:30:06.636590 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:06.636571 4522 scope.go:122] "RemoveContainer" containerID="cfe806aa8a857e31dc2098e563c375de1f5fa3a48ab64db2708255ea77b5e03a" Oct 30 17:30:06.636815 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:30:06.636792 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-d384dc-29889690-xxcd6_odh-ai-gateway-infra(e1d79713-dd75-4823-a8f8-0f252befa273)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889690-xxcd6" podUID="e1d79713-dd75-4823-a8f8-0f252befa273" Oct 30 17:30:06.638219 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:06.638204 4522 scope.go:122] "RemoveContainer" containerID="a8b39d35987154a599beec11a5836636daed52c08dbcb85e9374fa3a678fc446" Oct 30 17:30:06.638465 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:30:06.638446 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889690-ksf2d_odh-ai-gateway-infra(22ffe3de-e5e2-43a1-b276-20cf180c8b01)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889690-ksf2d" podUID="22ffe3de-e5e2-43a1-b276-20cf180c8b01" Oct 30 17:30:06.639764 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:06.639746 4522 scope.go:122] "RemoveContainer" containerID="d55ced492d901c4202a262b13348021163d3262badf6ee20e85bf87f635effbc" Oct 30 17:30:06.639967 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:30:06.639947 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889690-7kjml_odh-ai-gateway-infra(e2ce4a71-84b5-4817-b99c-9d56d4673869)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889690-7kjml" podUID="e2ce4a71-84b5-4817-b99c-9d56d4673869" Oct 30 17:30:06.641381 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:06.641364 4522 scope.go:122] "RemoveContainer" containerID="d95088e00c7af151fd2607ca2473d007fd1ff4cbe931c64c641a6a35de61ad01" Oct 30 17:30:06.641599 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:30:06.641581 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889690-qb6xz_odh-ai-gateway-infra(58bcea19-790f-486c-8994-21b1b4c20dc9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889690-qb6xz" podUID="58bcea19-790f-486c-8994-21b1b4c20dc9" Oct 30 17:30:09.785828 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:09.785793 4522 scope.go:122] "RemoveContainer" containerID="ee9f4e7957593b8bf86c4fdf1d77b859eb688bfc941d13eea29a66f418c59acc" Oct 30 17:30:09.797593 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:09.797565 4522 scope.go:122] "RemoveContainer" containerID="7d89b7a3711e881e790e16efb4c3848d3124bd85ef98d31e7167cbf1c3e4e826" Oct 30 17:30:09.808619 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:09.808596 4522 scope.go:122] "RemoveContainer" containerID="fb32183e6794aec6fe7c21ebf4470d6c972c0d7b719617a41a33c3479897ddf6" Oct 30 17:30:12.950927 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:30:12.950888 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:30:14.470567 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:14.470531 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8a7e4a-d55dd8b4d-hd4qr"] Oct 30 17:30:15.137515 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:15.137480 4522 scope.go:122] "RemoveContainer" containerID="639c74b533eca6c9c1402f41ecd2d3621675ac9b7f6cf085df50ec136d894c5e" Oct 30 17:30:15.769678 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:15.769590 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-469ad7-29889690-cfrlx" event={"ID":"c977d640-8303-4e86-8952-149525c0708c","Type":"ContainerStarted","Data":"3adbf5a9a331d35513386217e098fbc11d31a4955707585a03bbcddd26ea2c0c"} Oct 30 17:30:15.816578 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:15.816514 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-469ad7-29889690-cfrlx" podStartSLOduration=15.816494296 podStartE2EDuration="15.816494296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:15.812898397 +0000 UTC m=+2589.813653323" watchObservedRunningTime="2026-10-30 17:30:15.816494296 +0000 UTC m=+2589.817249201" Oct 30 17:30:16.145172 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:16.145143 4522 scope.go:122] "RemoveContainer" containerID="b7879c13a8c6607368d35f8cded5358593090fb7c14c2151c205b6e0028589fb" Oct 30 17:30:16.145437 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:16.145412 4522 scope.go:122] "RemoveContainer" containerID="a8b39d35987154a599beec11a5836636daed52c08dbcb85e9374fa3a678fc446" Oct 30 17:30:16.145752 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:16.145729 4522 scope.go:122] "RemoveContainer" containerID="7a757f64fb31fc0b1c5ee48e7b995d9c4bf9e3d14b101a880e406f022eb1fa35" Oct 30 17:30:16.145950 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:16.145898 4522 scope.go:122] "RemoveContainer" containerID="9669c5968e097f64d7c9ea61ac6cb237243516f6758676238ab92f34e302d1b2" Oct 30 17:30:16.146136 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:16.146117 4522 scope.go:122] "RemoveContainer" containerID="d0205f49f80bb681a77c60dfb1ead99c4685ddea5bb89e926dc7888f30b2c2f1" Oct 30 17:30:16.146207 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:16.146194 4522 scope.go:122] "RemoveContainer" containerID="c2cae4cd665f06504d1e6b45e6b19187e916d866f395b73038e5cece7776c751" Oct 30 17:30:16.781510 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:16.781415 4522 generic.go:356] "Generic (PLEG): container finished" podID="c977d640-8303-4e86-8952-149525c0708c" containerID="3adbf5a9a331d35513386217e098fbc11d31a4955707585a03bbcddd26ea2c0c" exitCode=60 Oct 30 17:30:16.781510 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:16.781491 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-469ad7-29889690-cfrlx" event={"ID":"c977d640-8303-4e86-8952-149525c0708c","Type":"ContainerDied","Data":"3adbf5a9a331d35513386217e098fbc11d31a4955707585a03bbcddd26ea2c0c"} Oct 30 17:30:16.782020 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:16.781564 4522 scope.go:122] "RemoveContainer" containerID="639c74b533eca6c9c1402f41ecd2d3621675ac9b7f6cf085df50ec136d894c5e" Oct 30 17:30:16.782020 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:16.782002 4522 scope.go:122] "RemoveContainer" containerID="3adbf5a9a331d35513386217e098fbc11d31a4955707585a03bbcddd26ea2c0c" Oct 30 17:30:16.782464 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:30:16.782440 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-469ad7-29889690-cfrlx_odh-ai-gateway-infra(c977d640-8303-4e86-8952-149525c0708c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-469ad7-29889690-cfrlx" podUID="c977d640-8303-4e86-8952-149525c0708c" Oct 30 17:30:16.828641 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:16.828609 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-469ad7-29889690-cfrlx"] Oct 30 17:30:17.138360 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:17.138243 4522 scope.go:122] "RemoveContainer" containerID="cfe806aa8a857e31dc2098e563c375de1f5fa3a48ab64db2708255ea77b5e03a" Oct 30 17:30:17.138513 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:17.138423 4522 scope.go:122] "RemoveContainer" containerID="734543e7d7f12c02b5b6dcf676c27ea7d207c0d705630b1aa03e034933faf934" Oct 30 17:30:17.138613 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:17.138590 4522 scope.go:122] "RemoveContainer" containerID="0c79a97f55e118fb1ec135a8bc0111d5de0b1e79eb6fb64a34fbae87378035de" Oct 30 17:30:17.793998 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:17.793966 4522 generic.go:356] "Generic (PLEG): container finished" podID="46fba0fa-d263-42d2-84fd-ddaefea25c63" containerID="c6b1f621b5c1b85a6d1ff8b62741b744f8900ecb180548c4b74b0f06c16dcc54" exitCode=60 Oct 30 17:30:17.794475 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:17.794030 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889690-s8ggr" event={"ID":"46fba0fa-d263-42d2-84fd-ddaefea25c63","Type":"ContainerDied","Data":"c6b1f621b5c1b85a6d1ff8b62741b744f8900ecb180548c4b74b0f06c16dcc54"} Oct 30 17:30:17.794475 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:17.794094 4522 scope.go:122] "RemoveContainer" containerID="c2cae4cd665f06504d1e6b45e6b19187e916d866f395b73038e5cece7776c751" Oct 30 17:30:17.794606 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:17.794574 4522 scope.go:122] "RemoveContainer" containerID="c6b1f621b5c1b85a6d1ff8b62741b744f8900ecb180548c4b74b0f06c16dcc54" Oct 30 17:30:17.794983 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:30:17.794959 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b80d83-29889690-s8ggr_odh-ai-gateway-infra(46fba0fa-d263-42d2-84fd-ddaefea25c63)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889690-s8ggr" podUID="46fba0fa-d263-42d2-84fd-ddaefea25c63" Oct 30 17:30:17.796533 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:17.796511 4522 generic.go:356] "Generic (PLEG): container finished" podID="f3c414cb-e4f0-4fbf-9bdf-946842e1297f" containerID="f963d73fb5012cd07e17c2fd2d7fc2c014dcc09c82f54ee7675301024e3c4f77" exitCode=60 Oct 30 17:30:17.796671 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:17.796552 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-001a96-29889690-j9qwv" event={"ID":"f3c414cb-e4f0-4fbf-9bdf-946842e1297f","Type":"ContainerDied","Data":"f963d73fb5012cd07e17c2fd2d7fc2c014dcc09c82f54ee7675301024e3c4f77"} Oct 30 17:30:17.797010 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:17.796988 4522 scope.go:122] "RemoveContainer" containerID="f963d73fb5012cd07e17c2fd2d7fc2c014dcc09c82f54ee7675301024e3c4f77" Oct 30 17:30:17.797461 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:30:17.797436 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-001a96-29889690-j9qwv_odh-ai-gateway-infra(f3c414cb-e4f0-4fbf-9bdf-946842e1297f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-001a96-29889690-j9qwv" podUID="f3c414cb-e4f0-4fbf-9bdf-946842e1297f" Oct 30 17:30:17.799864 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:17.799818 4522 generic.go:356] "Generic (PLEG): container finished" podID="b49aaf80-dff6-40d4-8404-d55de5b568ae" containerID="525ef17aa201bc4c2a575864b9a7e9ae90281e354311f703cacd50c6a7ecccd3" exitCode=60 Oct 30 17:30:17.799864 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:17.799851 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889690-d76sf" event={"ID":"b49aaf80-dff6-40d4-8404-d55de5b568ae","Type":"ContainerDied","Data":"525ef17aa201bc4c2a575864b9a7e9ae90281e354311f703cacd50c6a7ecccd3"} Oct 30 17:30:17.800373 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:17.800337 4522 scope.go:122] "RemoveContainer" containerID="525ef17aa201bc4c2a575864b9a7e9ae90281e354311f703cacd50c6a7ecccd3" Oct 30 17:30:17.803980 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:17.803004 4522 generic.go:356] "Generic (PLEG): container finished" podID="22ffe3de-e5e2-43a1-b276-20cf180c8b01" containerID="5208c9ea07bc24ca301adc9f68d8e22826daf2e25478049602e6e6cb8df6132a" exitCode=60 Oct 30 17:30:17.803980 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:17.803131 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889690-ksf2d" event={"ID":"22ffe3de-e5e2-43a1-b276-20cf180c8b01","Type":"ContainerDied","Data":"5208c9ea07bc24ca301adc9f68d8e22826daf2e25478049602e6e6cb8df6132a"} Oct 30 17:30:17.805593 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:17.805551 4522 scope.go:122] "RemoveContainer" containerID="5208c9ea07bc24ca301adc9f68d8e22826daf2e25478049602e6e6cb8df6132a" Oct 30 17:30:17.806077 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:30:17.806022 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889690-ksf2d_odh-ai-gateway-infra(22ffe3de-e5e2-43a1-b276-20cf180c8b01)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889690-ksf2d" podUID="22ffe3de-e5e2-43a1-b276-20cf180c8b01" Oct 30 17:30:17.806250 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:30:17.805964 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-04248e-29889690-d76sf_odh-ai-gateway-infra(b49aaf80-dff6-40d4-8404-d55de5b568ae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889690-d76sf" podUID="b49aaf80-dff6-40d4-8404-d55de5b568ae" Oct 30 17:30:17.810137 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:17.809171 4522 generic.go:356] "Generic (PLEG): container finished" podID="ddc0e647-bfb8-474d-b423-be4d72b31b32" containerID="c510e98af1284195dd00b9abfab396973170f52f04262534fe6ab86cddb66f78" exitCode=60 Oct 30 17:30:17.810137 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:17.809254 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d2d985-29889690-vlq47" event={"ID":"ddc0e647-bfb8-474d-b423-be4d72b31b32","Type":"ContainerDied","Data":"c510e98af1284195dd00b9abfab396973170f52f04262534fe6ab86cddb66f78"} Oct 30 17:30:17.810137 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:17.809885 4522 scope.go:122] "RemoveContainer" containerID="c510e98af1284195dd00b9abfab396973170f52f04262534fe6ab86cddb66f78" Oct 30 17:30:17.810458 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:30:17.810390 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-d2d985-29889690-vlq47_odh-ai-gateway-infra(ddc0e647-bfb8-474d-b423-be4d72b31b32)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d2d985-29889690-vlq47" podUID="ddc0e647-bfb8-474d-b423-be4d72b31b32" Oct 30 17:30:17.817687 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:17.817652 4522 generic.go:356] "Generic (PLEG): container finished" podID="427507de-33c6-409a-a68c-8e5a978b24ed" containerID="c12c290c95a06b384c1559d1b33f9a57642148a8f4142009ac2bef79a61c7d01" exitCode=60 Oct 30 17:30:17.817846 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:17.817697 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c17e8c-29889690-nvd7z" event={"ID":"427507de-33c6-409a-a68c-8e5a978b24ed","Type":"ContainerDied","Data":"c12c290c95a06b384c1559d1b33f9a57642148a8f4142009ac2bef79a61c7d01"} Oct 30 17:30:17.818115 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:17.818099 4522 scope.go:122] "RemoveContainer" containerID="c12c290c95a06b384c1559d1b33f9a57642148a8f4142009ac2bef79a61c7d01" Oct 30 17:30:17.818642 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:30:17.818614 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-c17e8c-29889690-nvd7z_odh-ai-gateway-infra(427507de-33c6-409a-a68c-8e5a978b24ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c17e8c-29889690-nvd7z" podUID="427507de-33c6-409a-a68c-8e5a978b24ed" Oct 30 17:30:17.830895 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:17.830862 4522 scope.go:122] "RemoveContainer" containerID="d0205f49f80bb681a77c60dfb1ead99c4685ddea5bb89e926dc7888f30b2c2f1" Oct 30 17:30:17.876168 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:17.876136 4522 scope.go:122] "RemoveContainer" containerID="7a757f64fb31fc0b1c5ee48e7b995d9c4bf9e3d14b101a880e406f022eb1fa35" Oct 30 17:30:17.895823 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:17.895795 4522 scope.go:122] "RemoveContainer" containerID="a8b39d35987154a599beec11a5836636daed52c08dbcb85e9374fa3a678fc446" Oct 30 17:30:17.914379 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:17.914356 4522 scope.go:122] "RemoveContainer" containerID="9669c5968e097f64d7c9ea61ac6cb237243516f6758676238ab92f34e302d1b2" Oct 30 17:30:17.935821 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:17.935789 4522 scope.go:122] "RemoveContainer" containerID="b7879c13a8c6607368d35f8cded5358593090fb7c14c2151c205b6e0028589fb" Oct 30 17:30:18.096768 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:18.096740 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-469ad7-29889690-cfrlx" Oct 30 17:30:18.137545 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:18.137505 4522 scope.go:122] "RemoveContainer" containerID="d55ced492d901c4202a262b13348021163d3262badf6ee20e85bf87f635effbc" Oct 30 17:30:18.137730 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:18.137713 4522 scope.go:122] "RemoveContainer" containerID="b46ac2d32779d757aa0dbdfcf22ba42e8a11531ad667a0d511e37ec3f2554be1" Oct 30 17:30:18.137856 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:18.137838 4522 scope.go:122] "RemoveContainer" containerID="2ba7c2777a6beecc4d3585d7275be8aed4e341dfa09df8ce1a4d275ea1e00a20" Oct 30 17:30:18.185406 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:18.185366 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c977d640-8303-4e86-8952-149525c0708c-kube-api-access-mksvh\" (UniqueName: \"kubernetes.io/projected/c977d640-8303-4e86-8952-149525c0708c-kube-api-access-mksvh\") pod \"c977d640-8303-4e86-8952-149525c0708c\" (UID: \"c977d640-8303-4e86-8952-149525c0708c\") " Oct 30 17:30:18.185818 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:18.185794 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c977d640-8303-4e86-8952-149525c0708c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c977d640-8303-4e86-8952-149525c0708c-maas-api-service-ca\") pod \"c977d640-8303-4e86-8952-149525c0708c\" (UID: \"c977d640-8303-4e86-8952-149525c0708c\") " Oct 30 17:30:18.188172 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:18.188148 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c977d640-8303-4e86-8952-149525c0708c-maas-api-service-ca" pod "c977d640-8303-4e86-8952-149525c0708c" (UID: "c977d640-8303-4e86-8952-149525c0708c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:30:18.191140 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:18.191118 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c977d640-8303-4e86-8952-149525c0708c-kube-api-access-mksvh" pod "c977d640-8303-4e86-8952-149525c0708c" (UID: "c977d640-8303-4e86-8952-149525c0708c"). InnerVolumeSpecName "kube-api-access-mksvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:30:18.288046 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:18.288015 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c977d640-8303-4e86-8952-149525c0708c-maas-api-service-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:30:18.288046 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:18.288044 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mksvh\" (UniqueName: \"kubernetes.io/projected/c977d640-8303-4e86-8952-149525c0708c-kube-api-access-mksvh\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:30:18.832212 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:18.832179 4522 generic.go:356] "Generic (PLEG): container finished" podID="e1d79713-dd75-4823-a8f8-0f252befa273" containerID="82dc799ba7bcc72701d1f44dd435cadc6f55b6063ca8264d79af8e6591770292" exitCode=60 Oct 30 17:30:18.832747 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:18.832252 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889690-xxcd6" event={"ID":"e1d79713-dd75-4823-a8f8-0f252befa273","Type":"ContainerDied","Data":"82dc799ba7bcc72701d1f44dd435cadc6f55b6063ca8264d79af8e6591770292"} Oct 30 17:30:18.832747 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:18.832373 4522 scope.go:122] "RemoveContainer" containerID="cfe806aa8a857e31dc2098e563c375de1f5fa3a48ab64db2708255ea77b5e03a" Oct 30 17:30:18.832906 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:18.832881 4522 scope.go:122] "RemoveContainer" containerID="82dc799ba7bcc72701d1f44dd435cadc6f55b6063ca8264d79af8e6591770292" Oct 30 17:30:18.833364 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:30:18.833337 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-d384dc-29889690-xxcd6_odh-ai-gateway-infra(e1d79713-dd75-4823-a8f8-0f252befa273)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889690-xxcd6" podUID="e1d79713-dd75-4823-a8f8-0f252befa273" Oct 30 17:30:18.837577 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:18.837546 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889690-7kjml" event={"ID":"e2ce4a71-84b5-4817-b99c-9d56d4673869","Type":"ContainerStarted","Data":"534d72f8def193e449fd2a2e9d82adffcb175383f5616bd53e7a93bda0240c5d"} Oct 30 17:30:18.840163 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:18.840142 4522 generic.go:356] "Generic (PLEG): container finished" podID="8643b717-c5ec-4194-92bb-e54de07fef00" containerID="1b908d71c8876e4255ec418018ad6e60eea76c03b9ec17a1bf79950ccc0faf1e" exitCode=60 Oct 30 17:30:18.840281 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:18.840211 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-987dfc-29889690-vpglf" event={"ID":"8643b717-c5ec-4194-92bb-e54de07fef00","Type":"ContainerDied","Data":"1b908d71c8876e4255ec418018ad6e60eea76c03b9ec17a1bf79950ccc0faf1e"} Oct 30 17:30:18.840802 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:18.840780 4522 scope.go:122] "RemoveContainer" containerID="1b908d71c8876e4255ec418018ad6e60eea76c03b9ec17a1bf79950ccc0faf1e" Oct 30 17:30:18.841219 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:30:18.841193 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-987dfc-29889690-vpglf_odh-ai-gateway-infra(8643b717-c5ec-4194-92bb-e54de07fef00)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-987dfc-29889690-vpglf" podUID="8643b717-c5ec-4194-92bb-e54de07fef00" Oct 30 17:30:18.842730 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:18.842708 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-82243b-29889690-dmgd9" event={"ID":"f6b0d3f6-2109-40b3-b86f-b895a4616558","Type":"ContainerStarted","Data":"14bd60669dc55d9551bb680ccd2764b4d2207bd0c6517423be412fe7439a217a"} Oct 30 17:30:18.846113 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:18.846093 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-469ad7-29889690-cfrlx" event={"ID":"c977d640-8303-4e86-8952-149525c0708c","Type":"ContainerDied","Data":"eeea0a3cc1f79ae6dfef84fe9c4ecab12440609bde6030b495717e3edb0527db"} Oct 30 17:30:18.846210 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:18.846098 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-469ad7-29889690-cfrlx" Oct 30 17:30:18.848154 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:18.848135 4522 generic.go:356] "Generic (PLEG): container finished" podID="d4fa40dd-b387-4513-aed9-8e11394d3454" containerID="31764ae7c08069ec0108cbc3a62fa5c91340c75eeba96d201f709c741029b66a" exitCode=60 Oct 30 17:30:18.848251 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:18.848203 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889690-csm2k" event={"ID":"d4fa40dd-b387-4513-aed9-8e11394d3454","Type":"ContainerDied","Data":"31764ae7c08069ec0108cbc3a62fa5c91340c75eeba96d201f709c741029b66a"} Oct 30 17:30:18.848649 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:18.848632 4522 scope.go:122] "RemoveContainer" containerID="31764ae7c08069ec0108cbc3a62fa5c91340c75eeba96d201f709c741029b66a" Oct 30 17:30:18.848956 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:30:18.848931 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-096d75-29889690-csm2k_odh-ai-gateway-infra(d4fa40dd-b387-4513-aed9-8e11394d3454)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889690-csm2k" podUID="d4fa40dd-b387-4513-aed9-8e11394d3454" Oct 30 17:30:18.855661 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:18.855635 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e13a51-29889690-dg6vt" event={"ID":"e4febc82-5717-4157-b46b-7ad8004d7f6a","Type":"ContainerStarted","Data":"a2b79f75da1879e8232c5018de5d4c88a4370d85db75e1b7701bcc80716efcdb"} Oct 30 17:30:18.870110 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:18.870077 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889690-s8ggr"] Oct 30 17:30:18.878861 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:18.878838 4522 scope.go:122] "RemoveContainer" containerID="734543e7d7f12c02b5b6dcf676c27ea7d207c0d705630b1aa03e034933faf934" Oct 30 17:30:18.899071 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:18.899047 4522 scope.go:122] "RemoveContainer" containerID="3adbf5a9a331d35513386217e098fbc11d31a4955707585a03bbcddd26ea2c0c" Oct 30 17:30:18.912232 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:18.912213 4522 scope.go:122] "RemoveContainer" containerID="0c79a97f55e118fb1ec135a8bc0111d5de0b1e79eb6fb64a34fbae87378035de" Oct 30 17:30:18.935237 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:18.935206 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d2d985-29889690-vlq47"] Oct 30 17:30:19.047852 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.047015 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889690-ksf2d"] Oct 30 17:30:19.142528 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.142496 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c17e8c-29889690-nvd7z"] Oct 30 17:30:19.143497 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.143473 4522 scope.go:122] "RemoveContainer" containerID="d95088e00c7af151fd2607ca2473d007fd1ff4cbe931c64c641a6a35de61ad01" Oct 30 17:30:19.338963 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.316765 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889690-s8ggr" Oct 30 17:30:19.338963 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.321636 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-469ad7-29889690-cfrlx"] Oct 30 17:30:19.349391 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.349357 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-469ad7-29889690-cfrlx"] Oct 30 17:30:19.359720 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.359618 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889690-d76sf"] Oct 30 17:30:19.382982 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.381988 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46fba0fa-d263-42d2-84fd-ddaefea25c63-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46fba0fa-d263-42d2-84fd-ddaefea25c63-maas-api-service-ca\") pod \"46fba0fa-d263-42d2-84fd-ddaefea25c63\" (UID: \"46fba0fa-d263-42d2-84fd-ddaefea25c63\") " Oct 30 17:30:19.382982 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.382292 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46fba0fa-d263-42d2-84fd-ddaefea25c63-kube-api-access-jqjkd\" (UniqueName: \"kubernetes.io/projected/46fba0fa-d263-42d2-84fd-ddaefea25c63-kube-api-access-jqjkd\") pod \"46fba0fa-d263-42d2-84fd-ddaefea25c63\" (UID: \"46fba0fa-d263-42d2-84fd-ddaefea25c63\") " Oct 30 17:30:19.384261 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.384089 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46fba0fa-d263-42d2-84fd-ddaefea25c63-maas-api-service-ca" pod "46fba0fa-d263-42d2-84fd-ddaefea25c63" (UID: "46fba0fa-d263-42d2-84fd-ddaefea25c63"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:30:19.393082 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.388056 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-001a96-29889690-j9qwv"] Oct 30 17:30:19.393082 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.392067 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46fba0fa-d263-42d2-84fd-ddaefea25c63-kube-api-access-jqjkd" pod "46fba0fa-d263-42d2-84fd-ddaefea25c63" (UID: "46fba0fa-d263-42d2-84fd-ddaefea25c63"). InnerVolumeSpecName "kube-api-access-jqjkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:30:19.557382 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.495908 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-8a7e4a-d55dd8b4d-hd4qr" podUID="901bc210-b26c-41df-b40f-e94a12c578c5" containerName="payload-processing" containerID="cri-o://86f928ca250d042a74c0658e77b9760e7fa6d5fef063db548206bcc856d499b7" gracePeriod=25 Oct 30 17:30:19.557382 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.497401 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46fba0fa-d263-42d2-84fd-ddaefea25c63-maas-api-service-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:30:19.557382 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.497423 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqjkd\" (UniqueName: \"kubernetes.io/projected/46fba0fa-d263-42d2-84fd-ddaefea25c63-kube-api-access-jqjkd\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:30:19.627644 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.627532 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d2d985-29889690-vlq47" Oct 30 17:30:19.708028 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.696270 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-63e60d-85c58c5cdd-6k2gk"] Oct 30 17:30:19.708028 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.697774 4522 cpu_manager.go:414] "containerMap: removing container" podUID="ddc0e647-bfb8-474d-b423-be4d72b31b32" containerName="cleanup" Oct 30 17:30:19.708028 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.697797 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddc0e647-bfb8-474d-b423-be4d72b31b32" containerName="cleanup" Oct 30 17:30:19.708028 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.697813 4522 cpu_manager.go:414] "containerMap: removing container" podUID="ddc0e647-bfb8-474d-b423-be4d72b31b32" containerName="cleanup" Oct 30 17:30:19.708028 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.697823 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddc0e647-bfb8-474d-b423-be4d72b31b32" containerName="cleanup" Oct 30 17:30:19.708028 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.697835 4522 cpu_manager.go:414] "containerMap: removing container" podUID="46fba0fa-d263-42d2-84fd-ddaefea25c63" containerName="cleanup" Oct 30 17:30:19.708028 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.697843 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46fba0fa-d263-42d2-84fd-ddaefea25c63" containerName="cleanup" Oct 30 17:30:19.708028 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.697861 4522 cpu_manager.go:414] "containerMap: removing container" podUID="c977d640-8303-4e86-8952-149525c0708c" containerName="cleanup" Oct 30 17:30:19.708028 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.697868 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c977d640-8303-4e86-8952-149525c0708c" containerName="cleanup" Oct 30 17:30:19.708028 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.697932 4522 cpu_manager.go:414] "containerMap: removing container" podUID="c977d640-8303-4e86-8952-149525c0708c" containerName="cleanup" Oct 30 17:30:19.708028 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.697940 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c977d640-8303-4e86-8952-149525c0708c" containerName="cleanup" Oct 30 17:30:19.708028 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.697963 4522 cpu_manager.go:414] "containerMap: removing container" podUID="46fba0fa-d263-42d2-84fd-ddaefea25c63" containerName="cleanup" Oct 30 17:30:19.708028 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.697970 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46fba0fa-d263-42d2-84fd-ddaefea25c63" containerName="cleanup" Oct 30 17:30:19.708028 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.698211 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="c977d640-8303-4e86-8952-149525c0708c" containerName="cleanup" Oct 30 17:30:19.708028 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.698229 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="c977d640-8303-4e86-8952-149525c0708c" containerName="cleanup" Oct 30 17:30:19.708028 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.698240 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="46fba0fa-d263-42d2-84fd-ddaefea25c63" containerName="cleanup" Oct 30 17:30:19.708028 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.698250 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="46fba0fa-d263-42d2-84fd-ddaefea25c63" containerName="cleanup" Oct 30 17:30:19.708028 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.698261 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddc0e647-bfb8-474d-b423-be4d72b31b32" containerName="cleanup" Oct 30 17:30:19.708028 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.698278 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddc0e647-bfb8-474d-b423-be4d72b31b32" containerName="cleanup" Oct 30 17:30:19.708028 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.698289 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddc0e647-bfb8-474d-b423-be4d72b31b32" containerName="cleanup" Oct 30 17:30:19.728459 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.728399 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddc0e647-bfb8-474d-b423-be4d72b31b32-maas-api-service-ca" pod "ddc0e647-bfb8-474d-b423-be4d72b31b32" (UID: "ddc0e647-bfb8-474d-b423-be4d72b31b32"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:30:19.728701 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.728681 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddc0e647-bfb8-474d-b423-be4d72b31b32-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddc0e647-bfb8-474d-b423-be4d72b31b32-maas-api-service-ca\") pod \"ddc0e647-bfb8-474d-b423-be4d72b31b32\" (UID: \"ddc0e647-bfb8-474d-b423-be4d72b31b32\") " Oct 30 17:30:19.738241 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.737650 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddc0e647-bfb8-474d-b423-be4d72b31b32-kube-api-access-h8fqh\" (UniqueName: \"kubernetes.io/projected/ddc0e647-bfb8-474d-b423-be4d72b31b32-kube-api-access-h8fqh\") pod \"ddc0e647-bfb8-474d-b423-be4d72b31b32\" (UID: \"ddc0e647-bfb8-474d-b423-be4d72b31b32\") " Oct 30 17:30:19.738685 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.738641 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddc0e647-bfb8-474d-b423-be4d72b31b32-maas-api-service-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:30:19.740996 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.740971 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-63e60d-85c58c5cdd-6k2gk"] Oct 30 17:30:19.741199 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.741093 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-63e60d-85c58c5cdd-6k2gk" Oct 30 17:30:19.741435 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.741297 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddc0e647-bfb8-474d-b423-be4d72b31b32-kube-api-access-h8fqh" pod "ddc0e647-bfb8-474d-b423-be4d72b31b32" (UID: "ddc0e647-bfb8-474d-b423-be4d72b31b32"). InnerVolumeSpecName "kube-api-access-h8fqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:30:19.761966 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.759075 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-63e60d\"" Oct 30 17:30:19.761966 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.760262 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-63e60d-dockercfg-ngmrn\"" Oct 30 17:30:19.812139 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.799816 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c17e8c-29889690-nvd7z" Oct 30 17:30:19.849725 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.844403 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqvms\" (UniqueName: \"kubernetes.io/projected/300450d2-c1b2-4a64-bd41-4a1ec5df3d77-kube-api-access-vqvms\") pod \"payload-processing-e2e-ipp-b-63e60d-85c58c5cdd-6k2gk\" (UID: \"300450d2-c1b2-4a64-bd41-4a1ec5df3d77\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-63e60d-85c58c5cdd-6k2gk" Oct 30 17:30:19.849725 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.844795 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/300450d2-c1b2-4a64-bd41-4a1ec5df3d77-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-63e60d-85c58c5cdd-6k2gk\" (UID: \"300450d2-c1b2-4a64-bd41-4a1ec5df3d77\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-63e60d-85c58c5cdd-6k2gk" Oct 30 17:30:19.849725 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.844962 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8fqh\" (UniqueName: \"kubernetes.io/projected/ddc0e647-bfb8-474d-b423-be4d72b31b32-kube-api-access-h8fqh\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:30:19.962838 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.916418 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-63e60d-5b4499f4c9-5j2lv"] Oct 30 17:30:19.962838 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.917844 4522 cpu_manager.go:414] "containerMap: removing container" podUID="427507de-33c6-409a-a68c-8e5a978b24ed" containerName="cleanup" Oct 30 17:30:19.962838 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.917867 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="427507de-33c6-409a-a68c-8e5a978b24ed" containerName="cleanup" Oct 30 17:30:19.962838 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.917881 4522 cpu_manager.go:414] "containerMap: removing container" podUID="ddc0e647-bfb8-474d-b423-be4d72b31b32" containerName="cleanup" Oct 30 17:30:19.962838 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.917889 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddc0e647-bfb8-474d-b423-be4d72b31b32" containerName="cleanup" Oct 30 17:30:19.962838 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.917912 4522 cpu_manager.go:414] "containerMap: removing container" podUID="c977d640-8303-4e86-8952-149525c0708c" containerName="cleanup" Oct 30 17:30:19.962838 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.917919 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c977d640-8303-4e86-8952-149525c0708c" containerName="cleanup" Oct 30 17:30:19.962838 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.917941 4522 cpu_manager.go:414] "containerMap: removing container" podUID="427507de-33c6-409a-a68c-8e5a978b24ed" containerName="cleanup" Oct 30 17:30:19.962838 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.917947 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="427507de-33c6-409a-a68c-8e5a978b24ed" containerName="cleanup" Oct 30 17:30:19.962838 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.918000 4522 cpu_manager.go:414] "containerMap: removing container" podUID="427507de-33c6-409a-a68c-8e5a978b24ed" containerName="cleanup" Oct 30 17:30:19.962838 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.918007 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="427507de-33c6-409a-a68c-8e5a978b24ed" containerName="cleanup" Oct 30 17:30:19.962838 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.918024 4522 cpu_manager.go:414] "containerMap: removing container" podUID="46fba0fa-d263-42d2-84fd-ddaefea25c63" containerName="cleanup" Oct 30 17:30:19.962838 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.918029 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46fba0fa-d263-42d2-84fd-ddaefea25c63" containerName="cleanup" Oct 30 17:30:19.962838 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.918250 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="427507de-33c6-409a-a68c-8e5a978b24ed" containerName="cleanup" Oct 30 17:30:19.962838 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.918268 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="c977d640-8303-4e86-8952-149525c0708c" containerName="cleanup" Oct 30 17:30:19.962838 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.918279 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="427507de-33c6-409a-a68c-8e5a978b24ed" containerName="cleanup" Oct 30 17:30:19.962838 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.918289 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="46fba0fa-d263-42d2-84fd-ddaefea25c63" containerName="cleanup" Oct 30 17:30:19.962838 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.944585 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e60d-5b4499f4c9-5j2lv" Oct 30 17:30:19.962838 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.949123 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/427507de-33c6-409a-a68c-8e5a978b24ed-kube-api-access-n9ln8\" (UniqueName: \"kubernetes.io/projected/427507de-33c6-409a-a68c-8e5a978b24ed-kube-api-access-n9ln8\") pod \"427507de-33c6-409a-a68c-8e5a978b24ed\" (UID: \"427507de-33c6-409a-a68c-8e5a978b24ed\") " Oct 30 17:30:19.962838 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.949329 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/427507de-33c6-409a-a68c-8e5a978b24ed-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/427507de-33c6-409a-a68c-8e5a978b24ed-maas-api-service-ca\") pod \"427507de-33c6-409a-a68c-8e5a978b24ed\" (UID: \"427507de-33c6-409a-a68c-8e5a978b24ed\") " Oct 30 17:30:19.962838 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.953599 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/427507de-33c6-409a-a68c-8e5a978b24ed-maas-api-service-ca" pod "427507de-33c6-409a-a68c-8e5a978b24ed" (UID: "427507de-33c6-409a-a68c-8e5a978b24ed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:30:19.962838 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.956724 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889690-ksf2d" Oct 30 17:30:19.962838 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.956830 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/300450d2-c1b2-4a64-bd41-4a1ec5df3d77-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-63e60d-85c58c5cdd-6k2gk\" (UID: \"300450d2-c1b2-4a64-bd41-4a1ec5df3d77\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-63e60d-85c58c5cdd-6k2gk" Oct 30 17:30:19.962838 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.957083 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqvms\" (UniqueName: \"kubernetes.io/projected/300450d2-c1b2-4a64-bd41-4a1ec5df3d77-kube-api-access-vqvms\") pod \"payload-processing-e2e-ipp-b-63e60d-85c58c5cdd-6k2gk\" (UID: \"300450d2-c1b2-4a64-bd41-4a1ec5df3d77\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-63e60d-85c58c5cdd-6k2gk" Oct 30 17:30:19.962838 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.957744 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/300450d2-c1b2-4a64-bd41-4a1ec5df3d77-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-63e60d-85c58c5cdd-6k2gk\" (UID: \"300450d2-c1b2-4a64-bd41-4a1ec5df3d77\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-63e60d-85c58c5cdd-6k2gk" Oct 30 17:30:19.962838 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.958594 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/427507de-33c6-409a-a68c-8e5a978b24ed-maas-api-service-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:30:19.962838 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.962618 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/427507de-33c6-409a-a68c-8e5a978b24ed-kube-api-access-n9ln8" pod "427507de-33c6-409a-a68c-8e5a978b24ed" (UID: "427507de-33c6-409a-a68c-8e5a978b24ed"). InnerVolumeSpecName "kube-api-access-n9ln8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:30:19.973707 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.966791 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-63e60d-5b4499f4c9-5j2lv"] Oct 30 17:30:19.976796 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.974707 4522 generic.go:356] "Generic (PLEG): container finished" podID="f6b0d3f6-2109-40b3-b86f-b895a4616558" containerID="14bd60669dc55d9551bb680ccd2764b4d2207bd0c6517423be412fe7439a217a" exitCode=60 Oct 30 17:30:19.976796 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.974796 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-82243b-29889690-dmgd9" event={"ID":"f6b0d3f6-2109-40b3-b86f-b895a4616558","Type":"ContainerDied","Data":"14bd60669dc55d9551bb680ccd2764b4d2207bd0c6517423be412fe7439a217a"} Oct 30 17:30:19.976796 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.974849 4522 scope.go:122] "RemoveContainer" containerID="b46ac2d32779d757aa0dbdfcf22ba42e8a11531ad667a0d511e37ec3f2554be1" Oct 30 17:30:19.976796 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.975389 4522 scope.go:122] "RemoveContainer" containerID="14bd60669dc55d9551bb680ccd2764b4d2207bd0c6517423be412fe7439a217a" Oct 30 17:30:19.976796 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:30:19.975846 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-82243b-29889690-dmgd9_odh-ai-gateway-infra(f6b0d3f6-2109-40b3-b86f-b895a4616558)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-82243b-29889690-dmgd9" podUID="f6b0d3f6-2109-40b3-b86f-b895a4616558" Oct 30 17:30:19.988338 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.981461 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d2d985-29889690-vlq47" event={"ID":"ddc0e647-bfb8-474d-b423-be4d72b31b32","Type":"ContainerDied","Data":"1afcd9d6953183f4e71d5bbfd8b522eb527cec5c9abc3819dd55871fe7c0782c"} Oct 30 17:30:19.988338 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.981565 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d2d985-29889690-vlq47" Oct 30 17:30:19.998095 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:19.996235 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqvms\" (UniqueName: \"kubernetes.io/projected/300450d2-c1b2-4a64-bd41-4a1ec5df3d77-kube-api-access-vqvms\") pod \"payload-processing-e2e-ipp-b-63e60d-85c58c5cdd-6k2gk\" (UID: \"300450d2-c1b2-4a64-bd41-4a1ec5df3d77\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-63e60d-85c58c5cdd-6k2gk" Oct 30 17:30:20.004265 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.003930 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889690-d76sf" Oct 30 17:30:20.004265 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.004017 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889690-xxcd6"] Oct 30 17:30:20.014372 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.010769 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c17e8c-29889690-nvd7z" event={"ID":"427507de-33c6-409a-a68c-8e5a978b24ed","Type":"ContainerDied","Data":"17a8fe6ae91377a1c207c9a220ea5856f0d905d30cb5a942c09c0563dfbfe966"} Oct 30 17:30:20.014372 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.010885 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c17e8c-29889690-nvd7z" Oct 30 17:30:20.020499 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.017701 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889690-s8ggr" event={"ID":"46fba0fa-d263-42d2-84fd-ddaefea25c63","Type":"ContainerDied","Data":"78966c1cf61cde7444fe08b5e2292924246278b005400960d78fe460df9ffbeb"} Oct 30 17:30:20.020499 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.017830 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889690-s8ggr" Oct 30 17:30:20.033346 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.028924 4522 scope.go:122] "RemoveContainer" containerID="c510e98af1284195dd00b9abfab396973170f52f04262534fe6ab86cddb66f78" Oct 30 17:30:20.033346 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.032195 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889690-7kjml"] Oct 30 17:30:20.037043 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.036605 4522 generic.go:356] "Generic (PLEG): container finished" podID="e4febc82-5717-4157-b46b-7ad8004d7f6a" containerID="a2b79f75da1879e8232c5018de5d4c88a4370d85db75e1b7701bcc80716efcdb" exitCode=60 Oct 30 17:30:20.037043 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.036788 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e13a51-29889690-dg6vt" event={"ID":"e4febc82-5717-4157-b46b-7ad8004d7f6a","Type":"ContainerDied","Data":"a2b79f75da1879e8232c5018de5d4c88a4370d85db75e1b7701bcc80716efcdb"} Oct 30 17:30:20.046032 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.039135 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-001a96-29889690-j9qwv" Oct 30 17:30:20.046032 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.044897 4522 scope.go:122] "RemoveContainer" containerID="a2b79f75da1879e8232c5018de5d4c88a4370d85db75e1b7701bcc80716efcdb" Oct 30 17:30:20.046032 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:30:20.045403 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-e13a51-29889690-dg6vt_odh-ai-gateway-infra(e4febc82-5717-4157-b46b-7ad8004d7f6a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e13a51-29889690-dg6vt" podUID="e4febc82-5717-4157-b46b-7ad8004d7f6a" Oct 30 17:30:20.055017 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.054748 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889690-ksf2d" event={"ID":"22ffe3de-e5e2-43a1-b276-20cf180c8b01","Type":"ContainerDied","Data":"79bfde61a27ec2c20849d366e7defc3a92b9742c62281b9eb791e619ef65482e"} Oct 30 17:30:20.055017 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.054844 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889690-ksf2d" Oct 30 17:30:20.056781 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.056531 4522 scope.go:122] "RemoveContainer" containerID="c12c290c95a06b384c1559d1b33f9a57642148a8f4142009ac2bef79a61c7d01" Oct 30 17:30:20.060429 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.059621 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22ffe3de-e5e2-43a1-b276-20cf180c8b01-kube-api-access-w25zl\" (UniqueName: \"kubernetes.io/projected/22ffe3de-e5e2-43a1-b276-20cf180c8b01-kube-api-access-w25zl\") pod \"22ffe3de-e5e2-43a1-b276-20cf180c8b01\" (UID: \"22ffe3de-e5e2-43a1-b276-20cf180c8b01\") " Oct 30 17:30:20.060429 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.059775 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22ffe3de-e5e2-43a1-b276-20cf180c8b01-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22ffe3de-e5e2-43a1-b276-20cf180c8b01-maas-api-service-ca\") pod \"22ffe3de-e5e2-43a1-b276-20cf180c8b01\" (UID: \"22ffe3de-e5e2-43a1-b276-20cf180c8b01\") " Oct 30 17:30:20.060429 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.060278 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85668031-55c5-458e-a623-26d81c8d4703-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-63e60d-5b4499f4c9-5j2lv\" (UID: \"85668031-55c5-458e-a623-26d81c8d4703\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e60d-5b4499f4c9-5j2lv" Oct 30 17:30:20.061598 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.060967 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pspc\" (UniqueName: \"kubernetes.io/projected/85668031-55c5-458e-a623-26d81c8d4703-kube-api-access-6pspc\") pod \"payload-pre-processing-e2e-ipp-b-63e60d-5b4499f4c9-5j2lv\" (UID: \"85668031-55c5-458e-a623-26d81c8d4703\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e60d-5b4499f4c9-5j2lv" Oct 30 17:30:20.061598 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.061237 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9ln8\" (UniqueName: \"kubernetes.io/projected/427507de-33c6-409a-a68c-8e5a978b24ed-kube-api-access-n9ln8\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:30:20.062486 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.061932 4522 generic.go:356] "Generic (PLEG): container finished" podID="e2ce4a71-84b5-4817-b99c-9d56d4673869" containerID="534d72f8def193e449fd2a2e9d82adffcb175383f5616bd53e7a93bda0240c5d" exitCode=60 Oct 30 17:30:20.062486 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.062045 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889690-7kjml" event={"ID":"e2ce4a71-84b5-4817-b99c-9d56d4673869","Type":"ContainerDied","Data":"534d72f8def193e449fd2a2e9d82adffcb175383f5616bd53e7a93bda0240c5d"} Oct 30 17:30:20.063298 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.062777 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22ffe3de-e5e2-43a1-b276-20cf180c8b01-maas-api-service-ca" pod "22ffe3de-e5e2-43a1-b276-20cf180c8b01" (UID: "22ffe3de-e5e2-43a1-b276-20cf180c8b01"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:30:20.064835 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.063793 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889690-csm2k"] Oct 30 17:30:20.078768 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.075251 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22ffe3de-e5e2-43a1-b276-20cf180c8b01-kube-api-access-w25zl" pod "22ffe3de-e5e2-43a1-b276-20cf180c8b01" (UID: "22ffe3de-e5e2-43a1-b276-20cf180c8b01"). InnerVolumeSpecName "kube-api-access-w25zl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:30:20.079253 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.079220 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889690-qb6xz" event={"ID":"58bcea19-790f-486c-8994-21b1b4c20dc9","Type":"ContainerStarted","Data":"4691fc40d7619feafbbc8d68786113ff31e0fabe10a9827fad62f52692688e3f"} Oct 30 17:30:20.079735 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.079712 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-63e60d-85c58c5cdd-6k2gk" Oct 30 17:30:20.131348 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.083188 4522 generic.go:356] "Generic (PLEG): container finished" podID="901bc210-b26c-41df-b40f-e94a12c578c5" containerID="86f928ca250d042a74c0658e77b9760e7fa6d5fef063db548206bcc856d499b7" exitCode=0 Oct 30 17:30:20.131348 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.083514 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8a7e4a-d55dd8b4d-hd4qr" event={"ID":"901bc210-b26c-41df-b40f-e94a12c578c5","Type":"ContainerDied","Data":"86f928ca250d042a74c0658e77b9760e7fa6d5fef063db548206bcc856d499b7"} Oct 30 17:30:20.165337 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.164087 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3c414cb-e4f0-4fbf-9bdf-946842e1297f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3c414cb-e4f0-4fbf-9bdf-946842e1297f-maas-api-service-ca\") pod \"f3c414cb-e4f0-4fbf-9bdf-946842e1297f\" (UID: \"f3c414cb-e4f0-4fbf-9bdf-946842e1297f\") " Oct 30 17:30:20.165337 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.164213 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b49aaf80-dff6-40d4-8404-d55de5b568ae-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b49aaf80-dff6-40d4-8404-d55de5b568ae-maas-api-service-ca\") pod \"b49aaf80-dff6-40d4-8404-d55de5b568ae\" (UID: \"b49aaf80-dff6-40d4-8404-d55de5b568ae\") " Oct 30 17:30:20.165337 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.164281 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b49aaf80-dff6-40d4-8404-d55de5b568ae-kube-api-access-5vbp2\" (UniqueName: \"kubernetes.io/projected/b49aaf80-dff6-40d4-8404-d55de5b568ae-kube-api-access-5vbp2\") pod \"b49aaf80-dff6-40d4-8404-d55de5b568ae\" (UID: \"b49aaf80-dff6-40d4-8404-d55de5b568ae\") " Oct 30 17:30:20.165337 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.164504 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3c414cb-e4f0-4fbf-9bdf-946842e1297f-kube-api-access-8cw4t\" (UniqueName: \"kubernetes.io/projected/f3c414cb-e4f0-4fbf-9bdf-946842e1297f-kube-api-access-8cw4t\") pod \"f3c414cb-e4f0-4fbf-9bdf-946842e1297f\" (UID: \"f3c414cb-e4f0-4fbf-9bdf-946842e1297f\") " Oct 30 17:30:20.165934 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.165415 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85668031-55c5-458e-a623-26d81c8d4703-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-63e60d-5b4499f4c9-5j2lv\" (UID: \"85668031-55c5-458e-a623-26d81c8d4703\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e60d-5b4499f4c9-5j2lv" Oct 30 17:30:20.165934 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.165658 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pspc\" (UniqueName: \"kubernetes.io/projected/85668031-55c5-458e-a623-26d81c8d4703-kube-api-access-6pspc\") pod \"payload-pre-processing-e2e-ipp-b-63e60d-5b4499f4c9-5j2lv\" (UID: \"85668031-55c5-458e-a623-26d81c8d4703\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e60d-5b4499f4c9-5j2lv" Oct 30 17:30:20.166064 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.166044 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22ffe3de-e5e2-43a1-b276-20cf180c8b01-maas-api-service-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:30:20.166125 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.166068 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w25zl\" (UniqueName: \"kubernetes.io/projected/22ffe3de-e5e2-43a1-b276-20cf180c8b01-kube-api-access-w25zl\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:30:20.167370 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.166986 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3c414cb-e4f0-4fbf-9bdf-946842e1297f-maas-api-service-ca" pod "f3c414cb-e4f0-4fbf-9bdf-946842e1297f" (UID: "f3c414cb-e4f0-4fbf-9bdf-946842e1297f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:30:20.167524 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.167416 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b49aaf80-dff6-40d4-8404-d55de5b568ae-maas-api-service-ca" pod "b49aaf80-dff6-40d4-8404-d55de5b568ae" (UID: "b49aaf80-dff6-40d4-8404-d55de5b568ae"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:30:20.168574 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.168548 4522 scope.go:122] "RemoveContainer" containerID="c6b1f621b5c1b85a6d1ff8b62741b744f8900ecb180548c4b74b0f06c16dcc54" Oct 30 17:30:20.171247 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.171186 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85668031-55c5-458e-a623-26d81c8d4703-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-63e60d-5b4499f4c9-5j2lv\" (UID: \"85668031-55c5-458e-a623-26d81c8d4703\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e60d-5b4499f4c9-5j2lv" Oct 30 17:30:20.187523 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.187455 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b49aaf80-dff6-40d4-8404-d55de5b568ae-kube-api-access-5vbp2" pod "b49aaf80-dff6-40d4-8404-d55de5b568ae" (UID: "b49aaf80-dff6-40d4-8404-d55de5b568ae"). InnerVolumeSpecName "kube-api-access-5vbp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:30:20.196425 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.193469 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3c414cb-e4f0-4fbf-9bdf-946842e1297f-kube-api-access-8cw4t" pod "f3c414cb-e4f0-4fbf-9bdf-946842e1297f" (UID: "f3c414cb-e4f0-4fbf-9bdf-946842e1297f"). InnerVolumeSpecName "kube-api-access-8cw4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:30:20.255056 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.248915 4522 scope.go:122] "RemoveContainer" containerID="2ba7c2777a6beecc4d3585d7275be8aed4e341dfa09df8ce1a4d275ea1e00a20" Oct 30 17:30:20.255056 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.253629 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pspc\" (UniqueName: \"kubernetes.io/projected/85668031-55c5-458e-a623-26d81c8d4703-kube-api-access-6pspc\") pod \"payload-pre-processing-e2e-ipp-b-63e60d-5b4499f4c9-5j2lv\" (UID: \"85668031-55c5-458e-a623-26d81c8d4703\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e60d-5b4499f4c9-5j2lv" Oct 30 17:30:20.258448 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.257055 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c977d640-8303-4e86-8952-149525c0708c" path="/var/lib/kubelet/pods/c977d640-8303-4e86-8952-149525c0708c/volumes" Oct 30 17:30:20.261856 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.260216 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e13a51-29889690-dg6vt"] Oct 30 17:30:20.277434 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.271585 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b49aaf80-dff6-40d4-8404-d55de5b568ae-maas-api-service-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:30:20.277434 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.271617 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vbp2\" (UniqueName: \"kubernetes.io/projected/b49aaf80-dff6-40d4-8404-d55de5b568ae-kube-api-access-5vbp2\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:30:20.277434 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.271634 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cw4t\" (UniqueName: \"kubernetes.io/projected/f3c414cb-e4f0-4fbf-9bdf-946842e1297f-kube-api-access-8cw4t\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:30:20.277434 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.271650 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3c414cb-e4f0-4fbf-9bdf-946842e1297f-maas-api-service-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:30:20.299971 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.299939 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e60d-5b4499f4c9-5j2lv" Oct 30 17:30:20.338666 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.337676 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-82243b-29889690-dmgd9"] Oct 30 17:30:20.350398 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.350334 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8a7e4a-d55dd8b4d-hd4qr" Oct 30 17:30:20.362920 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.362847 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d2d985-29889690-vlq47"] Oct 30 17:30:20.367382 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.366969 4522 scope.go:122] "RemoveContainer" containerID="5208c9ea07bc24ca301adc9f68d8e22826daf2e25478049602e6e6cb8df6132a" Oct 30 17:30:20.379388 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.379294 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d2d985-29889690-vlq47"] Oct 30 17:30:20.387650 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.387148 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-987dfc-29889690-vpglf"] Oct 30 17:30:20.433967 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.433902 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889690-s8ggr"] Oct 30 17:30:20.456694 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.456654 4522 scope.go:122] "RemoveContainer" containerID="d55ced492d901c4202a262b13348021163d3262badf6ee20e85bf87f635effbc" Oct 30 17:30:20.470826 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.470761 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889690-s8ggr"] Oct 30 17:30:20.480707 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.474823 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/901bc210-b26c-41df-b40f-e94a12c578c5-kube-api-access-6j5f4\" (UniqueName: \"kubernetes.io/projected/901bc210-b26c-41df-b40f-e94a12c578c5-kube-api-access-6j5f4\") pod \"901bc210-b26c-41df-b40f-e94a12c578c5\" (UID: \"901bc210-b26c-41df-b40f-e94a12c578c5\") " Oct 30 17:30:20.480707 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.475132 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/901bc210-b26c-41df-b40f-e94a12c578c5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/901bc210-b26c-41df-b40f-e94a12c578c5-plugins-config-volume\") pod \"901bc210-b26c-41df-b40f-e94a12c578c5\" (UID: \"901bc210-b26c-41df-b40f-e94a12c578c5\") " Oct 30 17:30:20.480707 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.476387 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/901bc210-b26c-41df-b40f-e94a12c578c5-plugins-config-volume" pod "901bc210-b26c-41df-b40f-e94a12c578c5" (UID: "901bc210-b26c-41df-b40f-e94a12c578c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:30:20.480707 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.482349 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889690-csm2k" Oct 30 17:30:20.494343 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.484491 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/901bc210-b26c-41df-b40f-e94a12c578c5-kube-api-access-6j5f4" pod "901bc210-b26c-41df-b40f-e94a12c578c5" (UID: "901bc210-b26c-41df-b40f-e94a12c578c5"). InnerVolumeSpecName "kube-api-access-6j5f4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:30:20.494343 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.487876 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889690-xxcd6" Oct 30 17:30:20.564002 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.563916 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889690-7kjml" Oct 30 17:30:20.572516 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.572487 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c17e8c-29889690-nvd7z"] Oct 30 17:30:20.577070 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.577042 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1d79713-dd75-4823-a8f8-0f252befa273-kube-api-access-9nl59\" (UniqueName: \"kubernetes.io/projected/e1d79713-dd75-4823-a8f8-0f252befa273-kube-api-access-9nl59\") pod \"e1d79713-dd75-4823-a8f8-0f252befa273\" (UID: \"e1d79713-dd75-4823-a8f8-0f252befa273\") " Oct 30 17:30:20.577662 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.577641 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4fa40dd-b387-4513-aed9-8e11394d3454-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4fa40dd-b387-4513-aed9-8e11394d3454-maas-api-service-ca\") pod \"d4fa40dd-b387-4513-aed9-8e11394d3454\" (UID: \"d4fa40dd-b387-4513-aed9-8e11394d3454\") " Oct 30 17:30:20.578276 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.578164 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4fa40dd-b387-4513-aed9-8e11394d3454-maas-api-service-ca" pod "d4fa40dd-b387-4513-aed9-8e11394d3454" (UID: "d4fa40dd-b387-4513-aed9-8e11394d3454"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:30:20.578749 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.578717 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1d79713-dd75-4823-a8f8-0f252befa273-maas-api-service-ca" pod "e1d79713-dd75-4823-a8f8-0f252befa273" (UID: "e1d79713-dd75-4823-a8f8-0f252befa273"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:30:20.578884 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.578868 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1d79713-dd75-4823-a8f8-0f252befa273-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1d79713-dd75-4823-a8f8-0f252befa273-maas-api-service-ca\") pod \"e1d79713-dd75-4823-a8f8-0f252befa273\" (UID: \"e1d79713-dd75-4823-a8f8-0f252befa273\") " Oct 30 17:30:20.579142 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.579125 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4fa40dd-b387-4513-aed9-8e11394d3454-kube-api-access-jsrxd\" (UniqueName: \"kubernetes.io/projected/d4fa40dd-b387-4513-aed9-8e11394d3454-kube-api-access-jsrxd\") pod \"d4fa40dd-b387-4513-aed9-8e11394d3454\" (UID: \"d4fa40dd-b387-4513-aed9-8e11394d3454\") " Oct 30 17:30:20.580546 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.580517 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1d79713-dd75-4823-a8f8-0f252befa273-kube-api-access-9nl59" pod "e1d79713-dd75-4823-a8f8-0f252befa273" (UID: "e1d79713-dd75-4823-a8f8-0f252befa273"). InnerVolumeSpecName "kube-api-access-9nl59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:30:20.580876 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.580859 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j5f4\" (UniqueName: \"kubernetes.io/projected/901bc210-b26c-41df-b40f-e94a12c578c5-kube-api-access-6j5f4\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:30:20.580989 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.580977 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nl59\" (UniqueName: \"kubernetes.io/projected/e1d79713-dd75-4823-a8f8-0f252befa273-kube-api-access-9nl59\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:30:20.581081 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.581069 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4fa40dd-b387-4513-aed9-8e11394d3454-maas-api-service-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:30:20.581171 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.581160 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/901bc210-b26c-41df-b40f-e94a12c578c5-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:30:20.581259 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.581245 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1d79713-dd75-4823-a8f8-0f252befa273-maas-api-service-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:30:20.581597 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.581577 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4fa40dd-b387-4513-aed9-8e11394d3454-kube-api-access-jsrxd" pod "d4fa40dd-b387-4513-aed9-8e11394d3454" (UID: "d4fa40dd-b387-4513-aed9-8e11394d3454"). InnerVolumeSpecName "kube-api-access-jsrxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:30:20.583105 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.583083 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c17e8c-29889690-nvd7z"] Oct 30 17:30:20.687224 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.687026 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2ce4a71-84b5-4817-b99c-9d56d4673869-kube-api-access-zjx5q\" (UniqueName: \"kubernetes.io/projected/e2ce4a71-84b5-4817-b99c-9d56d4673869-kube-api-access-zjx5q\") pod \"e2ce4a71-84b5-4817-b99c-9d56d4673869\" (UID: \"e2ce4a71-84b5-4817-b99c-9d56d4673869\") " Oct 30 17:30:20.688526 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.687336 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2ce4a71-84b5-4817-b99c-9d56d4673869-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2ce4a71-84b5-4817-b99c-9d56d4673869-maas-api-service-ca\") pod \"e2ce4a71-84b5-4817-b99c-9d56d4673869\" (UID: \"e2ce4a71-84b5-4817-b99c-9d56d4673869\") " Oct 30 17:30:20.688526 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.688282 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsrxd\" (UniqueName: \"kubernetes.io/projected/d4fa40dd-b387-4513-aed9-8e11394d3454-kube-api-access-jsrxd\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:30:20.688526 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.688321 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2ce4a71-84b5-4817-b99c-9d56d4673869-maas-api-service-ca" pod "e2ce4a71-84b5-4817-b99c-9d56d4673869" (UID: "e2ce4a71-84b5-4817-b99c-9d56d4673869"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:30:20.692057 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.692023 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2ce4a71-84b5-4817-b99c-9d56d4673869-kube-api-access-zjx5q" pod "e2ce4a71-84b5-4817-b99c-9d56d4673869" (UID: "e2ce4a71-84b5-4817-b99c-9d56d4673869"). InnerVolumeSpecName "kube-api-access-zjx5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:30:20.734007 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.733978 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-63e60d-5b4499f4c9-5j2lv"] Oct 30 17:30:20.747619 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.747079 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-987dfc-29889690-vpglf" Oct 30 17:30:20.789946 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.789912 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2ce4a71-84b5-4817-b99c-9d56d4673869-maas-api-service-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:30:20.790265 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.789948 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjx5q\" (UniqueName: \"kubernetes.io/projected/e2ce4a71-84b5-4817-b99c-9d56d4673869-kube-api-access-zjx5q\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:30:20.792774 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.792746 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889690-ksf2d"] Oct 30 17:30:20.798495 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.798466 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db8417-29889690-ksf2d"] Oct 30 17:30:20.866917 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.866884 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-63e60d-85c58c5cdd-6k2gk"] Oct 30 17:30:20.869390 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:30:20.869360 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod300450d2_c1b2_4a64_bd41_4a1ec5df3d77.slice/crio-0238c506e505b6a32a0c0eabbd69be3e47aedaffcdc9e0ecc3eb21fe06f5d3dc WatchSource:0}: Error finding container 0238c506e505b6a32a0c0eabbd69be3e47aedaffcdc9e0ecc3eb21fe06f5d3dc: Status 404 returned error can't find the container with id 0238c506e505b6a32a0c0eabbd69be3e47aedaffcdc9e0ecc3eb21fe06f5d3dc Oct 30 17:30:20.890922 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.890892 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8643b717-c5ec-4194-92bb-e54de07fef00-kube-api-access-9kvtc\" (UniqueName: \"kubernetes.io/projected/8643b717-c5ec-4194-92bb-e54de07fef00-kube-api-access-9kvtc\") pod \"8643b717-c5ec-4194-92bb-e54de07fef00\" (UID: \"8643b717-c5ec-4194-92bb-e54de07fef00\") " Oct 30 17:30:20.891217 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.891200 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8643b717-c5ec-4194-92bb-e54de07fef00-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8643b717-c5ec-4194-92bb-e54de07fef00-maas-api-service-ca\") pod \"8643b717-c5ec-4194-92bb-e54de07fef00\" (UID: \"8643b717-c5ec-4194-92bb-e54de07fef00\") " Oct 30 17:30:20.891617 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.891589 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8643b717-c5ec-4194-92bb-e54de07fef00-maas-api-service-ca" pod "8643b717-c5ec-4194-92bb-e54de07fef00" (UID: "8643b717-c5ec-4194-92bb-e54de07fef00"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:30:20.892099 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.892068 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8643b717-c5ec-4194-92bb-e54de07fef00-maas-api-service-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:30:20.894818 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.894779 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8643b717-c5ec-4194-92bb-e54de07fef00-kube-api-access-9kvtc" pod "8643b717-c5ec-4194-92bb-e54de07fef00" (UID: "8643b717-c5ec-4194-92bb-e54de07fef00"). InnerVolumeSpecName "kube-api-access-9kvtc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:30:20.994318 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:20.994203 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kvtc\" (UniqueName: \"kubernetes.io/projected/8643b717-c5ec-4194-92bb-e54de07fef00-kube-api-access-9kvtc\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:30:21.133555 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:21.133478 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889690-xxcd6" Oct 30 17:30:21.133555 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:21.133503 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889690-xxcd6" event={"ID":"e1d79713-dd75-4823-a8f8-0f252befa273","Type":"ContainerDied","Data":"236eec2f64ac673811dcb9357338c132b028d6fe418ca9c013f591269ab97fda"} Oct 30 17:30:21.133836 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:21.133577 4522 scope.go:122] "RemoveContainer" containerID="82dc799ba7bcc72701d1f44dd435cadc6f55b6063ca8264d79af8e6591770292" Oct 30 17:30:21.140361 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:21.136586 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889690-7kjml" Oct 30 17:30:21.140361 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:21.136970 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889690-7kjml" event={"ID":"e2ce4a71-84b5-4817-b99c-9d56d4673869","Type":"ContainerDied","Data":"f415498cde0e270c5754dd1509c440ddca20353ea1fe8c33c48b707f6c0a91e5"} Oct 30 17:30:21.142548 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:21.142526 4522 generic.go:356] "Generic (PLEG): container finished" podID="58bcea19-790f-486c-8994-21b1b4c20dc9" containerID="4691fc40d7619feafbbc8d68786113ff31e0fabe10a9827fad62f52692688e3f" exitCode=60 Oct 30 17:30:21.142659 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:21.142554 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889690-qb6xz" event={"ID":"58bcea19-790f-486c-8994-21b1b4c20dc9","Type":"ContainerDied","Data":"4691fc40d7619feafbbc8d68786113ff31e0fabe10a9827fad62f52692688e3f"} Oct 30 17:30:21.143789 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:21.143768 4522 scope.go:122] "RemoveContainer" containerID="4691fc40d7619feafbbc8d68786113ff31e0fabe10a9827fad62f52692688e3f" Oct 30 17:30:21.150137 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:30:21.149169 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889690-qb6xz_odh-ai-gateway-infra(58bcea19-790f-486c-8994-21b1b4c20dc9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889690-qb6xz" podUID="58bcea19-790f-486c-8994-21b1b4c20dc9" Oct 30 17:30:21.157427 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:21.157400 4522 scope.go:122] "RemoveContainer" containerID="534d72f8def193e449fd2a2e9d82adffcb175383f5616bd53e7a93bda0240c5d" Oct 30 17:30:21.159103 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:21.159083 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8a7e4a-d55dd8b4d-hd4qr" Oct 30 17:30:21.159196 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:21.159160 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8a7e4a-d55dd8b4d-hd4qr" event={"ID":"901bc210-b26c-41df-b40f-e94a12c578c5","Type":"ContainerDied","Data":"305e40cee1369994de2fb89da5ad57b093d1a5fd5e4c16bae46ff47defe022ab"} Oct 30 17:30:21.180355 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:21.180299 4522 scope.go:122] "RemoveContainer" containerID="d95088e00c7af151fd2607ca2473d007fd1ff4cbe931c64c641a6a35de61ad01" Oct 30 17:30:21.180689 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:21.180657 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e60d-5b4499f4c9-5j2lv" event={"ID":"85668031-55c5-458e-a623-26d81c8d4703","Type":"ContainerStarted","Data":"f4cf50a33c6238ee27b0122125a7d238fa16c070ec5656e5d4b65384511d2b60"} Oct 30 17:30:21.180796 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:21.180719 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e60d-5b4499f4c9-5j2lv" event={"ID":"85668031-55c5-458e-a623-26d81c8d4703","Type":"ContainerStarted","Data":"58a02dd946eb139a5457c1e1623208328f6fa07aad7eaebfc46d50cd3deefb5c"} Oct 30 17:30:21.180982 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:21.180956 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e60d-5b4499f4c9-5j2lv" Oct 30 17:30:21.184364 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:21.184292 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-63e60d-85c58c5cdd-6k2gk" event={"ID":"300450d2-c1b2-4a64-bd41-4a1ec5df3d77","Type":"ContainerStarted","Data":"4c14bf70328cefea257af5321836cfa1200ef5131d702d5aacdb427560bce077"} Oct 30 17:30:21.184513 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:21.184390 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-63e60d-85c58c5cdd-6k2gk" event={"ID":"300450d2-c1b2-4a64-bd41-4a1ec5df3d77","Type":"ContainerStarted","Data":"0238c506e505b6a32a0c0eabbd69be3e47aedaffcdc9e0ecc3eb21fe06f5d3dc"} Oct 30 17:30:21.184513 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:21.184437 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-63e60d-85c58c5cdd-6k2gk" Oct 30 17:30:21.185504 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:21.185486 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889690-qb6xz"] Oct 30 17:30:21.186167 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:21.186150 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-987dfc-29889690-vpglf" Oct 30 17:30:21.186268 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:21.186175 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-987dfc-29889690-vpglf" event={"ID":"8643b717-c5ec-4194-92bb-e54de07fef00","Type":"ContainerDied","Data":"dacb3345d05247a50f48623c812290949a69b273eff4dc01f3fe5e968e458ac8"} Oct 30 17:30:21.192520 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:21.192490 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889690-csm2k" Oct 30 17:30:21.192621 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:21.192540 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889690-csm2k" event={"ID":"d4fa40dd-b387-4513-aed9-8e11394d3454","Type":"ContainerDied","Data":"36728edab92c8c97c05007f642d1959b6079ed8865c7b6fd3e7698c24bbf5ab6"} Oct 30 17:30:21.203783 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:21.203762 4522 scope.go:122] "RemoveContainer" containerID="86f928ca250d042a74c0658e77b9760e7fa6d5fef063db548206bcc856d499b7" Oct 30 17:30:21.210475 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:21.210426 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-001a96-29889690-j9qwv" event={"ID":"f3c414cb-e4f0-4fbf-9bdf-946842e1297f","Type":"ContainerDied","Data":"cdbd0cdf61fd56dc15b734c0d93323b25757767745683db2346033f13aa98f4d"} Oct 30 17:30:21.210810 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:21.210793 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-001a96-29889690-j9qwv" Oct 30 17:30:21.222017 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:21.221985 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889690-d76sf" event={"ID":"b49aaf80-dff6-40d4-8404-d55de5b568ae","Type":"ContainerDied","Data":"06fe34004885f96ec33bf8bf8161a1908a1dfd07c438656534df37a46c1f57be"} Oct 30 17:30:21.222139 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:21.222113 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889690-d76sf" Oct 30 17:30:21.297349 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:21.289927 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-63e60d-85c58c5cdd-6k2gk" podStartSLOduration=2.289905148 podStartE2EDuration="2.289905148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:21.269241204 +0000 UTC m=+2595.269996123" watchObservedRunningTime="2026-10-30 17:30:21.289905148 +0000 UTC m=+2595.290660051" Oct 30 17:30:21.297349 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:21.293372 4522 scope.go:122] "RemoveContainer" containerID="1b908d71c8876e4255ec418018ad6e60eea76c03b9ec17a1bf79950ccc0faf1e" Oct 30 17:30:21.338035 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:21.336570 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e60d-5b4499f4c9-5j2lv" podStartSLOduration=2.336549905 podStartE2EDuration="2.336549905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:21.333349292 +0000 UTC m=+2595.334104189" watchObservedRunningTime="2026-10-30 17:30:21.336549905 +0000 UTC m=+2595.337304808" Oct 30 17:30:21.351147 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:21.351107 4522 scope.go:122] "RemoveContainer" containerID="31764ae7c08069ec0108cbc3a62fa5c91340c75eeba96d201f709c741029b66a" Oct 30 17:30:21.375526 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:21.375466 4522 scope.go:122] "RemoveContainer" containerID="f963d73fb5012cd07e17c2fd2d7fc2c014dcc09c82f54ee7675301024e3c4f77" Oct 30 17:30:21.389706 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:21.389682 4522 scope.go:122] "RemoveContainer" containerID="525ef17aa201bc4c2a575864b9a7e9ae90281e354311f703cacd50c6a7ecccd3" Oct 30 17:30:21.424545 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:21.424510 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-987dfc-29889690-vpglf"] Oct 30 17:30:21.463409 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:21.462407 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-987dfc-29889690-vpglf"] Oct 30 17:30:21.496008 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:21.495978 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889690-xxcd6"] Oct 30 17:30:21.513982 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:21.513895 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-82243b-29889690-dmgd9" Oct 30 17:30:21.526383 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:21.526345 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889690-xxcd6"] Oct 30 17:30:21.568177 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:21.568144 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8a7e4a-d55dd8b4d-hd4qr"] Oct 30 17:30:21.574864 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:21.574832 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8a7e4a-d55dd8b4d-hd4qr"] Oct 30 17:30:21.574864 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:21.574855 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e13a51-29889690-dg6vt" Oct 30 17:30:21.608955 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:21.608923 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889690-7kjml"] Oct 30 17:30:21.613273 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:21.613235 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6b0d3f6-2109-40b3-b86f-b895a4616558-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6b0d3f6-2109-40b3-b86f-b895a4616558-maas-api-service-ca\") pod \"f6b0d3f6-2109-40b3-b86f-b895a4616558\" (UID: \"f6b0d3f6-2109-40b3-b86f-b895a4616558\") " Oct 30 17:30:21.613561 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:21.613544 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6b0d3f6-2109-40b3-b86f-b895a4616558-kube-api-access-925th\" (UniqueName: \"kubernetes.io/projected/f6b0d3f6-2109-40b3-b86f-b895a4616558-kube-api-access-925th\") pod \"f6b0d3f6-2109-40b3-b86f-b895a4616558\" (UID: \"f6b0d3f6-2109-40b3-b86f-b895a4616558\") " Oct 30 17:30:21.613711 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:21.613682 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6b0d3f6-2109-40b3-b86f-b895a4616558-maas-api-service-ca" pod "f6b0d3f6-2109-40b3-b86f-b895a4616558" (UID: "f6b0d3f6-2109-40b3-b86f-b895a4616558"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:30:21.614179 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:21.614158 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6b0d3f6-2109-40b3-b86f-b895a4616558-maas-api-service-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:30:21.614259 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:21.614195 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889690-7kjml"] Oct 30 17:30:21.615654 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:21.615631 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6b0d3f6-2109-40b3-b86f-b895a4616558-kube-api-access-925th" pod "f6b0d3f6-2109-40b3-b86f-b895a4616558" (UID: "f6b0d3f6-2109-40b3-b86f-b895a4616558"). InnerVolumeSpecName "kube-api-access-925th". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:30:21.710196 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:21.710156 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889690-csm2k"] Oct 30 17:30:21.715222 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:21.715195 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889690-csm2k"] Oct 30 17:30:21.715736 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:21.715713 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4febc82-5717-4157-b46b-7ad8004d7f6a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4febc82-5717-4157-b46b-7ad8004d7f6a-maas-api-service-ca\") pod \"e4febc82-5717-4157-b46b-7ad8004d7f6a\" (UID: \"e4febc82-5717-4157-b46b-7ad8004d7f6a\") " Oct 30 17:30:21.715792 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:21.715778 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4febc82-5717-4157-b46b-7ad8004d7f6a-kube-api-access-6kgvk\" (UniqueName: \"kubernetes.io/projected/e4febc82-5717-4157-b46b-7ad8004d7f6a-kube-api-access-6kgvk\") pod \"e4febc82-5717-4157-b46b-7ad8004d7f6a\" (UID: \"e4febc82-5717-4157-b46b-7ad8004d7f6a\") " Oct 30 17:30:21.716151 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:21.716129 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4febc82-5717-4157-b46b-7ad8004d7f6a-maas-api-service-ca" pod "e4febc82-5717-4157-b46b-7ad8004d7f6a" (UID: "e4febc82-5717-4157-b46b-7ad8004d7f6a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:30:21.716493 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:21.716461 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4febc82-5717-4157-b46b-7ad8004d7f6a-maas-api-service-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:30:21.716493 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:21.716494 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-925th\" (UniqueName: \"kubernetes.io/projected/f6b0d3f6-2109-40b3-b86f-b895a4616558-kube-api-access-925th\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:30:21.718105 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:21.718081 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4febc82-5717-4157-b46b-7ad8004d7f6a-kube-api-access-6kgvk" pod "e4febc82-5717-4157-b46b-7ad8004d7f6a" (UID: "e4febc82-5717-4157-b46b-7ad8004d7f6a"). InnerVolumeSpecName "kube-api-access-6kgvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:30:21.796125 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:21.796032 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889690-d76sf"] Oct 30 17:30:21.801467 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:21.801434 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889690-d76sf"] Oct 30 17:30:21.818860 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:21.818822 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kgvk\" (UniqueName: \"kubernetes.io/projected/e4febc82-5717-4157-b46b-7ad8004d7f6a-kube-api-access-6kgvk\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:30:21.885269 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:21.885231 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-001a96-29889690-j9qwv"] Oct 30 17:30:21.890330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:21.890278 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-001a96-29889690-j9qwv"] Oct 30 17:30:22.148945 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:22.148866 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22ffe3de-e5e2-43a1-b276-20cf180c8b01" path="/var/lib/kubelet/pods/22ffe3de-e5e2-43a1-b276-20cf180c8b01/volumes" Oct 30 17:30:22.149513 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:22.149493 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="427507de-33c6-409a-a68c-8e5a978b24ed" path="/var/lib/kubelet/pods/427507de-33c6-409a-a68c-8e5a978b24ed/volumes" Oct 30 17:30:22.150001 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:22.149987 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46fba0fa-d263-42d2-84fd-ddaefea25c63" path="/var/lib/kubelet/pods/46fba0fa-d263-42d2-84fd-ddaefea25c63/volumes" Oct 30 17:30:22.150786 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:22.150771 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8643b717-c5ec-4194-92bb-e54de07fef00" path="/var/lib/kubelet/pods/8643b717-c5ec-4194-92bb-e54de07fef00/volumes" Oct 30 17:30:22.151250 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:22.151236 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="901bc210-b26c-41df-b40f-e94a12c578c5" path="/var/lib/kubelet/pods/901bc210-b26c-41df-b40f-e94a12c578c5/volumes" Oct 30 17:30:22.151698 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:22.151684 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b49aaf80-dff6-40d4-8404-d55de5b568ae" path="/var/lib/kubelet/pods/b49aaf80-dff6-40d4-8404-d55de5b568ae/volumes" Oct 30 17:30:22.152448 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:22.152433 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4fa40dd-b387-4513-aed9-8e11394d3454" path="/var/lib/kubelet/pods/d4fa40dd-b387-4513-aed9-8e11394d3454/volumes" Oct 30 17:30:22.152893 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:22.152876 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddc0e647-bfb8-474d-b423-be4d72b31b32" path="/var/lib/kubelet/pods/ddc0e647-bfb8-474d-b423-be4d72b31b32/volumes" Oct 30 17:30:22.153333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:22.153319 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1d79713-dd75-4823-a8f8-0f252befa273" path="/var/lib/kubelet/pods/e1d79713-dd75-4823-a8f8-0f252befa273/volumes" Oct 30 17:30:22.154010 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:22.153996 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2ce4a71-84b5-4817-b99c-9d56d4673869" path="/var/lib/kubelet/pods/e2ce4a71-84b5-4817-b99c-9d56d4673869/volumes" Oct 30 17:30:22.154416 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:22.154403 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3c414cb-e4f0-4fbf-9bdf-946842e1297f" path="/var/lib/kubelet/pods/f3c414cb-e4f0-4fbf-9bdf-946842e1297f/volumes" Oct 30 17:30:22.247346 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:22.247293 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-82243b-29889690-dmgd9" Oct 30 17:30:22.247499 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:22.247292 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-82243b-29889690-dmgd9" event={"ID":"f6b0d3f6-2109-40b3-b86f-b895a4616558","Type":"ContainerDied","Data":"762943ceeef17a1b4e12283a92b2790fa0acc76e86b253da7d8cdf610a191d82"} Oct 30 17:30:22.247499 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:22.247461 4522 scope.go:122] "RemoveContainer" containerID="14bd60669dc55d9551bb680ccd2764b4d2207bd0c6517423be412fe7439a217a" Oct 30 17:30:22.252026 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:22.252005 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e13a51-29889690-dg6vt" Oct 30 17:30:22.252152 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:22.252024 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e13a51-29889690-dg6vt" event={"ID":"e4febc82-5717-4157-b46b-7ad8004d7f6a","Type":"ContainerDied","Data":"8f82066950325ee9e821de864508d3134a8cfd09924dc28b572afc31fdfaccf2"} Oct 30 17:30:22.261695 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:22.261673 4522 scope.go:122] "RemoveContainer" containerID="a2b79f75da1879e8232c5018de5d4c88a4370d85db75e1b7701bcc80716efcdb" Oct 30 17:30:22.335116 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:22.335081 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e13a51-29889690-dg6vt"] Oct 30 17:30:22.355562 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:22.355530 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e13a51-29889690-dg6vt"] Oct 30 17:30:22.397434 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:22.397405 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-82243b-29889690-dmgd9"] Oct 30 17:30:22.402737 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:22.402673 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-82243b-29889690-dmgd9"] Oct 30 17:30:22.447955 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:22.447908 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889690-qb6xz" Oct 30 17:30:22.529240 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:22.529197 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58bcea19-790f-486c-8994-21b1b4c20dc9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58bcea19-790f-486c-8994-21b1b4c20dc9-maas-api-service-ca\") pod \"58bcea19-790f-486c-8994-21b1b4c20dc9\" (UID: \"58bcea19-790f-486c-8994-21b1b4c20dc9\") " Oct 30 17:30:22.529439 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:22.529344 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58bcea19-790f-486c-8994-21b1b4c20dc9-kube-api-access-nzdnr\" (UniqueName: \"kubernetes.io/projected/58bcea19-790f-486c-8994-21b1b4c20dc9-kube-api-access-nzdnr\") pod \"58bcea19-790f-486c-8994-21b1b4c20dc9\" (UID: \"58bcea19-790f-486c-8994-21b1b4c20dc9\") " Oct 30 17:30:22.529631 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:22.529601 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58bcea19-790f-486c-8994-21b1b4c20dc9-maas-api-service-ca" pod "58bcea19-790f-486c-8994-21b1b4c20dc9" (UID: "58bcea19-790f-486c-8994-21b1b4c20dc9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:30:22.530300 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:22.530272 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58bcea19-790f-486c-8994-21b1b4c20dc9-maas-api-service-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:30:22.532590 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:22.532547 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58bcea19-790f-486c-8994-21b1b4c20dc9-kube-api-access-nzdnr" pod "58bcea19-790f-486c-8994-21b1b4c20dc9" (UID: "58bcea19-790f-486c-8994-21b1b4c20dc9"). InnerVolumeSpecName "kube-api-access-nzdnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:30:22.632393 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:22.632352 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzdnr\" (UniqueName: \"kubernetes.io/projected/58bcea19-790f-486c-8994-21b1b4c20dc9-kube-api-access-nzdnr\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:30:23.271378 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:23.271339 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889690-qb6xz" Oct 30 17:30:23.271863 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:23.271338 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889690-qb6xz" event={"ID":"58bcea19-790f-486c-8994-21b1b4c20dc9","Type":"ContainerDied","Data":"1d3b149ca37f1fc8c8b9b72987a865d6faa0aa285d71fd65c6d1dfe59fe27323"} Oct 30 17:30:23.271863 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:23.271496 4522 scope.go:122] "RemoveContainer" containerID="4691fc40d7619feafbbc8d68786113ff31e0fabe10a9827fad62f52692688e3f" Oct 30 17:30:23.333633 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:23.333600 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889690-qb6xz"] Oct 30 17:30:23.338293 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:23.338272 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8681b5-29889690-qb6xz"] Oct 30 17:30:24.148007 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:24.147971 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58bcea19-790f-486c-8994-21b1b4c20dc9" path="/var/lib/kubelet/pods/58bcea19-790f-486c-8994-21b1b4c20dc9/volumes" Oct 30 17:30:24.148537 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:24.148521 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4febc82-5717-4157-b46b-7ad8004d7f6a" path="/var/lib/kubelet/pods/e4febc82-5717-4157-b46b-7ad8004d7f6a/volumes" Oct 30 17:30:24.148973 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:24.148960 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6b0d3f6-2109-40b3-b86f-b895a4616558" path="/var/lib/kubelet/pods/f6b0d3f6-2109-40b3-b86f-b895a4616558/volumes" Oct 30 17:30:32.258302 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.258265 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-63e60d-85c58c5cdd-6k2gk" Oct 30 17:30:32.258811 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.258376 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e60d-5b4499f4c9-5j2lv" Oct 30 17:30:32.943768 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.943727 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c17e8c-567d66ffff-l28mb"] Oct 30 17:30:32.944763 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.944739 4522 cpu_manager.go:414] "containerMap: removing container" podUID="e1d79713-dd75-4823-a8f8-0f252befa273" containerName="cleanup" Oct 30 17:30:32.944763 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.944761 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1d79713-dd75-4823-a8f8-0f252befa273" containerName="cleanup" Oct 30 17:30:32.944763 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.944776 4522 cpu_manager.go:414] "containerMap: removing container" podUID="58bcea19-790f-486c-8994-21b1b4c20dc9" containerName="cleanup" Oct 30 17:30:32.945058 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.944782 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58bcea19-790f-486c-8994-21b1b4c20dc9" containerName="cleanup" Oct 30 17:30:32.945058 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.944793 4522 cpu_manager.go:414] "containerMap: removing container" podUID="f3c414cb-e4f0-4fbf-9bdf-946842e1297f" containerName="cleanup" Oct 30 17:30:32.945058 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.944798 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3c414cb-e4f0-4fbf-9bdf-946842e1297f" containerName="cleanup" Oct 30 17:30:32.945058 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.944804 4522 cpu_manager.go:414] "containerMap: removing container" podUID="f6b0d3f6-2109-40b3-b86f-b895a4616558" containerName="cleanup" Oct 30 17:30:32.945058 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.944809 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6b0d3f6-2109-40b3-b86f-b895a4616558" containerName="cleanup" Oct 30 17:30:32.945058 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.944817 4522 cpu_manager.go:414] "containerMap: removing container" podUID="f3c414cb-e4f0-4fbf-9bdf-946842e1297f" containerName="cleanup" Oct 30 17:30:32.945058 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.944821 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3c414cb-e4f0-4fbf-9bdf-946842e1297f" containerName="cleanup" Oct 30 17:30:32.945058 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.944836 4522 cpu_manager.go:414] "containerMap: removing container" podUID="8643b717-c5ec-4194-92bb-e54de07fef00" containerName="cleanup" Oct 30 17:30:32.945058 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.944843 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8643b717-c5ec-4194-92bb-e54de07fef00" containerName="cleanup" Oct 30 17:30:32.945058 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.944852 4522 cpu_manager.go:414] "containerMap: removing container" podUID="e4febc82-5717-4157-b46b-7ad8004d7f6a" containerName="cleanup" Oct 30 17:30:32.945058 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.944857 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4febc82-5717-4157-b46b-7ad8004d7f6a" containerName="cleanup" Oct 30 17:30:32.945058 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.944868 4522 cpu_manager.go:414] "containerMap: removing container" podUID="f6b0d3f6-2109-40b3-b86f-b895a4616558" containerName="cleanup" Oct 30 17:30:32.945058 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.944872 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6b0d3f6-2109-40b3-b86f-b895a4616558" containerName="cleanup" Oct 30 17:30:32.945058 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.944882 4522 cpu_manager.go:414] "containerMap: removing container" podUID="b49aaf80-dff6-40d4-8404-d55de5b568ae" containerName="cleanup" Oct 30 17:30:32.945058 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.944886 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b49aaf80-dff6-40d4-8404-d55de5b568ae" containerName="cleanup" Oct 30 17:30:32.945058 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.944893 4522 cpu_manager.go:414] "containerMap: removing container" podUID="d4fa40dd-b387-4513-aed9-8e11394d3454" containerName="cleanup" Oct 30 17:30:32.945058 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.944897 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4fa40dd-b387-4513-aed9-8e11394d3454" containerName="cleanup" Oct 30 17:30:32.945058 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.944906 4522 cpu_manager.go:414] "containerMap: removing container" podUID="e1d79713-dd75-4823-a8f8-0f252befa273" containerName="cleanup" Oct 30 17:30:32.945058 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.944912 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1d79713-dd75-4823-a8f8-0f252befa273" containerName="cleanup" Oct 30 17:30:32.945058 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.944927 4522 cpu_manager.go:414] "containerMap: removing container" podUID="e2ce4a71-84b5-4817-b99c-9d56d4673869" containerName="cleanup" Oct 30 17:30:32.945058 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.944934 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2ce4a71-84b5-4817-b99c-9d56d4673869" containerName="cleanup" Oct 30 17:30:32.945058 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.944940 4522 cpu_manager.go:414] "containerMap: removing container" podUID="901bc210-b26c-41df-b40f-e94a12c578c5" containerName="payload-processing" Oct 30 17:30:32.945058 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.944944 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="901bc210-b26c-41df-b40f-e94a12c578c5" containerName="payload-processing" Oct 30 17:30:32.945058 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.944954 4522 cpu_manager.go:414] "containerMap: removing container" podUID="d4fa40dd-b387-4513-aed9-8e11394d3454" containerName="cleanup" Oct 30 17:30:32.945058 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.944958 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4fa40dd-b387-4513-aed9-8e11394d3454" containerName="cleanup" Oct 30 17:30:32.945058 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.944969 4522 cpu_manager.go:414] "containerMap: removing container" podUID="f6b0d3f6-2109-40b3-b86f-b895a4616558" containerName="cleanup" Oct 30 17:30:32.945058 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.944973 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6b0d3f6-2109-40b3-b86f-b895a4616558" containerName="cleanup" Oct 30 17:30:32.945058 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.944981 4522 cpu_manager.go:414] "containerMap: removing container" podUID="e4febc82-5717-4157-b46b-7ad8004d7f6a" containerName="cleanup" Oct 30 17:30:32.945058 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.944989 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4febc82-5717-4157-b46b-7ad8004d7f6a" containerName="cleanup" Oct 30 17:30:32.945058 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.945002 4522 cpu_manager.go:414] "containerMap: removing container" podUID="b49aaf80-dff6-40d4-8404-d55de5b568ae" containerName="cleanup" Oct 30 17:30:32.945058 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.945008 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b49aaf80-dff6-40d4-8404-d55de5b568ae" containerName="cleanup" Oct 30 17:30:32.945058 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.945018 4522 cpu_manager.go:414] "containerMap: removing container" podUID="e4febc82-5717-4157-b46b-7ad8004d7f6a" containerName="cleanup" Oct 30 17:30:32.945058 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.945024 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4febc82-5717-4157-b46b-7ad8004d7f6a" containerName="cleanup" Oct 30 17:30:32.945058 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.945035 4522 cpu_manager.go:414] "containerMap: removing container" podUID="22ffe3de-e5e2-43a1-b276-20cf180c8b01" containerName="cleanup" Oct 30 17:30:32.945058 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.945039 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22ffe3de-e5e2-43a1-b276-20cf180c8b01" containerName="cleanup" Oct 30 17:30:32.945058 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.945047 4522 cpu_manager.go:414] "containerMap: removing container" podUID="22ffe3de-e5e2-43a1-b276-20cf180c8b01" containerName="cleanup" Oct 30 17:30:32.945058 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.945051 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22ffe3de-e5e2-43a1-b276-20cf180c8b01" containerName="cleanup" Oct 30 17:30:32.945058 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.945063 4522 cpu_manager.go:414] "containerMap: removing container" podUID="8643b717-c5ec-4194-92bb-e54de07fef00" containerName="cleanup" Oct 30 17:30:32.945058 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.945071 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8643b717-c5ec-4194-92bb-e54de07fef00" containerName="cleanup" Oct 30 17:30:32.946361 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.945086 4522 cpu_manager.go:414] "containerMap: removing container" podUID="58bcea19-790f-486c-8994-21b1b4c20dc9" containerName="cleanup" Oct 30 17:30:32.946361 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.945091 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58bcea19-790f-486c-8994-21b1b4c20dc9" containerName="cleanup" Oct 30 17:30:32.946361 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.945099 4522 cpu_manager.go:414] "containerMap: removing container" podUID="d4fa40dd-b387-4513-aed9-8e11394d3454" containerName="cleanup" Oct 30 17:30:32.946361 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.945103 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4fa40dd-b387-4513-aed9-8e11394d3454" containerName="cleanup" Oct 30 17:30:32.946361 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.945111 4522 cpu_manager.go:414] "containerMap: removing container" podUID="e2ce4a71-84b5-4817-b99c-9d56d4673869" containerName="cleanup" Oct 30 17:30:32.946361 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.945115 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2ce4a71-84b5-4817-b99c-9d56d4673869" containerName="cleanup" Oct 30 17:30:32.946361 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.945123 4522 cpu_manager.go:414] "containerMap: removing container" podUID="e2ce4a71-84b5-4817-b99c-9d56d4673869" containerName="cleanup" Oct 30 17:30:32.946361 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.945127 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2ce4a71-84b5-4817-b99c-9d56d4673869" containerName="cleanup" Oct 30 17:30:32.946361 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.945140 4522 cpu_manager.go:414] "containerMap: removing container" podUID="22ffe3de-e5e2-43a1-b276-20cf180c8b01" containerName="cleanup" Oct 30 17:30:32.946361 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.945147 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22ffe3de-e5e2-43a1-b276-20cf180c8b01" containerName="cleanup" Oct 30 17:30:32.946361 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.945281 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6b0d3f6-2109-40b3-b86f-b895a4616558" containerName="cleanup" Oct 30 17:30:32.946361 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.945289 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4febc82-5717-4157-b46b-7ad8004d7f6a" containerName="cleanup" Oct 30 17:30:32.946361 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.945295 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="b49aaf80-dff6-40d4-8404-d55de5b568ae" containerName="cleanup" Oct 30 17:30:32.946361 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.945315 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4febc82-5717-4157-b46b-7ad8004d7f6a" containerName="cleanup" Oct 30 17:30:32.946361 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.945323 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="22ffe3de-e5e2-43a1-b276-20cf180c8b01" containerName="cleanup" Oct 30 17:30:32.946361 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.945333 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="22ffe3de-e5e2-43a1-b276-20cf180c8b01" containerName="cleanup" Oct 30 17:30:32.946361 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.945339 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="58bcea19-790f-486c-8994-21b1b4c20dc9" containerName="cleanup" Oct 30 17:30:32.946361 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.945345 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="8643b717-c5ec-4194-92bb-e54de07fef00" containerName="cleanup" Oct 30 17:30:32.946361 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.945349 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3c414cb-e4f0-4fbf-9bdf-946842e1297f" containerName="cleanup" Oct 30 17:30:32.946361 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.945356 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="58bcea19-790f-486c-8994-21b1b4c20dc9" containerName="cleanup" Oct 30 17:30:32.946361 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.945362 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1d79713-dd75-4823-a8f8-0f252befa273" containerName="cleanup" Oct 30 17:30:32.946361 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.945368 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4fa40dd-b387-4513-aed9-8e11394d3454" containerName="cleanup" Oct 30 17:30:32.946361 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.945373 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2ce4a71-84b5-4817-b99c-9d56d4673869" containerName="cleanup" Oct 30 17:30:32.946361 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.945378 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2ce4a71-84b5-4817-b99c-9d56d4673869" containerName="cleanup" Oct 30 17:30:32.946361 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.945387 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1d79713-dd75-4823-a8f8-0f252befa273" containerName="cleanup" Oct 30 17:30:32.946361 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.945393 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="58bcea19-790f-486c-8994-21b1b4c20dc9" containerName="cleanup" Oct 30 17:30:32.946361 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.945401 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6b0d3f6-2109-40b3-b86f-b895a4616558" containerName="cleanup" Oct 30 17:30:32.946361 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.945410 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3c414cb-e4f0-4fbf-9bdf-946842e1297f" containerName="cleanup" Oct 30 17:30:32.946361 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.945416 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="8643b717-c5ec-4194-92bb-e54de07fef00" containerName="cleanup" Oct 30 17:30:32.946361 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.945422 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6b0d3f6-2109-40b3-b86f-b895a4616558" containerName="cleanup" Oct 30 17:30:32.946361 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.945427 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="b49aaf80-dff6-40d4-8404-d55de5b568ae" containerName="cleanup" Oct 30 17:30:32.946361 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.945431 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4fa40dd-b387-4513-aed9-8e11394d3454" containerName="cleanup" Oct 30 17:30:32.946361 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.945436 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="427507de-33c6-409a-a68c-8e5a978b24ed" containerName="cleanup" Oct 30 17:30:32.946361 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.945442 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="901bc210-b26c-41df-b40f-e94a12c578c5" containerName="payload-processing" Oct 30 17:30:32.951082 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.951060 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c17e8c-567d66ffff-l28mb" Oct 30 17:30:32.976187 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.968784 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c17e8c\"" Oct 30 17:30:32.976187 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.969103 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c17e8c-567d66ffff-l28mb"] Oct 30 17:30:32.976187 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:32.969521 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-c17e8c-dockercfg-hvhcm\"" Oct 30 17:30:33.039695 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:33.039567 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/369893a4-34e9-48a2-8246-7abf4a51018b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c17e8c-567d66ffff-l28mb\" (UID: \"369893a4-34e9-48a2-8246-7abf4a51018b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c17e8c-567d66ffff-l28mb" Oct 30 17:30:33.039975 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:33.039951 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxj8x\" (UniqueName: \"kubernetes.io/projected/369893a4-34e9-48a2-8246-7abf4a51018b-kube-api-access-fxj8x\") pod \"payload-pre-processing-e2e-ipp-a-c17e8c-567d66ffff-l28mb\" (UID: \"369893a4-34e9-48a2-8246-7abf4a51018b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c17e8c-567d66ffff-l28mb" Oct 30 17:30:33.096175 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:33.096140 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c17e8c-84c889b56d-4rrvl"] Oct 30 17:30:33.097056 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:33.097033 4522 cpu_manager.go:414] "containerMap: removing container" podUID="58bcea19-790f-486c-8994-21b1b4c20dc9" containerName="cleanup" Oct 30 17:30:33.097056 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:33.097055 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58bcea19-790f-486c-8994-21b1b4c20dc9" containerName="cleanup" Oct 30 17:30:33.097150 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:33.097070 4522 cpu_manager.go:414] "containerMap: removing container" podUID="f3c414cb-e4f0-4fbf-9bdf-946842e1297f" containerName="cleanup" Oct 30 17:30:33.097150 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:33.097076 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3c414cb-e4f0-4fbf-9bdf-946842e1297f" containerName="cleanup" Oct 30 17:30:33.097150 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:33.097098 4522 cpu_manager.go:414] "containerMap: removing container" podUID="e1d79713-dd75-4823-a8f8-0f252befa273" containerName="cleanup" Oct 30 17:30:33.097150 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:33.097104 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1d79713-dd75-4823-a8f8-0f252befa273" containerName="cleanup" Oct 30 17:30:33.097279 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:33.097122 4522 cpu_manager.go:414] "containerMap: removing container" podUID="b49aaf80-dff6-40d4-8404-d55de5b568ae" containerName="cleanup" Oct 30 17:30:33.097279 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:33.097182 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b49aaf80-dff6-40d4-8404-d55de5b568ae" containerName="cleanup" Oct 30 17:30:33.097279 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:33.097213 4522 cpu_manager.go:414] "containerMap: removing container" podUID="8643b717-c5ec-4194-92bb-e54de07fef00" containerName="cleanup" Oct 30 17:30:33.097279 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:33.097220 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8643b717-c5ec-4194-92bb-e54de07fef00" containerName="cleanup" Oct 30 17:30:33.097488 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:33.097420 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1d79713-dd75-4823-a8f8-0f252befa273" containerName="cleanup" Oct 30 17:30:33.097488 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:33.097435 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3c414cb-e4f0-4fbf-9bdf-946842e1297f" containerName="cleanup" Oct 30 17:30:33.097488 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:33.097444 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="b49aaf80-dff6-40d4-8404-d55de5b568ae" containerName="cleanup" Oct 30 17:30:33.097488 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:33.097450 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4fa40dd-b387-4513-aed9-8e11394d3454" containerName="cleanup" Oct 30 17:30:33.097488 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:33.097460 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4febc82-5717-4157-b46b-7ad8004d7f6a" containerName="cleanup" Oct 30 17:30:33.097488 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:33.097468 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="22ffe3de-e5e2-43a1-b276-20cf180c8b01" containerName="cleanup" Oct 30 17:30:33.097488 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:33.097475 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="8643b717-c5ec-4194-92bb-e54de07fef00" containerName="cleanup" Oct 30 17:30:33.097488 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:33.097483 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2ce4a71-84b5-4817-b99c-9d56d4673869" containerName="cleanup" Oct 30 17:30:33.101827 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:33.101804 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c17e8c-84c889b56d-4rrvl" Oct 30 17:30:33.123358 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:33.123284 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c17e8c-84c889b56d-4rrvl"] Oct 30 17:30:33.141482 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:33.141452 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/369893a4-34e9-48a2-8246-7abf4a51018b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c17e8c-567d66ffff-l28mb\" (UID: \"369893a4-34e9-48a2-8246-7abf4a51018b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c17e8c-567d66ffff-l28mb" Oct 30 17:30:33.141667 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:33.141490 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxj8x\" (UniqueName: \"kubernetes.io/projected/369893a4-34e9-48a2-8246-7abf4a51018b-kube-api-access-fxj8x\") pod \"payload-pre-processing-e2e-ipp-a-c17e8c-567d66ffff-l28mb\" (UID: \"369893a4-34e9-48a2-8246-7abf4a51018b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c17e8c-567d66ffff-l28mb" Oct 30 17:30:33.142232 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:33.142208 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/369893a4-34e9-48a2-8246-7abf4a51018b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c17e8c-567d66ffff-l28mb\" (UID: \"369893a4-34e9-48a2-8246-7abf4a51018b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c17e8c-567d66ffff-l28mb" Oct 30 17:30:33.166834 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:33.166801 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxj8x\" (UniqueName: \"kubernetes.io/projected/369893a4-34e9-48a2-8246-7abf4a51018b-kube-api-access-fxj8x\") pod \"payload-pre-processing-e2e-ipp-a-c17e8c-567d66ffff-l28mb\" (UID: \"369893a4-34e9-48a2-8246-7abf4a51018b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c17e8c-567d66ffff-l28mb" Oct 30 17:30:33.243537 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:33.243502 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2676z\" (UniqueName: \"kubernetes.io/projected/dd7c8134-eff8-4014-8bb6-d10ca68651ba-kube-api-access-2676z\") pod \"payload-processing-e2e-ipp-a-c17e8c-84c889b56d-4rrvl\" (UID: \"dd7c8134-eff8-4014-8bb6-d10ca68651ba\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c17e8c-84c889b56d-4rrvl" Oct 30 17:30:33.243736 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:33.243585 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd7c8134-eff8-4014-8bb6-d10ca68651ba-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c17e8c-84c889b56d-4rrvl\" (UID: \"dd7c8134-eff8-4014-8bb6-d10ca68651ba\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c17e8c-84c889b56d-4rrvl" Oct 30 17:30:33.280658 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:33.280607 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c17e8c-567d66ffff-l28mb" Oct 30 17:30:33.345515 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:33.345468 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2676z\" (UniqueName: \"kubernetes.io/projected/dd7c8134-eff8-4014-8bb6-d10ca68651ba-kube-api-access-2676z\") pod \"payload-processing-e2e-ipp-a-c17e8c-84c889b56d-4rrvl\" (UID: \"dd7c8134-eff8-4014-8bb6-d10ca68651ba\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c17e8c-84c889b56d-4rrvl" Oct 30 17:30:33.345604 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:33.345549 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd7c8134-eff8-4014-8bb6-d10ca68651ba-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c17e8c-84c889b56d-4rrvl\" (UID: \"dd7c8134-eff8-4014-8bb6-d10ca68651ba\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c17e8c-84c889b56d-4rrvl" Oct 30 17:30:33.346652 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:33.346361 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd7c8134-eff8-4014-8bb6-d10ca68651ba-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c17e8c-84c889b56d-4rrvl\" (UID: \"dd7c8134-eff8-4014-8bb6-d10ca68651ba\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c17e8c-84c889b56d-4rrvl" Oct 30 17:30:33.384509 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:33.384432 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2676z\" (UniqueName: \"kubernetes.io/projected/dd7c8134-eff8-4014-8bb6-d10ca68651ba-kube-api-access-2676z\") pod \"payload-processing-e2e-ipp-a-c17e8c-84c889b56d-4rrvl\" (UID: \"dd7c8134-eff8-4014-8bb6-d10ca68651ba\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c17e8c-84c889b56d-4rrvl" Oct 30 17:30:33.417865 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:33.417829 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c17e8c-84c889b56d-4rrvl" Oct 30 17:30:33.601294 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:30:33.601266 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod369893a4_34e9_48a2_8246_7abf4a51018b.slice/crio-456bbc8cca23c478ab3e07d08ffe7376fcc58a182188ae6a1bd53195a585b7e9 WatchSource:0}: Error finding container 456bbc8cca23c478ab3e07d08ffe7376fcc58a182188ae6a1bd53195a585b7e9: Status 404 returned error can't find the container with id 456bbc8cca23c478ab3e07d08ffe7376fcc58a182188ae6a1bd53195a585b7e9 Oct 30 17:30:33.603781 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:33.603750 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c17e8c-567d66ffff-l28mb"] Oct 30 17:30:33.733064 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:30:33.733034 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd7c8134_eff8_4014_8bb6_d10ca68651ba.slice/crio-d97cf624d6cfc357cc93c716c10cc620d891a6d519482b54ae907ec11b132dee WatchSource:0}: Error finding container d97cf624d6cfc357cc93c716c10cc620d891a6d519482b54ae907ec11b132dee: Status 404 returned error can't find the container with id d97cf624d6cfc357cc93c716c10cc620d891a6d519482b54ae907ec11b132dee Oct 30 17:30:33.735995 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:33.735970 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c17e8c-84c889b56d-4rrvl"] Oct 30 17:30:34.404895 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:34.404858 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c17e8c-567d66ffff-l28mb" event={"ID":"369893a4-34e9-48a2-8246-7abf4a51018b","Type":"ContainerStarted","Data":"2e66da4df6a132e1f06115006b2db617b78b2bda7a647315133de520305b1ecc"} Oct 30 17:30:34.405429 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:34.404928 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c17e8c-567d66ffff-l28mb" event={"ID":"369893a4-34e9-48a2-8246-7abf4a51018b","Type":"ContainerStarted","Data":"456bbc8cca23c478ab3e07d08ffe7376fcc58a182188ae6a1bd53195a585b7e9"} Oct 30 17:30:34.405429 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:34.404963 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c17e8c-567d66ffff-l28mb" Oct 30 17:30:34.406916 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:34.406889 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c17e8c-84c889b56d-4rrvl" event={"ID":"dd7c8134-eff8-4014-8bb6-d10ca68651ba","Type":"ContainerStarted","Data":"9f251f3108eba00b372b4c75a18480dcadd2328fbd1ceba3d7d035ab18d74b75"} Oct 30 17:30:34.407039 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:34.406937 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c17e8c-84c889b56d-4rrvl" event={"ID":"dd7c8134-eff8-4014-8bb6-d10ca68651ba","Type":"ContainerStarted","Data":"d97cf624d6cfc357cc93c716c10cc620d891a6d519482b54ae907ec11b132dee"} Oct 30 17:30:34.407039 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:34.406973 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c17e8c-84c889b56d-4rrvl" Oct 30 17:30:34.450103 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:34.450051 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c17e8c-567d66ffff-l28mb" podStartSLOduration=2.450036349 podStartE2EDuration="2.450036349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:34.449338887 +0000 UTC m=+2608.450093790" watchObservedRunningTime="2026-10-30 17:30:34.450036349 +0000 UTC m=+2608.450791294" Oct 30 17:30:34.495436 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:34.495367 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-c17e8c-84c889b56d-4rrvl" podStartSLOduration=1.49534728 podStartE2EDuration="1.49534728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:34.493847957 +0000 UTC m=+2608.494602859" watchObservedRunningTime="2026-10-30 17:30:34.49534728 +0000 UTC m=+2608.496102183" Oct 30 17:30:36.551841 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:36.551809 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abe243-k9r5c"] Oct 30 17:30:36.556814 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:36.556792 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abe243-k9r5c" Oct 30 17:30:36.575714 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:36.575683 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abe243-k9r5c"] Oct 30 17:30:36.581528 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:36.581483 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:30:36.581528 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:36.581493 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:30:36.617363 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:36.617329 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a848b71b-676f-4d8c-8a12-3dc521b737f2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-abe243-k9r5c\" (UID: \"a848b71b-676f-4d8c-8a12-3dc521b737f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abe243-k9r5c" Oct 30 17:30:36.617631 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:36.617603 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpcdk\" (UniqueName: \"kubernetes.io/projected/a848b71b-676f-4d8c-8a12-3dc521b737f2-kube-api-access-rpcdk\") pod \"maas-api-revoke-keys-e2e-ipp-a-abe243-k9r5c\" (UID: \"a848b71b-676f-4d8c-8a12-3dc521b737f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abe243-k9r5c" Oct 30 17:30:36.617820 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:36.617800 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a848b71b-676f-4d8c-8a12-3dc521b737f2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-abe243-k9r5c\" (UID: \"a848b71b-676f-4d8c-8a12-3dc521b737f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abe243-k9r5c" Oct 30 17:30:36.719092 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:36.719058 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a848b71b-676f-4d8c-8a12-3dc521b737f2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-abe243-k9r5c\" (UID: \"a848b71b-676f-4d8c-8a12-3dc521b737f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abe243-k9r5c" Oct 30 17:30:36.719259 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:36.719155 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a848b71b-676f-4d8c-8a12-3dc521b737f2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-abe243-k9r5c\" (UID: \"a848b71b-676f-4d8c-8a12-3dc521b737f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abe243-k9r5c" Oct 30 17:30:36.719374 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:36.719356 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpcdk\" (UniqueName: \"kubernetes.io/projected/a848b71b-676f-4d8c-8a12-3dc521b737f2-kube-api-access-rpcdk\") pod \"maas-api-revoke-keys-e2e-ipp-a-abe243-k9r5c\" (UID: \"a848b71b-676f-4d8c-8a12-3dc521b737f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abe243-k9r5c" Oct 30 17:30:36.719912 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:36.719890 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a848b71b-676f-4d8c-8a12-3dc521b737f2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-abe243-k9r5c\" (UID: \"a848b71b-676f-4d8c-8a12-3dc521b737f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abe243-k9r5c" Oct 30 17:30:36.721668 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:36.721648 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a848b71b-676f-4d8c-8a12-3dc521b737f2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-abe243-k9r5c\" (UID: \"a848b71b-676f-4d8c-8a12-3dc521b737f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abe243-k9r5c" Oct 30 17:30:36.744988 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:36.744956 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpcdk\" (UniqueName: \"kubernetes.io/projected/a848b71b-676f-4d8c-8a12-3dc521b737f2-kube-api-access-rpcdk\") pod \"maas-api-revoke-keys-e2e-ipp-a-abe243-k9r5c\" (UID: \"a848b71b-676f-4d8c-8a12-3dc521b737f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abe243-k9r5c" Oct 30 17:30:36.875862 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:36.875783 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abe243-k9r5c" Oct 30 17:30:37.190299 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:30:37.190265 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda848b71b_676f_4d8c_8a12_3dc521b737f2.slice/crio-436fef9988e30834b92c979ce555f7b2f87d3242da1de0b3bb6b1da6ef230b29 WatchSource:0}: Error finding container 436fef9988e30834b92c979ce555f7b2f87d3242da1de0b3bb6b1da6ef230b29: Status 404 returned error can't find the container with id 436fef9988e30834b92c979ce555f7b2f87d3242da1de0b3bb6b1da6ef230b29 Oct 30 17:30:37.193110 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:37.193080 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abe243-k9r5c"] Oct 30 17:30:37.451428 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:37.451395 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abe243-k9r5c" event={"ID":"a848b71b-676f-4d8c-8a12-3dc521b737f2","Type":"ContainerStarted","Data":"436fef9988e30834b92c979ce555f7b2f87d3242da1de0b3bb6b1da6ef230b29"} Oct 30 17:30:38.463249 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:38.463216 4522 generic.go:356] "Generic (PLEG): container finished" podID="a848b71b-676f-4d8c-8a12-3dc521b737f2" containerID="572781c294d4c4998833ab61e73180cc1906d96a853ffdfd4759075e00f6aea5" exitCode=0 Oct 30 17:30:38.463649 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:38.463249 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abe243-k9r5c" event={"ID":"a848b71b-676f-4d8c-8a12-3dc521b737f2","Type":"ContainerDied","Data":"572781c294d4c4998833ab61e73180cc1906d96a853ffdfd4759075e00f6aea5"} Oct 30 17:30:39.654094 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:39.654068 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abe243-k9r5c" Oct 30 17:30:39.667530 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:39.667504 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a848b71b-676f-4d8c-8a12-3dc521b737f2-kube-api-access-rpcdk\" (UniqueName: \"kubernetes.io/projected/a848b71b-676f-4d8c-8a12-3dc521b737f2-kube-api-access-rpcdk\") pod \"a848b71b-676f-4d8c-8a12-3dc521b737f2\" (UID: \"a848b71b-676f-4d8c-8a12-3dc521b737f2\") " Oct 30 17:30:39.667677 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:39.667565 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a848b71b-676f-4d8c-8a12-3dc521b737f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a848b71b-676f-4d8c-8a12-3dc521b737f2-maas-api-openshift-ca\") pod \"a848b71b-676f-4d8c-8a12-3dc521b737f2\" (UID: \"a848b71b-676f-4d8c-8a12-3dc521b737f2\") " Oct 30 17:30:39.667745 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:39.667733 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a848b71b-676f-4d8c-8a12-3dc521b737f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a848b71b-676f-4d8c-8a12-3dc521b737f2-maas-api-xks-ca\") pod \"a848b71b-676f-4d8c-8a12-3dc521b737f2\" (UID: \"a848b71b-676f-4d8c-8a12-3dc521b737f2\") " Oct 30 17:30:39.667934 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:39.667905 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a848b71b-676f-4d8c-8a12-3dc521b737f2-maas-api-openshift-ca" pod "a848b71b-676f-4d8c-8a12-3dc521b737f2" (UID: "a848b71b-676f-4d8c-8a12-3dc521b737f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:30:39.668362 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:39.668338 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a848b71b-676f-4d8c-8a12-3dc521b737f2-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:30:39.669706 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:39.669684 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a848b71b-676f-4d8c-8a12-3dc521b737f2-kube-api-access-rpcdk" pod "a848b71b-676f-4d8c-8a12-3dc521b737f2" (UID: "a848b71b-676f-4d8c-8a12-3dc521b737f2"). InnerVolumeSpecName "kube-api-access-rpcdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:30:39.669808 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:39.669790 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a848b71b-676f-4d8c-8a12-3dc521b737f2-maas-api-xks-ca" pod "a848b71b-676f-4d8c-8a12-3dc521b737f2" (UID: "a848b71b-676f-4d8c-8a12-3dc521b737f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:30:39.769807 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:39.769729 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpcdk\" (UniqueName: \"kubernetes.io/projected/a848b71b-676f-4d8c-8a12-3dc521b737f2-kube-api-access-rpcdk\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:30:39.769807 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:39.769757 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a848b71b-676f-4d8c-8a12-3dc521b737f2-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:30:40.485158 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:40.485124 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abe243-k9r5c" Oct 30 17:30:40.485390 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:40.485166 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abe243-k9r5c" event={"ID":"a848b71b-676f-4d8c-8a12-3dc521b737f2","Type":"ContainerDied","Data":"436fef9988e30834b92c979ce555f7b2f87d3242da1de0b3bb6b1da6ef230b29"} Oct 30 17:30:40.485390 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:40.485225 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="436fef9988e30834b92c979ce555f7b2f87d3242da1de0b3bb6b1da6ef230b29" Oct 30 17:30:42.950066 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:30:42.950029 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:30:45.431994 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:45.431960 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c17e8c-567d66ffff-l28mb" Oct 30 17:30:45.432456 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:45.432031 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c17e8c-84c889b56d-4rrvl" Oct 30 17:30:49.210949 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:49.210908 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w6-e13a51/auth-test-model-e13a51-kserve-58fc6d6745-cbjrq"] Oct 30 17:30:49.211932 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:49.211911 4522 cpu_manager.go:414] "containerMap: removing container" podUID="a848b71b-676f-4d8c-8a12-3dc521b737f2" containerName="revoke-keys" Oct 30 17:30:49.211932 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:49.211931 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a848b71b-676f-4d8c-8a12-3dc521b737f2" containerName="revoke-keys" Oct 30 17:30:49.212110 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:49.212076 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="a848b71b-676f-4d8c-8a12-3dc521b737f2" containerName="revoke-keys" Oct 30 17:30:49.216635 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:49.216612 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-e13a51/auth-test-model-e13a51-kserve-58fc6d6745-cbjrq" Oct 30 17:30:49.239163 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:49.239134 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-e13a51/auth-test-model-e13a51-kserve-58fc6d6745-cbjrq"] Oct 30 17:30:49.245864 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:49.245837 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-e13a51\"/\"openshift-service-ca.crt\"" Oct 30 17:30:49.246059 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:49.245973 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-e13a51\"/\"default-dockercfg-hgrrc\"" Oct 30 17:30:49.246195 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:49.246014 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-e13a51\"/\"kube-root-ca.crt\"" Oct 30 17:30:49.246526 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:49.246508 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-e13a51\"/\"auth-test-model-e13a51-kserve-self-signed-certs\"" Oct 30 17:30:49.342955 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:49.342923 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1a21adc-645e-44a6-bb65-bb82f392c03d-home\") pod \"auth-test-model-e13a51-kserve-58fc6d6745-cbjrq\" (UID: \"e1a21adc-645e-44a6-bb65-bb82f392c03d\") " pod="ai-tenant-e2e-shared-b-w6-e13a51/auth-test-model-e13a51-kserve-58fc6d6745-cbjrq" Oct 30 17:30:49.343143 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:49.343054 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnnjn\" (UniqueName: \"kubernetes.io/projected/e1a21adc-645e-44a6-bb65-bb82f392c03d-kube-api-access-qnnjn\") pod \"auth-test-model-e13a51-kserve-58fc6d6745-cbjrq\" (UID: \"e1a21adc-645e-44a6-bb65-bb82f392c03d\") " pod="ai-tenant-e2e-shared-b-w6-e13a51/auth-test-model-e13a51-kserve-58fc6d6745-cbjrq" Oct 30 17:30:49.343185 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:49.343156 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1a21adc-645e-44a6-bb65-bb82f392c03d-tmp-dir\") pod \"auth-test-model-e13a51-kserve-58fc6d6745-cbjrq\" (UID: \"e1a21adc-645e-44a6-bb65-bb82f392c03d\") " pod="ai-tenant-e2e-shared-b-w6-e13a51/auth-test-model-e13a51-kserve-58fc6d6745-cbjrq" Oct 30 17:30:49.343226 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:49.343189 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1a21adc-645e-44a6-bb65-bb82f392c03d-tls-certs\") pod \"auth-test-model-e13a51-kserve-58fc6d6745-cbjrq\" (UID: \"e1a21adc-645e-44a6-bb65-bb82f392c03d\") " pod="ai-tenant-e2e-shared-b-w6-e13a51/auth-test-model-e13a51-kserve-58fc6d6745-cbjrq" Oct 30 17:30:49.343270 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:49.343246 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1a21adc-645e-44a6-bb65-bb82f392c03d-dshm\") pod \"auth-test-model-e13a51-kserve-58fc6d6745-cbjrq\" (UID: \"e1a21adc-645e-44a6-bb65-bb82f392c03d\") " pod="ai-tenant-e2e-shared-b-w6-e13a51/auth-test-model-e13a51-kserve-58fc6d6745-cbjrq" Oct 30 17:30:49.343428 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:49.343413 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1a21adc-645e-44a6-bb65-bb82f392c03d-model-cache\") pod \"auth-test-model-e13a51-kserve-58fc6d6745-cbjrq\" (UID: \"e1a21adc-645e-44a6-bb65-bb82f392c03d\") " pod="ai-tenant-e2e-shared-b-w6-e13a51/auth-test-model-e13a51-kserve-58fc6d6745-cbjrq" Oct 30 17:30:49.444891 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:49.444850 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1a21adc-645e-44a6-bb65-bb82f392c03d-model-cache\") pod \"auth-test-model-e13a51-kserve-58fc6d6745-cbjrq\" (UID: \"e1a21adc-645e-44a6-bb65-bb82f392c03d\") " pod="ai-tenant-e2e-shared-b-w6-e13a51/auth-test-model-e13a51-kserve-58fc6d6745-cbjrq" Oct 30 17:30:49.445073 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:49.444962 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1a21adc-645e-44a6-bb65-bb82f392c03d-home\") pod \"auth-test-model-e13a51-kserve-58fc6d6745-cbjrq\" (UID: \"e1a21adc-645e-44a6-bb65-bb82f392c03d\") " pod="ai-tenant-e2e-shared-b-w6-e13a51/auth-test-model-e13a51-kserve-58fc6d6745-cbjrq" Oct 30 17:30:49.445115 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:49.445074 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnnjn\" (UniqueName: \"kubernetes.io/projected/e1a21adc-645e-44a6-bb65-bb82f392c03d-kube-api-access-qnnjn\") pod \"auth-test-model-e13a51-kserve-58fc6d6745-cbjrq\" (UID: \"e1a21adc-645e-44a6-bb65-bb82f392c03d\") " pod="ai-tenant-e2e-shared-b-w6-e13a51/auth-test-model-e13a51-kserve-58fc6d6745-cbjrq" Oct 30 17:30:49.445215 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:49.445199 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1a21adc-645e-44a6-bb65-bb82f392c03d-tmp-dir\") pod \"auth-test-model-e13a51-kserve-58fc6d6745-cbjrq\" (UID: \"e1a21adc-645e-44a6-bb65-bb82f392c03d\") " pod="ai-tenant-e2e-shared-b-w6-e13a51/auth-test-model-e13a51-kserve-58fc6d6745-cbjrq" Oct 30 17:30:49.445265 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:49.445252 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1a21adc-645e-44a6-bb65-bb82f392c03d-tls-certs\") pod \"auth-test-model-e13a51-kserve-58fc6d6745-cbjrq\" (UID: \"e1a21adc-645e-44a6-bb65-bb82f392c03d\") " pod="ai-tenant-e2e-shared-b-w6-e13a51/auth-test-model-e13a51-kserve-58fc6d6745-cbjrq" Oct 30 17:30:49.445337 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:49.445287 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1a21adc-645e-44a6-bb65-bb82f392c03d-model-cache\") pod \"auth-test-model-e13a51-kserve-58fc6d6745-cbjrq\" (UID: \"e1a21adc-645e-44a6-bb65-bb82f392c03d\") " pod="ai-tenant-e2e-shared-b-w6-e13a51/auth-test-model-e13a51-kserve-58fc6d6745-cbjrq" Oct 30 17:30:49.445337 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:49.445301 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1a21adc-645e-44a6-bb65-bb82f392c03d-home\") pod \"auth-test-model-e13a51-kserve-58fc6d6745-cbjrq\" (UID: \"e1a21adc-645e-44a6-bb65-bb82f392c03d\") " pod="ai-tenant-e2e-shared-b-w6-e13a51/auth-test-model-e13a51-kserve-58fc6d6745-cbjrq" Oct 30 17:30:49.445460 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:49.445376 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1a21adc-645e-44a6-bb65-bb82f392c03d-dshm\") pod \"auth-test-model-e13a51-kserve-58fc6d6745-cbjrq\" (UID: \"e1a21adc-645e-44a6-bb65-bb82f392c03d\") " pod="ai-tenant-e2e-shared-b-w6-e13a51/auth-test-model-e13a51-kserve-58fc6d6745-cbjrq" Oct 30 17:30:49.445596 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:49.445575 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1a21adc-645e-44a6-bb65-bb82f392c03d-tmp-dir\") pod \"auth-test-model-e13a51-kserve-58fc6d6745-cbjrq\" (UID: \"e1a21adc-645e-44a6-bb65-bb82f392c03d\") " pod="ai-tenant-e2e-shared-b-w6-e13a51/auth-test-model-e13a51-kserve-58fc6d6745-cbjrq" Oct 30 17:30:49.447713 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:49.447694 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1a21adc-645e-44a6-bb65-bb82f392c03d-dshm\") pod \"auth-test-model-e13a51-kserve-58fc6d6745-cbjrq\" (UID: \"e1a21adc-645e-44a6-bb65-bb82f392c03d\") " pod="ai-tenant-e2e-shared-b-w6-e13a51/auth-test-model-e13a51-kserve-58fc6d6745-cbjrq" Oct 30 17:30:49.447912 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:49.447895 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1a21adc-645e-44a6-bb65-bb82f392c03d-tls-certs\") pod \"auth-test-model-e13a51-kserve-58fc6d6745-cbjrq\" (UID: \"e1a21adc-645e-44a6-bb65-bb82f392c03d\") " pod="ai-tenant-e2e-shared-b-w6-e13a51/auth-test-model-e13a51-kserve-58fc6d6745-cbjrq" Oct 30 17:30:49.470138 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:49.470069 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnnjn\" (UniqueName: \"kubernetes.io/projected/e1a21adc-645e-44a6-bb65-bb82f392c03d-kube-api-access-qnnjn\") pod \"auth-test-model-e13a51-kserve-58fc6d6745-cbjrq\" (UID: \"e1a21adc-645e-44a6-bb65-bb82f392c03d\") " pod="ai-tenant-e2e-shared-b-w6-e13a51/auth-test-model-e13a51-kserve-58fc6d6745-cbjrq" Oct 30 17:30:49.533201 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:49.533164 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-e13a51/auth-test-model-e13a51-kserve-58fc6d6745-cbjrq" Oct 30 17:30:49.935620 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:30:49.935591 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode1a21adc_645e_44a6_bb65_bb82f392c03d.slice/crio-1f043cb37570d39b5a87206d02e0701f1cef2b0a85385f5896b8b2af85514f3d WatchSource:0}: Error finding container 1f043cb37570d39b5a87206d02e0701f1cef2b0a85385f5896b8b2af85514f3d: Status 404 returned error can't find the container with id 1f043cb37570d39b5a87206d02e0701f1cef2b0a85385f5896b8b2af85514f3d Oct 30 17:30:49.938088 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:49.938064 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-e13a51/auth-test-model-e13a51-kserve-58fc6d6745-cbjrq"] Oct 30 17:30:50.638664 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:50.638627 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-e13a51/auth-test-model-e13a51-kserve-58fc6d6745-cbjrq" event={"ID":"e1a21adc-645e-44a6-bb65-bb82f392c03d","Type":"ContainerStarted","Data":"016c326d4d71767376a7f0ef99944bfa9536b9f2e19217c5015be6f0e8dc034f"} Oct 30 17:30:50.639224 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:50.638673 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-e13a51/auth-test-model-e13a51-kserve-58fc6d6745-cbjrq" event={"ID":"e1a21adc-645e-44a6-bb65-bb82f392c03d","Type":"ContainerStarted","Data":"1f043cb37570d39b5a87206d02e0701f1cef2b0a85385f5896b8b2af85514f3d"} Oct 30 17:30:50.686463 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:50.686401 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w6-e13a51/auth-test-model-e13a51-kserve-58fc6d6745-cbjrq" podStartSLOduration=1.686380895 podStartE2EDuration="1.686380895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:50.683369934 +0000 UTC m=+2624.684124860" watchObservedRunningTime="2026-10-30 17:30:50.686380895 +0000 UTC m=+2624.687135801" Oct 30 17:30:51.534340 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:51.534275 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w6-e13a51/auth-test-model-e13a51-kserve-58fc6d6745-cbjrq" Oct 30 17:30:53.806109 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:53.806072 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9e9ce7-699c4dcc55-fjmlj"] Oct 30 17:30:53.811684 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:53.811661 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9e9ce7-699c4dcc55-fjmlj" Oct 30 17:30:53.830002 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:53.829973 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9e9ce7-699c4dcc55-fjmlj"] Oct 30 17:30:53.837182 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:53.837152 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-9e9ce7\"" Oct 30 17:30:53.837389 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:53.837261 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-9e9ce7-dockercfg-rllrl\"" Oct 30 17:30:53.994920 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:53.994882 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c214620-08c5-4a95-aea8-b2d4cbae85a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9e9ce7-699c4dcc55-fjmlj\" (UID: \"1c214620-08c5-4a95-aea8-b2d4cbae85a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9e9ce7-699c4dcc55-fjmlj" Oct 30 17:30:53.995475 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:53.995448 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckxt6\" (UniqueName: \"kubernetes.io/projected/1c214620-08c5-4a95-aea8-b2d4cbae85a2-kube-api-access-ckxt6\") pod \"payload-pre-processing-e2e-ipp-b-9e9ce7-699c4dcc55-fjmlj\" (UID: \"1c214620-08c5-4a95-aea8-b2d4cbae85a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9e9ce7-699c4dcc55-fjmlj" Oct 30 17:30:54.097300 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:54.097211 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c214620-08c5-4a95-aea8-b2d4cbae85a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9e9ce7-699c4dcc55-fjmlj\" (UID: \"1c214620-08c5-4a95-aea8-b2d4cbae85a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9e9ce7-699c4dcc55-fjmlj" Oct 30 17:30:54.097539 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:54.097442 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckxt6\" (UniqueName: \"kubernetes.io/projected/1c214620-08c5-4a95-aea8-b2d4cbae85a2-kube-api-access-ckxt6\") pod \"payload-pre-processing-e2e-ipp-b-9e9ce7-699c4dcc55-fjmlj\" (UID: \"1c214620-08c5-4a95-aea8-b2d4cbae85a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9e9ce7-699c4dcc55-fjmlj" Oct 30 17:30:54.098215 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:54.098188 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c214620-08c5-4a95-aea8-b2d4cbae85a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9e9ce7-699c4dcc55-fjmlj\" (UID: \"1c214620-08c5-4a95-aea8-b2d4cbae85a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9e9ce7-699c4dcc55-fjmlj" Oct 30 17:30:54.123006 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:54.122975 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckxt6\" (UniqueName: \"kubernetes.io/projected/1c214620-08c5-4a95-aea8-b2d4cbae85a2-kube-api-access-ckxt6\") pod \"payload-pre-processing-e2e-ipp-b-9e9ce7-699c4dcc55-fjmlj\" (UID: \"1c214620-08c5-4a95-aea8-b2d4cbae85a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9e9ce7-699c4dcc55-fjmlj" Oct 30 17:30:54.131148 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:54.131118 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9e9ce7-699c4dcc55-fjmlj" Oct 30 17:30:54.456182 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:30:54.456153 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c214620_08c5_4a95_aea8_b2d4cbae85a2.slice/crio-fb719603742632b4f572af5af2be2d10df2d0eed976b1cff7c4f877ea1df10a5 WatchSource:0}: Error finding container fb719603742632b4f572af5af2be2d10df2d0eed976b1cff7c4f877ea1df10a5: Status 404 returned error can't find the container with id fb719603742632b4f572af5af2be2d10df2d0eed976b1cff7c4f877ea1df10a5 Oct 30 17:30:54.458822 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:54.458796 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9e9ce7-699c4dcc55-fjmlj"] Oct 30 17:30:54.677929 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:54.677893 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9e9ce7-699c4dcc55-fjmlj" event={"ID":"1c214620-08c5-4a95-aea8-b2d4cbae85a2","Type":"ContainerStarted","Data":"f28a669d967d6a8b39a9908c54c69f9ce6c16655c107ae1537b6fcdd60784dac"} Oct 30 17:30:54.678127 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:54.677946 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9e9ce7-699c4dcc55-fjmlj" event={"ID":"1c214620-08c5-4a95-aea8-b2d4cbae85a2","Type":"ContainerStarted","Data":"fb719603742632b4f572af5af2be2d10df2d0eed976b1cff7c4f877ea1df10a5"} Oct 30 17:30:54.678127 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:54.677971 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9e9ce7-699c4dcc55-fjmlj" Oct 30 17:30:54.721679 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:54.721593 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9e9ce7-699c4dcc55-fjmlj" podStartSLOduration=1.721580668 podStartE2EDuration="1.721580668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:54.721235198 +0000 UTC m=+2628.721990101" watchObservedRunningTime="2026-10-30 17:30:54.721580668 +0000 UTC m=+2628.722335568" Oct 30 17:30:59.534330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:59.533663 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w6-e13a51/auth-test-model-e13a51-kserve-58fc6d6745-cbjrq" Oct 30 17:30:59.552268 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:59.552238 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w6-e13a51/auth-test-model-e13a51-kserve-58fc6d6745-cbjrq" Oct 30 17:30:59.745466 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:30:59.745430 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w6-e13a51/auth-test-model-e13a51-kserve-58fc6d6745-cbjrq" Oct 30 17:31:04.042181 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:04.042140 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d5f5a1e-n8wrb"] Oct 30 17:31:04.056405 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:04.056370 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d5f5a1e-n8wrb"] Oct 30 17:31:04.159538 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:04.159497 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eea2f1a1-5fe2-4cf3-a643-12e22b0c69ca" path="/var/lib/kubelet/pods/eea2f1a1-5fe2-4cf3-a643-12e22b0c69ca/volumes" Oct 30 17:31:05.700624 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:05.700589 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9e9ce7-699c4dcc55-fjmlj" Oct 30 17:31:06.034817 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:06.034777 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6144f753-gfncd"] Oct 30 17:31:06.039981 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:06.039951 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6144f753-gfncd"] Oct 30 17:31:06.147918 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:06.147886 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a5af931-bbe4-4516-b808-86ca7efb37ad" path="/var/lib/kubelet/pods/7a5af931-bbe4-4516-b808-86ca7efb37ad/volumes" Oct 30 17:31:09.928640 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:09.928596 4522 scope.go:122] "RemoveContainer" containerID="372e3ea64bea5d0e271c12341eaeee27d682347f403ef0f3828d3da9c528bf12" Oct 30 17:31:10.000041 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:09.998050 4522 scope.go:122] "RemoveContainer" containerID="abe94397698f6cd8accd710d3e38f3e159517610dba84494000bfeeb5eb937e4" Oct 30 17:31:10.019150 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:10.019128 4522 scope.go:122] "RemoveContainer" containerID="0d9c839ee5d51fe5a54b648864216adaa78a312ddcdd403e991e9c77b52e3f2c" Oct 30 17:31:10.039118 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:10.039096 4522 scope.go:122] "RemoveContainer" containerID="ef0b13f2d389e6f7147a1e0b294bdbbc16bc6f5c7d1fed436ceb169e7d1b4410" Oct 30 17:31:10.052258 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:10.052232 4522 scope.go:122] "RemoveContainer" containerID="2fa6f955f412fa5b607f41e20027e3a922bdb06c152a16cc6abc20416e257648" Oct 30 17:31:10.064258 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:10.064238 4522 scope.go:122] "RemoveContainer" containerID="6e11a96e9bedc3c5a36e71649fcca22a349778bc515d1851c3a678bcc6e340c1" Oct 30 17:31:10.081269 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:10.081244 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8a7e4a-66cd879994-drkhg"] Oct 30 17:31:10.088868 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:10.088849 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8a7e4a-66cd879994-drkhg" Oct 30 17:31:10.121654 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:10.111883 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8a7e4a-66cd879994-drkhg"] Oct 30 17:31:10.121654 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:10.118881 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-8a7e4a\"" Oct 30 17:31:10.121654 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:10.119698 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-8a7e4a-dockercfg-p5hbd\"" Oct 30 17:31:10.171685 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:10.171657 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2abedac3-fc80-4dcf-9f92-c6aefa1a2cf7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8a7e4a-66cd879994-drkhg\" (UID: \"2abedac3-fc80-4dcf-9f92-c6aefa1a2cf7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8a7e4a-66cd879994-drkhg" Oct 30 17:31:10.171989 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:10.171960 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfwlq\" (UniqueName: \"kubernetes.io/projected/2abedac3-fc80-4dcf-9f92-c6aefa1a2cf7-kube-api-access-rfwlq\") pod \"payload-processing-e2e-ipp-a-8a7e4a-66cd879994-drkhg\" (UID: \"2abedac3-fc80-4dcf-9f92-c6aefa1a2cf7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8a7e4a-66cd879994-drkhg" Oct 30 17:31:10.266323 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:10.265491 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8a7e4a-5d55c87fcf-qf6bc"] Oct 30 17:31:10.273663 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:10.273627 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfwlq\" (UniqueName: \"kubernetes.io/projected/2abedac3-fc80-4dcf-9f92-c6aefa1a2cf7-kube-api-access-rfwlq\") pod \"payload-processing-e2e-ipp-a-8a7e4a-66cd879994-drkhg\" (UID: \"2abedac3-fc80-4dcf-9f92-c6aefa1a2cf7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8a7e4a-66cd879994-drkhg" Oct 30 17:31:10.273851 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:10.273796 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2abedac3-fc80-4dcf-9f92-c6aefa1a2cf7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8a7e4a-66cd879994-drkhg\" (UID: \"2abedac3-fc80-4dcf-9f92-c6aefa1a2cf7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8a7e4a-66cd879994-drkhg" Oct 30 17:31:10.275124 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:10.275096 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2abedac3-fc80-4dcf-9f92-c6aefa1a2cf7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8a7e4a-66cd879994-drkhg\" (UID: \"2abedac3-fc80-4dcf-9f92-c6aefa1a2cf7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8a7e4a-66cd879994-drkhg" Oct 30 17:31:10.279152 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:10.279132 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8a7e4a-5d55c87fcf-qf6bc" Oct 30 17:31:10.298725 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:10.297702 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8a7e4a-5d55c87fcf-qf6bc"] Oct 30 17:31:10.307507 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:10.307478 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfwlq\" (UniqueName: \"kubernetes.io/projected/2abedac3-fc80-4dcf-9f92-c6aefa1a2cf7-kube-api-access-rfwlq\") pod \"payload-processing-e2e-ipp-a-8a7e4a-66cd879994-drkhg\" (UID: \"2abedac3-fc80-4dcf-9f92-c6aefa1a2cf7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8a7e4a-66cd879994-drkhg" Oct 30 17:31:10.375509 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:10.375467 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcfedbe8-98e7-479a-8f7b-f8e6d1be8b1a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8a7e4a-5d55c87fcf-qf6bc\" (UID: \"fcfedbe8-98e7-479a-8f7b-f8e6d1be8b1a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8a7e4a-5d55c87fcf-qf6bc" Oct 30 17:31:10.375709 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:10.375599 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5rkg\" (UniqueName: \"kubernetes.io/projected/fcfedbe8-98e7-479a-8f7b-f8e6d1be8b1a-kube-api-access-g5rkg\") pod \"payload-pre-processing-e2e-ipp-a-8a7e4a-5d55c87fcf-qf6bc\" (UID: \"fcfedbe8-98e7-479a-8f7b-f8e6d1be8b1a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8a7e4a-5d55c87fcf-qf6bc" Oct 30 17:31:10.412172 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:10.412135 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8a7e4a-66cd879994-drkhg" Oct 30 17:31:10.481330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:10.480698 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcfedbe8-98e7-479a-8f7b-f8e6d1be8b1a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8a7e4a-5d55c87fcf-qf6bc\" (UID: \"fcfedbe8-98e7-479a-8f7b-f8e6d1be8b1a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8a7e4a-5d55c87fcf-qf6bc" Oct 30 17:31:10.481330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:10.480790 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5rkg\" (UniqueName: \"kubernetes.io/projected/fcfedbe8-98e7-479a-8f7b-f8e6d1be8b1a-kube-api-access-g5rkg\") pod \"payload-pre-processing-e2e-ipp-a-8a7e4a-5d55c87fcf-qf6bc\" (UID: \"fcfedbe8-98e7-479a-8f7b-f8e6d1be8b1a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8a7e4a-5d55c87fcf-qf6bc" Oct 30 17:31:10.485386 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:10.482516 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcfedbe8-98e7-479a-8f7b-f8e6d1be8b1a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8a7e4a-5d55c87fcf-qf6bc\" (UID: \"fcfedbe8-98e7-479a-8f7b-f8e6d1be8b1a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8a7e4a-5d55c87fcf-qf6bc" Oct 30 17:31:10.526330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:10.523739 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5rkg\" (UniqueName: \"kubernetes.io/projected/fcfedbe8-98e7-479a-8f7b-f8e6d1be8b1a-kube-api-access-g5rkg\") pod \"payload-pre-processing-e2e-ipp-a-8a7e4a-5d55c87fcf-qf6bc\" (UID: \"fcfedbe8-98e7-479a-8f7b-f8e6d1be8b1a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8a7e4a-5d55c87fcf-qf6bc" Oct 30 17:31:10.627473 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:10.627125 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8a7e4a-5d55c87fcf-qf6bc" Oct 30 17:31:10.870269 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:31:10.870239 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2abedac3_fc80_4dcf_9f92_c6aefa1a2cf7.slice/crio-b3e03a058d23fd4063ca377b5c42499ffa3143dd1d2c4b7d4ede6f035534955a WatchSource:0}: Error finding container b3e03a058d23fd4063ca377b5c42499ffa3143dd1d2c4b7d4ede6f035534955a: Status 404 returned error can't find the container with id b3e03a058d23fd4063ca377b5c42499ffa3143dd1d2c4b7d4ede6f035534955a Oct 30 17:31:10.879352 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:10.878891 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8a7e4a-66cd879994-drkhg"] Oct 30 17:31:11.053785 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:31:11.053749 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcfedbe8_98e7_479a_8f7b_f8e6d1be8b1a.slice/crio-a851f33e2ba6cab8c84b1e3c0f847608aa0de0f3956b115c89ec6986accc7757 WatchSource:0}: Error finding container a851f33e2ba6cab8c84b1e3c0f847608aa0de0f3956b115c89ec6986accc7757: Status 404 returned error can't find the container with id a851f33e2ba6cab8c84b1e3c0f847608aa0de0f3956b115c89ec6986accc7757 Oct 30 17:31:11.054459 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:11.054435 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8a7e4a-5d55c87fcf-qf6bc"] Oct 30 17:31:11.891081 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:11.891050 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8a7e4a-66cd879994-drkhg" event={"ID":"2abedac3-fc80-4dcf-9f92-c6aefa1a2cf7","Type":"ContainerStarted","Data":"bc4f2034dccfe0f060278b751d23e73ba8968d4c63e1a2aea11c6ddc24c438a6"} Oct 30 17:31:11.892197 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:11.892171 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8a7e4a-66cd879994-drkhg" event={"ID":"2abedac3-fc80-4dcf-9f92-c6aefa1a2cf7","Type":"ContainerStarted","Data":"b3e03a058d23fd4063ca377b5c42499ffa3143dd1d2c4b7d4ede6f035534955a"} Oct 30 17:31:11.898421 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:11.896098 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8a7e4a-66cd879994-drkhg" Oct 30 17:31:11.898421 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:11.896645 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8a7e4a-5d55c87fcf-qf6bc" event={"ID":"fcfedbe8-98e7-479a-8f7b-f8e6d1be8b1a","Type":"ContainerStarted","Data":"adfaa78121d678486c4a401fadaf11a38f6616f0258f7a83f623b13d8bf37292"} Oct 30 17:31:11.898421 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:11.896683 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8a7e4a-5d55c87fcf-qf6bc" event={"ID":"fcfedbe8-98e7-479a-8f7b-f8e6d1be8b1a","Type":"ContainerStarted","Data":"a851f33e2ba6cab8c84b1e3c0f847608aa0de0f3956b115c89ec6986accc7757"} Oct 30 17:31:11.898421 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:11.897496 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8a7e4a-5d55c87fcf-qf6bc" Oct 30 17:31:11.946696 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:11.946645 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-8a7e4a-66cd879994-drkhg" podStartSLOduration=1.946629867 podStartE2EDuration="1.946629867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:31:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:31:11.942367784 +0000 UTC m=+2645.943122688" watchObservedRunningTime="2026-10-30 17:31:11.946629867 +0000 UTC m=+2645.947384767" Oct 30 17:31:11.994699 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:11.994638 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8a7e4a-5d55c87fcf-qf6bc" podStartSLOduration=1.994617788 podStartE2EDuration="1.994617788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:31:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:31:11.989793619 +0000 UTC m=+2645.990548542" watchObservedRunningTime="2026-10-30 17:31:11.994617788 +0000 UTC m=+2645.995372692" Oct 30 17:31:13.109637 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:31:13.109556 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:31:15.035598 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:15.035557 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d2f4ecb-wfqwc"] Oct 30 17:31:15.049999 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:15.049969 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d2f4ecb-wfqwc"] Oct 30 17:31:15.788131 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:15.788101 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9722dad0-hrmt2"] Oct 30 17:31:15.795074 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:15.795050 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9722dad0-hrmt2" Oct 30 17:31:15.813555 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:15.813510 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9722dad0-hrmt2"] Oct 30 17:31:15.821503 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:15.821475 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:31:15.821715 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:15.821542 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:31:15.830527 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:15.830495 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjxwc\" (UniqueName: \"kubernetes.io/projected/52db46a9-175b-4cad-bc70-1345d8ddb498-kube-api-access-vjxwc\") pod \"maas-api-revoke-sub-9722dad0-hrmt2\" (UID: \"52db46a9-175b-4cad-bc70-1345d8ddb498\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9722dad0-hrmt2" Oct 30 17:31:15.830793 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:15.830765 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52db46a9-175b-4cad-bc70-1345d8ddb498-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9722dad0-hrmt2\" (UID: \"52db46a9-175b-4cad-bc70-1345d8ddb498\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9722dad0-hrmt2" Oct 30 17:31:15.831250 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:15.830839 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52db46a9-175b-4cad-bc70-1345d8ddb498-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9722dad0-hrmt2\" (UID: \"52db46a9-175b-4cad-bc70-1345d8ddb498\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9722dad0-hrmt2" Oct 30 17:31:15.932580 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:15.932547 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjxwc\" (UniqueName: \"kubernetes.io/projected/52db46a9-175b-4cad-bc70-1345d8ddb498-kube-api-access-vjxwc\") pod \"maas-api-revoke-sub-9722dad0-hrmt2\" (UID: \"52db46a9-175b-4cad-bc70-1345d8ddb498\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9722dad0-hrmt2" Oct 30 17:31:15.932763 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:15.932736 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52db46a9-175b-4cad-bc70-1345d8ddb498-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9722dad0-hrmt2\" (UID: \"52db46a9-175b-4cad-bc70-1345d8ddb498\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9722dad0-hrmt2" Oct 30 17:31:15.932817 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:15.932778 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52db46a9-175b-4cad-bc70-1345d8ddb498-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9722dad0-hrmt2\" (UID: \"52db46a9-175b-4cad-bc70-1345d8ddb498\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9722dad0-hrmt2" Oct 30 17:31:15.933576 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:15.933553 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52db46a9-175b-4cad-bc70-1345d8ddb498-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9722dad0-hrmt2\" (UID: \"52db46a9-175b-4cad-bc70-1345d8ddb498\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9722dad0-hrmt2" Oct 30 17:31:15.935406 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:15.935376 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52db46a9-175b-4cad-bc70-1345d8ddb498-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9722dad0-hrmt2\" (UID: \"52db46a9-175b-4cad-bc70-1345d8ddb498\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9722dad0-hrmt2" Oct 30 17:31:15.960175 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:15.960145 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjxwc\" (UniqueName: \"kubernetes.io/projected/52db46a9-175b-4cad-bc70-1345d8ddb498-kube-api-access-vjxwc\") pod \"maas-api-revoke-sub-9722dad0-hrmt2\" (UID: \"52db46a9-175b-4cad-bc70-1345d8ddb498\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9722dad0-hrmt2" Oct 30 17:31:16.113703 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:16.113588 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9722dad0-hrmt2" Oct 30 17:31:16.154047 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:16.154019 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="418d079d-aedd-408e-9d92-b354789e72fc" path="/var/lib/kubelet/pods/418d079d-aedd-408e-9d92-b354789e72fc/volumes" Oct 30 17:31:16.440080 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:31:16.440042 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52db46a9_175b_4cad_bc70_1345d8ddb498.slice/crio-a49613df73cbe0d4c36c445ef5da65401078f2a9cc113783322783df55688d04 WatchSource:0}: Error finding container a49613df73cbe0d4c36c445ef5da65401078f2a9cc113783322783df55688d04: Status 404 returned error can't find the container with id a49613df73cbe0d4c36c445ef5da65401078f2a9cc113783322783df55688d04 Oct 30 17:31:16.444065 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:16.444045 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9722dad0-hrmt2"] Oct 30 17:31:16.954414 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:16.954367 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9722dad0-hrmt2" event={"ID":"52db46a9-175b-4cad-bc70-1345d8ddb498","Type":"ContainerStarted","Data":"2d3854802a66fc1c4cccb6ec05e682b10c74f6dd082ffb802eb9d98413762c49"} Oct 30 17:31:16.954653 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:16.954442 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9722dad0-hrmt2" event={"ID":"52db46a9-175b-4cad-bc70-1345d8ddb498","Type":"ContainerStarted","Data":"a49613df73cbe0d4c36c445ef5da65401078f2a9cc113783322783df55688d04"} Oct 30 17:31:17.002678 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:17.002629 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9722dad0-hrmt2" podStartSLOduration=2.002616948 podStartE2EDuration="2.002616948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:31:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:31:17.000511417 +0000 UTC m=+2651.001266338" watchObservedRunningTime="2026-10-30 17:31:17.002616948 +0000 UTC m=+2651.003371848" Oct 30 17:31:17.044459 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:17.044420 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60c5652f-xch6m"] Oct 30 17:31:17.050263 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:17.050174 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60c5652f-xch6m"] Oct 30 17:31:17.205346 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:17.205301 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-93e020-6c9d97748-pnfp5"] Oct 30 17:31:17.387414 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:17.387381 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz"] Oct 30 17:31:17.387596 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:17.387572 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz" podUID="b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc" containerName="istio-proxy" containerID="cri-o://7f2c75141c115bd014075858751b48d33608e1a5877f0ba2ede04d93a3c899ad" gracePeriod=30 Oct 30 17:31:17.964711 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:17.964675 4522 generic.go:356] "Generic (PLEG): container finished" podID="52db46a9-175b-4cad-bc70-1345d8ddb498" containerID="2d3854802a66fc1c4cccb6ec05e682b10c74f6dd082ffb802eb9d98413762c49" exitCode=0 Oct 30 17:31:17.964924 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:17.964762 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9722dad0-hrmt2" event={"ID":"52db46a9-175b-4cad-bc70-1345d8ddb498","Type":"ContainerDied","Data":"2d3854802a66fc1c4cccb6ec05e682b10c74f6dd082ffb802eb9d98413762c49"} Oct 30 17:31:18.176810 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:18.176771 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4afca48-3a36-4718-a83f-314a67da8061" path="/var/lib/kubelet/pods/f4afca48-3a36-4718-a83f-314a67da8061/volumes" Oct 30 17:31:19.161557 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:19.161529 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9722dad0-hrmt2" Oct 30 17:31:19.203608 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:19.203579 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52db46a9-175b-4cad-bc70-1345d8ddb498-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52db46a9-175b-4cad-bc70-1345d8ddb498-maas-api-openshift-ca\") pod \"52db46a9-175b-4cad-bc70-1345d8ddb498\" (UID: \"52db46a9-175b-4cad-bc70-1345d8ddb498\") " Oct 30 17:31:19.204003 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:19.203980 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52db46a9-175b-4cad-bc70-1345d8ddb498-kube-api-access-vjxwc\" (UniqueName: \"kubernetes.io/projected/52db46a9-175b-4cad-bc70-1345d8ddb498-kube-api-access-vjxwc\") pod \"52db46a9-175b-4cad-bc70-1345d8ddb498\" (UID: \"52db46a9-175b-4cad-bc70-1345d8ddb498\") " Oct 30 17:31:19.204104 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:19.203995 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52db46a9-175b-4cad-bc70-1345d8ddb498-maas-api-openshift-ca" pod "52db46a9-175b-4cad-bc70-1345d8ddb498" (UID: "52db46a9-175b-4cad-bc70-1345d8ddb498"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:31:19.204104 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:19.204077 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52db46a9-175b-4cad-bc70-1345d8ddb498-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52db46a9-175b-4cad-bc70-1345d8ddb498-maas-api-xks-ca\") pod \"52db46a9-175b-4cad-bc70-1345d8ddb498\" (UID: \"52db46a9-175b-4cad-bc70-1345d8ddb498\") " Oct 30 17:31:19.204780 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:19.204758 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52db46a9-175b-4cad-bc70-1345d8ddb498-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:31:19.206291 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:19.206264 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52db46a9-175b-4cad-bc70-1345d8ddb498-maas-api-xks-ca" pod "52db46a9-175b-4cad-bc70-1345d8ddb498" (UID: "52db46a9-175b-4cad-bc70-1345d8ddb498"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:31:19.206427 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:19.206404 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52db46a9-175b-4cad-bc70-1345d8ddb498-kube-api-access-vjxwc" pod "52db46a9-175b-4cad-bc70-1345d8ddb498" (UID: "52db46a9-175b-4cad-bc70-1345d8ddb498"). InnerVolumeSpecName "kube-api-access-vjxwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:31:19.306403 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:19.306324 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjxwc\" (UniqueName: \"kubernetes.io/projected/52db46a9-175b-4cad-bc70-1345d8ddb498-kube-api-access-vjxwc\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:31:19.306403 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:19.306352 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52db46a9-175b-4cad-bc70-1345d8ddb498-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:31:19.985185 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:19.985150 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9722dad0-hrmt2" Oct 30 17:31:19.985406 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:19.985181 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9722dad0-hrmt2" event={"ID":"52db46a9-175b-4cad-bc70-1345d8ddb498","Type":"ContainerDied","Data":"a49613df73cbe0d4c36c445ef5da65401078f2a9cc113783322783df55688d04"} Oct 30 17:31:19.985406 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:19.985229 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a49613df73cbe0d4c36c445ef5da65401078f2a9cc113783322783df55688d04" Oct 30 17:31:22.206038 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:22.205998 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-93e020-6c9d97748-pnfp5" podUID="af795373-0e85-4f37-80ab-2caa6e2dd689" containerName="payload-processing" containerID="cri-o://2053bff1310a76a1aafcae8f1ccbcb25bc9843ee6d47361eca6bcd6e2393929e" gracePeriod=25 Oct 30 17:31:22.525366 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:22.525338 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-93e020-6c9d97748-pnfp5" Oct 30 17:31:22.661117 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:22.661086 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af795373-0e85-4f37-80ab-2caa6e2dd689-kube-api-access-n5l2k\" (UniqueName: \"kubernetes.io/projected/af795373-0e85-4f37-80ab-2caa6e2dd689-kube-api-access-n5l2k\") pod \"af795373-0e85-4f37-80ab-2caa6e2dd689\" (UID: \"af795373-0e85-4f37-80ab-2caa6e2dd689\") " Oct 30 17:31:22.661463 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:22.661444 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af795373-0e85-4f37-80ab-2caa6e2dd689-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af795373-0e85-4f37-80ab-2caa6e2dd689-plugins-config-volume\") pod \"af795373-0e85-4f37-80ab-2caa6e2dd689\" (UID: \"af795373-0e85-4f37-80ab-2caa6e2dd689\") " Oct 30 17:31:22.662731 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:22.662699 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af795373-0e85-4f37-80ab-2caa6e2dd689-plugins-config-volume" pod "af795373-0e85-4f37-80ab-2caa6e2dd689" (UID: "af795373-0e85-4f37-80ab-2caa6e2dd689"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:31:22.664628 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:22.664593 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af795373-0e85-4f37-80ab-2caa6e2dd689-kube-api-access-n5l2k" pod "af795373-0e85-4f37-80ab-2caa6e2dd689" (UID: "af795373-0e85-4f37-80ab-2caa6e2dd689"). InnerVolumeSpecName "kube-api-access-n5l2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:31:22.679168 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:22.679143 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz" Oct 30 17:31:22.763692 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:22.763613 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc-workload-socket\") pod \"b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc\" (UID: \"b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc\") " Oct 30 17:31:22.763692 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:22.763652 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc-workload-certs\") pod \"b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc\" (UID: \"b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc\") " Oct 30 17:31:22.763935 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:22.763696 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc-istio-data\") pod \"b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc\" (UID: \"b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc\") " Oct 30 17:31:22.763935 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:22.763736 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc-istio-podinfo\") pod \"b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc\" (UID: \"b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc\") " Oct 30 17:31:22.763935 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:22.763909 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc-workload-socket" pod "b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc" (UID: "b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:31:22.763935 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:22.763914 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc-kube-api-access-sm8q4\" (UniqueName: \"kubernetes.io/projected/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc-kube-api-access-sm8q4\") pod \"b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc\" (UID: \"b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc\") " Oct 30 17:31:22.764172 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:22.764029 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc-workload-certs" pod "b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc" (UID: "b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:31:22.764172 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:22.764039 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc-istio-envoy\") pod \"b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc\" (UID: \"b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc\") " Oct 30 17:31:22.764293 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:22.764212 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc-credential-socket\") pod \"b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc\" (UID: \"b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc\") " Oct 30 17:31:22.764293 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:22.764268 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc-istiod-ca-cert\") pod \"b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc\" (UID: \"b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc\") " Oct 30 17:31:22.764750 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:22.764438 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc-istio-token\" (UniqueName: \"kubernetes.io/projected/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc-istio-token\") pod \"b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc\" (UID: \"b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc\") " Oct 30 17:31:22.764750 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:22.764496 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc-credential-socket" pod "b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc" (UID: "b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:31:22.764972 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:22.764743 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc-istiod-ca-cert" pod "b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc" (UID: "b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:31:22.765035 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:22.765003 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc-istio-data" pod "b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc" (UID: "b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:31:22.765480 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:22.765458 4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc-workload-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:31:22.765589 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:22.765498 4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc-workload-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:31:22.765589 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:22.765514 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af795373-0e85-4f37-80ab-2caa6e2dd689-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:31:22.765589 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:22.765529 4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc-istio-data\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:31:22.765589 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:22.765542 4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc-credential-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:31:22.765589 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:22.765554 4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc-istiod-ca-cert\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:31:22.765589 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:22.765582 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5l2k\" (UniqueName: \"kubernetes.io/projected/af795373-0e85-4f37-80ab-2caa6e2dd689-kube-api-access-n5l2k\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:31:22.766767 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:22.766739 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc-istio-envoy" pod "b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc" (UID: "b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:31:22.766767 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:22.766757 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc-kube-api-access-sm8q4" pod "b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc" (UID: "b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc"). InnerVolumeSpecName "kube-api-access-sm8q4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:31:22.767298 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:22.767277 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc-istio-podinfo" pod "b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc" (UID: "b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:31:22.767900 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:22.767872 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc-istio-token" pod "b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc" (UID: "b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:31:22.868132 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:22.868096 4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc-istio-token\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:31:22.868132 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:22.868135 4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc-istio-podinfo\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:31:22.868366 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:22.868151 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sm8q4\" (UniqueName: \"kubernetes.io/projected/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc-kube-api-access-sm8q4\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:31:22.868366 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:22.868165 4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc-istio-envoy\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:31:23.013972 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:23.013884 4522 generic.go:356] "Generic (PLEG): container finished" podID="af795373-0e85-4f37-80ab-2caa6e2dd689" containerID="2053bff1310a76a1aafcae8f1ccbcb25bc9843ee6d47361eca6bcd6e2393929e" exitCode=0 Oct 30 17:31:23.013972 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:23.013930 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-93e020-6c9d97748-pnfp5" Oct 30 17:31:23.014181 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:23.013967 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-93e020-6c9d97748-pnfp5" event={"ID":"af795373-0e85-4f37-80ab-2caa6e2dd689","Type":"ContainerDied","Data":"2053bff1310a76a1aafcae8f1ccbcb25bc9843ee6d47361eca6bcd6e2393929e"} Oct 30 17:31:23.014181 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:23.014030 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-93e020-6c9d97748-pnfp5" event={"ID":"af795373-0e85-4f37-80ab-2caa6e2dd689","Type":"ContainerDied","Data":"1bd6ee6ccf3869f2481d598274672e20399b371751d95448d0d340d671a63e05"} Oct 30 17:31:23.014181 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:23.014055 4522 scope.go:122] "RemoveContainer" containerID="2053bff1310a76a1aafcae8f1ccbcb25bc9843ee6d47361eca6bcd6e2393929e" Oct 30 17:31:23.015835 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:23.015811 4522 generic.go:356] "Generic (PLEG): container finished" podID="b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc" containerID="7f2c75141c115bd014075858751b48d33608e1a5877f0ba2ede04d93a3c899ad" exitCode=0 Oct 30 17:31:23.015947 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:23.015861 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz" Oct 30 17:31:23.015947 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:23.015894 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz" event={"ID":"b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc","Type":"ContainerDied","Data":"7f2c75141c115bd014075858751b48d33608e1a5877f0ba2ede04d93a3c899ad"} Oct 30 17:31:23.015947 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:23.015932 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz" event={"ID":"b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc","Type":"ContainerDied","Data":"4fff6a76f7727a7f279c29bc13adc1d9c007cec815984b225686a4af716f5e15"} Oct 30 17:31:23.026830 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:23.026814 4522 scope.go:122] "RemoveContainer" containerID="2053bff1310a76a1aafcae8f1ccbcb25bc9843ee6d47361eca6bcd6e2393929e" Oct 30 17:31:23.027178 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:31:23.027064 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2053bff1310a76a1aafcae8f1ccbcb25bc9843ee6d47361eca6bcd6e2393929e\": container with ID starting with 2053bff1310a76a1aafcae8f1ccbcb25bc9843ee6d47361eca6bcd6e2393929e not found: ID does not exist" containerID="2053bff1310a76a1aafcae8f1ccbcb25bc9843ee6d47361eca6bcd6e2393929e" Oct 30 17:31:23.027178 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:23.027087 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2053bff1310a76a1aafcae8f1ccbcb25bc9843ee6d47361eca6bcd6e2393929e"} err="failed to get container status \"2053bff1310a76a1aafcae8f1ccbcb25bc9843ee6d47361eca6bcd6e2393929e\": rpc error: code = NotFound desc = could not find container \"2053bff1310a76a1aafcae8f1ccbcb25bc9843ee6d47361eca6bcd6e2393929e\": container with ID starting with 2053bff1310a76a1aafcae8f1ccbcb25bc9843ee6d47361eca6bcd6e2393929e not found: ID does not exist" Oct 30 17:31:23.027178 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:23.027109 4522 scope.go:122] "RemoveContainer" containerID="7f2c75141c115bd014075858751b48d33608e1a5877f0ba2ede04d93a3c899ad" Oct 30 17:31:23.038003 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:23.037984 4522 scope.go:122] "RemoveContainer" containerID="7f2c75141c115bd014075858751b48d33608e1a5877f0ba2ede04d93a3c899ad" Oct 30 17:31:23.038263 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:31:23.038246 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f2c75141c115bd014075858751b48d33608e1a5877f0ba2ede04d93a3c899ad\": container with ID starting with 7f2c75141c115bd014075858751b48d33608e1a5877f0ba2ede04d93a3c899ad not found: ID does not exist" containerID="7f2c75141c115bd014075858751b48d33608e1a5877f0ba2ede04d93a3c899ad" Oct 30 17:31:23.038355 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:23.038270 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f2c75141c115bd014075858751b48d33608e1a5877f0ba2ede04d93a3c899ad"} err="failed to get container status \"7f2c75141c115bd014075858751b48d33608e1a5877f0ba2ede04d93a3c899ad\": rpc error: code = NotFound desc = could not find container \"7f2c75141c115bd014075858751b48d33608e1a5877f0ba2ede04d93a3c899ad\": container with ID starting with 7f2c75141c115bd014075858751b48d33608e1a5877f0ba2ede04d93a3c899ad not found: ID does not exist" Oct 30 17:31:23.079627 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:23.079585 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz"] Oct 30 17:31:23.086469 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:23.086446 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-zftlz"] Oct 30 17:31:23.126287 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:23.126256 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-93e020-6c9d97748-pnfp5"] Oct 30 17:31:23.131294 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:23.131273 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-93e020-6c9d97748-pnfp5"] Oct 30 17:31:23.926442 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:23.926414 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8a7e4a-66cd879994-drkhg" Oct 30 17:31:23.926830 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:23.926480 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8a7e4a-5d55c87fcf-qf6bc" Oct 30 17:31:24.177265 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:24.175145 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af795373-0e85-4f37-80ab-2caa6e2dd689" path="/var/lib/kubelet/pods/af795373-0e85-4f37-80ab-2caa6e2dd689/volumes" Oct 30 17:31:24.177265 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:24.176155 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc" path="/var/lib/kubelet/pods/b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc/volumes" Oct 30 17:31:26.213375 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:26.213332 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a884078c-88sll"] Oct 30 17:31:26.214257 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:26.214242 4522 cpu_manager.go:414] "containerMap: removing container" podUID="b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc" containerName="istio-proxy" Oct 30 17:31:26.214352 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:26.214258 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc" containerName="istio-proxy" Oct 30 17:31:26.214352 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:26.214284 4522 cpu_manager.go:414] "containerMap: removing container" podUID="af795373-0e85-4f37-80ab-2caa6e2dd689" containerName="payload-processing" Oct 30 17:31:26.214352 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:26.214289 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af795373-0e85-4f37-80ab-2caa6e2dd689" containerName="payload-processing" Oct 30 17:31:26.214352 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:26.214298 4522 cpu_manager.go:414] "containerMap: removing container" podUID="52db46a9-175b-4cad-bc70-1345d8ddb498" containerName="revoke-keys" Oct 30 17:31:26.214352 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:26.214317 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52db46a9-175b-4cad-bc70-1345d8ddb498" containerName="revoke-keys" Oct 30 17:31:26.214554 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:26.214476 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="b86d7efc-e3a6-43f7-b3cc-e25a8c7b36fc" containerName="istio-proxy" Oct 30 17:31:26.214554 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:26.214488 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="af795373-0e85-4f37-80ab-2caa6e2dd689" containerName="payload-processing" Oct 30 17:31:26.214554 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:26.214495 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="52db46a9-175b-4cad-bc70-1345d8ddb498" containerName="revoke-keys" Oct 30 17:31:26.220197 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:26.220180 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a884078c-88sll" Oct 30 17:31:26.239245 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:26.239221 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a884078c-88sll"] Oct 30 17:31:26.246926 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:26.246900 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:31:26.247070 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:26.246989 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:31:26.326351 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:26.326290 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lz9t\" (UniqueName: \"kubernetes.io/projected/c01c53da-a670-4554-9aac-8948be829acf-kube-api-access-4lz9t\") pod \"maas-api-revoke-sub-a884078c-88sll\" (UID: \"c01c53da-a670-4554-9aac-8948be829acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a884078c-88sll" Oct 30 17:31:26.326531 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:26.326395 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c01c53da-a670-4554-9aac-8948be829acf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a884078c-88sll\" (UID: \"c01c53da-a670-4554-9aac-8948be829acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a884078c-88sll" Oct 30 17:31:26.326607 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:26.326588 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c01c53da-a670-4554-9aac-8948be829acf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a884078c-88sll\" (UID: \"c01c53da-a670-4554-9aac-8948be829acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a884078c-88sll" Oct 30 17:31:26.428223 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:26.428180 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lz9t\" (UniqueName: \"kubernetes.io/projected/c01c53da-a670-4554-9aac-8948be829acf-kube-api-access-4lz9t\") pod \"maas-api-revoke-sub-a884078c-88sll\" (UID: \"c01c53da-a670-4554-9aac-8948be829acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a884078c-88sll" Oct 30 17:31:26.428439 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:26.428321 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c01c53da-a670-4554-9aac-8948be829acf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a884078c-88sll\" (UID: \"c01c53da-a670-4554-9aac-8948be829acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a884078c-88sll" Oct 30 17:31:26.428439 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:26.428366 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c01c53da-a670-4554-9aac-8948be829acf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a884078c-88sll\" (UID: \"c01c53da-a670-4554-9aac-8948be829acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a884078c-88sll" Oct 30 17:31:26.429110 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:26.429082 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c01c53da-a670-4554-9aac-8948be829acf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a884078c-88sll\" (UID: \"c01c53da-a670-4554-9aac-8948be829acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a884078c-88sll" Oct 30 17:31:26.430957 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:26.430931 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c01c53da-a670-4554-9aac-8948be829acf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a884078c-88sll\" (UID: \"c01c53da-a670-4554-9aac-8948be829acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a884078c-88sll" Oct 30 17:31:26.457618 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:26.457585 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lz9t\" (UniqueName: \"kubernetes.io/projected/c01c53da-a670-4554-9aac-8948be829acf-kube-api-access-4lz9t\") pod \"maas-api-revoke-sub-a884078c-88sll\" (UID: \"c01c53da-a670-4554-9aac-8948be829acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a884078c-88sll" Oct 30 17:31:26.536258 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:26.536176 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a884078c-88sll" Oct 30 17:31:26.845195 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:31:26.845160 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc01c53da_a670_4554_9aac_8948be829acf.slice/crio-a30c8edae93f24e560657c568c92359633aed46750fc7f876fa4acfe6351e82f WatchSource:0}: Error finding container a30c8edae93f24e560657c568c92359633aed46750fc7f876fa4acfe6351e82f: Status 404 returned error can't find the container with id a30c8edae93f24e560657c568c92359633aed46750fc7f876fa4acfe6351e82f Oct 30 17:31:26.847977 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:26.847949 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a884078c-88sll"] Oct 30 17:31:27.059454 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:27.059417 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a884078c-88sll" event={"ID":"c01c53da-a670-4554-9aac-8948be829acf","Type":"ContainerStarted","Data":"a30c8edae93f24e560657c568c92359633aed46750fc7f876fa4acfe6351e82f"} Oct 30 17:31:28.039623 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:28.039543 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9656b4a-fmbjm"] Oct 30 17:31:28.054738 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:28.054707 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9656b4a-fmbjm"] Oct 30 17:31:28.070544 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:28.070515 4522 generic.go:356] "Generic (PLEG): container finished" podID="c01c53da-a670-4554-9aac-8948be829acf" containerID="d2025a855d019536d113422d396f034e729edc9861a5aed3a3a41374ba88005b" exitCode=0 Oct 30 17:31:28.070682 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:28.070600 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a884078c-88sll" event={"ID":"c01c53da-a670-4554-9aac-8948be829acf","Type":"ContainerDied","Data":"d2025a855d019536d113422d396f034e729edc9861a5aed3a3a41374ba88005b"} Oct 30 17:31:28.148044 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:28.148011 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ae3b4c0-03b3-4f19-b7e1-892f1dee51f3" path="/var/lib/kubelet/pods/5ae3b4c0-03b3-4f19-b7e1-892f1dee51f3/volumes" Oct 30 17:31:29.279323 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:29.273292 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a884078c-88sll" Oct 30 17:31:29.389110 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:29.389023 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c01c53da-a670-4554-9aac-8948be829acf-kube-api-access-4lz9t\" (UniqueName: \"kubernetes.io/projected/c01c53da-a670-4554-9aac-8948be829acf-kube-api-access-4lz9t\") pod \"c01c53da-a670-4554-9aac-8948be829acf\" (UID: \"c01c53da-a670-4554-9aac-8948be829acf\") " Oct 30 17:31:29.389259 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:29.389191 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c01c53da-a670-4554-9aac-8948be829acf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c01c53da-a670-4554-9aac-8948be829acf-maas-api-xks-ca\") pod \"c01c53da-a670-4554-9aac-8948be829acf\" (UID: \"c01c53da-a670-4554-9aac-8948be829acf\") " Oct 30 17:31:29.389351 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:29.389293 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c01c53da-a670-4554-9aac-8948be829acf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c01c53da-a670-4554-9aac-8948be829acf-maas-api-openshift-ca\") pod \"c01c53da-a670-4554-9aac-8948be829acf\" (UID: \"c01c53da-a670-4554-9aac-8948be829acf\") " Oct 30 17:31:29.389675 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:29.389651 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c01c53da-a670-4554-9aac-8948be829acf-maas-api-openshift-ca" pod "c01c53da-a670-4554-9aac-8948be829acf" (UID: "c01c53da-a670-4554-9aac-8948be829acf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:31:29.390083 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:29.390061 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c01c53da-a670-4554-9aac-8948be829acf-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:31:29.391389 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:29.391366 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c01c53da-a670-4554-9aac-8948be829acf-maas-api-xks-ca" pod "c01c53da-a670-4554-9aac-8948be829acf" (UID: "c01c53da-a670-4554-9aac-8948be829acf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:31:29.391478 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:29.391416 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c01c53da-a670-4554-9aac-8948be829acf-kube-api-access-4lz9t" pod "c01c53da-a670-4554-9aac-8948be829acf" (UID: "c01c53da-a670-4554-9aac-8948be829acf"). InnerVolumeSpecName "kube-api-access-4lz9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:31:29.492206 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:29.492170 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lz9t\" (UniqueName: \"kubernetes.io/projected/c01c53da-a670-4554-9aac-8948be829acf-kube-api-access-4lz9t\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:31:29.492206 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:29.492202 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c01c53da-a670-4554-9aac-8948be829acf-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:31:30.089966 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:30.089935 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a884078c-88sll" Oct 30 17:31:30.090170 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:30.089972 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a884078c-88sll" event={"ID":"c01c53da-a670-4554-9aac-8948be829acf","Type":"ContainerDied","Data":"a30c8edae93f24e560657c568c92359633aed46750fc7f876fa4acfe6351e82f"} Oct 30 17:31:30.090170 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:30.090029 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a30c8edae93f24e560657c568c92359633aed46750fc7f876fa4acfe6351e82f" Oct 30 17:31:34.401499 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:34.401459 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4fb821-57799f5bb7-bvf8b"] Oct 30 17:31:34.402387 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:34.402372 4522 cpu_manager.go:414] "containerMap: removing container" podUID="c01c53da-a670-4554-9aac-8948be829acf" containerName="revoke-keys" Oct 30 17:31:34.402452 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:34.402388 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c01c53da-a670-4554-9aac-8948be829acf" containerName="revoke-keys" Oct 30 17:31:34.402540 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:34.402531 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="c01c53da-a670-4554-9aac-8948be829acf" containerName="revoke-keys" Oct 30 17:31:34.406932 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:34.406914 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4fb821-57799f5bb7-bvf8b" Oct 30 17:31:34.427193 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:34.427164 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4fb821-57799f5bb7-bvf8b"] Oct 30 17:31:34.435419 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:34.435396 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-4fb821\"" Oct 30 17:31:34.435687 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:34.435644 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-4fb821-dockercfg-6x4xf\"" Oct 30 17:31:34.468643 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:34.468607 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8r59\" (UniqueName: \"kubernetes.io/projected/33b0923a-026d-4946-9e27-a6cb6fc5eb94-kube-api-access-t8r59\") pod \"payload-pre-processing-e2e-ipp-b-4fb821-57799f5bb7-bvf8b\" (UID: \"33b0923a-026d-4946-9e27-a6cb6fc5eb94\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4fb821-57799f5bb7-bvf8b" Oct 30 17:31:34.468836 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:34.468723 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33b0923a-026d-4946-9e27-a6cb6fc5eb94-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4fb821-57799f5bb7-bvf8b\" (UID: \"33b0923a-026d-4946-9e27-a6cb6fc5eb94\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4fb821-57799f5bb7-bvf8b" Oct 30 17:31:34.600328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:34.570620 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8r59\" (UniqueName: \"kubernetes.io/projected/33b0923a-026d-4946-9e27-a6cb6fc5eb94-kube-api-access-t8r59\") pod \"payload-pre-processing-e2e-ipp-b-4fb821-57799f5bb7-bvf8b\" (UID: \"33b0923a-026d-4946-9e27-a6cb6fc5eb94\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4fb821-57799f5bb7-bvf8b" Oct 30 17:31:34.600328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:34.570729 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33b0923a-026d-4946-9e27-a6cb6fc5eb94-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4fb821-57799f5bb7-bvf8b\" (UID: \"33b0923a-026d-4946-9e27-a6cb6fc5eb94\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4fb821-57799f5bb7-bvf8b" Oct 30 17:31:34.600328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:34.572078 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33b0923a-026d-4946-9e27-a6cb6fc5eb94-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4fb821-57799f5bb7-bvf8b\" (UID: \"33b0923a-026d-4946-9e27-a6cb6fc5eb94\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4fb821-57799f5bb7-bvf8b" Oct 30 17:31:34.640362 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:34.633808 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8r59\" (UniqueName: \"kubernetes.io/projected/33b0923a-026d-4946-9e27-a6cb6fc5eb94-kube-api-access-t8r59\") pod \"payload-pre-processing-e2e-ipp-b-4fb821-57799f5bb7-bvf8b\" (UID: \"33b0923a-026d-4946-9e27-a6cb6fc5eb94\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4fb821-57799f5bb7-bvf8b" Oct 30 17:31:34.722365 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:34.722056 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4fb821-57799f5bb7-bvf8b" Oct 30 17:31:35.034502 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:31:35.034470 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33b0923a_026d_4946_9e27_a6cb6fc5eb94.slice/crio-d50d71a47079106e25406d256a4fd2f50ad18361073c3ed71563274dd3392e8b WatchSource:0}: Error finding container d50d71a47079106e25406d256a4fd2f50ad18361073c3ed71563274dd3392e8b: Status 404 returned error can't find the container with id d50d71a47079106e25406d256a4fd2f50ad18361073c3ed71563274dd3392e8b Oct 30 17:31:35.037209 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:35.037186 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4fb821-57799f5bb7-bvf8b"] Oct 30 17:31:35.141086 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:35.141057 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4fb821-57799f5bb7-bvf8b" event={"ID":"33b0923a-026d-4946-9e27-a6cb6fc5eb94","Type":"ContainerStarted","Data":"f35e2212fb29f19a43959c4b856786650eb23a9fcb27875c823a8a8af397adf7"} Oct 30 17:31:35.141257 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:35.141110 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4fb821-57799f5bb7-bvf8b" Oct 30 17:31:35.141257 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:35.141129 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4fb821-57799f5bb7-bvf8b" event={"ID":"33b0923a-026d-4946-9e27-a6cb6fc5eb94","Type":"ContainerStarted","Data":"d50d71a47079106e25406d256a4fd2f50ad18361073c3ed71563274dd3392e8b"} Oct 30 17:31:35.190263 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:35.190212 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4fb821-57799f5bb7-bvf8b" podStartSLOduration=1.190199961 podStartE2EDuration="1.190199961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:31:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:31:35.186857935 +0000 UTC m=+2669.187612848" watchObservedRunningTime="2026-10-30 17:31:35.190199961 +0000 UTC m=+2669.190954863" Oct 30 17:31:38.034786 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:38.034751 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81a19108-gwbtm"] Oct 30 17:31:38.049548 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:38.049518 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81a19108-gwbtm"] Oct 30 17:31:38.149209 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:38.149169 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65354bf3-e5b6-4039-8dd5-c6573ac71059" path="/var/lib/kubelet/pods/65354bf3-e5b6-4039-8dd5-c6573ac71059/volumes" Oct 30 17:31:42.951420 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:31:42.951379 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:31:44.130717 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:44.130679 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-abe243-c6b579d8-t25w4"] Oct 30 17:31:44.131237 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:44.130992 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-abe243-c6b579d8-t25w4" podUID="3ea67f7d-62df-49ff-a3c2-6395420ae0b9" containerName="payload-processing" containerID="cri-o://5b00d19009eea15a0537dff45f879ab01140af5b2afeed01419530e4dd50b595" gracePeriod=30 Oct 30 17:31:44.610587 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:44.609225 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-abe243-c6b579d8-t25w4" Oct 30 17:31:44.662000 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:44.661965 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ea67f7d-62df-49ff-a3c2-6395420ae0b9-kube-api-access-8ct5t\" (UniqueName: \"kubernetes.io/projected/3ea67f7d-62df-49ff-a3c2-6395420ae0b9-kube-api-access-8ct5t\") pod \"3ea67f7d-62df-49ff-a3c2-6395420ae0b9\" (UID: \"3ea67f7d-62df-49ff-a3c2-6395420ae0b9\") " Oct 30 17:31:44.662179 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:44.662139 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ea67f7d-62df-49ff-a3c2-6395420ae0b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ea67f7d-62df-49ff-a3c2-6395420ae0b9-plugins-config-volume\") pod \"3ea67f7d-62df-49ff-a3c2-6395420ae0b9\" (UID: \"3ea67f7d-62df-49ff-a3c2-6395420ae0b9\") " Oct 30 17:31:44.662558 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:44.662522 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ea67f7d-62df-49ff-a3c2-6395420ae0b9-plugins-config-volume" pod "3ea67f7d-62df-49ff-a3c2-6395420ae0b9" (UID: "3ea67f7d-62df-49ff-a3c2-6395420ae0b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:31:44.662919 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:44.662899 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ea67f7d-62df-49ff-a3c2-6395420ae0b9-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:31:44.664466 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:44.664443 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ea67f7d-62df-49ff-a3c2-6395420ae0b9-kube-api-access-8ct5t" pod "3ea67f7d-62df-49ff-a3c2-6395420ae0b9" (UID: "3ea67f7d-62df-49ff-a3c2-6395420ae0b9"). InnerVolumeSpecName "kube-api-access-8ct5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:31:44.764246 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:44.764168 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ct5t\" (UniqueName: \"kubernetes.io/projected/3ea67f7d-62df-49ff-a3c2-6395420ae0b9-kube-api-access-8ct5t\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:31:44.890245 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:44.890212 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abe243-k9r5c"] Oct 30 17:31:44.894911 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:44.894884 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abe243-k9r5c"] Oct 30 17:31:45.247672 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:45.247642 4522 generic.go:356] "Generic (PLEG): container finished" podID="3ea67f7d-62df-49ff-a3c2-6395420ae0b9" containerID="5b00d19009eea15a0537dff45f879ab01140af5b2afeed01419530e4dd50b595" exitCode=0 Oct 30 17:31:45.248114 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:45.247702 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-abe243-c6b579d8-t25w4" Oct 30 17:31:45.248114 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:45.247729 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-abe243-c6b579d8-t25w4" event={"ID":"3ea67f7d-62df-49ff-a3c2-6395420ae0b9","Type":"ContainerDied","Data":"5b00d19009eea15a0537dff45f879ab01140af5b2afeed01419530e4dd50b595"} Oct 30 17:31:45.248114 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:45.247791 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-abe243-c6b579d8-t25w4" event={"ID":"3ea67f7d-62df-49ff-a3c2-6395420ae0b9","Type":"ContainerDied","Data":"aa9dc01910bb9a9b5ac17e7b756a4c1185331d47b0cde8a0bfec6259aaa6dafb"} Oct 30 17:31:45.248114 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:45.247819 4522 scope.go:122] "RemoveContainer" containerID="5b00d19009eea15a0537dff45f879ab01140af5b2afeed01419530e4dd50b595" Oct 30 17:31:45.267300 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:45.267276 4522 scope.go:122] "RemoveContainer" containerID="5b00d19009eea15a0537dff45f879ab01140af5b2afeed01419530e4dd50b595" Oct 30 17:31:45.267653 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:31:45.267627 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b00d19009eea15a0537dff45f879ab01140af5b2afeed01419530e4dd50b595\": container with ID starting with 5b00d19009eea15a0537dff45f879ab01140af5b2afeed01419530e4dd50b595 not found: ID does not exist" containerID="5b00d19009eea15a0537dff45f879ab01140af5b2afeed01419530e4dd50b595" Oct 30 17:31:45.267768 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:45.267660 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b00d19009eea15a0537dff45f879ab01140af5b2afeed01419530e4dd50b595"} err="failed to get container status \"5b00d19009eea15a0537dff45f879ab01140af5b2afeed01419530e4dd50b595\": rpc error: code = NotFound desc = could not find container \"5b00d19009eea15a0537dff45f879ab01140af5b2afeed01419530e4dd50b595\": container with ID starting with 5b00d19009eea15a0537dff45f879ab01140af5b2afeed01419530e4dd50b595 not found: ID does not exist" Oct 30 17:31:45.307409 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:45.307377 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-abe243-c6b579d8-t25w4"] Oct 30 17:31:45.315912 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:45.315888 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-abe243-c6b579d8-t25w4"] Oct 30 17:31:46.148933 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:46.148900 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ea67f7d-62df-49ff-a3c2-6395420ae0b9" path="/var/lib/kubelet/pods/3ea67f7d-62df-49ff-a3c2-6395420ae0b9/volumes" Oct 30 17:31:46.149390 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:46.149376 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a848b71b-676f-4d8c-8a12-3dc521b737f2" path="/var/lib/kubelet/pods/a848b71b-676f-4d8c-8a12-3dc521b737f2/volumes" Oct 30 17:31:46.164299 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:46.164279 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4fb821-57799f5bb7-bvf8b" Oct 30 17:31:48.036186 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:48.036151 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fea415b0-z26b7"] Oct 30 17:31:48.049548 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:48.049520 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fea415b0-z26b7"] Oct 30 17:31:48.148776 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:48.148743 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e3b947c-7748-4cf7-a326-e3187e2b2b1d" path="/var/lib/kubelet/pods/5e3b947c-7748-4cf7-a326-e3187e2b2b1d/volumes" Oct 30 17:31:50.583556 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:50.583525 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9e9ce7-5dd848f984-zhxlv"] Oct 30 17:31:50.584467 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:50.584441 4522 cpu_manager.go:414] "containerMap: removing container" podUID="3ea67f7d-62df-49ff-a3c2-6395420ae0b9" containerName="payload-processing" Oct 30 17:31:50.584467 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:50.584458 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ea67f7d-62df-49ff-a3c2-6395420ae0b9" containerName="payload-processing" Oct 30 17:31:50.584659 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:50.584647 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ea67f7d-62df-49ff-a3c2-6395420ae0b9" containerName="payload-processing" Oct 30 17:31:50.590251 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:50.590230 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9e9ce7-5dd848f984-zhxlv" Oct 30 17:31:50.608749 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:50.608724 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-9e9ce7-serving-cert\"" Oct 30 17:31:50.613855 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:50.613822 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9e9ce7-5dd848f984-zhxlv"] Oct 30 17:31:50.765331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:50.764714 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f335c19-096c-4904-ae7f-68883ae53596-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9e9ce7-5dd848f984-zhxlv\" (UID: \"3f335c19-096c-4904-ae7f-68883ae53596\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9e9ce7-5dd848f984-zhxlv" Oct 30 17:31:50.765331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:50.764955 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f335c19-096c-4904-ae7f-68883ae53596-metrics-certs\") pod \"maas-api-e2e-ipp-b-9e9ce7-5dd848f984-zhxlv\" (UID: \"3f335c19-096c-4904-ae7f-68883ae53596\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9e9ce7-5dd848f984-zhxlv" Oct 30 17:31:50.765331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:50.765102 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fvxs\" (UniqueName: \"kubernetes.io/projected/3f335c19-096c-4904-ae7f-68883ae53596-kube-api-access-2fvxs\") pod \"maas-api-e2e-ipp-b-9e9ce7-5dd848f984-zhxlv\" (UID: \"3f335c19-096c-4904-ae7f-68883ae53596\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9e9ce7-5dd848f984-zhxlv" Oct 30 17:31:50.765331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:50.765179 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f335c19-096c-4904-ae7f-68883ae53596-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-9e9ce7-5dd848f984-zhxlv\" (UID: \"3f335c19-096c-4904-ae7f-68883ae53596\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9e9ce7-5dd848f984-zhxlv" Oct 30 17:31:50.866923 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:50.866803 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f335c19-096c-4904-ae7f-68883ae53596-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9e9ce7-5dd848f984-zhxlv\" (UID: \"3f335c19-096c-4904-ae7f-68883ae53596\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9e9ce7-5dd848f984-zhxlv" Oct 30 17:31:50.867089 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:50.866953 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f335c19-096c-4904-ae7f-68883ae53596-metrics-certs\") pod \"maas-api-e2e-ipp-b-9e9ce7-5dd848f984-zhxlv\" (UID: \"3f335c19-096c-4904-ae7f-68883ae53596\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9e9ce7-5dd848f984-zhxlv" Oct 30 17:31:50.867089 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:31:50.866964 4522 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-9e9ce7-serving-cert: secret "maas-api-e2e-ipp-b-9e9ce7-serving-cert" not found Oct 30 17:31:50.867089 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:31:50.867048 4522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f335c19-096c-4904-ae7f-68883ae53596-maas-api-tls podName:3f335c19-096c-4904-ae7f-68883ae53596 nodeName:}" failed. No retries permitted until 2026-10-30 17:31:51.36703122 +0000 UTC m=+2685.367786112 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3f335c19-096c-4904-ae7f-68883ae53596-maas-api-tls") pod "maas-api-e2e-ipp-b-9e9ce7-5dd848f984-zhxlv" (UID: "3f335c19-096c-4904-ae7f-68883ae53596") : secret "maas-api-e2e-ipp-b-9e9ce7-serving-cert" not found Oct 30 17:31:50.867208 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:50.867144 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fvxs\" (UniqueName: \"kubernetes.io/projected/3f335c19-096c-4904-ae7f-68883ae53596-kube-api-access-2fvxs\") pod \"maas-api-e2e-ipp-b-9e9ce7-5dd848f984-zhxlv\" (UID: \"3f335c19-096c-4904-ae7f-68883ae53596\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9e9ce7-5dd848f984-zhxlv" Oct 30 17:31:50.867208 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:50.867186 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f335c19-096c-4904-ae7f-68883ae53596-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-9e9ce7-5dd848f984-zhxlv\" (UID: \"3f335c19-096c-4904-ae7f-68883ae53596\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9e9ce7-5dd848f984-zhxlv" Oct 30 17:31:50.867923 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:50.867882 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f335c19-096c-4904-ae7f-68883ae53596-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-9e9ce7-5dd848f984-zhxlv\" (UID: \"3f335c19-096c-4904-ae7f-68883ae53596\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9e9ce7-5dd848f984-zhxlv" Oct 30 17:31:50.869697 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:50.869677 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f335c19-096c-4904-ae7f-68883ae53596-metrics-certs\") pod \"maas-api-e2e-ipp-b-9e9ce7-5dd848f984-zhxlv\" (UID: \"3f335c19-096c-4904-ae7f-68883ae53596\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9e9ce7-5dd848f984-zhxlv" Oct 30 17:31:50.893989 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:50.893965 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fvxs\" (UniqueName: \"kubernetes.io/projected/3f335c19-096c-4904-ae7f-68883ae53596-kube-api-access-2fvxs\") pod \"maas-api-e2e-ipp-b-9e9ce7-5dd848f984-zhxlv\" (UID: \"3f335c19-096c-4904-ae7f-68883ae53596\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9e9ce7-5dd848f984-zhxlv" Oct 30 17:31:51.375645 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:51.375600 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f335c19-096c-4904-ae7f-68883ae53596-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9e9ce7-5dd848f984-zhxlv\" (UID: \"3f335c19-096c-4904-ae7f-68883ae53596\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9e9ce7-5dd848f984-zhxlv" Oct 30 17:31:51.378354 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:51.378286 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f335c19-096c-4904-ae7f-68883ae53596-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9e9ce7-5dd848f984-zhxlv\" (UID: \"3f335c19-096c-4904-ae7f-68883ae53596\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9e9ce7-5dd848f984-zhxlv" Oct 30 17:31:51.508703 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:51.508666 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9e9ce7-5dd848f984-zhxlv" Oct 30 17:31:51.831736 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:51.831705 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9e9ce7-5dd848f984-zhxlv"] Oct 30 17:31:51.835545 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:31:51.834505 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f335c19_096c_4904_ae7f_68883ae53596.slice/crio-84627766ce32c401c25ff2356acb5a41b6ebe5972633627768016b5a301807c2 WatchSource:0}: Error finding container 84627766ce32c401c25ff2356acb5a41b6ebe5972633627768016b5a301807c2: Status 404 returned error can't find the container with id 84627766ce32c401c25ff2356acb5a41b6ebe5972633627768016b5a301807c2 Oct 30 17:31:52.324492 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:52.324437 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9e9ce7-5dd848f984-zhxlv" event={"ID":"3f335c19-096c-4904-ae7f-68883ae53596","Type":"ContainerStarted","Data":"627f4775570cf128ceee713f2ad9215ef95b6ca3f48e5cadfd8777b02b2898bd"} Oct 30 17:31:52.324695 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:52.324522 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9e9ce7-5dd848f984-zhxlv" Oct 30 17:31:52.324695 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:52.324548 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9e9ce7-5dd848f984-zhxlv" event={"ID":"3f335c19-096c-4904-ae7f-68883ae53596","Type":"ContainerStarted","Data":"84627766ce32c401c25ff2356acb5a41b6ebe5972633627768016b5a301807c2"} Oct 30 17:31:52.438391 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:52.436279 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9e9ce7-5dd848f984-zhxlv" podStartSLOduration=2.142443317 podStartE2EDuration="2.436258836s" totalImagesPullingTime="293.815519ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:31:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:31:52.39570989 +0000 UTC m=+2686.396464829" watchObservedRunningTime="2026-10-30 17:31:52.436258836 +0000 UTC m=+2686.437013738" Oct 30 17:31:58.340127 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:58.340098 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9e9ce7-5dd848f984-zhxlv" Oct 30 17:31:59.037037 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:59.037005 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d678803a-22zr8"] Oct 30 17:31:59.041717 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:31:59.041687 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d678803a-22zr8"] Oct 30 17:32:00.152110 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:00.152035 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b122ae6b-31a7-4085-821e-6f3d69f2076e" path="/var/lib/kubelet/pods/b122ae6b-31a7-4085-821e-6f3d69f2076e/volumes" Oct 30 17:32:03.814255 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:03.814215 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-75586d/auth-test-model-75586d-kserve-76fdc47ddf-7j94j"] Oct 30 17:32:10.035844 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:10.035804 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac8bb032-kq9dm"] Oct 30 17:32:10.049341 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:10.049274 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac8bb032-kq9dm"] Oct 30 17:32:10.148082 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:10.148046 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e6bdbf9-3f04-49c9-b38f-9b7d20e14743" path="/var/lib/kubelet/pods/3e6bdbf9-3f04-49c9-b38f-9b7d20e14743/volumes" Oct 30 17:32:10.279560 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:10.279523 4522 scope.go:122] "RemoveContainer" containerID="1ecbce7b32cec1d9a0765cbb1b184dbd7938601f874e5a38efd39cac2a1a1d39" Oct 30 17:32:10.293160 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:10.293139 4522 scope.go:122] "RemoveContainer" containerID="969248328e2cccc45171c4e22cf438f16d1e4902170b4798aceac0f547959c13" Oct 30 17:32:10.303965 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:10.303939 4522 scope.go:122] "RemoveContainer" containerID="338dc6ef06443409e504a4e8f56126f8300f866ec5c80f28c15bd34aa3083ad3" Oct 30 17:32:12.957544 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:32:12.957503 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:32:15.594334 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:15.594265 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-93e020-c7c6648db-8x96v"] Oct 30 17:32:15.602895 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:15.602872 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-93e020-c7c6648db-8x96v" Oct 30 17:32:15.606844 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:15.606812 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdsxm\" (UniqueName: \"kubernetes.io/projected/a50af203-2bc5-40a0-b668-b166180d3518-kube-api-access-qdsxm\") pod \"payload-processing-e2e-ipp-b-93e020-c7c6648db-8x96v\" (UID: \"a50af203-2bc5-40a0-b668-b166180d3518\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-93e020-c7c6648db-8x96v" Oct 30 17:32:15.606984 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:15.606965 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a50af203-2bc5-40a0-b668-b166180d3518-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-93e020-c7c6648db-8x96v\" (UID: \"a50af203-2bc5-40a0-b668-b166180d3518\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-93e020-c7c6648db-8x96v" Oct 30 17:32:15.621118 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:15.621089 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-93e020-c7c6648db-8x96v"] Oct 30 17:32:15.627405 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:15.627371 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-93e020-dockercfg-5jpm8\"" Oct 30 17:32:15.627591 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:15.627570 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-93e020\"" Oct 30 17:32:15.708530 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:15.708493 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdsxm\" (UniqueName: \"kubernetes.io/projected/a50af203-2bc5-40a0-b668-b166180d3518-kube-api-access-qdsxm\") pod \"payload-processing-e2e-ipp-b-93e020-c7c6648db-8x96v\" (UID: \"a50af203-2bc5-40a0-b668-b166180d3518\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-93e020-c7c6648db-8x96v" Oct 30 17:32:15.708735 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:15.708610 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a50af203-2bc5-40a0-b668-b166180d3518-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-93e020-c7c6648db-8x96v\" (UID: \"a50af203-2bc5-40a0-b668-b166180d3518\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-93e020-c7c6648db-8x96v" Oct 30 17:32:15.709334 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:15.709292 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a50af203-2bc5-40a0-b668-b166180d3518-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-93e020-c7c6648db-8x96v\" (UID: \"a50af203-2bc5-40a0-b668-b166180d3518\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-93e020-c7c6648db-8x96v" Oct 30 17:32:15.736917 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:15.736883 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdsxm\" (UniqueName: \"kubernetes.io/projected/a50af203-2bc5-40a0-b668-b166180d3518-kube-api-access-qdsxm\") pod \"payload-processing-e2e-ipp-b-93e020-c7c6648db-8x96v\" (UID: \"a50af203-2bc5-40a0-b668-b166180d3518\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-93e020-c7c6648db-8x96v" Oct 30 17:32:15.740994 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:15.740960 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-93e020-6444c5bc99-xfnzj"] Oct 30 17:32:15.746250 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:15.746226 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-93e020-6444c5bc99-xfnzj" Oct 30 17:32:15.769582 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:15.769548 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-93e020-6444c5bc99-xfnzj"] Oct 30 17:32:15.810259 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:15.810209 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f46edcee-db0c-4a46-a956-0817cd0ee429-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-93e020-6444c5bc99-xfnzj\" (UID: \"f46edcee-db0c-4a46-a956-0817cd0ee429\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-93e020-6444c5bc99-xfnzj" Oct 30 17:32:15.810455 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:15.810289 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg9zc\" (UniqueName: \"kubernetes.io/projected/f46edcee-db0c-4a46-a956-0817cd0ee429-kube-api-access-dg9zc\") pod \"payload-pre-processing-e2e-ipp-b-93e020-6444c5bc99-xfnzj\" (UID: \"f46edcee-db0c-4a46-a956-0817cd0ee429\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-93e020-6444c5bc99-xfnzj" Oct 30 17:32:15.911742 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:15.911645 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg9zc\" (UniqueName: \"kubernetes.io/projected/f46edcee-db0c-4a46-a956-0817cd0ee429-kube-api-access-dg9zc\") pod \"payload-pre-processing-e2e-ipp-b-93e020-6444c5bc99-xfnzj\" (UID: \"f46edcee-db0c-4a46-a956-0817cd0ee429\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-93e020-6444c5bc99-xfnzj" Oct 30 17:32:15.912039 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:15.912022 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f46edcee-db0c-4a46-a956-0817cd0ee429-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-93e020-6444c5bc99-xfnzj\" (UID: \"f46edcee-db0c-4a46-a956-0817cd0ee429\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-93e020-6444c5bc99-xfnzj" Oct 30 17:32:15.912785 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:15.912767 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f46edcee-db0c-4a46-a956-0817cd0ee429-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-93e020-6444c5bc99-xfnzj\" (UID: \"f46edcee-db0c-4a46-a956-0817cd0ee429\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-93e020-6444c5bc99-xfnzj" Oct 30 17:32:15.920142 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:15.920115 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-93e020-c7c6648db-8x96v" Oct 30 17:32:15.937149 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:15.937115 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg9zc\" (UniqueName: \"kubernetes.io/projected/f46edcee-db0c-4a46-a956-0817cd0ee429-kube-api-access-dg9zc\") pod \"payload-pre-processing-e2e-ipp-b-93e020-6444c5bc99-xfnzj\" (UID: \"f46edcee-db0c-4a46-a956-0817cd0ee429\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-93e020-6444c5bc99-xfnzj" Oct 30 17:32:16.073641 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:16.073608 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-93e020-6444c5bc99-xfnzj" Oct 30 17:32:16.235568 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:32:16.235541 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda50af203_2bc5_40a0_b668_b166180d3518.slice/crio-46348444f1e385589655923d9ce0d77f123ddb9aeed3a4c084416a2dbeffc980 WatchSource:0}: Error finding container 46348444f1e385589655923d9ce0d77f123ddb9aeed3a4c084416a2dbeffc980: Status 404 returned error can't find the container with id 46348444f1e385589655923d9ce0d77f123ddb9aeed3a4c084416a2dbeffc980 Oct 30 17:32:16.239795 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:16.239771 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-93e020-c7c6648db-8x96v"] Oct 30 17:32:16.391709 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:32:16.391671 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf46edcee_db0c_4a46_a956_0817cd0ee429.slice/crio-aa33e8f211a25d5f1a1eb2ac9ed24b607031416395c25ca3b64648bd9921872e WatchSource:0}: Error finding container aa33e8f211a25d5f1a1eb2ac9ed24b607031416395c25ca3b64648bd9921872e: Status 404 returned error can't find the container with id aa33e8f211a25d5f1a1eb2ac9ed24b607031416395c25ca3b64648bd9921872e Oct 30 17:32:16.394252 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:16.394228 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-93e020-6444c5bc99-xfnzj"] Oct 30 17:32:16.722460 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:16.722422 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-93e020-c7c6648db-8x96v" event={"ID":"a50af203-2bc5-40a0-b668-b166180d3518","Type":"ContainerStarted","Data":"5c2e7964435349ea8f9fde92ea3fb0a9dad795d2797edc1286762bc15bb4783d"} Oct 30 17:32:16.722960 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:16.722486 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-93e020-c7c6648db-8x96v" event={"ID":"a50af203-2bc5-40a0-b668-b166180d3518","Type":"ContainerStarted","Data":"46348444f1e385589655923d9ce0d77f123ddb9aeed3a4c084416a2dbeffc980"} Oct 30 17:32:16.722960 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:16.722517 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-93e020-c7c6648db-8x96v" Oct 30 17:32:16.724621 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:16.724592 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-93e020-6444c5bc99-xfnzj" event={"ID":"f46edcee-db0c-4a46-a956-0817cd0ee429","Type":"ContainerStarted","Data":"ae90fd3a3c3e1e8ef02d527ac6e75234290b15ba3fcdc2bbbfb2bb1c84988e12"} Oct 30 17:32:16.724735 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:16.724641 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-93e020-6444c5bc99-xfnzj" event={"ID":"f46edcee-db0c-4a46-a956-0817cd0ee429","Type":"ContainerStarted","Data":"aa33e8f211a25d5f1a1eb2ac9ed24b607031416395c25ca3b64648bd9921872e"} Oct 30 17:32:16.724735 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:16.724684 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-93e020-6444c5bc99-xfnzj" Oct 30 17:32:16.771004 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:16.770944 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-93e020-c7c6648db-8x96v" podStartSLOduration=1.7709256770000001 podStartE2EDuration="1.770925677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:32:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:32:16.767599615 +0000 UTC m=+2710.768354535" watchObservedRunningTime="2026-10-30 17:32:16.770925677 +0000 UTC m=+2710.771680580" Oct 30 17:32:16.817036 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:16.816979 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-93e020-6444c5bc99-xfnzj" podStartSLOduration=1.816963326 podStartE2EDuration="1.816963326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:32:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:32:16.811690656 +0000 UTC m=+2710.812445558" watchObservedRunningTime="2026-10-30 17:32:16.816963326 +0000 UTC m=+2710.817718226" Oct 30 17:32:18.052234 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:18.052190 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-424ecfc9-ds2z7"] Oct 30 17:32:18.054989 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:18.054963 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-424ecfc9-ds2z7"] Oct 30 17:32:18.147642 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:18.147606 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b54956ec-9536-4d68-af21-67b316b5b169" path="/var/lib/kubelet/pods/b54956ec-9536-4d68-af21-67b316b5b169/volumes" Oct 30 17:32:18.838148 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:18.838102 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-75586d/auth-test-model-75586d-kserve-76fdc47ddf-7j94j" podUID="920b5d6f-bdb8-47a6-b573-627e25beb8c5" containerName="main" containerID="cri-o://d8f528ea5fd3a4efa3eb95b8d7e8b302db91ec092ff4a0b410da52b0087ce662" gracePeriod=45 Oct 30 17:32:19.160763 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:19.160735 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-75586d/auth-test-model-75586d-kserve-76fdc47ddf-7j94j" Oct 30 17:32:19.273150 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:19.273112 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/920b5d6f-bdb8-47a6-b573-627e25beb8c5-tls-certs\" (UniqueName: \"kubernetes.io/secret/920b5d6f-bdb8-47a6-b573-627e25beb8c5-tls-certs\") pod \"920b5d6f-bdb8-47a6-b573-627e25beb8c5\" (UID: \"920b5d6f-bdb8-47a6-b573-627e25beb8c5\") " Oct 30 17:32:19.273401 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:19.273220 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/920b5d6f-bdb8-47a6-b573-627e25beb8c5-kube-api-access-klb2q\" (UniqueName: \"kubernetes.io/projected/920b5d6f-bdb8-47a6-b573-627e25beb8c5-kube-api-access-klb2q\") pod \"920b5d6f-bdb8-47a6-b573-627e25beb8c5\" (UID: \"920b5d6f-bdb8-47a6-b573-627e25beb8c5\") " Oct 30 17:32:19.273401 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:19.273364 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/920b5d6f-bdb8-47a6-b573-627e25beb8c5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/920b5d6f-bdb8-47a6-b573-627e25beb8c5-dshm\") pod \"920b5d6f-bdb8-47a6-b573-627e25beb8c5\" (UID: \"920b5d6f-bdb8-47a6-b573-627e25beb8c5\") " Oct 30 17:32:19.273573 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:19.273554 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/920b5d6f-bdb8-47a6-b573-627e25beb8c5-home\" (UniqueName: \"kubernetes.io/empty-dir/920b5d6f-bdb8-47a6-b573-627e25beb8c5-home\") pod \"920b5d6f-bdb8-47a6-b573-627e25beb8c5\" (UID: \"920b5d6f-bdb8-47a6-b573-627e25beb8c5\") " Oct 30 17:32:19.273643 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:19.273590 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/920b5d6f-bdb8-47a6-b573-627e25beb8c5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/920b5d6f-bdb8-47a6-b573-627e25beb8c5-tmp-dir\") pod \"920b5d6f-bdb8-47a6-b573-627e25beb8c5\" (UID: \"920b5d6f-bdb8-47a6-b573-627e25beb8c5\") " Oct 30 17:32:19.273710 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:19.273662 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/920b5d6f-bdb8-47a6-b573-627e25beb8c5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/920b5d6f-bdb8-47a6-b573-627e25beb8c5-model-cache\") pod \"920b5d6f-bdb8-47a6-b573-627e25beb8c5\" (UID: \"920b5d6f-bdb8-47a6-b573-627e25beb8c5\") " Oct 30 17:32:19.274978 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:19.274946 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/920b5d6f-bdb8-47a6-b573-627e25beb8c5-home" pod "920b5d6f-bdb8-47a6-b573-627e25beb8c5" (UID: "920b5d6f-bdb8-47a6-b573-627e25beb8c5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:32:19.275177 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:19.275154 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/920b5d6f-bdb8-47a6-b573-627e25beb8c5-tmp-dir" pod "920b5d6f-bdb8-47a6-b573-627e25beb8c5" (UID: "920b5d6f-bdb8-47a6-b573-627e25beb8c5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:32:19.275382 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:19.275359 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/920b5d6f-bdb8-47a6-b573-627e25beb8c5-model-cache" pod "920b5d6f-bdb8-47a6-b573-627e25beb8c5" (UID: "920b5d6f-bdb8-47a6-b573-627e25beb8c5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:32:19.295396 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:19.283466 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/920b5d6f-bdb8-47a6-b573-627e25beb8c5-kube-api-access-klb2q" pod "920b5d6f-bdb8-47a6-b573-627e25beb8c5" (UID: "920b5d6f-bdb8-47a6-b573-627e25beb8c5"). InnerVolumeSpecName "kube-api-access-klb2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:32:19.295396 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:19.283535 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/920b5d6f-bdb8-47a6-b573-627e25beb8c5-tls-certs" pod "920b5d6f-bdb8-47a6-b573-627e25beb8c5" (UID: "920b5d6f-bdb8-47a6-b573-627e25beb8c5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:32:19.295396 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:19.283582 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/920b5d6f-bdb8-47a6-b573-627e25beb8c5-dshm" pod "920b5d6f-bdb8-47a6-b573-627e25beb8c5" (UID: "920b5d6f-bdb8-47a6-b573-627e25beb8c5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:32:19.375838 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:19.375759 4522 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/920b5d6f-bdb8-47a6-b573-627e25beb8c5-dshm\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:32:19.375838 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:19.375784 4522 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/920b5d6f-bdb8-47a6-b573-627e25beb8c5-home\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:32:19.375838 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:19.375793 4522 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/920b5d6f-bdb8-47a6-b573-627e25beb8c5-tmp-dir\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:32:19.375838 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:19.375802 4522 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/920b5d6f-bdb8-47a6-b573-627e25beb8c5-model-cache\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:32:19.375838 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:19.375809 4522 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/920b5d6f-bdb8-47a6-b573-627e25beb8c5-tls-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:32:19.375838 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:19.375818 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klb2q\" (UniqueName: \"kubernetes.io/projected/920b5d6f-bdb8-47a6-b573-627e25beb8c5-kube-api-access-klb2q\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:32:19.376166 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:19.375850 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-63e60d-85c58c5cdd-6k2gk"] Oct 30 17:32:19.376166 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:19.376033 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-63e60d-85c58c5cdd-6k2gk" podUID="300450d2-c1b2-4a64-bd41-4a1ec5df3d77" containerName="payload-processing" containerID="cri-o://4c14bf70328cefea257af5321836cfa1200ef5131d702d5aacdb427560bce077" gracePeriod=30 Oct 30 17:32:19.418367 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:19.416905 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-63e60d-5b4499f4c9-5j2lv"] Oct 30 17:32:19.418367 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:19.417265 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e60d-5b4499f4c9-5j2lv" podUID="85668031-55c5-458e-a623-26d81c8d4703" containerName="payload-pre-processing" containerID="cri-o://f4cf50a33c6238ee27b0122125a7d238fa16c070ec5656e5d4b65384511d2b60" gracePeriod=30 Oct 30 17:32:19.920706 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:19.920672 4522 generic.go:356] "Generic (PLEG): container finished" podID="920b5d6f-bdb8-47a6-b573-627e25beb8c5" containerID="d8f528ea5fd3a4efa3eb95b8d7e8b302db91ec092ff4a0b410da52b0087ce662" exitCode=0 Oct 30 17:32:19.921065 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:19.921042 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-75586d/auth-test-model-75586d-kserve-76fdc47ddf-7j94j" event={"ID":"920b5d6f-bdb8-47a6-b573-627e25beb8c5","Type":"ContainerDied","Data":"d8f528ea5fd3a4efa3eb95b8d7e8b302db91ec092ff4a0b410da52b0087ce662"} Oct 30 17:32:19.922678 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:19.922653 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-75586d/auth-test-model-75586d-kserve-76fdc47ddf-7j94j" event={"ID":"920b5d6f-bdb8-47a6-b573-627e25beb8c5","Type":"ContainerDied","Data":"be67b01f30353185439074b56e8fef6ad66bfd17e4bc238e620796b833f62813"} Oct 30 17:32:19.922846 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:19.922834 4522 scope.go:122] "RemoveContainer" containerID="d8f528ea5fd3a4efa3eb95b8d7e8b302db91ec092ff4a0b410da52b0087ce662" Oct 30 17:32:19.923049 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:19.922006 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-75586d/auth-test-model-75586d-kserve-76fdc47ddf-7j94j" Oct 30 17:32:19.929200 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:19.928667 4522 generic.go:356] "Generic (PLEG): container finished" podID="85668031-55c5-458e-a623-26d81c8d4703" containerID="f4cf50a33c6238ee27b0122125a7d238fa16c070ec5656e5d4b65384511d2b60" exitCode=0 Oct 30 17:32:19.929200 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:19.928744 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e60d-5b4499f4c9-5j2lv" event={"ID":"85668031-55c5-458e-a623-26d81c8d4703","Type":"ContainerDied","Data":"f4cf50a33c6238ee27b0122125a7d238fa16c070ec5656e5d4b65384511d2b60"} Oct 30 17:32:19.931017 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:19.930814 4522 generic.go:356] "Generic (PLEG): container finished" podID="300450d2-c1b2-4a64-bd41-4a1ec5df3d77" containerID="4c14bf70328cefea257af5321836cfa1200ef5131d702d5aacdb427560bce077" exitCode=0 Oct 30 17:32:19.931017 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:19.930872 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-63e60d-85c58c5cdd-6k2gk" event={"ID":"300450d2-c1b2-4a64-bd41-4a1ec5df3d77","Type":"ContainerDied","Data":"4c14bf70328cefea257af5321836cfa1200ef5131d702d5aacdb427560bce077"} Oct 30 17:32:19.931017 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:19.930914 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-63e60d-85c58c5cdd-6k2gk" event={"ID":"300450d2-c1b2-4a64-bd41-4a1ec5df3d77","Type":"ContainerDied","Data":"0238c506e505b6a32a0c0eabbd69be3e47aedaffcdc9e0ecc3eb21fe06f5d3dc"} Oct 30 17:32:19.931017 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:19.930999 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0238c506e505b6a32a0c0eabbd69be3e47aedaffcdc9e0ecc3eb21fe06f5d3dc" Oct 30 17:32:19.932907 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:19.932888 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-63e60d-85c58c5cdd-6k2gk" Oct 30 17:32:19.950078 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:19.950056 4522 scope.go:122] "RemoveContainer" containerID="d8f528ea5fd3a4efa3eb95b8d7e8b302db91ec092ff4a0b410da52b0087ce662" Oct 30 17:32:19.950995 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:32:19.950964 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8f528ea5fd3a4efa3eb95b8d7e8b302db91ec092ff4a0b410da52b0087ce662\": container with ID starting with d8f528ea5fd3a4efa3eb95b8d7e8b302db91ec092ff4a0b410da52b0087ce662 not found: ID does not exist" containerID="d8f528ea5fd3a4efa3eb95b8d7e8b302db91ec092ff4a0b410da52b0087ce662" Oct 30 17:32:19.951118 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:19.951010 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8f528ea5fd3a4efa3eb95b8d7e8b302db91ec092ff4a0b410da52b0087ce662"} err="failed to get container status \"d8f528ea5fd3a4efa3eb95b8d7e8b302db91ec092ff4a0b410da52b0087ce662\": rpc error: code = NotFound desc = could not find container \"d8f528ea5fd3a4efa3eb95b8d7e8b302db91ec092ff4a0b410da52b0087ce662\": container with ID starting with d8f528ea5fd3a4efa3eb95b8d7e8b302db91ec092ff4a0b410da52b0087ce662 not found: ID does not exist" Oct 30 17:32:20.002788 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:20.002705 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e60d-5b4499f4c9-5j2lv" Oct 30 17:32:20.034682 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:20.034648 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-75586d/auth-test-model-75586d-kserve-76fdc47ddf-7j94j"] Oct 30 17:32:20.039681 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:20.039649 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-75586d/auth-test-model-75586d-kserve-76fdc47ddf-7j94j"] Oct 30 17:32:20.053633 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:20.053601 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/300450d2-c1b2-4a64-bd41-4a1ec5df3d77-kube-api-access-vqvms\" (UniqueName: \"kubernetes.io/projected/300450d2-c1b2-4a64-bd41-4a1ec5df3d77-kube-api-access-vqvms\") pod \"300450d2-c1b2-4a64-bd41-4a1ec5df3d77\" (UID: \"300450d2-c1b2-4a64-bd41-4a1ec5df3d77\") " Oct 30 17:32:20.053835 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:20.053686 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/300450d2-c1b2-4a64-bd41-4a1ec5df3d77-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/300450d2-c1b2-4a64-bd41-4a1ec5df3d77-plugins-config-volume\") pod \"300450d2-c1b2-4a64-bd41-4a1ec5df3d77\" (UID: \"300450d2-c1b2-4a64-bd41-4a1ec5df3d77\") " Oct 30 17:32:20.054151 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:20.054123 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/300450d2-c1b2-4a64-bd41-4a1ec5df3d77-plugins-config-volume" pod "300450d2-c1b2-4a64-bd41-4a1ec5df3d77" (UID: "300450d2-c1b2-4a64-bd41-4a1ec5df3d77"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:32:20.054556 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:20.054524 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/300450d2-c1b2-4a64-bd41-4a1ec5df3d77-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:32:20.056110 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:20.056081 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/300450d2-c1b2-4a64-bd41-4a1ec5df3d77-kube-api-access-vqvms" pod "300450d2-c1b2-4a64-bd41-4a1ec5df3d77" (UID: "300450d2-c1b2-4a64-bd41-4a1ec5df3d77"). InnerVolumeSpecName "kube-api-access-vqvms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:32:20.150006 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:20.149973 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="920b5d6f-bdb8-47a6-b573-627e25beb8c5" path="/var/lib/kubelet/pods/920b5d6f-bdb8-47a6-b573-627e25beb8c5/volumes" Oct 30 17:32:20.155100 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:20.155071 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85668031-55c5-458e-a623-26d81c8d4703-kube-api-access-6pspc\" (UniqueName: \"kubernetes.io/projected/85668031-55c5-458e-a623-26d81c8d4703-kube-api-access-6pspc\") pod \"85668031-55c5-458e-a623-26d81c8d4703\" (UID: \"85668031-55c5-458e-a623-26d81c8d4703\") " Oct 30 17:32:20.155452 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:20.155434 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85668031-55c5-458e-a623-26d81c8d4703-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85668031-55c5-458e-a623-26d81c8d4703-plugins-config-volume\") pod \"85668031-55c5-458e-a623-26d81c8d4703\" (UID: \"85668031-55c5-458e-a623-26d81c8d4703\") " Oct 30 17:32:20.155859 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:20.155835 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85668031-55c5-458e-a623-26d81c8d4703-plugins-config-volume" pod "85668031-55c5-458e-a623-26d81c8d4703" (UID: "85668031-55c5-458e-a623-26d81c8d4703"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:32:20.155989 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:20.155972 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85668031-55c5-458e-a623-26d81c8d4703-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:32:20.156061 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:20.155992 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqvms\" (UniqueName: \"kubernetes.io/projected/300450d2-c1b2-4a64-bd41-4a1ec5df3d77-kube-api-access-vqvms\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:32:20.157262 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:20.157245 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85668031-55c5-458e-a623-26d81c8d4703-kube-api-access-6pspc" pod "85668031-55c5-458e-a623-26d81c8d4703" (UID: "85668031-55c5-458e-a623-26d81c8d4703"). InnerVolumeSpecName "kube-api-access-6pspc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:32:20.258190 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:20.258147 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pspc\" (UniqueName: \"kubernetes.io/projected/85668031-55c5-458e-a623-26d81c8d4703-kube-api-access-6pspc\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:32:20.942463 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:20.942425 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e60d-5b4499f4c9-5j2lv" Oct 30 17:32:20.942463 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:20.942442 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e60d-5b4499f4c9-5j2lv" event={"ID":"85668031-55c5-458e-a623-26d81c8d4703","Type":"ContainerDied","Data":"58a02dd946eb139a5457c1e1623208328f6fa07aad7eaebfc46d50cd3deefb5c"} Oct 30 17:32:20.942463 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:20.942464 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-63e60d-85c58c5cdd-6k2gk" Oct 30 17:32:20.942772 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:20.942526 4522 scope.go:122] "RemoveContainer" containerID="f4cf50a33c6238ee27b0122125a7d238fa16c070ec5656e5d4b65384511d2b60" Oct 30 17:32:21.017971 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:21.017937 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-63e60d-85c58c5cdd-6k2gk"] Oct 30 17:32:21.031148 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:21.031109 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-63e60d-85c58c5cdd-6k2gk"] Oct 30 17:32:21.041145 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:21.041117 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a20b19b-7krsq"] Oct 30 17:32:21.046464 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:21.046443 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a20b19b-7krsq"] Oct 30 17:32:21.068836 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:21.068800 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-63e60d-5b4499f4c9-5j2lv"] Oct 30 17:32:21.073279 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:21.073250 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-63e60d-5b4499f4c9-5j2lv"] Oct 30 17:32:22.148288 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:22.148259 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="300450d2-c1b2-4a64-bd41-4a1ec5df3d77" path="/var/lib/kubelet/pods/300450d2-c1b2-4a64-bd41-4a1ec5df3d77/volumes" Oct 30 17:32:22.148867 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:22.148850 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85668031-55c5-458e-a623-26d81c8d4703" path="/var/lib/kubelet/pods/85668031-55c5-458e-a623-26d81c8d4703/volumes" Oct 30 17:32:22.149374 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:22.149356 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5a593f9-c4b1-45a5-aef4-f88b8bc76e5f" path="/var/lib/kubelet/pods/e5a593f9-c4b1-45a5-aef4-f88b8bc76e5f/volumes" Oct 30 17:32:26.204131 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:26.204094 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2psr5"] Oct 30 17:32:26.205021 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:26.205004 4522 cpu_manager.go:414] "containerMap: removing container" podUID="300450d2-c1b2-4a64-bd41-4a1ec5df3d77" containerName="payload-processing" Oct 30 17:32:26.205021 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:26.205022 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="300450d2-c1b2-4a64-bd41-4a1ec5df3d77" containerName="payload-processing" Oct 30 17:32:26.205122 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:26.205038 4522 cpu_manager.go:414] "containerMap: removing container" podUID="85668031-55c5-458e-a623-26d81c8d4703" containerName="payload-pre-processing" Oct 30 17:32:26.205122 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:26.205043 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85668031-55c5-458e-a623-26d81c8d4703" containerName="payload-pre-processing" Oct 30 17:32:26.205122 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:26.205069 4522 cpu_manager.go:414] "containerMap: removing container" podUID="920b5d6f-bdb8-47a6-b573-627e25beb8c5" containerName="main" Oct 30 17:32:26.205122 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:26.205074 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="920b5d6f-bdb8-47a6-b573-627e25beb8c5" containerName="main" Oct 30 17:32:26.205262 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:26.205214 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="300450d2-c1b2-4a64-bd41-4a1ec5df3d77" containerName="payload-processing" Oct 30 17:32:26.205262 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:26.205223 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="85668031-55c5-458e-a623-26d81c8d4703" containerName="payload-pre-processing" Oct 30 17:32:26.205262 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:26.205234 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="920b5d6f-bdb8-47a6-b573-627e25beb8c5" containerName="main" Oct 30 17:32:26.210747 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:26.210728 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2psr5" Oct 30 17:32:26.236465 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:26.236424 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2psr5"] Oct 30 17:32:26.253481 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:26.253441 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7e69801d-f91f-47b2-96e6-fc4d9bf46678-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2psr5\" (UID: \"7e69801d-f91f-47b2-96e6-fc4d9bf46678\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2psr5" Oct 30 17:32:26.253798 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:26.253772 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7e69801d-f91f-47b2-96e6-fc4d9bf46678-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2psr5\" (UID: \"7e69801d-f91f-47b2-96e6-fc4d9bf46678\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2psr5" Oct 30 17:32:26.253919 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:26.253877 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7e69801d-f91f-47b2-96e6-fc4d9bf46678-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2psr5\" (UID: \"7e69801d-f91f-47b2-96e6-fc4d9bf46678\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2psr5" Oct 30 17:32:26.254006 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:26.253983 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9j7t\" (UniqueName: \"kubernetes.io/projected/7e69801d-f91f-47b2-96e6-fc4d9bf46678-kube-api-access-t9j7t\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2psr5\" (UID: \"7e69801d-f91f-47b2-96e6-fc4d9bf46678\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2psr5" Oct 30 17:32:26.254076 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:26.254047 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7e69801d-f91f-47b2-96e6-fc4d9bf46678-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2psr5\" (UID: \"7e69801d-f91f-47b2-96e6-fc4d9bf46678\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2psr5" Oct 30 17:32:26.254171 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:26.254150 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e69801d-f91f-47b2-96e6-fc4d9bf46678-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2psr5\" (UID: \"7e69801d-f91f-47b2-96e6-fc4d9bf46678\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2psr5" Oct 30 17:32:26.254413 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:26.254387 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e69801d-f91f-47b2-96e6-fc4d9bf46678-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2psr5\" (UID: \"7e69801d-f91f-47b2-96e6-fc4d9bf46678\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2psr5" Oct 30 17:32:26.327095 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:26.327063 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 17:32:26.327259 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:26.327187 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 17:32:26.327259 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:26.327215 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 17:32:26.327418 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:26.327277 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-xjmfv\"" Oct 30 17:32:26.355911 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:26.355875 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7e69801d-f91f-47b2-96e6-fc4d9bf46678-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2psr5\" (UID: \"7e69801d-f91f-47b2-96e6-fc4d9bf46678\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2psr5" Oct 30 17:32:26.356077 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:26.355932 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e69801d-f91f-47b2-96e6-fc4d9bf46678-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2psr5\" (UID: \"7e69801d-f91f-47b2-96e6-fc4d9bf46678\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2psr5" Oct 30 17:32:26.356077 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:26.356025 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e69801d-f91f-47b2-96e6-fc4d9bf46678-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2psr5\" (UID: \"7e69801d-f91f-47b2-96e6-fc4d9bf46678\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2psr5" Oct 30 17:32:26.356183 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:26.356170 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7e69801d-f91f-47b2-96e6-fc4d9bf46678-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2psr5\" (UID: \"7e69801d-f91f-47b2-96e6-fc4d9bf46678\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2psr5" Oct 30 17:32:26.356249 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:26.356235 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7e69801d-f91f-47b2-96e6-fc4d9bf46678-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2psr5\" (UID: \"7e69801d-f91f-47b2-96e6-fc4d9bf46678\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2psr5" Oct 30 17:32:26.356293 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:26.356272 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7e69801d-f91f-47b2-96e6-fc4d9bf46678-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2psr5\" (UID: \"7e69801d-f91f-47b2-96e6-fc4d9bf46678\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2psr5" Oct 30 17:32:26.356389 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:26.356330 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9j7t\" (UniqueName: \"kubernetes.io/projected/7e69801d-f91f-47b2-96e6-fc4d9bf46678-kube-api-access-t9j7t\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2psr5\" (UID: \"7e69801d-f91f-47b2-96e6-fc4d9bf46678\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2psr5" Oct 30 17:32:26.356907 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:26.356875 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7e69801d-f91f-47b2-96e6-fc4d9bf46678-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2psr5\" (UID: \"7e69801d-f91f-47b2-96e6-fc4d9bf46678\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2psr5" Oct 30 17:32:26.358516 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:26.358496 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7e69801d-f91f-47b2-96e6-fc4d9bf46678-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2psr5\" (UID: \"7e69801d-f91f-47b2-96e6-fc4d9bf46678\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2psr5" Oct 30 17:32:26.358690 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:26.358670 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7e69801d-f91f-47b2-96e6-fc4d9bf46678-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2psr5\" (UID: \"7e69801d-f91f-47b2-96e6-fc4d9bf46678\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2psr5" Oct 30 17:32:26.358812 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:26.358792 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e69801d-f91f-47b2-96e6-fc4d9bf46678-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2psr5\" (UID: \"7e69801d-f91f-47b2-96e6-fc4d9bf46678\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2psr5" Oct 30 17:32:26.358919 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:26.358898 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7e69801d-f91f-47b2-96e6-fc4d9bf46678-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2psr5\" (UID: \"7e69801d-f91f-47b2-96e6-fc4d9bf46678\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2psr5" Oct 30 17:32:26.382331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:26.382278 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e69801d-f91f-47b2-96e6-fc4d9bf46678-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2psr5\" (UID: \"7e69801d-f91f-47b2-96e6-fc4d9bf46678\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2psr5" Oct 30 17:32:26.382612 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:26.382589 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9j7t\" (UniqueName: \"kubernetes.io/projected/7e69801d-f91f-47b2-96e6-fc4d9bf46678-kube-api-access-t9j7t\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2psr5\" (UID: \"7e69801d-f91f-47b2-96e6-fc4d9bf46678\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2psr5" Oct 30 17:32:26.527603 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:26.527523 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2psr5" Oct 30 17:32:26.858876 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:32:26.858841 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e69801d_f91f_47b2_96e6_fc4d9bf46678.slice/crio-c2ef87155fa68e60d55b35d020f0570f0168200e118e54dd8c6382ca9dbd9b28 WatchSource:0}: Error finding container c2ef87155fa68e60d55b35d020f0570f0168200e118e54dd8c6382ca9dbd9b28: Status 404 returned error can't find the container with id c2ef87155fa68e60d55b35d020f0570f0168200e118e54dd8c6382ca9dbd9b28 Oct 30 17:32:26.860837 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:26.860813 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2psr5"] Oct 30 17:32:26.862033 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:26.861991 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:32:26.862122 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:26.862060 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:32:27.001338 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:27.001274 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2psr5" event={"ID":"7e69801d-f91f-47b2-96e6-fc4d9bf46678","Type":"ContainerStarted","Data":"f3303ff42eef707940516e2c1f1f9ecbe1d81d04fc19dc2db0bc0ceff8316e3b"} Oct 30 17:32:27.001520 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:27.001375 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2psr5" Oct 30 17:32:27.001520 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:27.001399 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2psr5" event={"ID":"7e69801d-f91f-47b2-96e6-fc4d9bf46678","Type":"ContainerStarted","Data":"c2ef87155fa68e60d55b35d020f0570f0168200e118e54dd8c6382ca9dbd9b28"} Oct 30 17:32:27.002951 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:27.002921 4522 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-2psr5 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.75:8080/ready\": dial tcp 10.134.1.75:8080: connect: connection refused" start-of-body= Oct 30 17:32:27.003075 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:27.002977 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2psr5" podUID="7e69801d-f91f-47b2-96e6-fc4d9bf46678" containerName="discovery" probeResult="failure" output="Get \"http://10.134.1.75:8080/ready\": dial tcp 10.134.1.75:8080: connect: connection refused" Oct 30 17:32:27.053445 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:27.053392 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2psr5" podStartSLOduration=1.053377707 podStartE2EDuration="1.053377707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:32:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:32:27.049126619 +0000 UTC m=+2721.049881551" watchObservedRunningTime="2026-10-30 17:32:27.053377707 +0000 UTC m=+2721.054132607" Oct 30 17:32:27.736571 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:27.736536 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-93e020-c7c6648db-8x96v" Oct 30 17:32:27.738040 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:27.738015 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-93e020-6444c5bc99-xfnzj" Oct 30 17:32:28.015345 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:28.015224 4522 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-2psr5 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:32:28.015345 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:28.015287 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2psr5" podUID="7e69801d-f91f-47b2-96e6-fc4d9bf46678" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:32:28.139327 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:28.138433 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9"] Oct 30 17:32:28.139327 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:28.138713 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9" podUID="e904bd74-db7c-4430-863a-5277580a6342" containerName="istio-proxy" containerID="cri-o://67401d16a05f49d8810f2279c1776d3596a4fd4967912d5a1abec21b8bac73c8" gracePeriod=30 Oct 30 17:32:28.439388 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:28.404938 4522 patch_prober.go:29] interesting pod/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:32:28.439836 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:28.439805 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9" podUID="e904bd74-db7c-4430-863a-5277580a6342" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:32:29.039552 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:29.039516 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02ca91a1-njlr4"] Oct 30 17:32:29.052480 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:29.052446 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02ca91a1-njlr4"] Oct 30 17:32:30.148930 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:30.148897 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16c3823f-3221-46bc-bbf6-bf3c00559c72" path="/var/lib/kubelet/pods/16c3823f-3221-46bc-bbf6-bf3c00559c72/volumes" Oct 30 17:32:31.014825 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:31.014784 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2psr5" Oct 30 17:32:33.447016 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:33.446986 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9" Oct 30 17:32:33.502650 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:33.502610 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e904bd74-db7c-4430-863a-5277580a6342-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e904bd74-db7c-4430-863a-5277580a6342-istio-podinfo\") pod \"e904bd74-db7c-4430-863a-5277580a6342\" (UID: \"e904bd74-db7c-4430-863a-5277580a6342\") " Oct 30 17:32:33.502828 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:33.502748 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e904bd74-db7c-4430-863a-5277580a6342-istio-token\" (UniqueName: \"kubernetes.io/projected/e904bd74-db7c-4430-863a-5277580a6342-istio-token\") pod \"e904bd74-db7c-4430-863a-5277580a6342\" (UID: \"e904bd74-db7c-4430-863a-5277580a6342\") " Oct 30 17:32:33.502828 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:33.502779 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e904bd74-db7c-4430-863a-5277580a6342-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e904bd74-db7c-4430-863a-5277580a6342-workload-certs\") pod \"e904bd74-db7c-4430-863a-5277580a6342\" (UID: \"e904bd74-db7c-4430-863a-5277580a6342\") " Oct 30 17:32:33.502828 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:33.502818 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e904bd74-db7c-4430-863a-5277580a6342-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e904bd74-db7c-4430-863a-5277580a6342-workload-socket\") pod \"e904bd74-db7c-4430-863a-5277580a6342\" (UID: \"e904bd74-db7c-4430-863a-5277580a6342\") " Oct 30 17:32:33.502949 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:33.502935 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e904bd74-db7c-4430-863a-5277580a6342-kube-api-access-v2rrn\" (UniqueName: \"kubernetes.io/projected/e904bd74-db7c-4430-863a-5277580a6342-kube-api-access-v2rrn\") pod \"e904bd74-db7c-4430-863a-5277580a6342\" (UID: \"e904bd74-db7c-4430-863a-5277580a6342\") " Oct 30 17:32:33.503004 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:33.502985 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e904bd74-db7c-4430-863a-5277580a6342-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e904bd74-db7c-4430-863a-5277580a6342-istio-data\") pod \"e904bd74-db7c-4430-863a-5277580a6342\" (UID: \"e904bd74-db7c-4430-863a-5277580a6342\") " Oct 30 17:32:33.503071 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:33.503016 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e904bd74-db7c-4430-863a-5277580a6342-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e904bd74-db7c-4430-863a-5277580a6342-credential-socket\") pod \"e904bd74-db7c-4430-863a-5277580a6342\" (UID: \"e904bd74-db7c-4430-863a-5277580a6342\") " Oct 30 17:32:33.503134 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:33.503082 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e904bd74-db7c-4430-863a-5277580a6342-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e904bd74-db7c-4430-863a-5277580a6342-istio-envoy\") pod \"e904bd74-db7c-4430-863a-5277580a6342\" (UID: \"e904bd74-db7c-4430-863a-5277580a6342\") " Oct 30 17:32:33.503192 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:33.503127 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e904bd74-db7c-4430-863a-5277580a6342-workload-certs" pod "e904bd74-db7c-4430-863a-5277580a6342" (UID: "e904bd74-db7c-4430-863a-5277580a6342"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:32:33.503192 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:33.503137 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e904bd74-db7c-4430-863a-5277580a6342-workload-socket" pod "e904bd74-db7c-4430-863a-5277580a6342" (UID: "e904bd74-db7c-4430-863a-5277580a6342"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:32:33.503349 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:33.503319 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e904bd74-db7c-4430-863a-5277580a6342-istio-data" pod "e904bd74-db7c-4430-863a-5277580a6342" (UID: "e904bd74-db7c-4430-863a-5277580a6342"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:32:33.503349 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:33.503332 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e904bd74-db7c-4430-863a-5277580a6342-credential-socket" pod "e904bd74-db7c-4430-863a-5277580a6342" (UID: "e904bd74-db7c-4430-863a-5277580a6342"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:32:33.503515 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:33.503499 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e904bd74-db7c-4430-863a-5277580a6342-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e904bd74-db7c-4430-863a-5277580a6342-istiod-ca-cert\") pod \"e904bd74-db7c-4430-863a-5277580a6342\" (UID: \"e904bd74-db7c-4430-863a-5277580a6342\") " Oct 30 17:32:33.503893 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:33.503865 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e904bd74-db7c-4430-863a-5277580a6342-istiod-ca-cert" pod "e904bd74-db7c-4430-863a-5277580a6342" (UID: "e904bd74-db7c-4430-863a-5277580a6342"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:32:33.504323 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:33.504282 4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e904bd74-db7c-4430-863a-5277580a6342-workload-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:32:33.504323 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:33.504315 4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e904bd74-db7c-4430-863a-5277580a6342-workload-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:32:33.504518 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:33.504330 4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e904bd74-db7c-4430-863a-5277580a6342-istio-data\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:32:33.504518 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:33.504344 4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e904bd74-db7c-4430-863a-5277580a6342-credential-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:32:33.504518 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:33.504352 4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e904bd74-db7c-4430-863a-5277580a6342-istiod-ca-cert\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:32:33.505217 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:33.505194 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e904bd74-db7c-4430-863a-5277580a6342-istio-podinfo" pod "e904bd74-db7c-4430-863a-5277580a6342" (UID: "e904bd74-db7c-4430-863a-5277580a6342"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:32:33.505217 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:33.505209 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e904bd74-db7c-4430-863a-5277580a6342-kube-api-access-v2rrn" pod "e904bd74-db7c-4430-863a-5277580a6342" (UID: "e904bd74-db7c-4430-863a-5277580a6342"). InnerVolumeSpecName "kube-api-access-v2rrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:32:33.505439 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:33.505423 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e904bd74-db7c-4430-863a-5277580a6342-istio-token" pod "e904bd74-db7c-4430-863a-5277580a6342" (UID: "e904bd74-db7c-4430-863a-5277580a6342"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:32:33.505547 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:33.505526 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e904bd74-db7c-4430-863a-5277580a6342-istio-envoy" pod "e904bd74-db7c-4430-863a-5277580a6342" (UID: "e904bd74-db7c-4430-863a-5277580a6342"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:32:33.606138 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:33.606106 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2rrn\" (UniqueName: \"kubernetes.io/projected/e904bd74-db7c-4430-863a-5277580a6342-kube-api-access-v2rrn\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:32:33.606138 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:33.606136 4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e904bd74-db7c-4430-863a-5277580a6342-istio-envoy\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:32:33.606138 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:33.606146 4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e904bd74-db7c-4430-863a-5277580a6342-istio-podinfo\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:32:33.606418 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:33.606154 4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e904bd74-db7c-4430-863a-5277580a6342-istio-token\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:32:33.975240 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:33.975205 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-93e020-c7c6648db-8x96v_a50af203-2bc5-40a0-b668-b166180d3518/payload-processing/0.log" Oct 30 17:32:34.081159 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:34.081129 4522 generic.go:356] "Generic (PLEG): container finished" podID="e904bd74-db7c-4430-863a-5277580a6342" containerID="67401d16a05f49d8810f2279c1776d3596a4fd4967912d5a1abec21b8bac73c8" exitCode=0 Oct 30 17:32:34.081399 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:34.081192 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9" Oct 30 17:32:34.081399 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:34.081212 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9" event={"ID":"e904bd74-db7c-4430-863a-5277580a6342","Type":"ContainerDied","Data":"67401d16a05f49d8810f2279c1776d3596a4fd4967912d5a1abec21b8bac73c8"} Oct 30 17:32:34.081399 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:34.081271 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9" event={"ID":"e904bd74-db7c-4430-863a-5277580a6342","Type":"ContainerDied","Data":"83625eef45b74b1fc6cca417339c3499b7e771df0702e252a4a6e62436462fb4"} Oct 30 17:32:34.081399 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:34.081295 4522 scope.go:122] "RemoveContainer" containerID="67401d16a05f49d8810f2279c1776d3596a4fd4967912d5a1abec21b8bac73c8" Oct 30 17:32:34.094171 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:34.094154 4522 scope.go:122] "RemoveContainer" containerID="9d6632e288587cddb15621dad4ff6694cd4720783e0a7dfa7b7d37923bdce115" Oct 30 17:32:34.105351 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:34.105331 4522 scope.go:122] "RemoveContainer" containerID="67401d16a05f49d8810f2279c1776d3596a4fd4967912d5a1abec21b8bac73c8" Oct 30 17:32:34.105594 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:32:34.105575 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67401d16a05f49d8810f2279c1776d3596a4fd4967912d5a1abec21b8bac73c8\": container with ID starting with 67401d16a05f49d8810f2279c1776d3596a4fd4967912d5a1abec21b8bac73c8 not found: ID does not exist" containerID="67401d16a05f49d8810f2279c1776d3596a4fd4967912d5a1abec21b8bac73c8" Oct 30 17:32:34.105646 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:34.105602 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67401d16a05f49d8810f2279c1776d3596a4fd4967912d5a1abec21b8bac73c8"} err="failed to get container status \"67401d16a05f49d8810f2279c1776d3596a4fd4967912d5a1abec21b8bac73c8\": rpc error: code = NotFound desc = could not find container \"67401d16a05f49d8810f2279c1776d3596a4fd4967912d5a1abec21b8bac73c8\": container with ID starting with 67401d16a05f49d8810f2279c1776d3596a4fd4967912d5a1abec21b8bac73c8 not found: ID does not exist" Oct 30 17:32:34.105646 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:34.105619 4522 scope.go:122] "RemoveContainer" containerID="9d6632e288587cddb15621dad4ff6694cd4720783e0a7dfa7b7d37923bdce115" Oct 30 17:32:34.105862 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:32:34.105842 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d6632e288587cddb15621dad4ff6694cd4720783e0a7dfa7b7d37923bdce115\": container with ID starting with 9d6632e288587cddb15621dad4ff6694cd4720783e0a7dfa7b7d37923bdce115 not found: ID does not exist" containerID="9d6632e288587cddb15621dad4ff6694cd4720783e0a7dfa7b7d37923bdce115" Oct 30 17:32:34.105904 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:34.105870 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d6632e288587cddb15621dad4ff6694cd4720783e0a7dfa7b7d37923bdce115"} err="failed to get container status \"9d6632e288587cddb15621dad4ff6694cd4720783e0a7dfa7b7d37923bdce115\": rpc error: code = NotFound desc = could not find container \"9d6632e288587cddb15621dad4ff6694cd4720783e0a7dfa7b7d37923bdce115\": container with ID starting with 9d6632e288587cddb15621dad4ff6694cd4720783e0a7dfa7b7d37923bdce115 not found: ID does not exist" Oct 30 17:32:34.150684 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:34.150654 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9"] Oct 30 17:32:34.159686 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:34.159660 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-63e60d-openshift-default-7c658c4845-gmkn9"] Oct 30 17:32:36.147485 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:36.147456 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e904bd74-db7c-4430-863a-5277580a6342" path="/var/lib/kubelet/pods/e904bd74-db7c-4430-863a-5277580a6342/volumes" Oct 30 17:32:38.882991 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:38.882956 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-987dfc-85f675bb7f-phjx7"] Oct 30 17:32:42.950079 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:32:42.950042 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:32:43.883844 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:43.883800 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-987dfc-85f675bb7f-phjx7" podUID="ceac6a2a-ea4a-45ca-a6f0-2beb70ca52d2" containerName="payload-pre-processing" containerID="cri-o://917a434c5fca903d113330d152bee12c3180782fa101ba7603c82ba4e1a5e8d7" gracePeriod=25 Oct 30 17:32:48.379605 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:48.379568 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph"] Oct 30 17:32:48.380171 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:48.379780 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph" podUID="8ca9280e-ce64-48e6-9a1e-6a104853002e" containerName="istio-proxy" containerID="cri-o://cd4e75e748e3bf0fcdc6f167eed9079be2947f3a176a7afd88868b483680512c" gracePeriod=30 Oct 30 17:32:48.554342 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:48.554293 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-987dfc-85f675bb7f-phjx7" podUID="ceac6a2a-ea4a-45ca-a6f0-2beb70ca52d2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:32:53.678346 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:53.678289 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph" Oct 30 17:32:53.750756 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:53.750716 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ca9280e-ce64-48e6-9a1e-6a104853002e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ca9280e-ce64-48e6-9a1e-6a104853002e-workload-socket\") pod \"8ca9280e-ce64-48e6-9a1e-6a104853002e\" (UID: \"8ca9280e-ce64-48e6-9a1e-6a104853002e\") " Oct 30 17:32:53.750974 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:53.750860 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ca9280e-ce64-48e6-9a1e-6a104853002e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ca9280e-ce64-48e6-9a1e-6a104853002e-workload-certs\") pod \"8ca9280e-ce64-48e6-9a1e-6a104853002e\" (UID: \"8ca9280e-ce64-48e6-9a1e-6a104853002e\") " Oct 30 17:32:53.750974 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:53.750903 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ca9280e-ce64-48e6-9a1e-6a104853002e-istio-token\" (UniqueName: \"kubernetes.io/projected/8ca9280e-ce64-48e6-9a1e-6a104853002e-istio-token\") pod \"8ca9280e-ce64-48e6-9a1e-6a104853002e\" (UID: \"8ca9280e-ce64-48e6-9a1e-6a104853002e\") " Oct 30 17:32:53.751124 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:53.750974 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ca9280e-ce64-48e6-9a1e-6a104853002e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ca9280e-ce64-48e6-9a1e-6a104853002e-istio-envoy\") pod \"8ca9280e-ce64-48e6-9a1e-6a104853002e\" (UID: \"8ca9280e-ce64-48e6-9a1e-6a104853002e\") " Oct 30 17:32:53.751124 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:53.751049 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ca9280e-ce64-48e6-9a1e-6a104853002e-kube-api-access-sdrzz\" (UniqueName: \"kubernetes.io/projected/8ca9280e-ce64-48e6-9a1e-6a104853002e-kube-api-access-sdrzz\") pod \"8ca9280e-ce64-48e6-9a1e-6a104853002e\" (UID: \"8ca9280e-ce64-48e6-9a1e-6a104853002e\") " Oct 30 17:32:53.751291 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:53.751153 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ca9280e-ce64-48e6-9a1e-6a104853002e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ca9280e-ce64-48e6-9a1e-6a104853002e-credential-socket\") pod \"8ca9280e-ce64-48e6-9a1e-6a104853002e\" (UID: \"8ca9280e-ce64-48e6-9a1e-6a104853002e\") " Oct 30 17:32:53.751291 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:53.751250 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ca9280e-ce64-48e6-9a1e-6a104853002e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ca9280e-ce64-48e6-9a1e-6a104853002e-istiod-ca-cert\") pod \"8ca9280e-ce64-48e6-9a1e-6a104853002e\" (UID: \"8ca9280e-ce64-48e6-9a1e-6a104853002e\") " Oct 30 17:32:53.751461 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:53.751359 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ca9280e-ce64-48e6-9a1e-6a104853002e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ca9280e-ce64-48e6-9a1e-6a104853002e-istio-data\") pod \"8ca9280e-ce64-48e6-9a1e-6a104853002e\" (UID: \"8ca9280e-ce64-48e6-9a1e-6a104853002e\") " Oct 30 17:32:53.751461 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:53.751442 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ca9280e-ce64-48e6-9a1e-6a104853002e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ca9280e-ce64-48e6-9a1e-6a104853002e-istio-podinfo\") pod \"8ca9280e-ce64-48e6-9a1e-6a104853002e\" (UID: \"8ca9280e-ce64-48e6-9a1e-6a104853002e\") " Oct 30 17:32:53.752618 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:53.751241 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ca9280e-ce64-48e6-9a1e-6a104853002e-workload-certs" pod "8ca9280e-ce64-48e6-9a1e-6a104853002e" (UID: "8ca9280e-ce64-48e6-9a1e-6a104853002e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:32:53.752753 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:53.751590 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ca9280e-ce64-48e6-9a1e-6a104853002e-workload-socket" pod "8ca9280e-ce64-48e6-9a1e-6a104853002e" (UID: "8ca9280e-ce64-48e6-9a1e-6a104853002e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:32:53.752753 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:53.752197 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ca9280e-ce64-48e6-9a1e-6a104853002e-istiod-ca-cert" pod "8ca9280e-ce64-48e6-9a1e-6a104853002e" (UID: "8ca9280e-ce64-48e6-9a1e-6a104853002e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:32:53.752881 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:53.752811 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ca9280e-ce64-48e6-9a1e-6a104853002e-istio-data" pod "8ca9280e-ce64-48e6-9a1e-6a104853002e" (UID: "8ca9280e-ce64-48e6-9a1e-6a104853002e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:32:53.752975 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:53.752952 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ca9280e-ce64-48e6-9a1e-6a104853002e-credential-socket" pod "8ca9280e-ce64-48e6-9a1e-6a104853002e" (UID: "8ca9280e-ce64-48e6-9a1e-6a104853002e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:32:53.755183 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:53.755151 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ca9280e-ce64-48e6-9a1e-6a104853002e-istio-podinfo" pod "8ca9280e-ce64-48e6-9a1e-6a104853002e" (UID: "8ca9280e-ce64-48e6-9a1e-6a104853002e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:32:53.757343 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:53.757275 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ca9280e-ce64-48e6-9a1e-6a104853002e-istio-envoy" pod "8ca9280e-ce64-48e6-9a1e-6a104853002e" (UID: "8ca9280e-ce64-48e6-9a1e-6a104853002e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:32:53.757469 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:53.757429 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ca9280e-ce64-48e6-9a1e-6a104853002e-kube-api-access-sdrzz" pod "8ca9280e-ce64-48e6-9a1e-6a104853002e" (UID: "8ca9280e-ce64-48e6-9a1e-6a104853002e"). InnerVolumeSpecName "kube-api-access-sdrzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:32:53.757708 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:53.757677 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ca9280e-ce64-48e6-9a1e-6a104853002e-istio-token" pod "8ca9280e-ce64-48e6-9a1e-6a104853002e" (UID: "8ca9280e-ce64-48e6-9a1e-6a104853002e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:32:53.855335 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:53.855281 4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ca9280e-ce64-48e6-9a1e-6a104853002e-credential-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:32:53.855537 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:53.855376 4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ca9280e-ce64-48e6-9a1e-6a104853002e-istiod-ca-cert\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:32:53.855537 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:53.855396 4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ca9280e-ce64-48e6-9a1e-6a104853002e-istio-data\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:32:53.855537 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:53.855408 4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ca9280e-ce64-48e6-9a1e-6a104853002e-istio-podinfo\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:32:53.855537 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:53.855420 4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ca9280e-ce64-48e6-9a1e-6a104853002e-workload-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:32:53.855537 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:53.855434 4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ca9280e-ce64-48e6-9a1e-6a104853002e-workload-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:32:53.855537 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:53.855446 4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ca9280e-ce64-48e6-9a1e-6a104853002e-istio-token\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:32:53.855537 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:53.855457 4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ca9280e-ce64-48e6-9a1e-6a104853002e-istio-envoy\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:32:53.855537 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:53.855471 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdrzz\" (UniqueName: \"kubernetes.io/projected/8ca9280e-ce64-48e6-9a1e-6a104853002e-kube-api-access-sdrzz\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:32:54.269509 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:54.269475 4522 generic.go:356] "Generic (PLEG): container finished" podID="8ca9280e-ce64-48e6-9a1e-6a104853002e" containerID="cd4e75e748e3bf0fcdc6f167eed9079be2947f3a176a7afd88868b483680512c" exitCode=0 Oct 30 17:32:54.269712 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:54.269523 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph" Oct 30 17:32:54.269712 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:54.269555 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph" event={"ID":"8ca9280e-ce64-48e6-9a1e-6a104853002e","Type":"ContainerDied","Data":"cd4e75e748e3bf0fcdc6f167eed9079be2947f3a176a7afd88868b483680512c"} Oct 30 17:32:54.269712 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:54.269622 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph" event={"ID":"8ca9280e-ce64-48e6-9a1e-6a104853002e","Type":"ContainerDied","Data":"2ef5de0c6f1f7cc73f0585836425190d0f06e78404590a68763693ae72225647"} Oct 30 17:32:54.269712 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:54.269635 4522 scope.go:122] "RemoveContainer" containerID="cd4e75e748e3bf0fcdc6f167eed9079be2947f3a176a7afd88868b483680512c" Oct 30 17:32:54.291389 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:54.291367 4522 scope.go:122] "RemoveContainer" containerID="cd4e75e748e3bf0fcdc6f167eed9079be2947f3a176a7afd88868b483680512c" Oct 30 17:32:54.291696 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:32:54.291675 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd4e75e748e3bf0fcdc6f167eed9079be2947f3a176a7afd88868b483680512c\": container with ID starting with cd4e75e748e3bf0fcdc6f167eed9079be2947f3a176a7afd88868b483680512c not found: ID does not exist" containerID="cd4e75e748e3bf0fcdc6f167eed9079be2947f3a176a7afd88868b483680512c" Oct 30 17:32:54.291752 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:54.291709 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd4e75e748e3bf0fcdc6f167eed9079be2947f3a176a7afd88868b483680512c"} err="failed to get container status \"cd4e75e748e3bf0fcdc6f167eed9079be2947f3a176a7afd88868b483680512c\": rpc error: code = NotFound desc = could not find container \"cd4e75e748e3bf0fcdc6f167eed9079be2947f3a176a7afd88868b483680512c\": container with ID starting with cd4e75e748e3bf0fcdc6f167eed9079be2947f3a176a7afd88868b483680512c not found: ID does not exist" Oct 30 17:32:54.344294 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:54.344260 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph"] Oct 30 17:32:54.357800 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:54.357764 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-9rjph"] Oct 30 17:32:54.935451 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:54.935415 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9e9ce7-699c4dcc55-fjmlj"] Oct 30 17:32:56.149133 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:56.149097 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ca9280e-ce64-48e6-9a1e-6a104853002e" path="/var/lib/kubelet/pods/8ca9280e-ce64-48e6-9a1e-6a104853002e/volumes" Oct 30 17:32:57.365713 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:57.365676 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc"] Oct 30 17:32:57.366149 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:57.365887 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc" podUID="4d44befd-705e-4589-b6d0-c944810a76bc" containerName="istio-proxy" containerID="cri-o://8e2803f8b7c3d8f4026c1c9193383ed799fb717c5629d680431eb3a3e6190c06" gracePeriod=30 Oct 30 17:32:58.554232 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:58.554194 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-987dfc-85f675bb7f-phjx7" podUID="ceac6a2a-ea4a-45ca-a6f0-2beb70ca52d2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:32:59.936611 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:32:59.936555 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9e9ce7-699c4dcc55-fjmlj" podUID="1c214620-08c5-4a95-aea8-b2d4cbae85a2" containerName="payload-pre-processing" containerID="cri-o://f28a669d967d6a8b39a9908c54c69f9ce6c16655c107ae1537b6fcdd60784dac" gracePeriod=25 Oct 30 17:33:00.243387 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:00.243359 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9e9ce7-699c4dcc55-fjmlj" Oct 30 17:33:00.347326 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:00.337698 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c214620-08c5-4a95-aea8-b2d4cbae85a2-kube-api-access-ckxt6\" (UniqueName: \"kubernetes.io/projected/1c214620-08c5-4a95-aea8-b2d4cbae85a2-kube-api-access-ckxt6\") pod \"1c214620-08c5-4a95-aea8-b2d4cbae85a2\" (UID: \"1c214620-08c5-4a95-aea8-b2d4cbae85a2\") " Oct 30 17:33:00.347326 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:00.337886 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c214620-08c5-4a95-aea8-b2d4cbae85a2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c214620-08c5-4a95-aea8-b2d4cbae85a2-plugins-config-volume\") pod \"1c214620-08c5-4a95-aea8-b2d4cbae85a2\" (UID: \"1c214620-08c5-4a95-aea8-b2d4cbae85a2\") " Oct 30 17:33:00.347326 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:00.339486 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c214620-08c5-4a95-aea8-b2d4cbae85a2-plugins-config-volume" pod "1c214620-08c5-4a95-aea8-b2d4cbae85a2" (UID: "1c214620-08c5-4a95-aea8-b2d4cbae85a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:33:00.347326 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:00.342210 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c214620-08c5-4a95-aea8-b2d4cbae85a2-kube-api-access-ckxt6" pod "1c214620-08c5-4a95-aea8-b2d4cbae85a2" (UID: "1c214620-08c5-4a95-aea8-b2d4cbae85a2"). InnerVolumeSpecName "kube-api-access-ckxt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:33:00.390267 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:00.390180 4522 generic.go:356] "Generic (PLEG): container finished" podID="1c214620-08c5-4a95-aea8-b2d4cbae85a2" containerID="f28a669d967d6a8b39a9908c54c69f9ce6c16655c107ae1537b6fcdd60784dac" exitCode=0 Oct 30 17:33:00.390267 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:00.390234 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9e9ce7-699c4dcc55-fjmlj" Oct 30 17:33:00.390480 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:00.390259 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9e9ce7-699c4dcc55-fjmlj" event={"ID":"1c214620-08c5-4a95-aea8-b2d4cbae85a2","Type":"ContainerDied","Data":"f28a669d967d6a8b39a9908c54c69f9ce6c16655c107ae1537b6fcdd60784dac"} Oct 30 17:33:00.390480 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:00.390353 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9e9ce7-699c4dcc55-fjmlj" event={"ID":"1c214620-08c5-4a95-aea8-b2d4cbae85a2","Type":"ContainerDied","Data":"fb719603742632b4f572af5af2be2d10df2d0eed976b1cff7c4f877ea1df10a5"} Oct 30 17:33:00.390480 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:00.390377 4522 scope.go:122] "RemoveContainer" containerID="f28a669d967d6a8b39a9908c54c69f9ce6c16655c107ae1537b6fcdd60784dac" Oct 30 17:33:00.403189 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:00.403166 4522 scope.go:122] "RemoveContainer" containerID="f28a669d967d6a8b39a9908c54c69f9ce6c16655c107ae1537b6fcdd60784dac" Oct 30 17:33:00.403528 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:33:00.403505 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f28a669d967d6a8b39a9908c54c69f9ce6c16655c107ae1537b6fcdd60784dac\": container with ID starting with f28a669d967d6a8b39a9908c54c69f9ce6c16655c107ae1537b6fcdd60784dac not found: ID does not exist" containerID="f28a669d967d6a8b39a9908c54c69f9ce6c16655c107ae1537b6fcdd60784dac" Oct 30 17:33:00.403583 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:00.403538 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f28a669d967d6a8b39a9908c54c69f9ce6c16655c107ae1537b6fcdd60784dac"} err="failed to get container status \"f28a669d967d6a8b39a9908c54c69f9ce6c16655c107ae1537b6fcdd60784dac\": rpc error: code = NotFound desc = could not find container \"f28a669d967d6a8b39a9908c54c69f9ce6c16655c107ae1537b6fcdd60784dac\": container with ID starting with f28a669d967d6a8b39a9908c54c69f9ce6c16655c107ae1537b6fcdd60784dac not found: ID does not exist" Oct 30 17:33:00.440712 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:00.440674 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckxt6\" (UniqueName: \"kubernetes.io/projected/1c214620-08c5-4a95-aea8-b2d4cbae85a2-kube-api-access-ckxt6\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:33:00.440712 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:00.440704 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c214620-08c5-4a95-aea8-b2d4cbae85a2-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:33:00.452004 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:00.451967 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9e9ce7-699c4dcc55-fjmlj"] Oct 30 17:33:00.458748 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:00.458720 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9e9ce7-699c4dcc55-fjmlj"] Oct 30 17:33:02.148849 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:02.148814 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c214620-08c5-4a95-aea8-b2d4cbae85a2" path="/var/lib/kubelet/pods/1c214620-08c5-4a95-aea8-b2d4cbae85a2/volumes" Oct 30 17:33:02.679715 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:02.679687 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc" Oct 30 17:33:02.794205 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:02.794106 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d44befd-705e-4589-b6d0-c944810a76bc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d44befd-705e-4589-b6d0-c944810a76bc-workload-socket\") pod \"4d44befd-705e-4589-b6d0-c944810a76bc\" (UID: \"4d44befd-705e-4589-b6d0-c944810a76bc\") " Oct 30 17:33:02.794205 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:02.794164 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d44befd-705e-4589-b6d0-c944810a76bc-istio-token\" (UniqueName: \"kubernetes.io/projected/4d44befd-705e-4589-b6d0-c944810a76bc-istio-token\") pod \"4d44befd-705e-4589-b6d0-c944810a76bc\" (UID: \"4d44befd-705e-4589-b6d0-c944810a76bc\") " Oct 30 17:33:02.794498 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:02.794234 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4d44befd-705e-4589-b6d0-c944810a76bc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d44befd-705e-4589-b6d0-c944810a76bc-istio-podinfo\") pod \"4d44befd-705e-4589-b6d0-c944810a76bc\" (UID: \"4d44befd-705e-4589-b6d0-c944810a76bc\") " Oct 30 17:33:02.794561 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:02.794522 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d44befd-705e-4589-b6d0-c944810a76bc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d44befd-705e-4589-b6d0-c944810a76bc-istio-data\") pod \"4d44befd-705e-4589-b6d0-c944810a76bc\" (UID: \"4d44befd-705e-4589-b6d0-c944810a76bc\") " Oct 30 17:33:02.794561 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:02.794524 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d44befd-705e-4589-b6d0-c944810a76bc-workload-socket" pod "4d44befd-705e-4589-b6d0-c944810a76bc" (UID: "4d44befd-705e-4589-b6d0-c944810a76bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:33:02.794681 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:02.794629 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d44befd-705e-4589-b6d0-c944810a76bc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d44befd-705e-4589-b6d0-c944810a76bc-istio-envoy\") pod \"4d44befd-705e-4589-b6d0-c944810a76bc\" (UID: \"4d44befd-705e-4589-b6d0-c944810a76bc\") " Oct 30 17:33:02.794743 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:02.794695 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d44befd-705e-4589-b6d0-c944810a76bc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d44befd-705e-4589-b6d0-c944810a76bc-workload-certs\") pod \"4d44befd-705e-4589-b6d0-c944810a76bc\" (UID: \"4d44befd-705e-4589-b6d0-c944810a76bc\") " Oct 30 17:33:02.794812 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:02.794749 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d44befd-705e-4589-b6d0-c944810a76bc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d44befd-705e-4589-b6d0-c944810a76bc-istiod-ca-cert\") pod \"4d44befd-705e-4589-b6d0-c944810a76bc\" (UID: \"4d44befd-705e-4589-b6d0-c944810a76bc\") " Oct 30 17:33:02.794812 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:02.794788 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d44befd-705e-4589-b6d0-c944810a76bc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d44befd-705e-4589-b6d0-c944810a76bc-credential-socket\") pod \"4d44befd-705e-4589-b6d0-c944810a76bc\" (UID: \"4d44befd-705e-4589-b6d0-c944810a76bc\") " Oct 30 17:33:02.794812 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:02.794801 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d44befd-705e-4589-b6d0-c944810a76bc-istio-data" pod "4d44befd-705e-4589-b6d0-c944810a76bc" (UID: "4d44befd-705e-4589-b6d0-c944810a76bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:33:02.794987 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:02.794840 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d44befd-705e-4589-b6d0-c944810a76bc-kube-api-access-sskxp\" (UniqueName: \"kubernetes.io/projected/4d44befd-705e-4589-b6d0-c944810a76bc-kube-api-access-sskxp\") pod \"4d44befd-705e-4589-b6d0-c944810a76bc\" (UID: \"4d44befd-705e-4589-b6d0-c944810a76bc\") " Oct 30 17:33:02.795053 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:02.794993 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d44befd-705e-4589-b6d0-c944810a76bc-workload-certs" pod "4d44befd-705e-4589-b6d0-c944810a76bc" (UID: "4d44befd-705e-4589-b6d0-c944810a76bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:33:02.795442 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:02.795405 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d44befd-705e-4589-b6d0-c944810a76bc-credential-socket" pod "4d44befd-705e-4589-b6d0-c944810a76bc" (UID: "4d44befd-705e-4589-b6d0-c944810a76bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:33:02.795650 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:02.795625 4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d44befd-705e-4589-b6d0-c944810a76bc-istio-data\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:33:02.795741 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:02.795658 4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d44befd-705e-4589-b6d0-c944810a76bc-workload-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:33:02.795741 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:02.795672 4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d44befd-705e-4589-b6d0-c944810a76bc-credential-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:33:02.795741 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:02.795685 4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d44befd-705e-4589-b6d0-c944810a76bc-workload-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:33:02.796239 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:02.796216 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d44befd-705e-4589-b6d0-c944810a76bc-istiod-ca-cert" pod "4d44befd-705e-4589-b6d0-c944810a76bc" (UID: "4d44befd-705e-4589-b6d0-c944810a76bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:33:02.797200 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:02.797168 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d44befd-705e-4589-b6d0-c944810a76bc-istio-podinfo" pod "4d44befd-705e-4589-b6d0-c944810a76bc" (UID: "4d44befd-705e-4589-b6d0-c944810a76bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:33:02.797386 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:02.797357 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d44befd-705e-4589-b6d0-c944810a76bc-istio-token" pod "4d44befd-705e-4589-b6d0-c944810a76bc" (UID: "4d44befd-705e-4589-b6d0-c944810a76bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:33:02.797715 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:02.797690 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d44befd-705e-4589-b6d0-c944810a76bc-istio-envoy" pod "4d44befd-705e-4589-b6d0-c944810a76bc" (UID: "4d44befd-705e-4589-b6d0-c944810a76bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:33:02.798290 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:02.798269 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d44befd-705e-4589-b6d0-c944810a76bc-kube-api-access-sskxp" pod "4d44befd-705e-4589-b6d0-c944810a76bc" (UID: "4d44befd-705e-4589-b6d0-c944810a76bc"). InnerVolumeSpecName "kube-api-access-sskxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:33:02.897021 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:02.896984 4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d44befd-705e-4589-b6d0-c944810a76bc-istio-envoy\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:33:02.897021 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:02.897016 4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d44befd-705e-4589-b6d0-c944810a76bc-istiod-ca-cert\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:33:02.897021 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:02.897025 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sskxp\" (UniqueName: \"kubernetes.io/projected/4d44befd-705e-4589-b6d0-c944810a76bc-kube-api-access-sskxp\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:33:02.897021 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:02.897033 4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d44befd-705e-4589-b6d0-c944810a76bc-istio-token\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:33:02.897326 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:02.897042 4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d44befd-705e-4589-b6d0-c944810a76bc-istio-podinfo\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:33:03.420035 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:03.420005 4522 generic.go:356] "Generic (PLEG): container finished" podID="4d44befd-705e-4589-b6d0-c944810a76bc" containerID="8e2803f8b7c3d8f4026c1c9193383ed799fb717c5629d680431eb3a3e6190c06" exitCode=0 Oct 30 17:33:03.420484 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:03.420059 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc" Oct 30 17:33:03.420484 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:03.420093 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc" event={"ID":"4d44befd-705e-4589-b6d0-c944810a76bc","Type":"ContainerDied","Data":"8e2803f8b7c3d8f4026c1c9193383ed799fb717c5629d680431eb3a3e6190c06"} Oct 30 17:33:03.420484 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:03.420148 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc" event={"ID":"4d44befd-705e-4589-b6d0-c944810a76bc","Type":"ContainerDied","Data":"ce0c4e7ff0af068eacad45ef0e0802884de0a23fe10009a8ab0d9a2358fbcb3c"} Oct 30 17:33:03.420484 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:03.420173 4522 scope.go:122] "RemoveContainer" containerID="8e2803f8b7c3d8f4026c1c9193383ed799fb717c5629d680431eb3a3e6190c06" Oct 30 17:33:03.432876 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:03.432858 4522 scope.go:122] "RemoveContainer" containerID="8e2803f8b7c3d8f4026c1c9193383ed799fb717c5629d680431eb3a3e6190c06" Oct 30 17:33:03.433169 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:33:03.433147 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e2803f8b7c3d8f4026c1c9193383ed799fb717c5629d680431eb3a3e6190c06\": container with ID starting with 8e2803f8b7c3d8f4026c1c9193383ed799fb717c5629d680431eb3a3e6190c06 not found: ID does not exist" containerID="8e2803f8b7c3d8f4026c1c9193383ed799fb717c5629d680431eb3a3e6190c06" Oct 30 17:33:03.433210 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:03.433178 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e2803f8b7c3d8f4026c1c9193383ed799fb717c5629d680431eb3a3e6190c06"} err="failed to get container status \"8e2803f8b7c3d8f4026c1c9193383ed799fb717c5629d680431eb3a3e6190c06\": rpc error: code = NotFound desc = could not find container \"8e2803f8b7c3d8f4026c1c9193383ed799fb717c5629d680431eb3a3e6190c06\": container with ID starting with 8e2803f8b7c3d8f4026c1c9193383ed799fb717c5629d680431eb3a3e6190c06 not found: ID does not exist" Oct 30 17:33:03.493363 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:03.493316 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc"] Oct 30 17:33:03.506875 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:03.506844 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hllsc"] Oct 30 17:33:04.160726 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:04.160195 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d44befd-705e-4589-b6d0-c944810a76bc" path="/var/lib/kubelet/pods/4d44befd-705e-4589-b6d0-c944810a76bc/volumes" Oct 30 17:33:04.219676 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:04.219648 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-987dfc-85f675bb7f-phjx7" Oct 30 17:33:04.347398 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:04.347348 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ceac6a2a-ea4a-45ca-a6f0-2beb70ca52d2-kube-api-access-jclr2\" (UniqueName: \"kubernetes.io/projected/ceac6a2a-ea4a-45ca-a6f0-2beb70ca52d2-kube-api-access-jclr2\") pod \"ceac6a2a-ea4a-45ca-a6f0-2beb70ca52d2\" (UID: \"ceac6a2a-ea4a-45ca-a6f0-2beb70ca52d2\") " Oct 30 17:33:04.347598 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:04.347511 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ceac6a2a-ea4a-45ca-a6f0-2beb70ca52d2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceac6a2a-ea4a-45ca-a6f0-2beb70ca52d2-plugins-config-volume\") pod \"ceac6a2a-ea4a-45ca-a6f0-2beb70ca52d2\" (UID: \"ceac6a2a-ea4a-45ca-a6f0-2beb70ca52d2\") " Oct 30 17:33:04.347914 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:04.347887 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceac6a2a-ea4a-45ca-a6f0-2beb70ca52d2-plugins-config-volume" pod "ceac6a2a-ea4a-45ca-a6f0-2beb70ca52d2" (UID: "ceac6a2a-ea4a-45ca-a6f0-2beb70ca52d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:33:04.348275 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:04.348257 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceac6a2a-ea4a-45ca-a6f0-2beb70ca52d2-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:33:04.349676 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:04.349656 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceac6a2a-ea4a-45ca-a6f0-2beb70ca52d2-kube-api-access-jclr2" pod "ceac6a2a-ea4a-45ca-a6f0-2beb70ca52d2" (UID: "ceac6a2a-ea4a-45ca-a6f0-2beb70ca52d2"). InnerVolumeSpecName "kube-api-access-jclr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:33:04.508374 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:04.494468 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jclr2\" (UniqueName: \"kubernetes.io/projected/ceac6a2a-ea4a-45ca-a6f0-2beb70ca52d2-kube-api-access-jclr2\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:33:04.518144 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:04.517162 4522 generic.go:356] "Generic (PLEG): container finished" podID="ceac6a2a-ea4a-45ca-a6f0-2beb70ca52d2" containerID="917a434c5fca903d113330d152bee12c3180782fa101ba7603c82ba4e1a5e8d7" exitCode=0 Oct 30 17:33:04.518144 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:04.517465 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-987dfc-85f675bb7f-phjx7" event={"ID":"ceac6a2a-ea4a-45ca-a6f0-2beb70ca52d2","Type":"ContainerDied","Data":"917a434c5fca903d113330d152bee12c3180782fa101ba7603c82ba4e1a5e8d7"} Oct 30 17:33:04.518144 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:04.517519 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-987dfc-85f675bb7f-phjx7" event={"ID":"ceac6a2a-ea4a-45ca-a6f0-2beb70ca52d2","Type":"ContainerDied","Data":"a1134d581c3f833d33bff30d9ff0c085bf077c710c8b2b56fbbe53035726cf75"} Oct 30 17:33:04.518144 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:04.517551 4522 scope.go:122] "RemoveContainer" containerID="917a434c5fca903d113330d152bee12c3180782fa101ba7603c82ba4e1a5e8d7" Oct 30 17:33:04.518144 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:04.517667 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-987dfc-85f675bb7f-phjx7" Oct 30 17:33:04.537976 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:04.537949 4522 scope.go:122] "RemoveContainer" containerID="917a434c5fca903d113330d152bee12c3180782fa101ba7603c82ba4e1a5e8d7" Oct 30 17:33:04.538351 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:33:04.538291 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"917a434c5fca903d113330d152bee12c3180782fa101ba7603c82ba4e1a5e8d7\": container with ID starting with 917a434c5fca903d113330d152bee12c3180782fa101ba7603c82ba4e1a5e8d7 not found: ID does not exist" containerID="917a434c5fca903d113330d152bee12c3180782fa101ba7603c82ba4e1a5e8d7" Oct 30 17:33:04.538417 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:04.538362 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"917a434c5fca903d113330d152bee12c3180782fa101ba7603c82ba4e1a5e8d7"} err="failed to get container status \"917a434c5fca903d113330d152bee12c3180782fa101ba7603c82ba4e1a5e8d7\": rpc error: code = NotFound desc = could not find container \"917a434c5fca903d113330d152bee12c3180782fa101ba7603c82ba4e1a5e8d7\": container with ID starting with 917a434c5fca903d113330d152bee12c3180782fa101ba7603c82ba4e1a5e8d7 not found: ID does not exist" Oct 30 17:33:04.579246 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:04.579214 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-987dfc-85f675bb7f-phjx7"] Oct 30 17:33:04.583602 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:04.583573 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-987dfc-85f675bb7f-phjx7"] Oct 30 17:33:06.148003 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:06.147971 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ceac6a2a-ea4a-45ca-a6f0-2beb70ca52d2" path="/var/lib/kubelet/pods/ceac6a2a-ea4a-45ca-a6f0-2beb70ca52d2/volumes" Oct 30 17:33:10.367445 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:10.367413 4522 scope.go:122] "RemoveContainer" containerID="64223a986e42ea78327a08be763269195539b3f95570a50478fa0c6fc479d26b" Oct 30 17:33:10.378679 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:10.378656 4522 scope.go:122] "RemoveContainer" containerID="8d0cb6649488b13c2b1c766f7c9823719429746ad887aaebd0efdeeb4b9a8bd8" Oct 30 17:33:10.389674 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:10.389653 4522 scope.go:122] "RemoveContainer" containerID="fa6fd7a5e4882d23fe0a196263449de591104a39ce48281617bd78108fca7616" Oct 30 17:33:10.399674 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:10.399653 4522 scope.go:122] "RemoveContainer" containerID="f1701401ed6d9598cd6669dc3fa3369cebe01b01e9db40fc027dd8272841ba51" Oct 30 17:33:10.409650 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:10.409630 4522 scope.go:122] "RemoveContainer" containerID="f26db7a2a5a0c1026063ed919bedccbd2b5e524e47512553dfd4dd02d1e37a99" Oct 30 17:33:10.422513 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:10.421191 4522 scope.go:122] "RemoveContainer" containerID="9fb0e8c52b35a00a4ba1e4c5044bba645048229b0f5a156cee5f775f2a14ea81" Oct 30 17:33:10.433238 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:10.433216 4522 scope.go:122] "RemoveContainer" containerID="d2440bf73ef85a9d632c435af42acdaa93b2f53f9bd837325d0d496ec059acc7" Oct 30 17:33:12.979881 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:33:12.979841 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:33:14.923017 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:14.922973 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-001a96-67fb95b8d4-qkhqq"] Oct 30 17:33:14.959990 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:14.959956 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-001a96-54c64574d7-txw2m"] Oct 30 17:33:19.923632 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:19.923570 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-001a96-67fb95b8d4-qkhqq" podUID="e4c8fa39-e889-46e8-9afc-d1bbefbe57c3" containerName="payload-processing" containerID="cri-o://d99275c569a87658b0e39c88e0376e7de68f50f04068319f3f4d182d149d260d" gracePeriod=25 Oct 30 17:33:19.961082 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:19.961043 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-001a96-54c64574d7-txw2m" podUID="05959f4f-5f66-4a9e-8050-851b8fce2c5b" containerName="payload-pre-processing" containerID="cri-o://3f964de1e10177334e5dadcc6593cfd3f99f5a83e3e322871b2ffd413b993270" gracePeriod=25 Oct 30 17:33:26.991482 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:26.991440 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-001a96-54c64574d7-txw2m" podUID="05959f4f-5f66-4a9e-8050-851b8fce2c5b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:33:26.992775 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:26.992748 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-001a96-67fb95b8d4-qkhqq" podUID="e4c8fa39-e889-46e8-9afc-d1bbefbe57c3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:33:31.034626 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:31.034589 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw"] Oct 30 17:33:31.049201 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:31.049171 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5"] Oct 30 17:33:35.298048 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:35.298016 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4fb821-57799f5bb7-bvf8b"] Oct 30 17:33:36.992163 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:36.992130 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-001a96-67fb95b8d4-qkhqq" podUID="e4c8fa39-e889-46e8-9afc-d1bbefbe57c3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:33:36.992980 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:36.992130 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-001a96-54c64574d7-txw2m" podUID="05959f4f-5f66-4a9e-8050-851b8fce2c5b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:33:37.044003 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:37.043969 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c7b9df-56764597fb-q9kdh"] Oct 30 17:33:37.075208 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:37.075170 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c7b9df-7498894d5f-vk5db"] Oct 30 17:33:38.703925 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:38.703892 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-41cb25-54df857bfd-mljxr"] Oct 30 17:33:38.704463 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:38.704102 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41cb25-54df857bfd-mljxr" podUID="02a70782-464d-476c-916d-7f539cc59183" containerName="maas-api" containerID="cri-o://b352e0bfa19421d1536baa494d03b6d7abd33f1ca2480e13d9bcd0c9c8312d09" gracePeriod=30 Oct 30 17:33:38.870170 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:38.870136 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-41cb25-6c785598f6-vcrjm"] Oct 30 17:33:38.870378 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:38.870356 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-41cb25-6c785598f6-vcrjm" podUID="cf81982b-e4fd-456c-99fe-52b89db4cced" containerName="payload-processing" containerID="cri-o://a70da96341342c94ddb60330fc268c520019d2244772f71bc6ef6831d2ea6650" gracePeriod=30 Oct 30 17:33:38.873757 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:38.873733 4522 generic.go:356] "Generic (PLEG): container finished" podID="02a70782-464d-476c-916d-7f539cc59183" containerID="b352e0bfa19421d1536baa494d03b6d7abd33f1ca2480e13d9bcd0c9c8312d09" exitCode=0 Oct 30 17:33:38.873891 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:38.873864 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41cb25-54df857bfd-mljxr" event={"ID":"02a70782-464d-476c-916d-7f539cc59183","Type":"ContainerDied","Data":"b352e0bfa19421d1536baa494d03b6d7abd33f1ca2480e13d9bcd0c9c8312d09"} Oct 30 17:33:38.901242 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:38.901207 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-41cb25-755479dd79-dj7jc"] Oct 30 17:33:38.901518 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:38.901459 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41cb25-755479dd79-dj7jc" podUID="3913f59f-4c2f-4163-9080-b9557dbd3987" containerName="payload-pre-processing" containerID="cri-o://fd99fd82304258eac2fd4074b58787637aff9988c0d362aba89a79c107e83682" gracePeriod=30 Oct 30 17:33:39.063891 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:39.063859 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41cb25-54df857bfd-mljxr" Oct 30 17:33:39.139638 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:39.138785 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02a70782-464d-476c-916d-7f539cc59183-metrics-certs\" (UniqueName: \"kubernetes.io/secret/02a70782-464d-476c-916d-7f539cc59183-metrics-certs\") pod \"02a70782-464d-476c-916d-7f539cc59183\" (UID: \"02a70782-464d-476c-916d-7f539cc59183\") " Oct 30 17:33:39.139638 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:39.138848 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02a70782-464d-476c-916d-7f539cc59183-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02a70782-464d-476c-916d-7f539cc59183-openshift-service-ca\") pod \"02a70782-464d-476c-916d-7f539cc59183\" (UID: \"02a70782-464d-476c-916d-7f539cc59183\") " Oct 30 17:33:39.139638 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:39.138941 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02a70782-464d-476c-916d-7f539cc59183-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02a70782-464d-476c-916d-7f539cc59183-maas-api-tls\") pod \"02a70782-464d-476c-916d-7f539cc59183\" (UID: \"02a70782-464d-476c-916d-7f539cc59183\") " Oct 30 17:33:39.139638 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:39.138990 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02a70782-464d-476c-916d-7f539cc59183-kube-api-access-z9f52\" (UniqueName: \"kubernetes.io/projected/02a70782-464d-476c-916d-7f539cc59183-kube-api-access-z9f52\") pod \"02a70782-464d-476c-916d-7f539cc59183\" (UID: \"02a70782-464d-476c-916d-7f539cc59183\") " Oct 30 17:33:39.140029 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:39.139720 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02a70782-464d-476c-916d-7f539cc59183-openshift-service-ca" pod "02a70782-464d-476c-916d-7f539cc59183" (UID: "02a70782-464d-476c-916d-7f539cc59183"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:33:39.148279 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:39.148225 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02a70782-464d-476c-916d-7f539cc59183-maas-api-tls" pod "02a70782-464d-476c-916d-7f539cc59183" (UID: "02a70782-464d-476c-916d-7f539cc59183"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:33:39.148493 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:39.148438 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02a70782-464d-476c-916d-7f539cc59183-kube-api-access-z9f52" pod "02a70782-464d-476c-916d-7f539cc59183" (UID: "02a70782-464d-476c-916d-7f539cc59183"). InnerVolumeSpecName "kube-api-access-z9f52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:33:39.149790 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:39.149750 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02a70782-464d-476c-916d-7f539cc59183-metrics-certs" pod "02a70782-464d-476c-916d-7f539cc59183" (UID: "02a70782-464d-476c-916d-7f539cc59183"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:33:39.203061 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:39.203032 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41cb25-755479dd79-dj7jc" Oct 30 17:33:39.237368 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:39.237258 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-41cb25-6c785598f6-vcrjm" Oct 30 17:33:39.241953 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:39.241926 4522 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02a70782-464d-476c-916d-7f539cc59183-metrics-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:33:39.241953 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:39.241954 4522 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02a70782-464d-476c-916d-7f539cc59183-openshift-service-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:33:39.242141 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:39.241968 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02a70782-464d-476c-916d-7f539cc59183-maas-api-tls\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:33:39.242141 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:39.241982 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9f52\" (UniqueName: \"kubernetes.io/projected/02a70782-464d-476c-916d-7f539cc59183-kube-api-access-z9f52\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:33:39.366326 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:39.363232 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf81982b-e4fd-456c-99fe-52b89db4cced-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf81982b-e4fd-456c-99fe-52b89db4cced-plugins-config-volume\") pod \"cf81982b-e4fd-456c-99fe-52b89db4cced\" (UID: \"cf81982b-e4fd-456c-99fe-52b89db4cced\") " Oct 30 17:33:39.366326 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:39.363418 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3913f59f-4c2f-4163-9080-b9557dbd3987-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3913f59f-4c2f-4163-9080-b9557dbd3987-plugins-config-volume\") pod \"3913f59f-4c2f-4163-9080-b9557dbd3987\" (UID: \"3913f59f-4c2f-4163-9080-b9557dbd3987\") " Oct 30 17:33:39.369607 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:39.367251 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3913f59f-4c2f-4163-9080-b9557dbd3987-kube-api-access-b7ftf\" (UniqueName: \"kubernetes.io/projected/3913f59f-4c2f-4163-9080-b9557dbd3987-kube-api-access-b7ftf\") pod \"3913f59f-4c2f-4163-9080-b9557dbd3987\" (UID: \"3913f59f-4c2f-4163-9080-b9557dbd3987\") " Oct 30 17:33:39.369607 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:39.367494 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf81982b-e4fd-456c-99fe-52b89db4cced-kube-api-access-8mcj7\" (UniqueName: \"kubernetes.io/projected/cf81982b-e4fd-456c-99fe-52b89db4cced-kube-api-access-8mcj7\") pod \"cf81982b-e4fd-456c-99fe-52b89db4cced\" (UID: \"cf81982b-e4fd-456c-99fe-52b89db4cced\") " Oct 30 17:33:39.373206 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:39.371635 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf81982b-e4fd-456c-99fe-52b89db4cced-kube-api-access-8mcj7" pod "cf81982b-e4fd-456c-99fe-52b89db4cced" (UID: "cf81982b-e4fd-456c-99fe-52b89db4cced"). InnerVolumeSpecName "kube-api-access-8mcj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:33:39.373206 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:39.372010 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf81982b-e4fd-456c-99fe-52b89db4cced-plugins-config-volume" pod "cf81982b-e4fd-456c-99fe-52b89db4cced" (UID: "cf81982b-e4fd-456c-99fe-52b89db4cced"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:33:39.373206 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:39.372590 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3913f59f-4c2f-4163-9080-b9557dbd3987-plugins-config-volume" pod "3913f59f-4c2f-4163-9080-b9557dbd3987" (UID: "3913f59f-4c2f-4163-9080-b9557dbd3987"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:33:39.377673 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:39.377647 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3913f59f-4c2f-4163-9080-b9557dbd3987-kube-api-access-b7ftf" pod "3913f59f-4c2f-4163-9080-b9557dbd3987" (UID: "3913f59f-4c2f-4163-9080-b9557dbd3987"). InnerVolumeSpecName "kube-api-access-b7ftf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:33:39.470240 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:39.470210 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mcj7\" (UniqueName: \"kubernetes.io/projected/cf81982b-e4fd-456c-99fe-52b89db4cced-kube-api-access-8mcj7\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:33:39.470240 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:39.470237 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf81982b-e4fd-456c-99fe-52b89db4cced-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:33:39.470478 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:39.470248 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3913f59f-4c2f-4163-9080-b9557dbd3987-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:33:39.470478 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:39.470267 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7ftf\" (UniqueName: \"kubernetes.io/projected/3913f59f-4c2f-4163-9080-b9557dbd3987-kube-api-access-b7ftf\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:33:39.884399 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:39.884366 4522 generic.go:356] "Generic (PLEG): container finished" podID="3913f59f-4c2f-4163-9080-b9557dbd3987" containerID="fd99fd82304258eac2fd4074b58787637aff9988c0d362aba89a79c107e83682" exitCode=0 Oct 30 17:33:39.884858 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:39.884469 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41cb25-755479dd79-dj7jc" Oct 30 17:33:39.884858 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:39.884507 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41cb25-755479dd79-dj7jc" event={"ID":"3913f59f-4c2f-4163-9080-b9557dbd3987","Type":"ContainerDied","Data":"fd99fd82304258eac2fd4074b58787637aff9988c0d362aba89a79c107e83682"} Oct 30 17:33:39.884858 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:39.884566 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41cb25-755479dd79-dj7jc" event={"ID":"3913f59f-4c2f-4163-9080-b9557dbd3987","Type":"ContainerDied","Data":"5624cdc848d0322874c16385492009426e53fd3718c639a2831820db9d02e874"} Oct 30 17:33:39.884858 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:39.884593 4522 scope.go:122] "RemoveContainer" containerID="fd99fd82304258eac2fd4074b58787637aff9988c0d362aba89a79c107e83682" Oct 30 17:33:39.886204 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:39.886181 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41cb25-54df857bfd-mljxr" Oct 30 17:33:39.886348 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:39.886213 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41cb25-54df857bfd-mljxr" event={"ID":"02a70782-464d-476c-916d-7f539cc59183","Type":"ContainerDied","Data":"83a94b901aaefb699af1b5e59e4c73393f99287fc3bb7cefb7774160619e2da9"} Oct 30 17:33:39.887869 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:39.887849 4522 generic.go:356] "Generic (PLEG): container finished" podID="cf81982b-e4fd-456c-99fe-52b89db4cced" containerID="a70da96341342c94ddb60330fc268c520019d2244772f71bc6ef6831d2ea6650" exitCode=0 Oct 30 17:33:39.887997 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:39.887893 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-41cb25-6c785598f6-vcrjm" event={"ID":"cf81982b-e4fd-456c-99fe-52b89db4cced","Type":"ContainerDied","Data":"a70da96341342c94ddb60330fc268c520019d2244772f71bc6ef6831d2ea6650"} Oct 30 17:33:39.887997 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:39.887902 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-41cb25-6c785598f6-vcrjm" Oct 30 17:33:39.887997 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:39.887924 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-41cb25-6c785598f6-vcrjm" event={"ID":"cf81982b-e4fd-456c-99fe-52b89db4cced","Type":"ContainerDied","Data":"fced5024e53ca9977989381216da14edcf7f945f7c2cd9285fba18a40d02e645"} Oct 30 17:33:39.899675 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:39.899650 4522 scope.go:122] "RemoveContainer" containerID="fd99fd82304258eac2fd4074b58787637aff9988c0d362aba89a79c107e83682" Oct 30 17:33:39.900163 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:33:39.900065 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd99fd82304258eac2fd4074b58787637aff9988c0d362aba89a79c107e83682\": container with ID starting with fd99fd82304258eac2fd4074b58787637aff9988c0d362aba89a79c107e83682 not found: ID does not exist" containerID="fd99fd82304258eac2fd4074b58787637aff9988c0d362aba89a79c107e83682" Oct 30 17:33:39.900163 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:39.900105 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd99fd82304258eac2fd4074b58787637aff9988c0d362aba89a79c107e83682"} err="failed to get container status \"fd99fd82304258eac2fd4074b58787637aff9988c0d362aba89a79c107e83682\": rpc error: code = NotFound desc = could not find container \"fd99fd82304258eac2fd4074b58787637aff9988c0d362aba89a79c107e83682\": container with ID starting with fd99fd82304258eac2fd4074b58787637aff9988c0d362aba89a79c107e83682 not found: ID does not exist" Oct 30 17:33:39.900163 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:39.900135 4522 scope.go:122] "RemoveContainer" containerID="b352e0bfa19421d1536baa494d03b6d7abd33f1ca2480e13d9bcd0c9c8312d09" Oct 30 17:33:39.912157 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:39.912139 4522 scope.go:122] "RemoveContainer" containerID="a70da96341342c94ddb60330fc268c520019d2244772f71bc6ef6831d2ea6650" Oct 30 17:33:39.923193 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:39.923171 4522 scope.go:122] "RemoveContainer" containerID="a70da96341342c94ddb60330fc268c520019d2244772f71bc6ef6831d2ea6650" Oct 30 17:33:39.923475 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:33:39.923453 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a70da96341342c94ddb60330fc268c520019d2244772f71bc6ef6831d2ea6650\": container with ID starting with a70da96341342c94ddb60330fc268c520019d2244772f71bc6ef6831d2ea6650 not found: ID does not exist" containerID="a70da96341342c94ddb60330fc268c520019d2244772f71bc6ef6831d2ea6650" Oct 30 17:33:39.923555 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:39.923482 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a70da96341342c94ddb60330fc268c520019d2244772f71bc6ef6831d2ea6650"} err="failed to get container status \"a70da96341342c94ddb60330fc268c520019d2244772f71bc6ef6831d2ea6650\": rpc error: code = NotFound desc = could not find container \"a70da96341342c94ddb60330fc268c520019d2244772f71bc6ef6831d2ea6650\": container with ID starting with a70da96341342c94ddb60330fc268c520019d2244772f71bc6ef6831d2ea6650 not found: ID does not exist" Oct 30 17:33:39.983567 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:39.983492 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-41cb25-6c785598f6-vcrjm"] Oct 30 17:33:39.988034 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:39.988004 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-41cb25-6c785598f6-vcrjm"] Oct 30 17:33:40.028481 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:40.028432 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-41cb25-755479dd79-dj7jc"] Oct 30 17:33:40.034665 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:40.034640 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-41cb25-755479dd79-dj7jc"] Oct 30 17:33:40.079736 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:40.079654 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-41cb25-54df857bfd-mljxr"] Oct 30 17:33:40.085357 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:40.085330 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-41cb25-54df857bfd-mljxr"] Oct 30 17:33:40.188491 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:40.186561 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02a70782-464d-476c-916d-7f539cc59183" path="/var/lib/kubelet/pods/02a70782-464d-476c-916d-7f539cc59183/volumes" Oct 30 17:33:40.188491 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:40.187026 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3913f59f-4c2f-4163-9080-b9557dbd3987" path="/var/lib/kubelet/pods/3913f59f-4c2f-4163-9080-b9557dbd3987/volumes" Oct 30 17:33:40.188491 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:40.187445 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf81982b-e4fd-456c-99fe-52b89db4cced" path="/var/lib/kubelet/pods/cf81982b-e4fd-456c-99fe-52b89db4cced/volumes" Oct 30 17:33:40.295875 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:40.295845 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-001a96-67fb95b8d4-qkhqq" Oct 30 17:33:40.299999 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:40.299938 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4fb821-57799f5bb7-bvf8b" podUID="33b0923a-026d-4946-9e27-a6cb6fc5eb94" containerName="payload-pre-processing" containerID="cri-o://f35e2212fb29f19a43959c4b856786650eb23a9fcb27875c823a8a8af397adf7" gracePeriod=25 Oct 30 17:33:40.413083 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:40.413055 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-001a96-54c64574d7-txw2m" Oct 30 17:33:40.489588 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:40.489560 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4c8fa39-e889-46e8-9afc-d1bbefbe57c3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4c8fa39-e889-46e8-9afc-d1bbefbe57c3-plugins-config-volume\") pod \"e4c8fa39-e889-46e8-9afc-d1bbefbe57c3\" (UID: \"e4c8fa39-e889-46e8-9afc-d1bbefbe57c3\") " Oct 30 17:33:40.489965 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:40.489931 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4c8fa39-e889-46e8-9afc-d1bbefbe57c3-plugins-config-volume" pod "e4c8fa39-e889-46e8-9afc-d1bbefbe57c3" (UID: "e4c8fa39-e889-46e8-9afc-d1bbefbe57c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:33:40.490176 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:40.490158 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4c8fa39-e889-46e8-9afc-d1bbefbe57c3-kube-api-access-zj6g9\" (UniqueName: \"kubernetes.io/projected/e4c8fa39-e889-46e8-9afc-d1bbefbe57c3-kube-api-access-zj6g9\") pod \"e4c8fa39-e889-46e8-9afc-d1bbefbe57c3\" (UID: \"e4c8fa39-e889-46e8-9afc-d1bbefbe57c3\") " Oct 30 17:33:40.491153 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:40.491127 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4c8fa39-e889-46e8-9afc-d1bbefbe57c3-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:33:40.494341 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:40.493683 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4c8fa39-e889-46e8-9afc-d1bbefbe57c3-kube-api-access-zj6g9" pod "e4c8fa39-e889-46e8-9afc-d1bbefbe57c3" (UID: "e4c8fa39-e889-46e8-9afc-d1bbefbe57c3"). InnerVolumeSpecName "kube-api-access-zj6g9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:33:40.591481 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:40.591452 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4fb821-57799f5bb7-bvf8b" Oct 30 17:33:40.591968 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:40.591945 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05959f4f-5f66-4a9e-8050-851b8fce2c5b-kube-api-access-njjjw\" (UniqueName: \"kubernetes.io/projected/05959f4f-5f66-4a9e-8050-851b8fce2c5b-kube-api-access-njjjw\") pod \"05959f4f-5f66-4a9e-8050-851b8fce2c5b\" (UID: \"05959f4f-5f66-4a9e-8050-851b8fce2c5b\") " Oct 30 17:33:40.592148 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:40.592135 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05959f4f-5f66-4a9e-8050-851b8fce2c5b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05959f4f-5f66-4a9e-8050-851b8fce2c5b-plugins-config-volume\") pod \"05959f4f-5f66-4a9e-8050-851b8fce2c5b\" (UID: \"05959f4f-5f66-4a9e-8050-851b8fce2c5b\") " Oct 30 17:33:40.592526 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:40.592502 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05959f4f-5f66-4a9e-8050-851b8fce2c5b-plugins-config-volume" pod "05959f4f-5f66-4a9e-8050-851b8fce2c5b" (UID: "05959f4f-5f66-4a9e-8050-851b8fce2c5b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:33:40.592756 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:40.592741 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj6g9\" (UniqueName: \"kubernetes.io/projected/e4c8fa39-e889-46e8-9afc-d1bbefbe57c3-kube-api-access-zj6g9\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:33:40.592795 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:40.592759 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05959f4f-5f66-4a9e-8050-851b8fce2c5b-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:33:40.594147 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:40.594123 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05959f4f-5f66-4a9e-8050-851b8fce2c5b-kube-api-access-njjjw" pod "05959f4f-5f66-4a9e-8050-851b8fce2c5b" (UID: "05959f4f-5f66-4a9e-8050-851b8fce2c5b"). InnerVolumeSpecName "kube-api-access-njjjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:33:40.694344 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:40.694296 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33b0923a-026d-4946-9e27-a6cb6fc5eb94-kube-api-access-t8r59\" (UniqueName: \"kubernetes.io/projected/33b0923a-026d-4946-9e27-a6cb6fc5eb94-kube-api-access-t8r59\") pod \"33b0923a-026d-4946-9e27-a6cb6fc5eb94\" (UID: \"33b0923a-026d-4946-9e27-a6cb6fc5eb94\") " Oct 30 17:33:40.694532 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:40.694406 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33b0923a-026d-4946-9e27-a6cb6fc5eb94-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33b0923a-026d-4946-9e27-a6cb6fc5eb94-plugins-config-volume\") pod \"33b0923a-026d-4946-9e27-a6cb6fc5eb94\" (UID: \"33b0923a-026d-4946-9e27-a6cb6fc5eb94\") " Oct 30 17:33:40.694791 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:40.694762 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33b0923a-026d-4946-9e27-a6cb6fc5eb94-plugins-config-volume" pod "33b0923a-026d-4946-9e27-a6cb6fc5eb94" (UID: "33b0923a-026d-4946-9e27-a6cb6fc5eb94"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:33:40.695190 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:40.695171 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33b0923a-026d-4946-9e27-a6cb6fc5eb94-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:33:40.695239 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:40.695193 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njjjw\" (UniqueName: \"kubernetes.io/projected/05959f4f-5f66-4a9e-8050-851b8fce2c5b-kube-api-access-njjjw\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:33:40.696584 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:40.696567 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33b0923a-026d-4946-9e27-a6cb6fc5eb94-kube-api-access-t8r59" pod "33b0923a-026d-4946-9e27-a6cb6fc5eb94" (UID: "33b0923a-026d-4946-9e27-a6cb6fc5eb94"). InnerVolumeSpecName "kube-api-access-t8r59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:33:40.796240 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:40.796193 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8r59\" (UniqueName: \"kubernetes.io/projected/33b0923a-026d-4946-9e27-a6cb6fc5eb94-kube-api-access-t8r59\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:33:40.899088 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:40.898224 4522 generic.go:356] "Generic (PLEG): container finished" podID="e4c8fa39-e889-46e8-9afc-d1bbefbe57c3" containerID="d99275c569a87658b0e39c88e0376e7de68f50f04068319f3f4d182d149d260d" exitCode=0 Oct 30 17:33:40.899088 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:40.898345 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-001a96-67fb95b8d4-qkhqq" event={"ID":"e4c8fa39-e889-46e8-9afc-d1bbefbe57c3","Type":"ContainerDied","Data":"d99275c569a87658b0e39c88e0376e7de68f50f04068319f3f4d182d149d260d"} Oct 30 17:33:40.899088 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:40.898397 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-001a96-67fb95b8d4-qkhqq" event={"ID":"e4c8fa39-e889-46e8-9afc-d1bbefbe57c3","Type":"ContainerDied","Data":"782b0a995930b18b98715326ce6e36344eede328c19087fec26ead7542d8b25f"} Oct 30 17:33:40.899088 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:40.898427 4522 scope.go:122] "RemoveContainer" containerID="d99275c569a87658b0e39c88e0376e7de68f50f04068319f3f4d182d149d260d" Oct 30 17:33:40.899088 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:40.898567 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-001a96-67fb95b8d4-qkhqq" Oct 30 17:33:40.903049 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:40.903025 4522 generic.go:356] "Generic (PLEG): container finished" podID="05959f4f-5f66-4a9e-8050-851b8fce2c5b" containerID="3f964de1e10177334e5dadcc6593cfd3f99f5a83e3e322871b2ffd413b993270" exitCode=0 Oct 30 17:33:40.903204 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:40.903074 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-001a96-54c64574d7-txw2m" Oct 30 17:33:40.903204 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:40.903122 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-001a96-54c64574d7-txw2m" event={"ID":"05959f4f-5f66-4a9e-8050-851b8fce2c5b","Type":"ContainerDied","Data":"3f964de1e10177334e5dadcc6593cfd3f99f5a83e3e322871b2ffd413b993270"} Oct 30 17:33:40.903204 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:40.903182 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-001a96-54c64574d7-txw2m" event={"ID":"05959f4f-5f66-4a9e-8050-851b8fce2c5b","Type":"ContainerDied","Data":"0ef8c9a18ca8eccaa0c39042c8526fce9b17b1e47d7eca9f04c904ad367e2932"} Oct 30 17:33:40.909009 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:40.908981 4522 generic.go:356] "Generic (PLEG): container finished" podID="33b0923a-026d-4946-9e27-a6cb6fc5eb94" containerID="f35e2212fb29f19a43959c4b856786650eb23a9fcb27875c823a8a8af397adf7" exitCode=0 Oct 30 17:33:40.909152 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:40.909025 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4fb821-57799f5bb7-bvf8b" Oct 30 17:33:40.909152 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:40.909022 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4fb821-57799f5bb7-bvf8b" event={"ID":"33b0923a-026d-4946-9e27-a6cb6fc5eb94","Type":"ContainerDied","Data":"f35e2212fb29f19a43959c4b856786650eb23a9fcb27875c823a8a8af397adf7"} Oct 30 17:33:40.909152 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:40.909090 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4fb821-57799f5bb7-bvf8b" event={"ID":"33b0923a-026d-4946-9e27-a6cb6fc5eb94","Type":"ContainerDied","Data":"d50d71a47079106e25406d256a4fd2f50ad18361073c3ed71563274dd3392e8b"} Oct 30 17:33:40.915540 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:40.915519 4522 scope.go:122] "RemoveContainer" containerID="d99275c569a87658b0e39c88e0376e7de68f50f04068319f3f4d182d149d260d" Oct 30 17:33:40.915803 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:33:40.915784 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d99275c569a87658b0e39c88e0376e7de68f50f04068319f3f4d182d149d260d\": container with ID starting with d99275c569a87658b0e39c88e0376e7de68f50f04068319f3f4d182d149d260d not found: ID does not exist" containerID="d99275c569a87658b0e39c88e0376e7de68f50f04068319f3f4d182d149d260d" Oct 30 17:33:40.915856 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:40.915812 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d99275c569a87658b0e39c88e0376e7de68f50f04068319f3f4d182d149d260d"} err="failed to get container status \"d99275c569a87658b0e39c88e0376e7de68f50f04068319f3f4d182d149d260d\": rpc error: code = NotFound desc = could not find container \"d99275c569a87658b0e39c88e0376e7de68f50f04068319f3f4d182d149d260d\": container with ID starting with d99275c569a87658b0e39c88e0376e7de68f50f04068319f3f4d182d149d260d not found: ID does not exist" Oct 30 17:33:40.915856 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:40.915830 4522 scope.go:122] "RemoveContainer" containerID="3f964de1e10177334e5dadcc6593cfd3f99f5a83e3e322871b2ffd413b993270" Oct 30 17:33:40.927562 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:40.927544 4522 scope.go:122] "RemoveContainer" containerID="3f964de1e10177334e5dadcc6593cfd3f99f5a83e3e322871b2ffd413b993270" Oct 30 17:33:40.927839 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:33:40.927819 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f964de1e10177334e5dadcc6593cfd3f99f5a83e3e322871b2ffd413b993270\": container with ID starting with 3f964de1e10177334e5dadcc6593cfd3f99f5a83e3e322871b2ffd413b993270 not found: ID does not exist" containerID="3f964de1e10177334e5dadcc6593cfd3f99f5a83e3e322871b2ffd413b993270" Oct 30 17:33:40.927896 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:40.927848 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f964de1e10177334e5dadcc6593cfd3f99f5a83e3e322871b2ffd413b993270"} err="failed to get container status \"3f964de1e10177334e5dadcc6593cfd3f99f5a83e3e322871b2ffd413b993270\": rpc error: code = NotFound desc = could not find container \"3f964de1e10177334e5dadcc6593cfd3f99f5a83e3e322871b2ffd413b993270\": container with ID starting with 3f964de1e10177334e5dadcc6593cfd3f99f5a83e3e322871b2ffd413b993270 not found: ID does not exist" Oct 30 17:33:40.927896 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:40.927865 4522 scope.go:122] "RemoveContainer" containerID="f35e2212fb29f19a43959c4b856786650eb23a9fcb27875c823a8a8af397adf7" Oct 30 17:33:40.939474 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:40.939452 4522 scope.go:122] "RemoveContainer" containerID="f35e2212fb29f19a43959c4b856786650eb23a9fcb27875c823a8a8af397adf7" Oct 30 17:33:40.939771 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:33:40.939751 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f35e2212fb29f19a43959c4b856786650eb23a9fcb27875c823a8a8af397adf7\": container with ID starting with f35e2212fb29f19a43959c4b856786650eb23a9fcb27875c823a8a8af397adf7 not found: ID does not exist" containerID="f35e2212fb29f19a43959c4b856786650eb23a9fcb27875c823a8a8af397adf7" Oct 30 17:33:40.939823 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:40.939781 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f35e2212fb29f19a43959c4b856786650eb23a9fcb27875c823a8a8af397adf7"} err="failed to get container status \"f35e2212fb29f19a43959c4b856786650eb23a9fcb27875c823a8a8af397adf7\": rpc error: code = NotFound desc = could not find container \"f35e2212fb29f19a43959c4b856786650eb23a9fcb27875c823a8a8af397adf7\": container with ID starting with f35e2212fb29f19a43959c4b856786650eb23a9fcb27875c823a8a8af397adf7 not found: ID does not exist" Oct 30 17:33:40.963069 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:40.963037 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-001a96-54c64574d7-txw2m"] Oct 30 17:33:40.977821 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:40.977787 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-001a96-54c64574d7-txw2m"] Oct 30 17:33:41.029466 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:41.029433 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-001a96-67fb95b8d4-qkhqq"] Oct 30 17:33:41.033594 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:41.033573 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-001a96-67fb95b8d4-qkhqq"] Oct 30 17:33:41.074945 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:41.074917 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4fb821-57799f5bb7-bvf8b"] Oct 30 17:33:41.078911 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:41.078890 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4fb821-57799f5bb7-bvf8b"] Oct 30 17:33:41.145718 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:41.145687 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d2d985-bdd4b595-zmgvf"] Oct 30 17:33:41.145935 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:41.145885 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d2d985-bdd4b595-zmgvf" podUID="6197c2d3-3465-4091-857f-9d8ccdcc2eb3" containerName="maas-api" containerID="cri-o://7271aee885831857cb9bb7b58ce053d46f2dfc5df5723400b9a634ffeff702e4" gracePeriod=30 Oct 30 17:33:41.315687 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:41.315644 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d2d985-698fbdbbdb-z4wjk"] Oct 30 17:33:41.315874 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:41.315836 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-d2d985-698fbdbbdb-z4wjk" podUID="9db299c4-22b3-4237-8d6c-b8dc9c20b117" containerName="payload-processing" containerID="cri-o://e0ba3abdae1db74c4f59777844c113213ae0618eb8c35130c23f35e4c726ca65" gracePeriod=30 Oct 30 17:33:41.335790 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:41.335756 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d2d985-688fcd588c-45k5x"] Oct 30 17:33:41.336004 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:41.335980 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d2d985-688fcd588c-45k5x" podUID="93279e37-0139-4f89-a5a0-55372047919c" containerName="payload-pre-processing" containerID="cri-o://3ac0a2388198a40a87a2708d8ef6a474f824c69ddcc2122af06b51ed7ec4d12b" gracePeriod=30 Oct 30 17:33:41.522541 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:41.522507 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d2d985-bdd4b595-zmgvf" Oct 30 17:33:41.607817 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:41.607784 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6197c2d3-3465-4091-857f-9d8ccdcc2eb3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6197c2d3-3465-4091-857f-9d8ccdcc2eb3-maas-api-tls\") pod \"6197c2d3-3465-4091-857f-9d8ccdcc2eb3\" (UID: \"6197c2d3-3465-4091-857f-9d8ccdcc2eb3\") " Oct 30 17:33:41.608026 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:41.607942 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6197c2d3-3465-4091-857f-9d8ccdcc2eb3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6197c2d3-3465-4091-857f-9d8ccdcc2eb3-metrics-certs\") pod \"6197c2d3-3465-4091-857f-9d8ccdcc2eb3\" (UID: \"6197c2d3-3465-4091-857f-9d8ccdcc2eb3\") " Oct 30 17:33:41.608185 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:41.608164 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6197c2d3-3465-4091-857f-9d8ccdcc2eb3-kube-api-access-755cj\" (UniqueName: \"kubernetes.io/projected/6197c2d3-3465-4091-857f-9d8ccdcc2eb3-kube-api-access-755cj\") pod \"6197c2d3-3465-4091-857f-9d8ccdcc2eb3\" (UID: \"6197c2d3-3465-4091-857f-9d8ccdcc2eb3\") " Oct 30 17:33:41.608276 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:41.608207 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6197c2d3-3465-4091-857f-9d8ccdcc2eb3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6197c2d3-3465-4091-857f-9d8ccdcc2eb3-openshift-service-ca\") pod \"6197c2d3-3465-4091-857f-9d8ccdcc2eb3\" (UID: \"6197c2d3-3465-4091-857f-9d8ccdcc2eb3\") " Oct 30 17:33:41.608912 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:41.608863 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6197c2d3-3465-4091-857f-9d8ccdcc2eb3-openshift-service-ca" pod "6197c2d3-3465-4091-857f-9d8ccdcc2eb3" (UID: "6197c2d3-3465-4091-857f-9d8ccdcc2eb3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:33:41.612788 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:41.612756 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6197c2d3-3465-4091-857f-9d8ccdcc2eb3-metrics-certs" pod "6197c2d3-3465-4091-857f-9d8ccdcc2eb3" (UID: "6197c2d3-3465-4091-857f-9d8ccdcc2eb3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:33:41.614864 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:41.614763 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6197c2d3-3465-4091-857f-9d8ccdcc2eb3-kube-api-access-755cj" pod "6197c2d3-3465-4091-857f-9d8ccdcc2eb3" (UID: "6197c2d3-3465-4091-857f-9d8ccdcc2eb3"). InnerVolumeSpecName "kube-api-access-755cj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:33:41.635925 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:41.622538 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6197c2d3-3465-4091-857f-9d8ccdcc2eb3-maas-api-tls" pod "6197c2d3-3465-4091-857f-9d8ccdcc2eb3" (UID: "6197c2d3-3465-4091-857f-9d8ccdcc2eb3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:33:41.724291 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:41.722470 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d2d985-698fbdbbdb-z4wjk" Oct 30 17:33:41.728244 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:41.728132 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-755cj\" (UniqueName: \"kubernetes.io/projected/6197c2d3-3465-4091-857f-9d8ccdcc2eb3-kube-api-access-755cj\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:33:41.728244 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:41.728168 4522 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6197c2d3-3465-4091-857f-9d8ccdcc2eb3-openshift-service-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:33:41.728244 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:41.728182 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6197c2d3-3465-4091-857f-9d8ccdcc2eb3-maas-api-tls\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:33:41.728244 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:41.728197 4522 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6197c2d3-3465-4091-857f-9d8ccdcc2eb3-metrics-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:33:41.830756 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:41.830722 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9db299c4-22b3-4237-8d6c-b8dc9c20b117-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9db299c4-22b3-4237-8d6c-b8dc9c20b117-plugins-config-volume\") pod \"9db299c4-22b3-4237-8d6c-b8dc9c20b117\" (UID: \"9db299c4-22b3-4237-8d6c-b8dc9c20b117\") " Oct 30 17:33:41.830938 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:41.830896 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9db299c4-22b3-4237-8d6c-b8dc9c20b117-kube-api-access-rf5tw\" (UniqueName: \"kubernetes.io/projected/9db299c4-22b3-4237-8d6c-b8dc9c20b117-kube-api-access-rf5tw\") pod \"9db299c4-22b3-4237-8d6c-b8dc9c20b117\" (UID: \"9db299c4-22b3-4237-8d6c-b8dc9c20b117\") " Oct 30 17:33:41.832129 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:41.832099 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9db299c4-22b3-4237-8d6c-b8dc9c20b117-plugins-config-volume" pod "9db299c4-22b3-4237-8d6c-b8dc9c20b117" (UID: "9db299c4-22b3-4237-8d6c-b8dc9c20b117"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:33:41.833533 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:41.833513 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9db299c4-22b3-4237-8d6c-b8dc9c20b117-kube-api-access-rf5tw" pod "9db299c4-22b3-4237-8d6c-b8dc9c20b117" (UID: "9db299c4-22b3-4237-8d6c-b8dc9c20b117"). InnerVolumeSpecName "kube-api-access-rf5tw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:33:41.860909 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:41.860887 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d2d985-688fcd588c-45k5x" Oct 30 17:33:41.926033 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:41.925883 4522 generic.go:356] "Generic (PLEG): container finished" podID="9db299c4-22b3-4237-8d6c-b8dc9c20b117" containerID="e0ba3abdae1db74c4f59777844c113213ae0618eb8c35130c23f35e4c726ca65" exitCode=0 Oct 30 17:33:41.926033 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:41.925946 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d2d985-698fbdbbdb-z4wjk" Oct 30 17:33:41.926033 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:41.925945 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d2d985-698fbdbbdb-z4wjk" event={"ID":"9db299c4-22b3-4237-8d6c-b8dc9c20b117","Type":"ContainerDied","Data":"e0ba3abdae1db74c4f59777844c113213ae0618eb8c35130c23f35e4c726ca65"} Oct 30 17:33:41.926701 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:41.926044 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d2d985-698fbdbbdb-z4wjk" event={"ID":"9db299c4-22b3-4237-8d6c-b8dc9c20b117","Type":"ContainerDied","Data":"96e73121dc4d9d8e1fbb9681d66b95c4dfba7dddb7d09bf4a58ba06f0561a184"} Oct 30 17:33:41.926701 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:41.926075 4522 scope.go:122] "RemoveContainer" containerID="e0ba3abdae1db74c4f59777844c113213ae0618eb8c35130c23f35e4c726ca65" Oct 30 17:33:41.934097 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:41.934066 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93279e37-0139-4f89-a5a0-55372047919c-kube-api-access-xgxft\" (UniqueName: \"kubernetes.io/projected/93279e37-0139-4f89-a5a0-55372047919c-kube-api-access-xgxft\") pod \"93279e37-0139-4f89-a5a0-55372047919c\" (UID: \"93279e37-0139-4f89-a5a0-55372047919c\") " Oct 30 17:33:41.934097 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:41.934077 4522 generic.go:356] "Generic (PLEG): container finished" podID="6197c2d3-3465-4091-857f-9d8ccdcc2eb3" containerID="7271aee885831857cb9bb7b58ce053d46f2dfc5df5723400b9a634ffeff702e4" exitCode=0 Oct 30 17:33:41.934334 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:41.934246 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93279e37-0139-4f89-a5a0-55372047919c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93279e37-0139-4f89-a5a0-55372047919c-plugins-config-volume\") pod \"93279e37-0139-4f89-a5a0-55372047919c\" (UID: \"93279e37-0139-4f89-a5a0-55372047919c\") " Oct 30 17:33:41.934415 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:41.934391 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d2d985-bdd4b595-zmgvf" event={"ID":"6197c2d3-3465-4091-857f-9d8ccdcc2eb3","Type":"ContainerDied","Data":"7271aee885831857cb9bb7b58ce053d46f2dfc5df5723400b9a634ffeff702e4"} Oct 30 17:33:41.934537 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:41.934440 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d2d985-bdd4b595-zmgvf" event={"ID":"6197c2d3-3465-4091-857f-9d8ccdcc2eb3","Type":"ContainerDied","Data":"5fb0dd07039c231002535a2c72a5bf2e31cc13e51bb66c115630338364b05e8e"} Oct 30 17:33:41.934537 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:41.934522 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d2d985-bdd4b595-zmgvf" Oct 30 17:33:41.936382 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:41.936359 4522 generic.go:356] "Generic (PLEG): container finished" podID="93279e37-0139-4f89-a5a0-55372047919c" containerID="3ac0a2388198a40a87a2708d8ef6a474f824c69ddcc2122af06b51ed7ec4d12b" exitCode=0 Oct 30 17:33:41.936507 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:41.936485 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d2d985-688fcd588c-45k5x" event={"ID":"93279e37-0139-4f89-a5a0-55372047919c","Type":"ContainerDied","Data":"3ac0a2388198a40a87a2708d8ef6a474f824c69ddcc2122af06b51ed7ec4d12b"} Oct 30 17:33:41.936575 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:41.936532 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d2d985-688fcd588c-45k5x" event={"ID":"93279e37-0139-4f89-a5a0-55372047919c","Type":"ContainerDied","Data":"55bc2394915daa66f731801ca13c18ab878c42d4132a09d11d1720b06b10a748"} Oct 30 17:33:41.936638 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:41.936607 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d2d985-688fcd588c-45k5x" Oct 30 17:33:41.937289 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:41.937168 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93279e37-0139-4f89-a5a0-55372047919c-plugins-config-volume" pod "93279e37-0139-4f89-a5a0-55372047919c" (UID: "93279e37-0139-4f89-a5a0-55372047919c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:33:41.937289 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:41.937224 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf5tw\" (UniqueName: \"kubernetes.io/projected/9db299c4-22b3-4237-8d6c-b8dc9c20b117-kube-api-access-rf5tw\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:33:41.937476 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:41.937359 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9db299c4-22b3-4237-8d6c-b8dc9c20b117-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:33:41.942323 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:41.942276 4522 scope.go:122] "RemoveContainer" containerID="e0ba3abdae1db74c4f59777844c113213ae0618eb8c35130c23f35e4c726ca65" Oct 30 17:33:41.942631 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:33:41.942611 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0ba3abdae1db74c4f59777844c113213ae0618eb8c35130c23f35e4c726ca65\": container with ID starting with e0ba3abdae1db74c4f59777844c113213ae0618eb8c35130c23f35e4c726ca65 not found: ID does not exist" containerID="e0ba3abdae1db74c4f59777844c113213ae0618eb8c35130c23f35e4c726ca65" Oct 30 17:33:41.942731 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:41.942636 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0ba3abdae1db74c4f59777844c113213ae0618eb8c35130c23f35e4c726ca65"} err="failed to get container status \"e0ba3abdae1db74c4f59777844c113213ae0618eb8c35130c23f35e4c726ca65\": rpc error: code = NotFound desc = could not find container \"e0ba3abdae1db74c4f59777844c113213ae0618eb8c35130c23f35e4c726ca65\": container with ID starting with e0ba3abdae1db74c4f59777844c113213ae0618eb8c35130c23f35e4c726ca65 not found: ID does not exist" Oct 30 17:33:41.942731 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:41.942653 4522 scope.go:122] "RemoveContainer" containerID="7271aee885831857cb9bb7b58ce053d46f2dfc5df5723400b9a634ffeff702e4" Oct 30 17:33:41.947744 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:41.947722 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93279e37-0139-4f89-a5a0-55372047919c-kube-api-access-xgxft" pod "93279e37-0139-4f89-a5a0-55372047919c" (UID: "93279e37-0139-4f89-a5a0-55372047919c"). InnerVolumeSpecName "kube-api-access-xgxft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:33:41.964066 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:41.964047 4522 scope.go:122] "RemoveContainer" containerID="7271aee885831857cb9bb7b58ce053d46f2dfc5df5723400b9a634ffeff702e4" Oct 30 17:33:41.964396 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:33:41.964375 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7271aee885831857cb9bb7b58ce053d46f2dfc5df5723400b9a634ffeff702e4\": container with ID starting with 7271aee885831857cb9bb7b58ce053d46f2dfc5df5723400b9a634ffeff702e4 not found: ID does not exist" containerID="7271aee885831857cb9bb7b58ce053d46f2dfc5df5723400b9a634ffeff702e4" Oct 30 17:33:41.964499 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:41.964401 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7271aee885831857cb9bb7b58ce053d46f2dfc5df5723400b9a634ffeff702e4"} err="failed to get container status \"7271aee885831857cb9bb7b58ce053d46f2dfc5df5723400b9a634ffeff702e4\": rpc error: code = NotFound desc = could not find container \"7271aee885831857cb9bb7b58ce053d46f2dfc5df5723400b9a634ffeff702e4\": container with ID starting with 7271aee885831857cb9bb7b58ce053d46f2dfc5df5723400b9a634ffeff702e4 not found: ID does not exist" Oct 30 17:33:41.964499 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:41.964418 4522 scope.go:122] "RemoveContainer" containerID="3ac0a2388198a40a87a2708d8ef6a474f824c69ddcc2122af06b51ed7ec4d12b" Oct 30 17:33:41.975643 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:41.975625 4522 scope.go:122] "RemoveContainer" containerID="3ac0a2388198a40a87a2708d8ef6a474f824c69ddcc2122af06b51ed7ec4d12b" Oct 30 17:33:41.975917 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:33:41.975896 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ac0a2388198a40a87a2708d8ef6a474f824c69ddcc2122af06b51ed7ec4d12b\": container with ID starting with 3ac0a2388198a40a87a2708d8ef6a474f824c69ddcc2122af06b51ed7ec4d12b not found: ID does not exist" containerID="3ac0a2388198a40a87a2708d8ef6a474f824c69ddcc2122af06b51ed7ec4d12b" Oct 30 17:33:41.975968 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:41.975922 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ac0a2388198a40a87a2708d8ef6a474f824c69ddcc2122af06b51ed7ec4d12b"} err="failed to get container status \"3ac0a2388198a40a87a2708d8ef6a474f824c69ddcc2122af06b51ed7ec4d12b\": rpc error: code = NotFound desc = could not find container \"3ac0a2388198a40a87a2708d8ef6a474f824c69ddcc2122af06b51ed7ec4d12b\": container with ID starting with 3ac0a2388198a40a87a2708d8ef6a474f824c69ddcc2122af06b51ed7ec4d12b not found: ID does not exist" Oct 30 17:33:42.024690 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:42.024659 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d2d985-698fbdbbdb-z4wjk"] Oct 30 17:33:42.028714 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:42.028687 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d2d985-698fbdbbdb-z4wjk"] Oct 30 17:33:42.038619 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:42.038595 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgxft\" (UniqueName: \"kubernetes.io/projected/93279e37-0139-4f89-a5a0-55372047919c-kube-api-access-xgxft\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:33:42.038619 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:42.038620 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93279e37-0139-4f89-a5a0-55372047919c-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:33:42.044289 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:42.044260 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-c7b9df-56764597fb-q9kdh" podUID="9a7fe95c-0ed3-4017-9002-20507d4c405c" containerName="payload-processing" containerID="cri-o://6d4e171435498f7a9813960b8eb2ec2cb61b1be3334916615f8c014ba1dab57a" gracePeriod=25 Oct 30 17:33:42.068337 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:42.068296 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d2d985-bdd4b595-zmgvf"] Oct 30 17:33:42.075647 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:42.075619 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c7b9df-7498894d5f-vk5db" podUID="5a9cf6b7-657b-4e3a-8f83-2946c56c43c5" containerName="payload-pre-processing" containerID="cri-o://f7d72e766210ac59bbddd541aba9c085d7f533e58310b5ddd456a3b053dfa4ca" gracePeriod=25 Oct 30 17:33:42.075744 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:42.075728 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d2d985-bdd4b595-zmgvf"] Oct 30 17:33:42.147926 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:42.147888 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05959f4f-5f66-4a9e-8050-851b8fce2c5b" path="/var/lib/kubelet/pods/05959f4f-5f66-4a9e-8050-851b8fce2c5b/volumes" Oct 30 17:33:42.148514 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:42.148490 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33b0923a-026d-4946-9e27-a6cb6fc5eb94" path="/var/lib/kubelet/pods/33b0923a-026d-4946-9e27-a6cb6fc5eb94/volumes" Oct 30 17:33:42.149048 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:42.149030 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6197c2d3-3465-4091-857f-9d8ccdcc2eb3" path="/var/lib/kubelet/pods/6197c2d3-3465-4091-857f-9d8ccdcc2eb3/volumes" Oct 30 17:33:42.149836 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:42.149816 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9db299c4-22b3-4237-8d6c-b8dc9c20b117" path="/var/lib/kubelet/pods/9db299c4-22b3-4237-8d6c-b8dc9c20b117/volumes" Oct 30 17:33:42.150287 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:42.150268 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4c8fa39-e889-46e8-9afc-d1bbefbe57c3" path="/var/lib/kubelet/pods/e4c8fa39-e889-46e8-9afc-d1bbefbe57c3/volumes" Oct 30 17:33:42.292390 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:42.292132 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d2d985-688fcd588c-45k5x"] Oct 30 17:33:42.301542 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:42.301509 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d2d985-688fcd588c-45k5x"] Oct 30 17:33:42.949966 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:33:42.949931 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:33:44.119320 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:44.119274 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e220f21c-5bcb2"] Oct 30 17:33:44.120166 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:44.120141 4522 cpu_manager.go:414] "containerMap: removing container" podUID="cf81982b-e4fd-456c-99fe-52b89db4cced" containerName="payload-processing" Oct 30 17:33:44.120166 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:44.120166 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf81982b-e4fd-456c-99fe-52b89db4cced" containerName="payload-processing" Oct 30 17:33:44.120359 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:44.120192 4522 cpu_manager.go:414] "containerMap: removing container" podUID="e904bd74-db7c-4430-863a-5277580a6342" containerName="istio-proxy" Oct 30 17:33:44.120359 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:44.120202 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e904bd74-db7c-4430-863a-5277580a6342" containerName="istio-proxy" Oct 30 17:33:44.120359 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:44.120229 4522 cpu_manager.go:414] "containerMap: removing container" podUID="9db299c4-22b3-4237-8d6c-b8dc9c20b117" containerName="payload-processing" Oct 30 17:33:44.120359 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:44.120237 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9db299c4-22b3-4237-8d6c-b8dc9c20b117" containerName="payload-processing" Oct 30 17:33:44.120359 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:44.120251 4522 cpu_manager.go:414] "containerMap: removing container" podUID="1c214620-08c5-4a95-aea8-b2d4cbae85a2" containerName="payload-pre-processing" Oct 30 17:33:44.120359 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:44.120258 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c214620-08c5-4a95-aea8-b2d4cbae85a2" containerName="payload-pre-processing" Oct 30 17:33:44.120359 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:44.120273 4522 cpu_manager.go:414] "containerMap: removing container" podUID="ceac6a2a-ea4a-45ca-a6f0-2beb70ca52d2" containerName="payload-pre-processing" Oct 30 17:33:44.120359 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:44.120279 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ceac6a2a-ea4a-45ca-a6f0-2beb70ca52d2" containerName="payload-pre-processing" Oct 30 17:33:44.120359 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:44.120286 4522 cpu_manager.go:414] "containerMap: removing container" podUID="3913f59f-4c2f-4163-9080-b9557dbd3987" containerName="payload-pre-processing" Oct 30 17:33:44.120359 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:44.120291 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3913f59f-4c2f-4163-9080-b9557dbd3987" containerName="payload-pre-processing" Oct 30 17:33:44.120359 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:44.120299 4522 cpu_manager.go:414] "containerMap: removing container" podUID="33b0923a-026d-4946-9e27-a6cb6fc5eb94" containerName="payload-pre-processing" Oct 30 17:33:44.120359 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:44.120318 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33b0923a-026d-4946-9e27-a6cb6fc5eb94" containerName="payload-pre-processing" Oct 30 17:33:44.120359 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:44.120337 4522 cpu_manager.go:414] "containerMap: removing container" podUID="e4c8fa39-e889-46e8-9afc-d1bbefbe57c3" containerName="payload-processing" Oct 30 17:33:44.120359 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:44.120345 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4c8fa39-e889-46e8-9afc-d1bbefbe57c3" containerName="payload-processing" Oct 30 17:33:44.120359 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:44.120362 4522 cpu_manager.go:414] "containerMap: removing container" podUID="4d44befd-705e-4589-b6d0-c944810a76bc" containerName="istio-proxy" Oct 30 17:33:44.120359 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:44.120367 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d44befd-705e-4589-b6d0-c944810a76bc" containerName="istio-proxy" Oct 30 17:33:44.121159 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:44.120384 4522 cpu_manager.go:414] "containerMap: removing container" podUID="02a70782-464d-476c-916d-7f539cc59183" containerName="maas-api" Oct 30 17:33:44.121159 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:44.120390 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02a70782-464d-476c-916d-7f539cc59183" containerName="maas-api" Oct 30 17:33:44.121159 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:44.120415 4522 cpu_manager.go:414] "containerMap: removing container" podUID="05959f4f-5f66-4a9e-8050-851b8fce2c5b" containerName="payload-pre-processing" Oct 30 17:33:44.121159 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:44.120420 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05959f4f-5f66-4a9e-8050-851b8fce2c5b" containerName="payload-pre-processing" Oct 30 17:33:44.121159 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:44.120433 4522 cpu_manager.go:414] "containerMap: removing container" podUID="8ca9280e-ce64-48e6-9a1e-6a104853002e" containerName="istio-proxy" Oct 30 17:33:44.121159 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:44.120438 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ca9280e-ce64-48e6-9a1e-6a104853002e" containerName="istio-proxy" Oct 30 17:33:44.121159 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:44.120448 4522 cpu_manager.go:414] "containerMap: removing container" podUID="93279e37-0139-4f89-a5a0-55372047919c" containerName="payload-pre-processing" Oct 30 17:33:44.121159 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:44.120453 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93279e37-0139-4f89-a5a0-55372047919c" containerName="payload-pre-processing" Oct 30 17:33:44.121159 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:44.120527 4522 cpu_manager.go:414] "containerMap: removing container" podUID="6197c2d3-3465-4091-857f-9d8ccdcc2eb3" containerName="maas-api" Oct 30 17:33:44.121159 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:44.120537 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6197c2d3-3465-4091-857f-9d8ccdcc2eb3" containerName="maas-api" Oct 30 17:33:44.121159 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:44.120682 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="05959f4f-5f66-4a9e-8050-851b8fce2c5b" containerName="payload-pre-processing" Oct 30 17:33:44.121159 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:44.120694 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ca9280e-ce64-48e6-9a1e-6a104853002e" containerName="istio-proxy" Oct 30 17:33:44.121159 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:44.120702 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="93279e37-0139-4f89-a5a0-55372047919c" containerName="payload-pre-processing" Oct 30 17:33:44.121159 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:44.120709 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="6197c2d3-3465-4091-857f-9d8ccdcc2eb3" containerName="maas-api" Oct 30 17:33:44.121159 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:44.120716 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf81982b-e4fd-456c-99fe-52b89db4cced" containerName="payload-processing" Oct 30 17:33:44.121159 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:44.120751 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="e904bd74-db7c-4430-863a-5277580a6342" containerName="istio-proxy" Oct 30 17:33:44.121159 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:44.120760 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="9db299c4-22b3-4237-8d6c-b8dc9c20b117" containerName="payload-processing" Oct 30 17:33:44.121159 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:44.120765 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c214620-08c5-4a95-aea8-b2d4cbae85a2" containerName="payload-pre-processing" Oct 30 17:33:44.121159 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:44.120772 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="ceac6a2a-ea4a-45ca-a6f0-2beb70ca52d2" containerName="payload-pre-processing" Oct 30 17:33:44.121159 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:44.120778 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="3913f59f-4c2f-4163-9080-b9557dbd3987" containerName="payload-pre-processing" Oct 30 17:33:44.121159 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:44.120783 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="33b0923a-026d-4946-9e27-a6cb6fc5eb94" containerName="payload-pre-processing" Oct 30 17:33:44.121159 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:44.120790 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4c8fa39-e889-46e8-9afc-d1bbefbe57c3" containerName="payload-processing" Oct 30 17:33:44.121159 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:44.120815 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d44befd-705e-4589-b6d0-c944810a76bc" containerName="istio-proxy" Oct 30 17:33:44.121159 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:44.120821 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="02a70782-464d-476c-916d-7f539cc59183" containerName="maas-api" Oct 30 17:33:44.126492 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:44.126471 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e220f21c-5bcb2" Oct 30 17:33:44.153363 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:44.153333 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93279e37-0139-4f89-a5a0-55372047919c" path="/var/lib/kubelet/pods/93279e37-0139-4f89-a5a0-55372047919c/volumes" Oct 30 17:33:44.153773 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:44.153755 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e220f21c-5bcb2"] Oct 30 17:33:44.158736 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:44.158710 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:33:44.158870 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:44.158735 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:33:44.171397 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:44.171357 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxg94\" (UniqueName: \"kubernetes.io/projected/7f18989d-1278-4f5f-a6b7-89179eff0676-kube-api-access-gxg94\") pod \"maas-api-revoke-sub-e220f21c-5bcb2\" (UID: \"7f18989d-1278-4f5f-a6b7-89179eff0676\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e220f21c-5bcb2" Oct 30 17:33:44.171549 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:44.171498 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f18989d-1278-4f5f-a6b7-89179eff0676-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e220f21c-5bcb2\" (UID: \"7f18989d-1278-4f5f-a6b7-89179eff0676\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e220f21c-5bcb2" Oct 30 17:33:44.171636 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:44.171564 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f18989d-1278-4f5f-a6b7-89179eff0676-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e220f21c-5bcb2\" (UID: \"7f18989d-1278-4f5f-a6b7-89179eff0676\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e220f21c-5bcb2" Oct 30 17:33:44.272828 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:44.272787 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f18989d-1278-4f5f-a6b7-89179eff0676-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e220f21c-5bcb2\" (UID: \"7f18989d-1278-4f5f-a6b7-89179eff0676\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e220f21c-5bcb2" Oct 30 17:33:44.273591 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:44.273556 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxg94\" (UniqueName: \"kubernetes.io/projected/7f18989d-1278-4f5f-a6b7-89179eff0676-kube-api-access-gxg94\") pod \"maas-api-revoke-sub-e220f21c-5bcb2\" (UID: \"7f18989d-1278-4f5f-a6b7-89179eff0676\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e220f21c-5bcb2" Oct 30 17:33:44.273734 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:44.273717 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f18989d-1278-4f5f-a6b7-89179eff0676-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e220f21c-5bcb2\" (UID: \"7f18989d-1278-4f5f-a6b7-89179eff0676\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e220f21c-5bcb2" Oct 30 17:33:44.274819 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:44.274790 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f18989d-1278-4f5f-a6b7-89179eff0676-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e220f21c-5bcb2\" (UID: \"7f18989d-1278-4f5f-a6b7-89179eff0676\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e220f21c-5bcb2" Oct 30 17:33:44.296360 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:44.288010 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f18989d-1278-4f5f-a6b7-89179eff0676-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e220f21c-5bcb2\" (UID: \"7f18989d-1278-4f5f-a6b7-89179eff0676\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e220f21c-5bcb2" Oct 30 17:33:44.300696 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:44.300672 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxg94\" (UniqueName: \"kubernetes.io/projected/7f18989d-1278-4f5f-a6b7-89179eff0676-kube-api-access-gxg94\") pod \"maas-api-revoke-sub-e220f21c-5bcb2\" (UID: \"7f18989d-1278-4f5f-a6b7-89179eff0676\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e220f21c-5bcb2" Oct 30 17:33:44.452746 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:44.452714 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e220f21c-5bcb2" Oct 30 17:33:44.770426 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:33:44.770399 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f18989d_1278_4f5f_a6b7_89179eff0676.slice/crio-29bc3117eeb9785c2214727b1495e37789c7468d6fc91bfc8caf88a55b25407c WatchSource:0}: Error finding container 29bc3117eeb9785c2214727b1495e37789c7468d6fc91bfc8caf88a55b25407c: Status 404 returned error can't find the container with id 29bc3117eeb9785c2214727b1495e37789c7468d6fc91bfc8caf88a55b25407c Oct 30 17:33:44.773792 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:44.773761 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e220f21c-5bcb2"] Oct 30 17:33:44.970773 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:44.970735 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e220f21c-5bcb2" event={"ID":"7f18989d-1278-4f5f-a6b7-89179eff0676","Type":"ContainerStarted","Data":"29bc3117eeb9785c2214727b1495e37789c7468d6fc91bfc8caf88a55b25407c"} Oct 30 17:33:45.981275 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:45.981188 4522 generic.go:356] "Generic (PLEG): container finished" podID="7f18989d-1278-4f5f-a6b7-89179eff0676" containerID="bde599dc3027ee90e60fa454f6f62a3e7546fad9b132f1c576d6afbc7812a3d3" exitCode=0 Oct 30 17:33:45.981687 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:45.981273 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e220f21c-5bcb2" event={"ID":"7f18989d-1278-4f5f-a6b7-89179eff0676","Type":"ContainerDied","Data":"bde599dc3027ee90e60fa454f6f62a3e7546fad9b132f1c576d6afbc7812a3d3"} Oct 30 17:33:46.057172 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:46.057131 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" podUID="43ca764d-ae0e-4858-9841-5bbd0ffdbb85" containerName="main" containerID="cri-o://45e88413ae21492a881ddb38f38e1ccb7482d6fd2927208f40ef2f93b1ee8928" gracePeriod=45 Oct 30 17:33:46.071024 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:46.070992 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" podUID="0b832781-3d41-4496-aa86-e4d92792e590" containerName="main" containerID="cri-o://a946f686eb0efbce447453509ce825ef866102dce48f34cc92a65830b0f6ab7f" gracePeriod=45 Oct 30 17:33:46.145705 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:46.145670 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c7b9df-7498894d5f-vk5db" podUID="5a9cf6b7-657b-4e3a-8f83-2946c56c43c5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:33:46.269973 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:46.269701 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" podUID="0b832781-3d41-4496-aa86-e4d92792e590" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": dial tcp 10.134.0.53:8000: connect: connection refused" Oct 30 17:33:46.372147 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:46.372111 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" Oct 30 17:33:46.420885 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:46.420860 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" Oct 30 17:33:46.509215 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:46.509126 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b832781-3d41-4496-aa86-e4d92792e590-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b832781-3d41-4496-aa86-e4d92792e590-model-cache\") pod \"0b832781-3d41-4496-aa86-e4d92792e590\" (UID: \"0b832781-3d41-4496-aa86-e4d92792e590\") " Oct 30 17:33:46.509215 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:46.509181 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b832781-3d41-4496-aa86-e4d92792e590-kube-api-access-h2lgv\" (UniqueName: \"kubernetes.io/projected/0b832781-3d41-4496-aa86-e4d92792e590-kube-api-access-h2lgv\") pod \"0b832781-3d41-4496-aa86-e4d92792e590\" (UID: \"0b832781-3d41-4496-aa86-e4d92792e590\") " Oct 30 17:33:46.509488 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:46.509244 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b832781-3d41-4496-aa86-e4d92792e590-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b832781-3d41-4496-aa86-e4d92792e590-dshm\") pod \"0b832781-3d41-4496-aa86-e4d92792e590\" (UID: \"0b832781-3d41-4496-aa86-e4d92792e590\") " Oct 30 17:33:46.509488 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:46.509342 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43ca764d-ae0e-4858-9841-5bbd0ffdbb85-home\" (UniqueName: \"kubernetes.io/empty-dir/43ca764d-ae0e-4858-9841-5bbd0ffdbb85-home\") pod \"43ca764d-ae0e-4858-9841-5bbd0ffdbb85\" (UID: \"43ca764d-ae0e-4858-9841-5bbd0ffdbb85\") " Oct 30 17:33:46.509488 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:46.509442 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43ca764d-ae0e-4858-9841-5bbd0ffdbb85-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43ca764d-ae0e-4858-9841-5bbd0ffdbb85-model-cache\") pod \"43ca764d-ae0e-4858-9841-5bbd0ffdbb85\" (UID: \"43ca764d-ae0e-4858-9841-5bbd0ffdbb85\") " Oct 30 17:33:46.509488 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:46.509463 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b832781-3d41-4496-aa86-e4d92792e590-model-cache" pod "0b832781-3d41-4496-aa86-e4d92792e590" (UID: "0b832781-3d41-4496-aa86-e4d92792e590"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:33:46.509488 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:46.509486 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b832781-3d41-4496-aa86-e4d92792e590-tls-certs\" (UniqueName: \"kubernetes.io/secret/0b832781-3d41-4496-aa86-e4d92792e590-tls-certs\") pod \"0b832781-3d41-4496-aa86-e4d92792e590\" (UID: \"0b832781-3d41-4496-aa86-e4d92792e590\") " Oct 30 17:33:46.509783 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:46.509552 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43ca764d-ae0e-4858-9841-5bbd0ffdbb85-kube-api-access-cjmdc\" (UniqueName: \"kubernetes.io/projected/43ca764d-ae0e-4858-9841-5bbd0ffdbb85-kube-api-access-cjmdc\") pod \"43ca764d-ae0e-4858-9841-5bbd0ffdbb85\" (UID: \"43ca764d-ae0e-4858-9841-5bbd0ffdbb85\") " Oct 30 17:33:46.509783 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:46.509586 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43ca764d-ae0e-4858-9841-5bbd0ffdbb85-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43ca764d-ae0e-4858-9841-5bbd0ffdbb85-tmp-dir\") pod \"43ca764d-ae0e-4858-9841-5bbd0ffdbb85\" (UID: \"43ca764d-ae0e-4858-9841-5bbd0ffdbb85\") " Oct 30 17:33:46.509783 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:46.509637 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43ca764d-ae0e-4858-9841-5bbd0ffdbb85-tls-certs\" (UniqueName: \"kubernetes.io/secret/43ca764d-ae0e-4858-9841-5bbd0ffdbb85-tls-certs\") pod \"43ca764d-ae0e-4858-9841-5bbd0ffdbb85\" (UID: \"43ca764d-ae0e-4858-9841-5bbd0ffdbb85\") " Oct 30 17:33:46.509783 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:46.509672 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b832781-3d41-4496-aa86-e4d92792e590-home\" (UniqueName: \"kubernetes.io/empty-dir/0b832781-3d41-4496-aa86-e4d92792e590-home\") pod \"0b832781-3d41-4496-aa86-e4d92792e590\" (UID: \"0b832781-3d41-4496-aa86-e4d92792e590\") " Oct 30 17:33:46.509783 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:46.509690 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43ca764d-ae0e-4858-9841-5bbd0ffdbb85-home" pod "43ca764d-ae0e-4858-9841-5bbd0ffdbb85" (UID: "43ca764d-ae0e-4858-9841-5bbd0ffdbb85"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:33:46.509783 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:46.509767 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b832781-3d41-4496-aa86-e4d92792e590-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b832781-3d41-4496-aa86-e4d92792e590-tmp-dir\") pod \"0b832781-3d41-4496-aa86-e4d92792e590\" (UID: \"0b832781-3d41-4496-aa86-e4d92792e590\") " Oct 30 17:33:46.510127 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:46.509823 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43ca764d-ae0e-4858-9841-5bbd0ffdbb85-dshm\" (UniqueName: \"kubernetes.io/empty-dir/43ca764d-ae0e-4858-9841-5bbd0ffdbb85-dshm\") pod \"43ca764d-ae0e-4858-9841-5bbd0ffdbb85\" (UID: \"43ca764d-ae0e-4858-9841-5bbd0ffdbb85\") " Oct 30 17:33:46.510633 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:46.510610 4522 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b832781-3d41-4496-aa86-e4d92792e590-model-cache\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:33:46.510633 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:46.510635 4522 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43ca764d-ae0e-4858-9841-5bbd0ffdbb85-home\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:33:46.510811 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:46.509763 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43ca764d-ae0e-4858-9841-5bbd0ffdbb85-model-cache" pod "43ca764d-ae0e-4858-9841-5bbd0ffdbb85" (UID: "43ca764d-ae0e-4858-9841-5bbd0ffdbb85"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:33:46.510811 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:46.509882 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43ca764d-ae0e-4858-9841-5bbd0ffdbb85-tmp-dir" pod "43ca764d-ae0e-4858-9841-5bbd0ffdbb85" (UID: "43ca764d-ae0e-4858-9841-5bbd0ffdbb85"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:33:46.510811 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:46.509981 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b832781-3d41-4496-aa86-e4d92792e590-home" pod "0b832781-3d41-4496-aa86-e4d92792e590" (UID: "0b832781-3d41-4496-aa86-e4d92792e590"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:33:46.510984 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:46.510894 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b832781-3d41-4496-aa86-e4d92792e590-tmp-dir" pod "0b832781-3d41-4496-aa86-e4d92792e590" (UID: "0b832781-3d41-4496-aa86-e4d92792e590"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:33:46.512992 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:46.512971 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b832781-3d41-4496-aa86-e4d92792e590-kube-api-access-h2lgv" pod "0b832781-3d41-4496-aa86-e4d92792e590" (UID: "0b832781-3d41-4496-aa86-e4d92792e590"). InnerVolumeSpecName "kube-api-access-h2lgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:33:46.513111 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:46.513006 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43ca764d-ae0e-4858-9841-5bbd0ffdbb85-dshm" pod "43ca764d-ae0e-4858-9841-5bbd0ffdbb85" (UID: "43ca764d-ae0e-4858-9841-5bbd0ffdbb85"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:33:46.518351 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:46.513752 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b832781-3d41-4496-aa86-e4d92792e590-tls-certs" pod "0b832781-3d41-4496-aa86-e4d92792e590" (UID: "0b832781-3d41-4496-aa86-e4d92792e590"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:33:46.518351 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:46.514048 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43ca764d-ae0e-4858-9841-5bbd0ffdbb85-kube-api-access-cjmdc" pod "43ca764d-ae0e-4858-9841-5bbd0ffdbb85" (UID: "43ca764d-ae0e-4858-9841-5bbd0ffdbb85"). InnerVolumeSpecName "kube-api-access-cjmdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:33:46.518351 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:46.514071 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b832781-3d41-4496-aa86-e4d92792e590-dshm" pod "0b832781-3d41-4496-aa86-e4d92792e590" (UID: "0b832781-3d41-4496-aa86-e4d92792e590"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:33:46.518351 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:46.514659 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43ca764d-ae0e-4858-9841-5bbd0ffdbb85-tls-certs" pod "43ca764d-ae0e-4858-9841-5bbd0ffdbb85" (UID: "43ca764d-ae0e-4858-9841-5bbd0ffdbb85"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:33:46.612828 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:46.612786 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2lgv\" (UniqueName: \"kubernetes.io/projected/0b832781-3d41-4496-aa86-e4d92792e590-kube-api-access-h2lgv\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:33:46.612828 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:46.612817 4522 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b832781-3d41-4496-aa86-e4d92792e590-dshm\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:33:46.612828 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:46.612828 4522 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43ca764d-ae0e-4858-9841-5bbd0ffdbb85-model-cache\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:33:46.612828 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:46.612838 4522 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b832781-3d41-4496-aa86-e4d92792e590-tls-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:33:46.613171 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:46.612850 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjmdc\" (UniqueName: \"kubernetes.io/projected/43ca764d-ae0e-4858-9841-5bbd0ffdbb85-kube-api-access-cjmdc\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:33:46.613171 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:46.612862 4522 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43ca764d-ae0e-4858-9841-5bbd0ffdbb85-tmp-dir\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:33:46.613171 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:46.612875 4522 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43ca764d-ae0e-4858-9841-5bbd0ffdbb85-tls-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:33:46.613171 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:46.612886 4522 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b832781-3d41-4496-aa86-e4d92792e590-home\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:33:46.613171 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:46.612895 4522 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b832781-3d41-4496-aa86-e4d92792e590-tmp-dir\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:33:46.613171 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:46.612905 4522 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43ca764d-ae0e-4858-9841-5bbd0ffdbb85-dshm\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:33:46.996592 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:46.996557 4522 generic.go:356] "Generic (PLEG): container finished" podID="43ca764d-ae0e-4858-9841-5bbd0ffdbb85" containerID="45e88413ae21492a881ddb38f38e1ccb7482d6fd2927208f40ef2f93b1ee8928" exitCode=0 Oct 30 17:33:46.997051 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:46.996623 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" Oct 30 17:33:46.997051 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:46.996655 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" event={"ID":"43ca764d-ae0e-4858-9841-5bbd0ffdbb85","Type":"ContainerDied","Data":"45e88413ae21492a881ddb38f38e1ccb7482d6fd2927208f40ef2f93b1ee8928"} Oct 30 17:33:46.997051 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:46.996731 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw" event={"ID":"43ca764d-ae0e-4858-9841-5bbd0ffdbb85","Type":"ContainerDied","Data":"81eb5c139949d65f6ec60db66498efa287a1b2a434ce8f91d626772bdec48053"} Oct 30 17:33:46.997051 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:46.996776 4522 scope.go:122] "RemoveContainer" containerID="45e88413ae21492a881ddb38f38e1ccb7482d6fd2927208f40ef2f93b1ee8928" Oct 30 17:33:46.998812 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:46.998784 4522 generic.go:356] "Generic (PLEG): container finished" podID="0b832781-3d41-4496-aa86-e4d92792e590" containerID="a946f686eb0efbce447453509ce825ef866102dce48f34cc92a65830b0f6ab7f" exitCode=0 Oct 30 17:33:46.999124 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:46.999103 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" Oct 30 17:33:46.999415 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:46.999382 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" event={"ID":"0b832781-3d41-4496-aa86-e4d92792e590","Type":"ContainerDied","Data":"a946f686eb0efbce447453509ce825ef866102dce48f34cc92a65830b0f6ab7f"} Oct 30 17:33:46.999512 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:46.999432 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5" event={"ID":"0b832781-3d41-4496-aa86-e4d92792e590","Type":"ContainerDied","Data":"7448356e0670fd2168213b48d3234fded53f4d2cfccdde2939741e6ac043c4d8"} Oct 30 17:33:47.010498 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:47.010427 4522 scope.go:122] "RemoveContainer" containerID="45e88413ae21492a881ddb38f38e1ccb7482d6fd2927208f40ef2f93b1ee8928" Oct 30 17:33:47.010869 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:33:47.010843 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45e88413ae21492a881ddb38f38e1ccb7482d6fd2927208f40ef2f93b1ee8928\": container with ID starting with 45e88413ae21492a881ddb38f38e1ccb7482d6fd2927208f40ef2f93b1ee8928 not found: ID does not exist" containerID="45e88413ae21492a881ddb38f38e1ccb7482d6fd2927208f40ef2f93b1ee8928" Oct 30 17:33:47.010996 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:47.010880 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45e88413ae21492a881ddb38f38e1ccb7482d6fd2927208f40ef2f93b1ee8928"} err="failed to get container status \"45e88413ae21492a881ddb38f38e1ccb7482d6fd2927208f40ef2f93b1ee8928\": rpc error: code = NotFound desc = could not find container \"45e88413ae21492a881ddb38f38e1ccb7482d6fd2927208f40ef2f93b1ee8928\": container with ID starting with 45e88413ae21492a881ddb38f38e1ccb7482d6fd2927208f40ef2f93b1ee8928 not found: ID does not exist" Oct 30 17:33:47.010996 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:47.010904 4522 scope.go:122] "RemoveContainer" containerID="a946f686eb0efbce447453509ce825ef866102dce48f34cc92a65830b0f6ab7f" Oct 30 17:33:47.022406 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:47.022382 4522 scope.go:122] "RemoveContainer" containerID="a946f686eb0efbce447453509ce825ef866102dce48f34cc92a65830b0f6ab7f" Oct 30 17:33:47.022726 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:33:47.022703 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a946f686eb0efbce447453509ce825ef866102dce48f34cc92a65830b0f6ab7f\": container with ID starting with a946f686eb0efbce447453509ce825ef866102dce48f34cc92a65830b0f6ab7f not found: ID does not exist" containerID="a946f686eb0efbce447453509ce825ef866102dce48f34cc92a65830b0f6ab7f" Oct 30 17:33:47.022806 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:47.022732 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a946f686eb0efbce447453509ce825ef866102dce48f34cc92a65830b0f6ab7f"} err="failed to get container status \"a946f686eb0efbce447453509ce825ef866102dce48f34cc92a65830b0f6ab7f\": rpc error: code = NotFound desc = could not find container \"a946f686eb0efbce447453509ce825ef866102dce48f34cc92a65830b0f6ab7f\": container with ID starting with a946f686eb0efbce447453509ce825ef866102dce48f34cc92a65830b0f6ab7f not found: ID does not exist" Oct 30 17:33:47.105768 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:47.104457 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw"] Oct 30 17:33:47.139883 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:47.139839 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-c7b9df/premium-w4-c7b9df-kserve-7cd7bb565-gjrvw"] Oct 30 17:33:47.265410 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:47.261478 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-c7b9df-56764597fb-q9kdh" podUID="9a7fe95c-0ed3-4017-9002-20507d4c405c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:33:47.275335 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:47.272267 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5"] Oct 30 17:33:47.286421 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:47.286393 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-c7b9df/simulator-w4-c7b9df-kserve-7d4fc798b4-kqrd5"] Oct 30 17:33:47.310728 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:47.307596 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e220f21c-5bcb2" Oct 30 17:33:47.454331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:47.443526 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f18989d-1278-4f5f-a6b7-89179eff0676-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f18989d-1278-4f5f-a6b7-89179eff0676-maas-api-xks-ca\") pod \"7f18989d-1278-4f5f-a6b7-89179eff0676\" (UID: \"7f18989d-1278-4f5f-a6b7-89179eff0676\") " Oct 30 17:33:47.454331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:47.443585 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f18989d-1278-4f5f-a6b7-89179eff0676-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f18989d-1278-4f5f-a6b7-89179eff0676-maas-api-openshift-ca\") pod \"7f18989d-1278-4f5f-a6b7-89179eff0676\" (UID: \"7f18989d-1278-4f5f-a6b7-89179eff0676\") " Oct 30 17:33:47.454331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:47.443702 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f18989d-1278-4f5f-a6b7-89179eff0676-kube-api-access-gxg94\" (UniqueName: \"kubernetes.io/projected/7f18989d-1278-4f5f-a6b7-89179eff0676-kube-api-access-gxg94\") pod \"7f18989d-1278-4f5f-a6b7-89179eff0676\" (UID: \"7f18989d-1278-4f5f-a6b7-89179eff0676\") " Oct 30 17:33:47.454331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:47.445050 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f18989d-1278-4f5f-a6b7-89179eff0676-maas-api-openshift-ca" pod "7f18989d-1278-4f5f-a6b7-89179eff0676" (UID: "7f18989d-1278-4f5f-a6b7-89179eff0676"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:33:47.454331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:47.445847 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f18989d-1278-4f5f-a6b7-89179eff0676-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:33:47.462673 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:47.462580 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f18989d-1278-4f5f-a6b7-89179eff0676-maas-api-xks-ca" pod "7f18989d-1278-4f5f-a6b7-89179eff0676" (UID: "7f18989d-1278-4f5f-a6b7-89179eff0676"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:33:47.462902 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:47.462788 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f18989d-1278-4f5f-a6b7-89179eff0676-kube-api-access-gxg94" pod "7f18989d-1278-4f5f-a6b7-89179eff0676" (UID: "7f18989d-1278-4f5f-a6b7-89179eff0676"). InnerVolumeSpecName "kube-api-access-gxg94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:33:47.547196 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:47.547108 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f18989d-1278-4f5f-a6b7-89179eff0676-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:33:47.547196 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:47.547141 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxg94\" (UniqueName: \"kubernetes.io/projected/7f18989d-1278-4f5f-a6b7-89179eff0676-kube-api-access-gxg94\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:33:47.757113 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:47.757075 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9e9ce7-fc6b84c5f-qwmnv"] Oct 30 17:33:47.757856 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:47.757834 4522 cpu_manager.go:414] "containerMap: removing container" podUID="0b832781-3d41-4496-aa86-e4d92792e590" containerName="main" Oct 30 17:33:47.757856 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:47.757853 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b832781-3d41-4496-aa86-e4d92792e590" containerName="main" Oct 30 17:33:47.758072 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:47.757896 4522 cpu_manager.go:414] "containerMap: removing container" podUID="43ca764d-ae0e-4858-9841-5bbd0ffdbb85" containerName="main" Oct 30 17:33:47.758072 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:47.757904 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43ca764d-ae0e-4858-9841-5bbd0ffdbb85" containerName="main" Oct 30 17:33:47.758072 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:47.757914 4522 cpu_manager.go:414] "containerMap: removing container" podUID="7f18989d-1278-4f5f-a6b7-89179eff0676" containerName="revoke-keys" Oct 30 17:33:47.758072 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:47.757921 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f18989d-1278-4f5f-a6b7-89179eff0676" containerName="revoke-keys" Oct 30 17:33:47.758072 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:47.757934 4522 cpu_manager.go:414] "containerMap: removing container" podUID="e904bd74-db7c-4430-863a-5277580a6342" containerName="istio-proxy" Oct 30 17:33:47.758072 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:47.757942 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e904bd74-db7c-4430-863a-5277580a6342" containerName="istio-proxy" Oct 30 17:33:47.758413 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:47.758177 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b832781-3d41-4496-aa86-e4d92792e590" containerName="main" Oct 30 17:33:47.758413 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:47.758201 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="43ca764d-ae0e-4858-9841-5bbd0ffdbb85" containerName="main" Oct 30 17:33:47.758413 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:47.758236 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f18989d-1278-4f5f-a6b7-89179eff0676" containerName="revoke-keys" Oct 30 17:33:47.758413 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:47.758247 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="e904bd74-db7c-4430-863a-5277580a6342" containerName="istio-proxy" Oct 30 17:33:47.762456 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:47.762421 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9e9ce7-fc6b84c5f-qwmnv" Oct 30 17:33:47.780145 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:47.780101 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-9e9ce7\"" Oct 30 17:33:47.784693 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:47.784394 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9e9ce7-fc6b84c5f-qwmnv"] Oct 30 17:33:47.852783 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:47.852681 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89ddq\" (UniqueName: \"kubernetes.io/projected/3c4810b7-b48e-41a5-9366-b9bcfec990ea-kube-api-access-89ddq\") pod \"payload-pre-processing-e2e-ipp-b-9e9ce7-fc6b84c5f-qwmnv\" (UID: \"3c4810b7-b48e-41a5-9366-b9bcfec990ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9e9ce7-fc6b84c5f-qwmnv" Oct 30 17:33:47.853040 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:47.853012 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c4810b7-b48e-41a5-9366-b9bcfec990ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9e9ce7-fc6b84c5f-qwmnv\" (UID: \"3c4810b7-b48e-41a5-9366-b9bcfec990ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9e9ce7-fc6b84c5f-qwmnv" Oct 30 17:33:47.864832 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:47.864799 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-9e9ce7-dockercfg-lksw8\"" Oct 30 17:33:47.909357 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:47.903643 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9e9ce7-779c78d4b4-2997b"] Oct 30 17:33:47.912579 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:47.912550 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9e9ce7-779c78d4b4-2997b" Oct 30 17:33:47.923129 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:47.923099 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9e9ce7-779c78d4b4-2997b"] Oct 30 17:33:47.954647 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:47.954597 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89ddq\" (UniqueName: \"kubernetes.io/projected/3c4810b7-b48e-41a5-9366-b9bcfec990ea-kube-api-access-89ddq\") pod \"payload-pre-processing-e2e-ipp-b-9e9ce7-fc6b84c5f-qwmnv\" (UID: \"3c4810b7-b48e-41a5-9366-b9bcfec990ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9e9ce7-fc6b84c5f-qwmnv" Oct 30 17:33:47.954854 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:47.954760 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3ece1ba-f9cb-4d06-93ad-ab7b4165b1cd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9e9ce7-779c78d4b4-2997b\" (UID: \"a3ece1ba-f9cb-4d06-93ad-ab7b4165b1cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9e9ce7-779c78d4b4-2997b" Oct 30 17:33:47.954854 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:47.954818 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snt4g\" (UniqueName: \"kubernetes.io/projected/a3ece1ba-f9cb-4d06-93ad-ab7b4165b1cd-kube-api-access-snt4g\") pod \"payload-processing-e2e-ipp-b-9e9ce7-779c78d4b4-2997b\" (UID: \"a3ece1ba-f9cb-4d06-93ad-ab7b4165b1cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9e9ce7-779c78d4b4-2997b" Oct 30 17:33:47.955175 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:47.955148 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c4810b7-b48e-41a5-9366-b9bcfec990ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9e9ce7-fc6b84c5f-qwmnv\" (UID: \"3c4810b7-b48e-41a5-9366-b9bcfec990ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9e9ce7-fc6b84c5f-qwmnv" Oct 30 17:33:47.956511 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:47.956486 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c4810b7-b48e-41a5-9366-b9bcfec990ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9e9ce7-fc6b84c5f-qwmnv\" (UID: \"3c4810b7-b48e-41a5-9366-b9bcfec990ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9e9ce7-fc6b84c5f-qwmnv" Oct 30 17:33:47.981424 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:47.981392 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89ddq\" (UniqueName: \"kubernetes.io/projected/3c4810b7-b48e-41a5-9366-b9bcfec990ea-kube-api-access-89ddq\") pod \"payload-pre-processing-e2e-ipp-b-9e9ce7-fc6b84c5f-qwmnv\" (UID: \"3c4810b7-b48e-41a5-9366-b9bcfec990ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9e9ce7-fc6b84c5f-qwmnv" Oct 30 17:33:48.009457 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:48.009427 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e220f21c-5bcb2" Oct 30 17:33:48.009875 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:48.009463 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e220f21c-5bcb2" event={"ID":"7f18989d-1278-4f5f-a6b7-89179eff0676","Type":"ContainerDied","Data":"29bc3117eeb9785c2214727b1495e37789c7468d6fc91bfc8caf88a55b25407c"} Oct 30 17:33:48.009875 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:48.009515 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29bc3117eeb9785c2214727b1495e37789c7468d6fc91bfc8caf88a55b25407c" Oct 30 17:33:48.056388 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:48.056357 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3ece1ba-f9cb-4d06-93ad-ab7b4165b1cd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9e9ce7-779c78d4b4-2997b\" (UID: \"a3ece1ba-f9cb-4d06-93ad-ab7b4165b1cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9e9ce7-779c78d4b4-2997b" Oct 30 17:33:48.056519 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:48.056401 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snt4g\" (UniqueName: \"kubernetes.io/projected/a3ece1ba-f9cb-4d06-93ad-ab7b4165b1cd-kube-api-access-snt4g\") pod \"payload-processing-e2e-ipp-b-9e9ce7-779c78d4b4-2997b\" (UID: \"a3ece1ba-f9cb-4d06-93ad-ab7b4165b1cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9e9ce7-779c78d4b4-2997b" Oct 30 17:33:48.057088 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:48.057068 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3ece1ba-f9cb-4d06-93ad-ab7b4165b1cd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9e9ce7-779c78d4b4-2997b\" (UID: \"a3ece1ba-f9cb-4d06-93ad-ab7b4165b1cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9e9ce7-779c78d4b4-2997b" Oct 30 17:33:48.081769 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:48.081734 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9e9ce7-fc6b84c5f-qwmnv" Oct 30 17:33:48.082736 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:48.082715 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snt4g\" (UniqueName: \"kubernetes.io/projected/a3ece1ba-f9cb-4d06-93ad-ab7b4165b1cd-kube-api-access-snt4g\") pod \"payload-processing-e2e-ipp-b-9e9ce7-779c78d4b4-2997b\" (UID: \"a3ece1ba-f9cb-4d06-93ad-ab7b4165b1cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9e9ce7-779c78d4b4-2997b" Oct 30 17:33:48.151510 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:48.151483 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b832781-3d41-4496-aa86-e4d92792e590" path="/var/lib/kubelet/pods/0b832781-3d41-4496-aa86-e4d92792e590/volumes" Oct 30 17:33:48.152158 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:48.152135 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43ca764d-ae0e-4858-9841-5bbd0ffdbb85" path="/var/lib/kubelet/pods/43ca764d-ae0e-4858-9841-5bbd0ffdbb85/volumes" Oct 30 17:33:48.243440 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:48.243408 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9e9ce7-779c78d4b4-2997b" Oct 30 17:33:48.394288 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:33:48.394250 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c4810b7_b48e_41a5_9366_b9bcfec990ea.slice/crio-a2aa7ef03ac93a263701ac772c8a4c72075fa1c1ccbc63e8be5d62d2d758bab6 WatchSource:0}: Error finding container a2aa7ef03ac93a263701ac772c8a4c72075fa1c1ccbc63e8be5d62d2d758bab6: Status 404 returned error can't find the container with id a2aa7ef03ac93a263701ac772c8a4c72075fa1c1ccbc63e8be5d62d2d758bab6 Oct 30 17:33:48.396639 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:48.396614 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9e9ce7-fc6b84c5f-qwmnv"] Oct 30 17:33:48.570904 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:48.569781 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9e9ce7-779c78d4b4-2997b"] Oct 30 17:33:49.021607 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:49.021569 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9e9ce7-fc6b84c5f-qwmnv" event={"ID":"3c4810b7-b48e-41a5-9366-b9bcfec990ea","Type":"ContainerStarted","Data":"a975c055bec194df6583f72e98db67558e66dfdd97d9fda65bb68a6a8ea66584"} Oct 30 17:33:49.022402 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:49.021634 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9e9ce7-fc6b84c5f-qwmnv" event={"ID":"3c4810b7-b48e-41a5-9366-b9bcfec990ea","Type":"ContainerStarted","Data":"a2aa7ef03ac93a263701ac772c8a4c72075fa1c1ccbc63e8be5d62d2d758bab6"} Oct 30 17:33:49.022402 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:49.021668 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9e9ce7-fc6b84c5f-qwmnv" Oct 30 17:33:49.023710 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:49.023683 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9e9ce7-779c78d4b4-2997b" event={"ID":"a3ece1ba-f9cb-4d06-93ad-ab7b4165b1cd","Type":"ContainerStarted","Data":"e3dccd30adadd4db62e923c4898ce9f7e3b2ff6375be7c8a2c09851c5263d6cb"} Oct 30 17:33:49.023861 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:49.023730 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9e9ce7-779c78d4b4-2997b" event={"ID":"a3ece1ba-f9cb-4d06-93ad-ab7b4165b1cd","Type":"ContainerStarted","Data":"c33d6b2db83e449e75efa173bf7726cef75e39d14d74385a9578a6c1860d6191"} Oct 30 17:33:49.023861 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:49.023777 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-9e9ce7-779c78d4b4-2997b" Oct 30 17:33:49.072607 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:49.072559 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9e9ce7-fc6b84c5f-qwmnv" podStartSLOduration=2.072544676 podStartE2EDuration="2.072544676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:33:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:33:49.068645475 +0000 UTC m=+2803.069400377" watchObservedRunningTime="2026-10-30 17:33:49.072544676 +0000 UTC m=+2803.073299577" Oct 30 17:33:49.118336 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:49.118242 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-9e9ce7-779c78d4b4-2997b" podStartSLOduration=2.118228562 podStartE2EDuration="2.118228562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:33:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:33:49.115195431 +0000 UTC m=+2803.115950332" watchObservedRunningTime="2026-10-30 17:33:49.118228562 +0000 UTC m=+2803.118983463" Oct 30 17:33:50.369018 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:50.368981 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-93e020-l295g"] Oct 30 17:33:50.373911 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:50.373891 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-93e020-l295g" Oct 30 17:33:50.395081 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:50.395050 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-93e020-l295g"] Oct 30 17:33:50.416267 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:50.416237 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:33:50.416521 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:50.416341 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:33:50.493148 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:50.493108 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0774819e-a31d-408a-9eab-c760433e4d71-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-93e020-l295g\" (UID: \"0774819e-a31d-408a-9eab-c760433e4d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-93e020-l295g" Oct 30 17:33:50.493380 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:50.493358 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m6d5\" (UniqueName: \"kubernetes.io/projected/0774819e-a31d-408a-9eab-c760433e4d71-kube-api-access-7m6d5\") pod \"maas-api-revoke-keys-e2e-ipp-b-93e020-l295g\" (UID: \"0774819e-a31d-408a-9eab-c760433e4d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-93e020-l295g" Oct 30 17:33:50.493480 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:50.493402 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0774819e-a31d-408a-9eab-c760433e4d71-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-93e020-l295g\" (UID: \"0774819e-a31d-408a-9eab-c760433e4d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-93e020-l295g" Oct 30 17:33:50.594452 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:50.594410 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0774819e-a31d-408a-9eab-c760433e4d71-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-93e020-l295g\" (UID: \"0774819e-a31d-408a-9eab-c760433e4d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-93e020-l295g" Oct 30 17:33:50.594675 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:50.594575 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m6d5\" (UniqueName: \"kubernetes.io/projected/0774819e-a31d-408a-9eab-c760433e4d71-kube-api-access-7m6d5\") pod \"maas-api-revoke-keys-e2e-ipp-b-93e020-l295g\" (UID: \"0774819e-a31d-408a-9eab-c760433e4d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-93e020-l295g" Oct 30 17:33:50.594675 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:50.594604 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0774819e-a31d-408a-9eab-c760433e4d71-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-93e020-l295g\" (UID: \"0774819e-a31d-408a-9eab-c760433e4d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-93e020-l295g" Oct 30 17:33:50.595227 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:50.595200 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0774819e-a31d-408a-9eab-c760433e4d71-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-93e020-l295g\" (UID: \"0774819e-a31d-408a-9eab-c760433e4d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-93e020-l295g" Oct 30 17:33:50.597052 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:50.597028 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0774819e-a31d-408a-9eab-c760433e4d71-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-93e020-l295g\" (UID: \"0774819e-a31d-408a-9eab-c760433e4d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-93e020-l295g" Oct 30 17:33:50.620244 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:50.620178 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m6d5\" (UniqueName: \"kubernetes.io/projected/0774819e-a31d-408a-9eab-c760433e4d71-kube-api-access-7m6d5\") pod \"maas-api-revoke-keys-e2e-ipp-b-93e020-l295g\" (UID: \"0774819e-a31d-408a-9eab-c760433e4d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-93e020-l295g" Oct 30 17:33:50.689446 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:50.689412 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-93e020-l295g" Oct 30 17:33:51.005648 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:33:51.005615 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0774819e_a31d_408a_9eab_c760433e4d71.slice/crio-f3b1c5478c9054535975d86056ebaa2c824dd0222c47de84ac6ee62f167921f2 WatchSource:0}: Error finding container f3b1c5478c9054535975d86056ebaa2c824dd0222c47de84ac6ee62f167921f2: Status 404 returned error can't find the container with id f3b1c5478c9054535975d86056ebaa2c824dd0222c47de84ac6ee62f167921f2 Oct 30 17:33:51.007973 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:51.007949 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-93e020-l295g"] Oct 30 17:33:51.049129 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:51.049100 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-93e020-l295g" event={"ID":"0774819e-a31d-408a-9eab-c760433e4d71","Type":"ContainerStarted","Data":"f3b1c5478c9054535975d86056ebaa2c824dd0222c47de84ac6ee62f167921f2"} Oct 30 17:33:52.058917 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:52.058885 4522 generic.go:356] "Generic (PLEG): container finished" podID="0774819e-a31d-408a-9eab-c760433e4d71" containerID="fda6cfd8cdade92e9c49fa92721df406af53f5aaea9f8f525ec2a0e288e6e582" exitCode=0 Oct 30 17:33:52.059400 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:52.058969 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-93e020-l295g" event={"ID":"0774819e-a31d-408a-9eab-c760433e4d71","Type":"ContainerDied","Data":"fda6cfd8cdade92e9c49fa92721df406af53f5aaea9f8f525ec2a0e288e6e582"} Oct 30 17:33:53.243380 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:53.243352 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-93e020-l295g" Oct 30 17:33:53.337867 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:53.337787 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0774819e-a31d-408a-9eab-c760433e4d71-kube-api-access-7m6d5\" (UniqueName: \"kubernetes.io/projected/0774819e-a31d-408a-9eab-c760433e4d71-kube-api-access-7m6d5\") pod \"0774819e-a31d-408a-9eab-c760433e4d71\" (UID: \"0774819e-a31d-408a-9eab-c760433e4d71\") " Oct 30 17:33:53.338035 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:53.337888 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0774819e-a31d-408a-9eab-c760433e4d71-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0774819e-a31d-408a-9eab-c760433e4d71-maas-api-openshift-ca\") pod \"0774819e-a31d-408a-9eab-c760433e4d71\" (UID: \"0774819e-a31d-408a-9eab-c760433e4d71\") " Oct 30 17:33:53.338121 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:53.338056 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0774819e-a31d-408a-9eab-c760433e4d71-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0774819e-a31d-408a-9eab-c760433e4d71-maas-api-xks-ca\") pod \"0774819e-a31d-408a-9eab-c760433e4d71\" (UID: \"0774819e-a31d-408a-9eab-c760433e4d71\") " Oct 30 17:33:53.338556 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:53.338338 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0774819e-a31d-408a-9eab-c760433e4d71-maas-api-openshift-ca" pod "0774819e-a31d-408a-9eab-c760433e4d71" (UID: "0774819e-a31d-408a-9eab-c760433e4d71"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:33:53.338865 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:53.338841 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0774819e-a31d-408a-9eab-c760433e4d71-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:33:53.340398 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:53.340369 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0774819e-a31d-408a-9eab-c760433e4d71-kube-api-access-7m6d5" pod "0774819e-a31d-408a-9eab-c760433e4d71" (UID: "0774819e-a31d-408a-9eab-c760433e4d71"). InnerVolumeSpecName "kube-api-access-7m6d5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:33:53.340500 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:53.340374 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0774819e-a31d-408a-9eab-c760433e4d71-maas-api-xks-ca" pod "0774819e-a31d-408a-9eab-c760433e4d71" (UID: "0774819e-a31d-408a-9eab-c760433e4d71"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:33:53.440321 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:53.440278 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7m6d5\" (UniqueName: \"kubernetes.io/projected/0774819e-a31d-408a-9eab-c760433e4d71-kube-api-access-7m6d5\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:33:53.440510 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:53.440330 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0774819e-a31d-408a-9eab-c760433e4d71-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:33:54.085049 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:54.085017 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-93e020-l295g" Oct 30 17:33:54.085381 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:54.085105 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-93e020-l295g" event={"ID":"0774819e-a31d-408a-9eab-c760433e4d71","Type":"ContainerDied","Data":"f3b1c5478c9054535975d86056ebaa2c824dd0222c47de84ac6ee62f167921f2"} Oct 30 17:33:54.085485 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:54.085435 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3b1c5478c9054535975d86056ebaa2c824dd0222c47de84ac6ee62f167921f2" Oct 30 17:33:56.145900 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:56.145868 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c7b9df-7498894d5f-vk5db" podUID="5a9cf6b7-657b-4e3a-8f83-2946c56c43c5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:33:57.151558 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:57.151518 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-c7b9df-56764597fb-q9kdh" podUID="9a7fe95c-0ed3-4017-9002-20507d4c405c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:33:58.093645 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:58.093611 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d63772ed-6sfht"] Oct 30 17:33:58.094419 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:58.094404 4522 cpu_manager.go:414] "containerMap: removing container" podUID="0774819e-a31d-408a-9eab-c760433e4d71" containerName="revoke-keys" Oct 30 17:33:58.094494 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:58.094420 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0774819e-a31d-408a-9eab-c760433e4d71" containerName="revoke-keys" Oct 30 17:33:58.094655 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:58.094638 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="0774819e-a31d-408a-9eab-c760433e4d71" containerName="revoke-keys" Oct 30 17:33:58.098764 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:58.098746 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d63772ed-6sfht" Oct 30 17:33:58.121122 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:58.121090 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d63772ed-6sfht"] Oct 30 17:33:58.128447 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:58.128423 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:33:58.128584 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:58.128454 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:33:58.216136 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:58.216093 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjsqc\" (UniqueName: \"kubernetes.io/projected/2f5abdf5-256f-425f-b4bf-09cd1cefe7d9-kube-api-access-qjsqc\") pod \"maas-api-revoke-sub-d63772ed-6sfht\" (UID: \"2f5abdf5-256f-425f-b4bf-09cd1cefe7d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d63772ed-6sfht" Oct 30 17:33:58.216673 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:58.216238 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f5abdf5-256f-425f-b4bf-09cd1cefe7d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d63772ed-6sfht\" (UID: \"2f5abdf5-256f-425f-b4bf-09cd1cefe7d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d63772ed-6sfht" Oct 30 17:33:58.216673 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:58.216271 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f5abdf5-256f-425f-b4bf-09cd1cefe7d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d63772ed-6sfht\" (UID: \"2f5abdf5-256f-425f-b4bf-09cd1cefe7d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d63772ed-6sfht" Oct 30 17:33:58.317823 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:58.317779 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjsqc\" (UniqueName: \"kubernetes.io/projected/2f5abdf5-256f-425f-b4bf-09cd1cefe7d9-kube-api-access-qjsqc\") pod \"maas-api-revoke-sub-d63772ed-6sfht\" (UID: \"2f5abdf5-256f-425f-b4bf-09cd1cefe7d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d63772ed-6sfht" Oct 30 17:33:58.318019 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:58.317943 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f5abdf5-256f-425f-b4bf-09cd1cefe7d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d63772ed-6sfht\" (UID: \"2f5abdf5-256f-425f-b4bf-09cd1cefe7d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d63772ed-6sfht" Oct 30 17:33:58.318019 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:58.317971 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f5abdf5-256f-425f-b4bf-09cd1cefe7d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d63772ed-6sfht\" (UID: \"2f5abdf5-256f-425f-b4bf-09cd1cefe7d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d63772ed-6sfht" Oct 30 17:33:58.318751 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:58.318726 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f5abdf5-256f-425f-b4bf-09cd1cefe7d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d63772ed-6sfht\" (UID: \"2f5abdf5-256f-425f-b4bf-09cd1cefe7d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d63772ed-6sfht" Oct 30 17:33:58.320726 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:58.320706 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f5abdf5-256f-425f-b4bf-09cd1cefe7d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d63772ed-6sfht\" (UID: \"2f5abdf5-256f-425f-b4bf-09cd1cefe7d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d63772ed-6sfht" Oct 30 17:33:58.342376 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:58.342345 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjsqc\" (UniqueName: \"kubernetes.io/projected/2f5abdf5-256f-425f-b4bf-09cd1cefe7d9-kube-api-access-qjsqc\") pod \"maas-api-revoke-sub-d63772ed-6sfht\" (UID: \"2f5abdf5-256f-425f-b4bf-09cd1cefe7d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d63772ed-6sfht" Oct 30 17:33:58.413802 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:58.413721 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d63772ed-6sfht" Oct 30 17:33:58.732744 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:33:58.732714 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f5abdf5_256f_425f_b4bf_09cd1cefe7d9.slice/crio-f6d0a8fedd73c6ce1155ff6751a1c02697ca987fb6dbc493659834c487712e94 WatchSource:0}: Error finding container f6d0a8fedd73c6ce1155ff6751a1c02697ca987fb6dbc493659834c487712e94: Status 404 returned error can't find the container with id f6d0a8fedd73c6ce1155ff6751a1c02697ca987fb6dbc493659834c487712e94 Oct 30 17:33:58.735368 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:58.735340 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d63772ed-6sfht"] Oct 30 17:33:59.143544 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:59.143503 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d63772ed-6sfht" event={"ID":"2f5abdf5-256f-425f-b4bf-09cd1cefe7d9","Type":"ContainerStarted","Data":"7a19f6cbc73c2b4a8348b60670c5ee1637fa625b7259f4f4caf90e726feb4286"} Oct 30 17:33:59.143786 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:59.143596 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d63772ed-6sfht" event={"ID":"2f5abdf5-256f-425f-b4bf-09cd1cefe7d9","Type":"ContainerStarted","Data":"f6d0a8fedd73c6ce1155ff6751a1c02697ca987fb6dbc493659834c487712e94"} Oct 30 17:33:59.190973 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:33:59.190919 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d63772ed-6sfht" podStartSLOduration=1.190905176 podStartE2EDuration="1.190905176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:33:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:33:59.186830855 +0000 UTC m=+2813.187585757" watchObservedRunningTime="2026-10-30 17:33:59.190905176 +0000 UTC m=+2813.191660107" Oct 30 17:34:00.040119 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:00.040085 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9e9ce7-fc6b84c5f-qwmnv" Oct 30 17:34:00.040648 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:00.040629 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-9e9ce7-779c78d4b4-2997b" Oct 30 17:34:00.157151 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:00.157119 4522 generic.go:356] "Generic (PLEG): container finished" podID="2f5abdf5-256f-425f-b4bf-09cd1cefe7d9" containerID="7a19f6cbc73c2b4a8348b60670c5ee1637fa625b7259f4f4caf90e726feb4286" exitCode=0 Oct 30 17:34:00.157481 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:00.157376 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d63772ed-6sfht" event={"ID":"2f5abdf5-256f-425f-b4bf-09cd1cefe7d9","Type":"ContainerDied","Data":"7a19f6cbc73c2b4a8348b60670c5ee1637fa625b7259f4f4caf90e726feb4286"} Oct 30 17:34:01.344637 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:01.344607 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d63772ed-6sfht" Oct 30 17:34:01.471100 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:01.471067 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f5abdf5-256f-425f-b4bf-09cd1cefe7d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f5abdf5-256f-425f-b4bf-09cd1cefe7d9-maas-api-openshift-ca\") pod \"2f5abdf5-256f-425f-b4bf-09cd1cefe7d9\" (UID: \"2f5abdf5-256f-425f-b4bf-09cd1cefe7d9\") " Oct 30 17:34:01.471254 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:01.471143 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f5abdf5-256f-425f-b4bf-09cd1cefe7d9-kube-api-access-qjsqc\" (UniqueName: \"kubernetes.io/projected/2f5abdf5-256f-425f-b4bf-09cd1cefe7d9-kube-api-access-qjsqc\") pod \"2f5abdf5-256f-425f-b4bf-09cd1cefe7d9\" (UID: \"2f5abdf5-256f-425f-b4bf-09cd1cefe7d9\") " Oct 30 17:34:01.471332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:01.471286 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f5abdf5-256f-425f-b4bf-09cd1cefe7d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f5abdf5-256f-425f-b4bf-09cd1cefe7d9-maas-api-xks-ca\") pod \"2f5abdf5-256f-425f-b4bf-09cd1cefe7d9\" (UID: \"2f5abdf5-256f-425f-b4bf-09cd1cefe7d9\") " Oct 30 17:34:01.471507 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:01.471478 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f5abdf5-256f-425f-b4bf-09cd1cefe7d9-maas-api-openshift-ca" pod "2f5abdf5-256f-425f-b4bf-09cd1cefe7d9" (UID: "2f5abdf5-256f-425f-b4bf-09cd1cefe7d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:34:01.471865 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:01.471840 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f5abdf5-256f-425f-b4bf-09cd1cefe7d9-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:34:01.473345 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:01.473298 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f5abdf5-256f-425f-b4bf-09cd1cefe7d9-maas-api-xks-ca" pod "2f5abdf5-256f-425f-b4bf-09cd1cefe7d9" (UID: "2f5abdf5-256f-425f-b4bf-09cd1cefe7d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:34:01.473455 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:01.473351 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f5abdf5-256f-425f-b4bf-09cd1cefe7d9-kube-api-access-qjsqc" pod "2f5abdf5-256f-425f-b4bf-09cd1cefe7d9" (UID: "2f5abdf5-256f-425f-b4bf-09cd1cefe7d9"). InnerVolumeSpecName "kube-api-access-qjsqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:34:01.573575 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:01.573543 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjsqc\" (UniqueName: \"kubernetes.io/projected/2f5abdf5-256f-425f-b4bf-09cd1cefe7d9-kube-api-access-qjsqc\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:34:01.573575 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:01.573576 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f5abdf5-256f-425f-b4bf-09cd1cefe7d9-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:34:02.180569 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:02.180537 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d63772ed-6sfht" Oct 30 17:34:02.180725 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:02.180572 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d63772ed-6sfht" event={"ID":"2f5abdf5-256f-425f-b4bf-09cd1cefe7d9","Type":"ContainerDied","Data":"f6d0a8fedd73c6ce1155ff6751a1c02697ca987fb6dbc493659834c487712e94"} Oct 30 17:34:02.180725 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:02.180638 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6d0a8fedd73c6ce1155ff6751a1c02697ca987fb6dbc493659834c487712e94" Oct 30 17:34:02.386260 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:02.386230 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-c7b9df-56764597fb-q9kdh" Oct 30 17:34:02.429826 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:02.429798 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c7b9df-7498894d5f-vk5db" Oct 30 17:34:02.492915 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:02.492786 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a7fe95c-0ed3-4017-9002-20507d4c405c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a7fe95c-0ed3-4017-9002-20507d4c405c-plugins-config-volume\") pod \"9a7fe95c-0ed3-4017-9002-20507d4c405c\" (UID: \"9a7fe95c-0ed3-4017-9002-20507d4c405c\") " Oct 30 17:34:02.493584 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:02.493542 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a7fe95c-0ed3-4017-9002-20507d4c405c-kube-api-access-k8xbg\" (UniqueName: \"kubernetes.io/projected/9a7fe95c-0ed3-4017-9002-20507d4c405c-kube-api-access-k8xbg\") pod \"9a7fe95c-0ed3-4017-9002-20507d4c405c\" (UID: \"9a7fe95c-0ed3-4017-9002-20507d4c405c\") " Oct 30 17:34:02.495827 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:02.495802 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a7fe95c-0ed3-4017-9002-20507d4c405c-plugins-config-volume" pod "9a7fe95c-0ed3-4017-9002-20507d4c405c" (UID: "9a7fe95c-0ed3-4017-9002-20507d4c405c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:34:02.505507 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:02.505474 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a7fe95c-0ed3-4017-9002-20507d4c405c-kube-api-access-k8xbg" pod "9a7fe95c-0ed3-4017-9002-20507d4c405c" (UID: "9a7fe95c-0ed3-4017-9002-20507d4c405c"). InnerVolumeSpecName "kube-api-access-k8xbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:34:02.596723 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:02.596625 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a9cf6b7-657b-4e3a-8f83-2946c56c43c5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a9cf6b7-657b-4e3a-8f83-2946c56c43c5-plugins-config-volume\") pod \"5a9cf6b7-657b-4e3a-8f83-2946c56c43c5\" (UID: \"5a9cf6b7-657b-4e3a-8f83-2946c56c43c5\") " Oct 30 17:34:02.596924 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:02.596902 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a9cf6b7-657b-4e3a-8f83-2946c56c43c5-kube-api-access-lv4ph\" (UniqueName: \"kubernetes.io/projected/5a9cf6b7-657b-4e3a-8f83-2946c56c43c5-kube-api-access-lv4ph\") pod \"5a9cf6b7-657b-4e3a-8f83-2946c56c43c5\" (UID: \"5a9cf6b7-657b-4e3a-8f83-2946c56c43c5\") " Oct 30 17:34:02.597011 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:02.596978 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a9cf6b7-657b-4e3a-8f83-2946c56c43c5-plugins-config-volume" pod "5a9cf6b7-657b-4e3a-8f83-2946c56c43c5" (UID: "5a9cf6b7-657b-4e3a-8f83-2946c56c43c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:34:02.597517 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:02.597494 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8xbg\" (UniqueName: \"kubernetes.io/projected/9a7fe95c-0ed3-4017-9002-20507d4c405c-kube-api-access-k8xbg\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:34:02.597517 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:02.597518 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a9cf6b7-657b-4e3a-8f83-2946c56c43c5-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:34:02.597659 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:02.597531 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a7fe95c-0ed3-4017-9002-20507d4c405c-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:34:02.598932 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:02.598913 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a9cf6b7-657b-4e3a-8f83-2946c56c43c5-kube-api-access-lv4ph" pod "5a9cf6b7-657b-4e3a-8f83-2946c56c43c5" (UID: "5a9cf6b7-657b-4e3a-8f83-2946c56c43c5"). InnerVolumeSpecName "kube-api-access-lv4ph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:34:02.698975 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:02.698936 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lv4ph\" (UniqueName: \"kubernetes.io/projected/5a9cf6b7-657b-4e3a-8f83-2946c56c43c5-kube-api-access-lv4ph\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:34:03.191400 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:03.191357 4522 generic.go:356] "Generic (PLEG): container finished" podID="5a9cf6b7-657b-4e3a-8f83-2946c56c43c5" containerID="f7d72e766210ac59bbddd541aba9c085d7f533e58310b5ddd456a3b053dfa4ca" exitCode=0 Oct 30 17:34:03.191607 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:03.191420 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c7b9df-7498894d5f-vk5db" Oct 30 17:34:03.191607 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:03.191424 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c7b9df-7498894d5f-vk5db" event={"ID":"5a9cf6b7-657b-4e3a-8f83-2946c56c43c5","Type":"ContainerDied","Data":"f7d72e766210ac59bbddd541aba9c085d7f533e58310b5ddd456a3b053dfa4ca"} Oct 30 17:34:03.191607 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:03.191483 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c7b9df-7498894d5f-vk5db" event={"ID":"5a9cf6b7-657b-4e3a-8f83-2946c56c43c5","Type":"ContainerDied","Data":"8ff0e410f9add26d3d1ea6e115f040c0ae3d5c19429b093d92c2775a039a4526"} Oct 30 17:34:03.191607 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:03.191514 4522 scope.go:122] "RemoveContainer" containerID="f7d72e766210ac59bbddd541aba9c085d7f533e58310b5ddd456a3b053dfa4ca" Oct 30 17:34:03.193237 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:03.193215 4522 generic.go:356] "Generic (PLEG): container finished" podID="9a7fe95c-0ed3-4017-9002-20507d4c405c" containerID="6d4e171435498f7a9813960b8eb2ec2cb61b1be3334916615f8c014ba1dab57a" exitCode=0 Oct 30 17:34:03.193399 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:03.193278 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-c7b9df-56764597fb-q9kdh" Oct 30 17:34:03.193399 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:03.193332 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-c7b9df-56764597fb-q9kdh" event={"ID":"9a7fe95c-0ed3-4017-9002-20507d4c405c","Type":"ContainerDied","Data":"6d4e171435498f7a9813960b8eb2ec2cb61b1be3334916615f8c014ba1dab57a"} Oct 30 17:34:03.193482 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:03.193408 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-c7b9df-56764597fb-q9kdh" event={"ID":"9a7fe95c-0ed3-4017-9002-20507d4c405c","Type":"ContainerDied","Data":"9953ca2d7c2082e8a9b1e3dbdba99facd0c59ca4bef34deb4f2cb26be0968da0"} Oct 30 17:34:03.204827 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:03.204801 4522 scope.go:122] "RemoveContainer" containerID="f7d72e766210ac59bbddd541aba9c085d7f533e58310b5ddd456a3b053dfa4ca" Oct 30 17:34:03.205173 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:34:03.205149 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7d72e766210ac59bbddd541aba9c085d7f533e58310b5ddd456a3b053dfa4ca\": container with ID starting with f7d72e766210ac59bbddd541aba9c085d7f533e58310b5ddd456a3b053dfa4ca not found: ID does not exist" containerID="f7d72e766210ac59bbddd541aba9c085d7f533e58310b5ddd456a3b053dfa4ca" Oct 30 17:34:03.205221 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:03.205182 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7d72e766210ac59bbddd541aba9c085d7f533e58310b5ddd456a3b053dfa4ca"} err="failed to get container status \"f7d72e766210ac59bbddd541aba9c085d7f533e58310b5ddd456a3b053dfa4ca\": rpc error: code = NotFound desc = could not find container \"f7d72e766210ac59bbddd541aba9c085d7f533e58310b5ddd456a3b053dfa4ca\": container with ID starting with f7d72e766210ac59bbddd541aba9c085d7f533e58310b5ddd456a3b053dfa4ca not found: ID does not exist" Oct 30 17:34:03.205221 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:03.205201 4522 scope.go:122] "RemoveContainer" containerID="6d4e171435498f7a9813960b8eb2ec2cb61b1be3334916615f8c014ba1dab57a" Oct 30 17:34:03.215696 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:03.215664 4522 scope.go:122] "RemoveContainer" containerID="6d4e171435498f7a9813960b8eb2ec2cb61b1be3334916615f8c014ba1dab57a" Oct 30 17:34:03.216016 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:34:03.215996 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d4e171435498f7a9813960b8eb2ec2cb61b1be3334916615f8c014ba1dab57a\": container with ID starting with 6d4e171435498f7a9813960b8eb2ec2cb61b1be3334916615f8c014ba1dab57a not found: ID does not exist" containerID="6d4e171435498f7a9813960b8eb2ec2cb61b1be3334916615f8c014ba1dab57a" Oct 30 17:34:03.216105 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:03.216028 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d4e171435498f7a9813960b8eb2ec2cb61b1be3334916615f8c014ba1dab57a"} err="failed to get container status \"6d4e171435498f7a9813960b8eb2ec2cb61b1be3334916615f8c014ba1dab57a\": rpc error: code = NotFound desc = could not find container \"6d4e171435498f7a9813960b8eb2ec2cb61b1be3334916615f8c014ba1dab57a\": container with ID starting with 6d4e171435498f7a9813960b8eb2ec2cb61b1be3334916615f8c014ba1dab57a not found: ID does not exist" Oct 30 17:34:03.249476 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:03.249442 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c7b9df-56764597fb-q9kdh"] Oct 30 17:34:03.262635 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:03.262605 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c7b9df-56764597fb-q9kdh"] Oct 30 17:34:03.292570 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:03.292539 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c7b9df-7498894d5f-vk5db"] Oct 30 17:34:03.296902 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:03.296878 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c7b9df-7498894d5f-vk5db"] Oct 30 17:34:03.571771 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:03.571733 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c17e8c-pcv9r"] Oct 30 17:34:03.572915 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:03.572896 4522 cpu_manager.go:414] "containerMap: removing container" podUID="5a9cf6b7-657b-4e3a-8f83-2946c56c43c5" containerName="payload-pre-processing" Oct 30 17:34:03.572984 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:03.572918 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a9cf6b7-657b-4e3a-8f83-2946c56c43c5" containerName="payload-pre-processing" Oct 30 17:34:03.572984 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:03.572975 4522 cpu_manager.go:414] "containerMap: removing container" podUID="2f5abdf5-256f-425f-b4bf-09cd1cefe7d9" containerName="revoke-keys" Oct 30 17:34:03.573055 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:03.572983 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f5abdf5-256f-425f-b4bf-09cd1cefe7d9" containerName="revoke-keys" Oct 30 17:34:03.573055 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:03.573037 4522 cpu_manager.go:414] "containerMap: removing container" podUID="9a7fe95c-0ed3-4017-9002-20507d4c405c" containerName="payload-processing" Oct 30 17:34:03.573055 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:03.573045 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a7fe95c-0ed3-4017-9002-20507d4c405c" containerName="payload-processing" Oct 30 17:34:03.573201 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:03.573190 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a7fe95c-0ed3-4017-9002-20507d4c405c" containerName="payload-processing" Oct 30 17:34:03.573235 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:03.573208 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a9cf6b7-657b-4e3a-8f83-2946c56c43c5" containerName="payload-pre-processing" Oct 30 17:34:03.573235 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:03.573226 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f5abdf5-256f-425f-b4bf-09cd1cefe7d9" containerName="revoke-keys" Oct 30 17:34:03.578824 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:03.578802 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c17e8c-pcv9r" Oct 30 17:34:03.594903 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:03.594876 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:34:03.595161 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:03.595136 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:34:03.595470 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:03.595447 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c17e8c-pcv9r"] Oct 30 17:34:03.712232 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:03.712193 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba6bb50d-e558-461f-853f-b33f6f0a8402-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c17e8c-pcv9r\" (UID: \"ba6bb50d-e558-461f-853f-b33f6f0a8402\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c17e8c-pcv9r" Oct 30 17:34:03.712438 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:03.712417 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba6bb50d-e558-461f-853f-b33f6f0a8402-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c17e8c-pcv9r\" (UID: \"ba6bb50d-e558-461f-853f-b33f6f0a8402\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c17e8c-pcv9r" Oct 30 17:34:03.712490 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:03.712457 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtg9l\" (UniqueName: \"kubernetes.io/projected/ba6bb50d-e558-461f-853f-b33f6f0a8402-kube-api-access-mtg9l\") pod \"maas-api-revoke-keys-e2e-ipp-a-c17e8c-pcv9r\" (UID: \"ba6bb50d-e558-461f-853f-b33f6f0a8402\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c17e8c-pcv9r" Oct 30 17:34:03.814479 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:03.814447 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba6bb50d-e558-461f-853f-b33f6f0a8402-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c17e8c-pcv9r\" (UID: \"ba6bb50d-e558-461f-853f-b33f6f0a8402\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c17e8c-pcv9r" Oct 30 17:34:03.814673 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:03.814566 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba6bb50d-e558-461f-853f-b33f6f0a8402-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c17e8c-pcv9r\" (UID: \"ba6bb50d-e558-461f-853f-b33f6f0a8402\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c17e8c-pcv9r" Oct 30 17:34:03.814673 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:03.814594 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtg9l\" (UniqueName: \"kubernetes.io/projected/ba6bb50d-e558-461f-853f-b33f6f0a8402-kube-api-access-mtg9l\") pod \"maas-api-revoke-keys-e2e-ipp-a-c17e8c-pcv9r\" (UID: \"ba6bb50d-e558-461f-853f-b33f6f0a8402\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c17e8c-pcv9r" Oct 30 17:34:03.815238 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:03.815211 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba6bb50d-e558-461f-853f-b33f6f0a8402-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c17e8c-pcv9r\" (UID: \"ba6bb50d-e558-461f-853f-b33f6f0a8402\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c17e8c-pcv9r" Oct 30 17:34:03.816817 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:03.816790 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba6bb50d-e558-461f-853f-b33f6f0a8402-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c17e8c-pcv9r\" (UID: \"ba6bb50d-e558-461f-853f-b33f6f0a8402\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c17e8c-pcv9r" Oct 30 17:34:03.840147 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:03.840084 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtg9l\" (UniqueName: \"kubernetes.io/projected/ba6bb50d-e558-461f-853f-b33f6f0a8402-kube-api-access-mtg9l\") pod \"maas-api-revoke-keys-e2e-ipp-a-c17e8c-pcv9r\" (UID: \"ba6bb50d-e558-461f-853f-b33f6f0a8402\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c17e8c-pcv9r" Oct 30 17:34:03.894847 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:03.894810 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c17e8c-pcv9r" Oct 30 17:34:04.147813 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:04.147723 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a9cf6b7-657b-4e3a-8f83-2946c56c43c5" path="/var/lib/kubelet/pods/5a9cf6b7-657b-4e3a-8f83-2946c56c43c5/volumes" Oct 30 17:34:04.148403 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:04.148381 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a7fe95c-0ed3-4017-9002-20507d4c405c" path="/var/lib/kubelet/pods/9a7fe95c-0ed3-4017-9002-20507d4c405c/volumes" Oct 30 17:34:04.203490 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:34:04.203443 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba6bb50d_e558_461f_853f_b33f6f0a8402.slice/crio-33cbc051e4e35c5914398f55bcad3d8d1eaabf7018d2047f98dbdb282444f644 WatchSource:0}: Error finding container 33cbc051e4e35c5914398f55bcad3d8d1eaabf7018d2047f98dbdb282444f644: Status 404 returned error can't find the container with id 33cbc051e4e35c5914398f55bcad3d8d1eaabf7018d2047f98dbdb282444f644 Oct 30 17:34:04.209842 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:04.209808 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c17e8c-pcv9r"] Oct 30 17:34:05.244653 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:05.244615 4522 generic.go:356] "Generic (PLEG): container finished" podID="ba6bb50d-e558-461f-853f-b33f6f0a8402" containerID="6f38426a25b5f146e8f2eac0b0e85935e9a1f3a40b8ba2b80345d94031215b56" exitCode=0 Oct 30 17:34:05.245054 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:05.244687 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c17e8c-pcv9r" event={"ID":"ba6bb50d-e558-461f-853f-b33f6f0a8402","Type":"ContainerDied","Data":"6f38426a25b5f146e8f2eac0b0e85935e9a1f3a40b8ba2b80345d94031215b56"} Oct 30 17:34:05.245054 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:05.244749 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c17e8c-pcv9r" event={"ID":"ba6bb50d-e558-461f-853f-b33f6f0a8402","Type":"ContainerStarted","Data":"33cbc051e4e35c5914398f55bcad3d8d1eaabf7018d2047f98dbdb282444f644"} Oct 30 17:34:06.330079 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:06.330042 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-93e020-c7c6648db-8x96v"] Oct 30 17:34:06.330674 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:06.330342 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-93e020-c7c6648db-8x96v" podUID="a50af203-2bc5-40a0-b668-b166180d3518" containerName="payload-processing" containerID="cri-o://5c2e7964435349ea8f9fde92ea3fb0a9dad795d2797edc1286762bc15bb4783d" gracePeriod=30 Oct 30 17:34:06.355370 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:06.353500 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-93e020-6444c5bc99-xfnzj"] Oct 30 17:34:06.355370 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:06.353732 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-93e020-6444c5bc99-xfnzj" podUID="f46edcee-db0c-4a46-a956-0817cd0ee429" containerName="payload-pre-processing" containerID="cri-o://ae90fd3a3c3e1e8ef02d527ac6e75234290b15ba3fcdc2bbbfb2bb1c84988e12" gracePeriod=30 Oct 30 17:34:06.463159 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:34:06.463122 4522 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf46edcee_db0c_4a46_a956_0817cd0ee429.slice/crio-conmon-ae90fd3a3c3e1e8ef02d527ac6e75234290b15ba3fcdc2bbbfb2bb1c84988e12.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf46edcee_db0c_4a46_a956_0817cd0ee429.slice/crio-ae90fd3a3c3e1e8ef02d527ac6e75234290b15ba3fcdc2bbbfb2bb1c84988e12.scope\": RecentStats: unable to find data in memory cache]" Oct 30 17:34:06.535377 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:06.535282 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c17e8c-pcv9r" Oct 30 17:34:06.558252 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:06.558211 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba6bb50d-e558-461f-853f-b33f6f0a8402-kube-api-access-mtg9l\" (UniqueName: \"kubernetes.io/projected/ba6bb50d-e558-461f-853f-b33f6f0a8402-kube-api-access-mtg9l\") pod \"ba6bb50d-e558-461f-853f-b33f6f0a8402\" (UID: \"ba6bb50d-e558-461f-853f-b33f6f0a8402\") " Oct 30 17:34:06.558443 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:06.558268 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba6bb50d-e558-461f-853f-b33f6f0a8402-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba6bb50d-e558-461f-853f-b33f6f0a8402-maas-api-xks-ca\") pod \"ba6bb50d-e558-461f-853f-b33f6f0a8402\" (UID: \"ba6bb50d-e558-461f-853f-b33f6f0a8402\") " Oct 30 17:34:06.558443 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:06.558344 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba6bb50d-e558-461f-853f-b33f6f0a8402-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba6bb50d-e558-461f-853f-b33f6f0a8402-maas-api-openshift-ca\") pod \"ba6bb50d-e558-461f-853f-b33f6f0a8402\" (UID: \"ba6bb50d-e558-461f-853f-b33f6f0a8402\") " Oct 30 17:34:06.559598 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:06.559567 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba6bb50d-e558-461f-853f-b33f6f0a8402-maas-api-openshift-ca" pod "ba6bb50d-e558-461f-853f-b33f6f0a8402" (UID: "ba6bb50d-e558-461f-853f-b33f6f0a8402"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:34:06.574998 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:06.574964 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba6bb50d-e558-461f-853f-b33f6f0a8402-kube-api-access-mtg9l" pod "ba6bb50d-e558-461f-853f-b33f6f0a8402" (UID: "ba6bb50d-e558-461f-853f-b33f6f0a8402"). InnerVolumeSpecName "kube-api-access-mtg9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:34:06.577266 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:06.577237 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba6bb50d-e558-461f-853f-b33f6f0a8402-maas-api-xks-ca" pod "ba6bb50d-e558-461f-853f-b33f6f0a8402" (UID: "ba6bb50d-e558-461f-853f-b33f6f0a8402"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:34:06.635654 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:06.634337 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-93e020-c7c6648db-8x96v" Oct 30 17:34:06.689344 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:06.689257 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a50af203-2bc5-40a0-b668-b166180d3518-kube-api-access-qdsxm\" (UniqueName: \"kubernetes.io/projected/a50af203-2bc5-40a0-b668-b166180d3518-kube-api-access-qdsxm\") pod \"a50af203-2bc5-40a0-b668-b166180d3518\" (UID: \"a50af203-2bc5-40a0-b668-b166180d3518\") " Oct 30 17:34:06.690787 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:06.690763 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a50af203-2bc5-40a0-b668-b166180d3518-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a50af203-2bc5-40a0-b668-b166180d3518-plugins-config-volume\") pod \"a50af203-2bc5-40a0-b668-b166180d3518\" (UID: \"a50af203-2bc5-40a0-b668-b166180d3518\") " Oct 30 17:34:06.691912 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:06.691888 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtg9l\" (UniqueName: \"kubernetes.io/projected/ba6bb50d-e558-461f-853f-b33f6f0a8402-kube-api-access-mtg9l\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:34:06.692066 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:06.692050 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba6bb50d-e558-461f-853f-b33f6f0a8402-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:34:06.692193 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:06.692178 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba6bb50d-e558-461f-853f-b33f6f0a8402-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:34:06.692665 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:06.692644 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a50af203-2bc5-40a0-b668-b166180d3518-plugins-config-volume" pod "a50af203-2bc5-40a0-b668-b166180d3518" (UID: "a50af203-2bc5-40a0-b668-b166180d3518"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:34:06.696151 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:06.696116 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a50af203-2bc5-40a0-b668-b166180d3518-kube-api-access-qdsxm" pod "a50af203-2bc5-40a0-b668-b166180d3518" (UID: "a50af203-2bc5-40a0-b668-b166180d3518"). InnerVolumeSpecName "kube-api-access-qdsxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:34:06.797918 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:06.797846 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-93e020-6444c5bc99-xfnzj" Oct 30 17:34:06.798615 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:06.798583 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdsxm\" (UniqueName: \"kubernetes.io/projected/a50af203-2bc5-40a0-b668-b166180d3518-kube-api-access-qdsxm\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:34:06.798758 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:06.798620 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a50af203-2bc5-40a0-b668-b166180d3518-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:34:06.899796 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:06.899764 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f46edcee-db0c-4a46-a956-0817cd0ee429-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f46edcee-db0c-4a46-a956-0817cd0ee429-plugins-config-volume\") pod \"f46edcee-db0c-4a46-a956-0817cd0ee429\" (UID: \"f46edcee-db0c-4a46-a956-0817cd0ee429\") " Oct 30 17:34:06.899973 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:06.899876 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f46edcee-db0c-4a46-a956-0817cd0ee429-kube-api-access-dg9zc\" (UniqueName: \"kubernetes.io/projected/f46edcee-db0c-4a46-a956-0817cd0ee429-kube-api-access-dg9zc\") pod \"f46edcee-db0c-4a46-a956-0817cd0ee429\" (UID: \"f46edcee-db0c-4a46-a956-0817cd0ee429\") " Oct 30 17:34:06.900159 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:06.900137 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f46edcee-db0c-4a46-a956-0817cd0ee429-plugins-config-volume" pod "f46edcee-db0c-4a46-a956-0817cd0ee429" (UID: "f46edcee-db0c-4a46-a956-0817cd0ee429"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:34:06.900618 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:06.900593 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f46edcee-db0c-4a46-a956-0817cd0ee429-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:34:06.902104 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:06.902078 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f46edcee-db0c-4a46-a956-0817cd0ee429-kube-api-access-dg9zc" pod "f46edcee-db0c-4a46-a956-0817cd0ee429" (UID: "f46edcee-db0c-4a46-a956-0817cd0ee429"). InnerVolumeSpecName "kube-api-access-dg9zc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:34:07.002443 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:07.002405 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg9zc\" (UniqueName: \"kubernetes.io/projected/f46edcee-db0c-4a46-a956-0817cd0ee429-kube-api-access-dg9zc\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:34:07.256732 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:07.256696 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-93e020-l295g"] Oct 30 17:34:07.261296 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:07.261265 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-93e020-l295g"] Oct 30 17:34:07.267274 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:07.267251 4522 generic.go:356] "Generic (PLEG): container finished" podID="a50af203-2bc5-40a0-b668-b166180d3518" containerID="5c2e7964435349ea8f9fde92ea3fb0a9dad795d2797edc1286762bc15bb4783d" exitCode=0 Oct 30 17:34:07.267411 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:07.267299 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-93e020-c7c6648db-8x96v" Oct 30 17:34:07.267411 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:07.267346 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-93e020-c7c6648db-8x96v" event={"ID":"a50af203-2bc5-40a0-b668-b166180d3518","Type":"ContainerDied","Data":"5c2e7964435349ea8f9fde92ea3fb0a9dad795d2797edc1286762bc15bb4783d"} Oct 30 17:34:07.267411 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:07.267401 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-93e020-c7c6648db-8x96v" event={"ID":"a50af203-2bc5-40a0-b668-b166180d3518","Type":"ContainerDied","Data":"46348444f1e385589655923d9ce0d77f123ddb9aeed3a4c084416a2dbeffc980"} Oct 30 17:34:07.267585 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:07.267417 4522 scope.go:122] "RemoveContainer" containerID="5c2e7964435349ea8f9fde92ea3fb0a9dad795d2797edc1286762bc15bb4783d" Oct 30 17:34:07.270381 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:07.270274 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c17e8c-pcv9r" event={"ID":"ba6bb50d-e558-461f-853f-b33f6f0a8402","Type":"ContainerDied","Data":"33cbc051e4e35c5914398f55bcad3d8d1eaabf7018d2047f98dbdb282444f644"} Oct 30 17:34:07.270381 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:07.270333 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33cbc051e4e35c5914398f55bcad3d8d1eaabf7018d2047f98dbdb282444f644" Oct 30 17:34:07.270381 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:07.270383 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c17e8c-pcv9r" Oct 30 17:34:07.281462 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:07.278716 4522 generic.go:356] "Generic (PLEG): container finished" podID="f46edcee-db0c-4a46-a956-0817cd0ee429" containerID="ae90fd3a3c3e1e8ef02d527ac6e75234290b15ba3fcdc2bbbfb2bb1c84988e12" exitCode=0 Oct 30 17:34:07.281462 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:07.278879 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-93e020-6444c5bc99-xfnzj" event={"ID":"f46edcee-db0c-4a46-a956-0817cd0ee429","Type":"ContainerDied","Data":"ae90fd3a3c3e1e8ef02d527ac6e75234290b15ba3fcdc2bbbfb2bb1c84988e12"} Oct 30 17:34:07.281462 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:07.278919 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-93e020-6444c5bc99-xfnzj" event={"ID":"f46edcee-db0c-4a46-a956-0817cd0ee429","Type":"ContainerDied","Data":"aa33e8f211a25d5f1a1eb2ac9ed24b607031416395c25ca3b64648bd9921872e"} Oct 30 17:34:07.281462 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:07.278934 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-93e020-6444c5bc99-xfnzj" Oct 30 17:34:07.288464 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:07.288444 4522 scope.go:122] "RemoveContainer" containerID="5c2e7964435349ea8f9fde92ea3fb0a9dad795d2797edc1286762bc15bb4783d" Oct 30 17:34:07.288928 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:34:07.288901 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c2e7964435349ea8f9fde92ea3fb0a9dad795d2797edc1286762bc15bb4783d\": container with ID starting with 5c2e7964435349ea8f9fde92ea3fb0a9dad795d2797edc1286762bc15bb4783d not found: ID does not exist" containerID="5c2e7964435349ea8f9fde92ea3fb0a9dad795d2797edc1286762bc15bb4783d" Oct 30 17:34:07.289036 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:07.288937 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c2e7964435349ea8f9fde92ea3fb0a9dad795d2797edc1286762bc15bb4783d"} err="failed to get container status \"5c2e7964435349ea8f9fde92ea3fb0a9dad795d2797edc1286762bc15bb4783d\": rpc error: code = NotFound desc = could not find container \"5c2e7964435349ea8f9fde92ea3fb0a9dad795d2797edc1286762bc15bb4783d\": container with ID starting with 5c2e7964435349ea8f9fde92ea3fb0a9dad795d2797edc1286762bc15bb4783d not found: ID does not exist" Oct 30 17:34:07.289036 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:07.288963 4522 scope.go:122] "RemoveContainer" containerID="ae90fd3a3c3e1e8ef02d527ac6e75234290b15ba3fcdc2bbbfb2bb1c84988e12" Oct 30 17:34:07.303169 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:07.303148 4522 scope.go:122] "RemoveContainer" containerID="ae90fd3a3c3e1e8ef02d527ac6e75234290b15ba3fcdc2bbbfb2bb1c84988e12" Oct 30 17:34:07.303479 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:34:07.303458 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae90fd3a3c3e1e8ef02d527ac6e75234290b15ba3fcdc2bbbfb2bb1c84988e12\": container with ID starting with ae90fd3a3c3e1e8ef02d527ac6e75234290b15ba3fcdc2bbbfb2bb1c84988e12 not found: ID does not exist" containerID="ae90fd3a3c3e1e8ef02d527ac6e75234290b15ba3fcdc2bbbfb2bb1c84988e12" Oct 30 17:34:07.303538 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:07.303487 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae90fd3a3c3e1e8ef02d527ac6e75234290b15ba3fcdc2bbbfb2bb1c84988e12"} err="failed to get container status \"ae90fd3a3c3e1e8ef02d527ac6e75234290b15ba3fcdc2bbbfb2bb1c84988e12\": rpc error: code = NotFound desc = could not find container \"ae90fd3a3c3e1e8ef02d527ac6e75234290b15ba3fcdc2bbbfb2bb1c84988e12\": container with ID starting with ae90fd3a3c3e1e8ef02d527ac6e75234290b15ba3fcdc2bbbfb2bb1c84988e12 not found: ID does not exist" Oct 30 17:34:07.367136 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:07.367105 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-93e020-6444c5bc99-xfnzj"] Oct 30 17:34:07.372250 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:07.372225 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-93e020-6444c5bc99-xfnzj"] Oct 30 17:34:07.409624 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:07.409598 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-93e020-c7c6648db-8x96v"] Oct 30 17:34:07.413333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:07.413297 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-93e020-c7c6648db-8x96v"] Oct 30 17:34:08.146987 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:08.146954 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0774819e-a31d-408a-9eab-c760433e4d71" path="/var/lib/kubelet/pods/0774819e-a31d-408a-9eab-c760433e4d71/volumes" Oct 30 17:34:08.147470 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:08.147455 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a50af203-2bc5-40a0-b668-b166180d3518" path="/var/lib/kubelet/pods/a50af203-2bc5-40a0-b668-b166180d3518/volumes" Oct 30 17:34:08.147873 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:08.147860 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f46edcee-db0c-4a46-a956-0817cd0ee429" path="/var/lib/kubelet/pods/f46edcee-db0c-4a46-a956-0817cd0ee429/volumes" Oct 30 17:34:10.537144 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:10.537106 4522 scope.go:122] "RemoveContainer" containerID="e8208bb4a261b3b389bc8f0348a9bac057893075eddcc66bb1adc0ffe7c5d767" Oct 30 17:34:10.548006 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:10.547978 4522 scope.go:122] "RemoveContainer" containerID="23d6552ae62ccd4b24db33537b2f72820d0335076fd9b1a8ba821dab10a91ee0" Oct 30 17:34:10.558523 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:10.558498 4522 scope.go:122] "RemoveContainer" containerID="bd4aade4914249fb22638a2d7ac8557f7d3a3d5a1c17187a63ff949abecc3f5a" Oct 30 17:34:10.568542 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:10.568523 4522 scope.go:122] "RemoveContainer" containerID="d7a1be5fc1bea3d37a890fb9341706d8b5987c6b3fae634f18228407b7315d69" Oct 30 17:34:10.578134 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:10.578112 4522 scope.go:122] "RemoveContainer" containerID="d6642267e3bea08551ca1d203d17f3d74686a4d3a72d988c521c7bb33b3479af" Oct 30 17:34:12.974925 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:34:12.974781 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:34:13.526390 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:13.522861 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-e13a51/auth-test-model-e13a51-kserve-58fc6d6745-cbjrq"] Oct 30 17:34:16.127114 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:16.127076 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5d8947-mnjf2"] Oct 30 17:34:16.128052 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:16.128027 4522 cpu_manager.go:414] "containerMap: removing container" podUID="f46edcee-db0c-4a46-a956-0817cd0ee429" containerName="payload-pre-processing" Oct 30 17:34:16.128173 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:16.128053 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f46edcee-db0c-4a46-a956-0817cd0ee429" containerName="payload-pre-processing" Oct 30 17:34:16.128173 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:16.128126 4522 cpu_manager.go:414] "containerMap: removing container" podUID="ba6bb50d-e558-461f-853f-b33f6f0a8402" containerName="revoke-keys" Oct 30 17:34:16.128173 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:16.128135 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba6bb50d-e558-461f-853f-b33f6f0a8402" containerName="revoke-keys" Oct 30 17:34:16.128173 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:16.128157 4522 cpu_manager.go:414] "containerMap: removing container" podUID="a50af203-2bc5-40a0-b668-b166180d3518" containerName="payload-processing" Oct 30 17:34:16.128173 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:16.128164 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a50af203-2bc5-40a0-b668-b166180d3518" containerName="payload-processing" Oct 30 17:34:16.128456 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:16.128377 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba6bb50d-e558-461f-853f-b33f6f0a8402" containerName="revoke-keys" Oct 30 17:34:16.128456 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:16.128391 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="a50af203-2bc5-40a0-b668-b166180d3518" containerName="payload-processing" Oct 30 17:34:16.128456 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:16.128409 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="f46edcee-db0c-4a46-a956-0817cd0ee429" containerName="payload-pre-processing" Oct 30 17:34:16.135356 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:16.135337 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5d8947-mnjf2" Oct 30 17:34:16.152213 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:16.152184 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5d8947-mnjf2"] Oct 30 17:34:16.159905 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:16.159885 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:34:16.160041 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:16.159936 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:34:16.206840 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:16.206811 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7nrf\" (UniqueName: \"kubernetes.io/projected/ac1d35a2-1aee-4407-8d21-7f1bdfdbc3dc-kube-api-access-m7nrf\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-5d8947-mnjf2\" (UID: \"ac1d35a2-1aee-4407-8d21-7f1bdfdbc3dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5d8947-mnjf2" Oct 30 17:34:16.207014 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:16.206888 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac1d35a2-1aee-4407-8d21-7f1bdfdbc3dc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-5d8947-mnjf2\" (UID: \"ac1d35a2-1aee-4407-8d21-7f1bdfdbc3dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5d8947-mnjf2" Oct 30 17:34:16.207084 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:16.207060 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac1d35a2-1aee-4407-8d21-7f1bdfdbc3dc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-5d8947-mnjf2\" (UID: \"ac1d35a2-1aee-4407-8d21-7f1bdfdbc3dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5d8947-mnjf2" Oct 30 17:34:16.308989 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:16.308954 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7nrf\" (UniqueName: \"kubernetes.io/projected/ac1d35a2-1aee-4407-8d21-7f1bdfdbc3dc-kube-api-access-m7nrf\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-5d8947-mnjf2\" (UID: \"ac1d35a2-1aee-4407-8d21-7f1bdfdbc3dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5d8947-mnjf2" Oct 30 17:34:16.309172 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:16.309040 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac1d35a2-1aee-4407-8d21-7f1bdfdbc3dc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-5d8947-mnjf2\" (UID: \"ac1d35a2-1aee-4407-8d21-7f1bdfdbc3dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5d8947-mnjf2" Oct 30 17:34:16.309172 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:16.309120 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac1d35a2-1aee-4407-8d21-7f1bdfdbc3dc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-5d8947-mnjf2\" (UID: \"ac1d35a2-1aee-4407-8d21-7f1bdfdbc3dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5d8947-mnjf2" Oct 30 17:34:16.309833 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:16.309808 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac1d35a2-1aee-4407-8d21-7f1bdfdbc3dc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-5d8947-mnjf2\" (UID: \"ac1d35a2-1aee-4407-8d21-7f1bdfdbc3dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5d8947-mnjf2" Oct 30 17:34:16.311509 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:16.311491 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac1d35a2-1aee-4407-8d21-7f1bdfdbc3dc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-5d8947-mnjf2\" (UID: \"ac1d35a2-1aee-4407-8d21-7f1bdfdbc3dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5d8947-mnjf2" Oct 30 17:34:16.336548 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:16.336518 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7nrf\" (UniqueName: \"kubernetes.io/projected/ac1d35a2-1aee-4407-8d21-7f1bdfdbc3dc-kube-api-access-m7nrf\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-5d8947-mnjf2\" (UID: \"ac1d35a2-1aee-4407-8d21-7f1bdfdbc3dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5d8947-mnjf2" Oct 30 17:34:16.452220 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:16.452192 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5d8947-mnjf2" Oct 30 17:34:16.765025 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:16.764995 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5d8947-mnjf2"] Oct 30 17:34:17.373542 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:17.373429 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5d8947-mnjf2" event={"ID":"ac1d35a2-1aee-4407-8d21-7f1bdfdbc3dc","Type":"ContainerStarted","Data":"88c4d3c2ab6164876b1e6185e120ff62aff4d63fe3cc56a2894876386aaceef3"} Oct 30 17:34:17.373542 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:17.373497 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5d8947-mnjf2" event={"ID":"ac1d35a2-1aee-4407-8d21-7f1bdfdbc3dc","Type":"ContainerStarted","Data":"41adfc538043ceba2c7dc77786584c8d9ab5de8e828ea400581d65ddf3213b56"} Oct 30 17:34:17.423364 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:17.423294 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5d8947-mnjf2" podStartSLOduration=1.423279451 podStartE2EDuration="1.423279451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:34:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:34:17.420395382 +0000 UTC m=+2831.421150309" watchObservedRunningTime="2026-10-30 17:34:17.423279451 +0000 UTC m=+2831.424034408" Oct 30 17:34:18.384497 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:18.384453 4522 generic.go:356] "Generic (PLEG): container finished" podID="ac1d35a2-1aee-4407-8d21-7f1bdfdbc3dc" containerID="88c4d3c2ab6164876b1e6185e120ff62aff4d63fe3cc56a2894876386aaceef3" exitCode=0 Oct 30 17:34:18.384901 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:18.384538 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5d8947-mnjf2" event={"ID":"ac1d35a2-1aee-4407-8d21-7f1bdfdbc3dc","Type":"ContainerDied","Data":"88c4d3c2ab6164876b1e6185e120ff62aff4d63fe3cc56a2894876386aaceef3"} Oct 30 17:34:19.559967 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:19.559942 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5d8947-mnjf2" Oct 30 17:34:19.663752 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:19.663674 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac1d35a2-1aee-4407-8d21-7f1bdfdbc3dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac1d35a2-1aee-4407-8d21-7f1bdfdbc3dc-maas-api-openshift-ca\") pod \"ac1d35a2-1aee-4407-8d21-7f1bdfdbc3dc\" (UID: \"ac1d35a2-1aee-4407-8d21-7f1bdfdbc3dc\") " Oct 30 17:34:19.663928 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:19.663761 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac1d35a2-1aee-4407-8d21-7f1bdfdbc3dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac1d35a2-1aee-4407-8d21-7f1bdfdbc3dc-maas-api-xks-ca\") pod \"ac1d35a2-1aee-4407-8d21-7f1bdfdbc3dc\" (UID: \"ac1d35a2-1aee-4407-8d21-7f1bdfdbc3dc\") " Oct 30 17:34:19.664085 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:19.664067 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac1d35a2-1aee-4407-8d21-7f1bdfdbc3dc-kube-api-access-m7nrf\" (UniqueName: \"kubernetes.io/projected/ac1d35a2-1aee-4407-8d21-7f1bdfdbc3dc-kube-api-access-m7nrf\") pod \"ac1d35a2-1aee-4407-8d21-7f1bdfdbc3dc\" (UID: \"ac1d35a2-1aee-4407-8d21-7f1bdfdbc3dc\") " Oct 30 17:34:19.664164 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:19.664077 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac1d35a2-1aee-4407-8d21-7f1bdfdbc3dc-maas-api-openshift-ca" pod "ac1d35a2-1aee-4407-8d21-7f1bdfdbc3dc" (UID: "ac1d35a2-1aee-4407-8d21-7f1bdfdbc3dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:34:19.664547 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:19.664526 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac1d35a2-1aee-4407-8d21-7f1bdfdbc3dc-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:34:19.666037 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:19.666005 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac1d35a2-1aee-4407-8d21-7f1bdfdbc3dc-maas-api-xks-ca" pod "ac1d35a2-1aee-4407-8d21-7f1bdfdbc3dc" (UID: "ac1d35a2-1aee-4407-8d21-7f1bdfdbc3dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:34:19.666244 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:19.666223 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac1d35a2-1aee-4407-8d21-7f1bdfdbc3dc-kube-api-access-m7nrf" pod "ac1d35a2-1aee-4407-8d21-7f1bdfdbc3dc" (UID: "ac1d35a2-1aee-4407-8d21-7f1bdfdbc3dc"). InnerVolumeSpecName "kube-api-access-m7nrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:34:19.766143 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:19.766108 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac1d35a2-1aee-4407-8d21-7f1bdfdbc3dc-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:34:19.766143 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:19.766137 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7nrf\" (UniqueName: \"kubernetes.io/projected/ac1d35a2-1aee-4407-8d21-7f1bdfdbc3dc-kube-api-access-m7nrf\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:34:20.403278 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:20.403249 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5d8947-mnjf2" Oct 30 17:34:20.403561 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:20.403279 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5d8947-mnjf2" event={"ID":"ac1d35a2-1aee-4407-8d21-7f1bdfdbc3dc","Type":"ContainerDied","Data":"41adfc538043ceba2c7dc77786584c8d9ab5de8e828ea400581d65ddf3213b56"} Oct 30 17:34:20.403561 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:20.403341 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41adfc538043ceba2c7dc77786584c8d9ab5de8e828ea400581d65ddf3213b56" Oct 30 17:34:22.852296 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:22.852260 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4fb821-77ff87d6fd-pq2qr"] Oct 30 17:34:22.853046 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:22.853029 4522 cpu_manager.go:414] "containerMap: removing container" podUID="ac1d35a2-1aee-4407-8d21-7f1bdfdbc3dc" containerName="revoke-keys" Oct 30 17:34:22.853088 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:22.853048 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac1d35a2-1aee-4407-8d21-7f1bdfdbc3dc" containerName="revoke-keys" Oct 30 17:34:22.853197 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:22.853188 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac1d35a2-1aee-4407-8d21-7f1bdfdbc3dc" containerName="revoke-keys" Oct 30 17:34:22.858173 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:22.858152 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4fb821-77ff87d6fd-pq2qr" Oct 30 17:34:22.878726 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:22.878691 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4fb821-77ff87d6fd-pq2qr"] Oct 30 17:34:22.886975 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:22.886944 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-4fb821-dockercfg-4mt7v\"" Oct 30 17:34:22.970701 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:22.970675 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-4fb821\"" Oct 30 17:34:23.050054 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:23.050018 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnzgm\" (UniqueName: \"kubernetes.io/projected/b9e25d51-dd1c-419c-b938-656d13abb61a-kube-api-access-xnzgm\") pod \"payload-processing-e2e-ipp-b-4fb821-77ff87d6fd-pq2qr\" (UID: \"b9e25d51-dd1c-419c-b938-656d13abb61a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4fb821-77ff87d6fd-pq2qr" Oct 30 17:34:23.050231 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:23.050134 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9e25d51-dd1c-419c-b938-656d13abb61a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4fb821-77ff87d6fd-pq2qr\" (UID: \"b9e25d51-dd1c-419c-b938-656d13abb61a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4fb821-77ff87d6fd-pq2qr" Oct 30 17:34:23.152031 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:23.151940 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnzgm\" (UniqueName: \"kubernetes.io/projected/b9e25d51-dd1c-419c-b938-656d13abb61a-kube-api-access-xnzgm\") pod \"payload-processing-e2e-ipp-b-4fb821-77ff87d6fd-pq2qr\" (UID: \"b9e25d51-dd1c-419c-b938-656d13abb61a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4fb821-77ff87d6fd-pq2qr" Oct 30 17:34:23.152031 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:23.152016 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9e25d51-dd1c-419c-b938-656d13abb61a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4fb821-77ff87d6fd-pq2qr\" (UID: \"b9e25d51-dd1c-419c-b938-656d13abb61a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4fb821-77ff87d6fd-pq2qr" Oct 30 17:34:23.152787 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:23.152762 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9e25d51-dd1c-419c-b938-656d13abb61a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4fb821-77ff87d6fd-pq2qr\" (UID: \"b9e25d51-dd1c-419c-b938-656d13abb61a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4fb821-77ff87d6fd-pq2qr" Oct 30 17:34:23.180188 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:23.180157 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnzgm\" (UniqueName: \"kubernetes.io/projected/b9e25d51-dd1c-419c-b938-656d13abb61a-kube-api-access-xnzgm\") pod \"payload-processing-e2e-ipp-b-4fb821-77ff87d6fd-pq2qr\" (UID: \"b9e25d51-dd1c-419c-b938-656d13abb61a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4fb821-77ff87d6fd-pq2qr" Oct 30 17:34:23.314340 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:23.314290 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c17e8c-84c889b56d-4rrvl"] Oct 30 17:34:23.314525 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:23.314494 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-c17e8c-84c889b56d-4rrvl" podUID="dd7c8134-eff8-4014-8bb6-d10ca68651ba" containerName="payload-processing" containerID="cri-o://9f251f3108eba00b372b4c75a18480dcadd2328fbd1ceba3d7d035ab18d74b75" gracePeriod=30 Oct 30 17:34:23.339931 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:23.339899 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c17e8c-567d66ffff-l28mb"] Oct 30 17:34:23.340168 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:23.340143 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c17e8c-567d66ffff-l28mb" podUID="369893a4-34e9-48a2-8246-7abf4a51018b" containerName="payload-pre-processing" containerID="cri-o://2e66da4df6a132e1f06115006b2db617b78b2bda7a647315133de520305b1ecc" gracePeriod=30 Oct 30 17:34:23.452835 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:23.452802 4522 generic.go:356] "Generic (PLEG): container finished" podID="dd7c8134-eff8-4014-8bb6-d10ca68651ba" containerID="9f251f3108eba00b372b4c75a18480dcadd2328fbd1ceba3d7d035ab18d74b75" exitCode=0 Oct 30 17:34:23.452975 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:23.452874 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c17e8c-84c889b56d-4rrvl" event={"ID":"dd7c8134-eff8-4014-8bb6-d10ca68651ba","Type":"ContainerDied","Data":"9f251f3108eba00b372b4c75a18480dcadd2328fbd1ceba3d7d035ab18d74b75"} Oct 30 17:34:23.475863 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:23.475607 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4fb821-77ff87d6fd-pq2qr" Oct 30 17:34:23.631236 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:23.631200 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c17e8c-84c889b56d-4rrvl" Oct 30 17:34:23.669402 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:23.664408 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd7c8134-eff8-4014-8bb6-d10ca68651ba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd7c8134-eff8-4014-8bb6-d10ca68651ba-plugins-config-volume\") pod \"dd7c8134-eff8-4014-8bb6-d10ca68651ba\" (UID: \"dd7c8134-eff8-4014-8bb6-d10ca68651ba\") " Oct 30 17:34:23.669402 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:23.664505 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd7c8134-eff8-4014-8bb6-d10ca68651ba-kube-api-access-2676z\" (UniqueName: \"kubernetes.io/projected/dd7c8134-eff8-4014-8bb6-d10ca68651ba-kube-api-access-2676z\") pod \"dd7c8134-eff8-4014-8bb6-d10ca68651ba\" (UID: \"dd7c8134-eff8-4014-8bb6-d10ca68651ba\") " Oct 30 17:34:23.669402 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:23.665606 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd7c8134-eff8-4014-8bb6-d10ca68651ba-plugins-config-volume" pod "dd7c8134-eff8-4014-8bb6-d10ca68651ba" (UID: "dd7c8134-eff8-4014-8bb6-d10ca68651ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:34:23.669402 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:23.668742 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd7c8134-eff8-4014-8bb6-d10ca68651ba-kube-api-access-2676z" pod "dd7c8134-eff8-4014-8bb6-d10ca68651ba" (UID: "dd7c8134-eff8-4014-8bb6-d10ca68651ba"). InnerVolumeSpecName "kube-api-access-2676z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:34:23.734445 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:23.731792 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c17e8c-567d66ffff-l28mb" Oct 30 17:34:23.767347 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:23.767205 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd7c8134-eff8-4014-8bb6-d10ca68651ba-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:34:23.767347 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:23.767240 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2676z\" (UniqueName: \"kubernetes.io/projected/dd7c8134-eff8-4014-8bb6-d10ca68651ba-kube-api-access-2676z\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:34:23.868296 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:23.868263 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/369893a4-34e9-48a2-8246-7abf4a51018b-kube-api-access-fxj8x\" (UniqueName: \"kubernetes.io/projected/369893a4-34e9-48a2-8246-7abf4a51018b-kube-api-access-fxj8x\") pod \"369893a4-34e9-48a2-8246-7abf4a51018b\" (UID: \"369893a4-34e9-48a2-8246-7abf4a51018b\") " Oct 30 17:34:23.868751 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:23.868444 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/369893a4-34e9-48a2-8246-7abf4a51018b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/369893a4-34e9-48a2-8246-7abf4a51018b-plugins-config-volume\") pod \"369893a4-34e9-48a2-8246-7abf4a51018b\" (UID: \"369893a4-34e9-48a2-8246-7abf4a51018b\") " Oct 30 17:34:23.868888 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:23.868863 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/369893a4-34e9-48a2-8246-7abf4a51018b-plugins-config-volume" pod "369893a4-34e9-48a2-8246-7abf4a51018b" (UID: "369893a4-34e9-48a2-8246-7abf4a51018b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:34:23.869063 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:23.869043 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/369893a4-34e9-48a2-8246-7abf4a51018b-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:34:23.870723 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:23.870697 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/369893a4-34e9-48a2-8246-7abf4a51018b-kube-api-access-fxj8x" pod "369893a4-34e9-48a2-8246-7abf4a51018b" (UID: "369893a4-34e9-48a2-8246-7abf4a51018b"). InnerVolumeSpecName "kube-api-access-fxj8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:34:23.970301 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:23.970266 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxj8x\" (UniqueName: \"kubernetes.io/projected/369893a4-34e9-48a2-8246-7abf4a51018b-kube-api-access-fxj8x\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:34:24.036525 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:34:24.036488 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9e25d51_dd1c_419c_b938_656d13abb61a.slice/crio-7cf6491a3cc97d0628648ed23438dbb8557cbf8a94502d33e9429ff41a05b89e WatchSource:0}: Error finding container 7cf6491a3cc97d0628648ed23438dbb8557cbf8a94502d33e9429ff41a05b89e: Status 404 returned error can't find the container with id 7cf6491a3cc97d0628648ed23438dbb8557cbf8a94502d33e9429ff41a05b89e Oct 30 17:34:24.038865 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:24.038830 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4fb821-77ff87d6fd-pq2qr"] Oct 30 17:34:24.043535 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:24.043502 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c17e8c-pcv9r"] Oct 30 17:34:24.047158 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:24.047131 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c17e8c-pcv9r"] Oct 30 17:34:24.151035 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:24.150998 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba6bb50d-e558-461f-853f-b33f6f0a8402" path="/var/lib/kubelet/pods/ba6bb50d-e558-461f-853f-b33f6f0a8402/volumes" Oct 30 17:34:24.462630 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:24.462599 4522 generic.go:356] "Generic (PLEG): container finished" podID="369893a4-34e9-48a2-8246-7abf4a51018b" containerID="2e66da4df6a132e1f06115006b2db617b78b2bda7a647315133de520305b1ecc" exitCode=0 Oct 30 17:34:24.462806 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:24.462671 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c17e8c-567d66ffff-l28mb" Oct 30 17:34:24.462806 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:24.462682 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c17e8c-567d66ffff-l28mb" event={"ID":"369893a4-34e9-48a2-8246-7abf4a51018b","Type":"ContainerDied","Data":"2e66da4df6a132e1f06115006b2db617b78b2bda7a647315133de520305b1ecc"} Oct 30 17:34:24.462806 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:24.462763 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c17e8c-567d66ffff-l28mb" event={"ID":"369893a4-34e9-48a2-8246-7abf4a51018b","Type":"ContainerDied","Data":"456bbc8cca23c478ab3e07d08ffe7376fcc58a182188ae6a1bd53195a585b7e9"} Oct 30 17:34:24.462806 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:24.462787 4522 scope.go:122] "RemoveContainer" containerID="2e66da4df6a132e1f06115006b2db617b78b2bda7a647315133de520305b1ecc" Oct 30 17:34:24.464533 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:24.464222 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c17e8c-84c889b56d-4rrvl" Oct 30 17:34:24.464533 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:24.464247 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c17e8c-84c889b56d-4rrvl" event={"ID":"dd7c8134-eff8-4014-8bb6-d10ca68651ba","Type":"ContainerDied","Data":"d97cf624d6cfc357cc93c716c10cc620d891a6d519482b54ae907ec11b132dee"} Oct 30 17:34:24.466475 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:24.466448 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4fb821-77ff87d6fd-pq2qr" event={"ID":"b9e25d51-dd1c-419c-b938-656d13abb61a","Type":"ContainerStarted","Data":"79535d25881a691d062a665025a4587551844b7d014f86b0f516ba680d641fa1"} Oct 30 17:34:24.466612 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:24.466500 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4fb821-77ff87d6fd-pq2qr" event={"ID":"b9e25d51-dd1c-419c-b938-656d13abb61a","Type":"ContainerStarted","Data":"7cf6491a3cc97d0628648ed23438dbb8557cbf8a94502d33e9429ff41a05b89e"} Oct 30 17:34:24.466612 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:24.466527 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-4fb821-77ff87d6fd-pq2qr" Oct 30 17:34:24.475604 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:24.475572 4522 scope.go:122] "RemoveContainer" containerID="2e66da4df6a132e1f06115006b2db617b78b2bda7a647315133de520305b1ecc" Oct 30 17:34:24.475891 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:34:24.475870 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e66da4df6a132e1f06115006b2db617b78b2bda7a647315133de520305b1ecc\": container with ID starting with 2e66da4df6a132e1f06115006b2db617b78b2bda7a647315133de520305b1ecc not found: ID does not exist" containerID="2e66da4df6a132e1f06115006b2db617b78b2bda7a647315133de520305b1ecc" Oct 30 17:34:24.475955 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:24.475902 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e66da4df6a132e1f06115006b2db617b78b2bda7a647315133de520305b1ecc"} err="failed to get container status \"2e66da4df6a132e1f06115006b2db617b78b2bda7a647315133de520305b1ecc\": rpc error: code = NotFound desc = could not find container \"2e66da4df6a132e1f06115006b2db617b78b2bda7a647315133de520305b1ecc\": container with ID starting with 2e66da4df6a132e1f06115006b2db617b78b2bda7a647315133de520305b1ecc not found: ID does not exist" Oct 30 17:34:24.475955 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:24.475919 4522 scope.go:122] "RemoveContainer" containerID="9f251f3108eba00b372b4c75a18480dcadd2328fbd1ceba3d7d035ab18d74b75" Oct 30 17:34:24.524241 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:24.524211 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c17e8c-84c889b56d-4rrvl"] Oct 30 17:34:24.530904 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:24.530875 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c17e8c-84c889b56d-4rrvl"] Oct 30 17:34:24.575094 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:24.575057 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c17e8c-567d66ffff-l28mb"] Oct 30 17:34:24.581157 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:24.581129 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c17e8c-567d66ffff-l28mb"] Oct 30 17:34:24.622509 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:24.622459 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-4fb821-77ff87d6fd-pq2qr" podStartSLOduration=2.622445467 podStartE2EDuration="2.622445467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:34:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:34:24.621194597 +0000 UTC m=+2838.621949498" watchObservedRunningTime="2026-10-30 17:34:24.622445467 +0000 UTC m=+2838.623200404" Oct 30 17:34:26.147394 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:26.147354 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="369893a4-34e9-48a2-8246-7abf4a51018b" path="/var/lib/kubelet/pods/369893a4-34e9-48a2-8246-7abf4a51018b/volumes" Oct 30 17:34:26.148069 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:26.148045 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd7c8134-eff8-4014-8bb6-d10ca68651ba" path="/var/lib/kubelet/pods/dd7c8134-eff8-4014-8bb6-d10ca68651ba/volumes" Oct 30 17:34:28.553356 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:28.553292 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w6-e13a51/auth-test-model-e13a51-kserve-58fc6d6745-cbjrq" podUID="e1a21adc-645e-44a6-bb65-bb82f392c03d" containerName="main" containerID="cri-o://016c326d4d71767376a7f0ef99944bfa9536b9f2e19217c5015be6f0e8dc034f" gracePeriod=45 Oct 30 17:34:28.855696 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:28.855664 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-e13a51/auth-test-model-e13a51-kserve-58fc6d6745-cbjrq" Oct 30 17:34:28.952226 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:28.952194 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1a21adc-645e-44a6-bb65-bb82f392c03d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1a21adc-645e-44a6-bb65-bb82f392c03d-dshm\") pod \"e1a21adc-645e-44a6-bb65-bb82f392c03d\" (UID: \"e1a21adc-645e-44a6-bb65-bb82f392c03d\") " Oct 30 17:34:28.952432 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:28.952265 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1a21adc-645e-44a6-bb65-bb82f392c03d-tls-certs\" (UniqueName: \"kubernetes.io/secret/e1a21adc-645e-44a6-bb65-bb82f392c03d-tls-certs\") pod \"e1a21adc-645e-44a6-bb65-bb82f392c03d\" (UID: \"e1a21adc-645e-44a6-bb65-bb82f392c03d\") " Oct 30 17:34:28.952432 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:28.952299 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1a21adc-645e-44a6-bb65-bb82f392c03d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1a21adc-645e-44a6-bb65-bb82f392c03d-tmp-dir\") pod \"e1a21adc-645e-44a6-bb65-bb82f392c03d\" (UID: \"e1a21adc-645e-44a6-bb65-bb82f392c03d\") " Oct 30 17:34:28.952432 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:28.952377 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1a21adc-645e-44a6-bb65-bb82f392c03d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1a21adc-645e-44a6-bb65-bb82f392c03d-model-cache\") pod \"e1a21adc-645e-44a6-bb65-bb82f392c03d\" (UID: \"e1a21adc-645e-44a6-bb65-bb82f392c03d\") " Oct 30 17:34:28.952432 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:28.952415 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1a21adc-645e-44a6-bb65-bb82f392c03d-home\" (UniqueName: \"kubernetes.io/empty-dir/e1a21adc-645e-44a6-bb65-bb82f392c03d-home\") pod \"e1a21adc-645e-44a6-bb65-bb82f392c03d\" (UID: \"e1a21adc-645e-44a6-bb65-bb82f392c03d\") " Oct 30 17:34:28.952651 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:28.952509 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1a21adc-645e-44a6-bb65-bb82f392c03d-kube-api-access-qnnjn\" (UniqueName: \"kubernetes.io/projected/e1a21adc-645e-44a6-bb65-bb82f392c03d-kube-api-access-qnnjn\") pod \"e1a21adc-645e-44a6-bb65-bb82f392c03d\" (UID: \"e1a21adc-645e-44a6-bb65-bb82f392c03d\") " Oct 30 17:34:28.952725 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:28.952653 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1a21adc-645e-44a6-bb65-bb82f392c03d-tmp-dir" pod "e1a21adc-645e-44a6-bb65-bb82f392c03d" (UID: "e1a21adc-645e-44a6-bb65-bb82f392c03d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:34:28.952725 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:28.952669 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1a21adc-645e-44a6-bb65-bb82f392c03d-model-cache" pod "e1a21adc-645e-44a6-bb65-bb82f392c03d" (UID: "e1a21adc-645e-44a6-bb65-bb82f392c03d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:34:28.952839 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:28.952776 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1a21adc-645e-44a6-bb65-bb82f392c03d-home" pod "e1a21adc-645e-44a6-bb65-bb82f392c03d" (UID: "e1a21adc-645e-44a6-bb65-bb82f392c03d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:34:28.953209 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:28.953187 4522 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1a21adc-645e-44a6-bb65-bb82f392c03d-tmp-dir\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:34:28.953286 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:28.953213 4522 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1a21adc-645e-44a6-bb65-bb82f392c03d-model-cache\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:34:28.953286 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:28.953226 4522 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1a21adc-645e-44a6-bb65-bb82f392c03d-home\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:34:28.954621 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:28.954601 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1a21adc-645e-44a6-bb65-bb82f392c03d-dshm" pod "e1a21adc-645e-44a6-bb65-bb82f392c03d" (UID: "e1a21adc-645e-44a6-bb65-bb82f392c03d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:34:28.954694 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:28.954673 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1a21adc-645e-44a6-bb65-bb82f392c03d-tls-certs" pod "e1a21adc-645e-44a6-bb65-bb82f392c03d" (UID: "e1a21adc-645e-44a6-bb65-bb82f392c03d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:34:28.954795 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:28.954779 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1a21adc-645e-44a6-bb65-bb82f392c03d-kube-api-access-qnnjn" pod "e1a21adc-645e-44a6-bb65-bb82f392c03d" (UID: "e1a21adc-645e-44a6-bb65-bb82f392c03d"). InnerVolumeSpecName "kube-api-access-qnnjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:34:29.054318 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:29.054273 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnnjn\" (UniqueName: \"kubernetes.io/projected/e1a21adc-645e-44a6-bb65-bb82f392c03d-kube-api-access-qnnjn\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:34:29.054490 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:29.054354 4522 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1a21adc-645e-44a6-bb65-bb82f392c03d-dshm\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:34:29.054490 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:29.054376 4522 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1a21adc-645e-44a6-bb65-bb82f392c03d-tls-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:34:29.515248 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:29.515214 4522 generic.go:356] "Generic (PLEG): container finished" podID="e1a21adc-645e-44a6-bb65-bb82f392c03d" containerID="016c326d4d71767376a7f0ef99944bfa9536b9f2e19217c5015be6f0e8dc034f" exitCode=0 Oct 30 17:34:29.515494 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:29.515261 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-e13a51/auth-test-model-e13a51-kserve-58fc6d6745-cbjrq" Oct 30 17:34:29.515494 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:29.515301 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-e13a51/auth-test-model-e13a51-kserve-58fc6d6745-cbjrq" event={"ID":"e1a21adc-645e-44a6-bb65-bb82f392c03d","Type":"ContainerDied","Data":"016c326d4d71767376a7f0ef99944bfa9536b9f2e19217c5015be6f0e8dc034f"} Oct 30 17:34:29.515494 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:29.515387 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-e13a51/auth-test-model-e13a51-kserve-58fc6d6745-cbjrq" event={"ID":"e1a21adc-645e-44a6-bb65-bb82f392c03d","Type":"ContainerDied","Data":"1f043cb37570d39b5a87206d02e0701f1cef2b0a85385f5896b8b2af85514f3d"} Oct 30 17:34:29.515494 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:29.515411 4522 scope.go:122] "RemoveContainer" containerID="016c326d4d71767376a7f0ef99944bfa9536b9f2e19217c5015be6f0e8dc034f" Oct 30 17:34:29.527400 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:29.527384 4522 scope.go:122] "RemoveContainer" containerID="016c326d4d71767376a7f0ef99944bfa9536b9f2e19217c5015be6f0e8dc034f" Oct 30 17:34:29.527636 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:34:29.527621 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"016c326d4d71767376a7f0ef99944bfa9536b9f2e19217c5015be6f0e8dc034f\": container with ID starting with 016c326d4d71767376a7f0ef99944bfa9536b9f2e19217c5015be6f0e8dc034f not found: ID does not exist" containerID="016c326d4d71767376a7f0ef99944bfa9536b9f2e19217c5015be6f0e8dc034f" Oct 30 17:34:29.527684 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:29.527644 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"016c326d4d71767376a7f0ef99944bfa9536b9f2e19217c5015be6f0e8dc034f"} err="failed to get container status \"016c326d4d71767376a7f0ef99944bfa9536b9f2e19217c5015be6f0e8dc034f\": rpc error: code = NotFound desc = could not find container \"016c326d4d71767376a7f0ef99944bfa9536b9f2e19217c5015be6f0e8dc034f\": container with ID starting with 016c326d4d71767376a7f0ef99944bfa9536b9f2e19217c5015be6f0e8dc034f not found: ID does not exist" Oct 30 17:34:29.572919 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:29.572887 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-e13a51/auth-test-model-e13a51-kserve-58fc6d6745-cbjrq"] Oct 30 17:34:29.577274 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:29.577255 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w6-e13a51/auth-test-model-e13a51-kserve-58fc6d6745-cbjrq"] Oct 30 17:34:29.735721 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:29.735675 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w6-e13a51/auth-test-model-e13a51-kserve-58fc6d6745-cbjrq" podUID="e1a21adc-645e-44a6-bb65-bb82f392c03d" containerName="main" probeResult="failure" output="Get \"https://10.134.1.65:8000/ready\": context deadline exceeded" Oct 30 17:34:30.147586 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:30.147543 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1a21adc-645e-44a6-bb65-bb82f392c03d" path="/var/lib/kubelet/pods/e1a21adc-645e-44a6-bb65-bb82f392c03d/volumes" Oct 30 17:34:31.251210 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:31.251176 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd"] Oct 30 17:34:31.251717 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:31.251401 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd" podUID="1e03b6f6-d0a5-45ac-af7f-3f1c50faf173" containerName="istio-proxy" containerID="cri-o://ada327d8a66a878d51ad3cfb2ee3c115b90383176a43c6db6f271bb03f62220d" gracePeriod=30 Oct 30 17:34:31.793529 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:31.793492 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9e9ce7-v789d"] Oct 30 17:34:31.794245 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:31.794231 4522 cpu_manager.go:414] "containerMap: removing container" podUID="369893a4-34e9-48a2-8246-7abf4a51018b" containerName="payload-pre-processing" Oct 30 17:34:31.794339 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:31.794246 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="369893a4-34e9-48a2-8246-7abf4a51018b" containerName="payload-pre-processing" Oct 30 17:34:31.794339 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:31.794289 4522 cpu_manager.go:414] "containerMap: removing container" podUID="dd7c8134-eff8-4014-8bb6-d10ca68651ba" containerName="payload-processing" Oct 30 17:34:31.794339 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:31.794295 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd7c8134-eff8-4014-8bb6-d10ca68651ba" containerName="payload-processing" Oct 30 17:34:31.794339 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:31.794318 4522 cpu_manager.go:414] "containerMap: removing container" podUID="e1a21adc-645e-44a6-bb65-bb82f392c03d" containerName="main" Oct 30 17:34:31.794339 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:31.794323 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1a21adc-645e-44a6-bb65-bb82f392c03d" containerName="main" Oct 30 17:34:31.794512 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:31.794426 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd7c8134-eff8-4014-8bb6-d10ca68651ba" containerName="payload-processing" Oct 30 17:34:31.794512 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:31.794435 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1a21adc-645e-44a6-bb65-bb82f392c03d" containerName="main" Oct 30 17:34:31.794512 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:31.794445 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="369893a4-34e9-48a2-8246-7abf4a51018b" containerName="payload-pre-processing" Oct 30 17:34:31.799650 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:31.799633 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9e9ce7-v789d" Oct 30 17:34:31.817576 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:31.817549 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9e9ce7-v789d"] Oct 30 17:34:31.852059 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:31.852027 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:34:31.852256 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:31.852093 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:34:31.908901 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:31.908855 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50e2c4b5-f884-480d-a6ae-8f215ec9d392-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-9e9ce7-v789d\" (UID: \"50e2c4b5-f884-480d-a6ae-8f215ec9d392\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9e9ce7-v789d" Oct 30 17:34:31.909119 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:31.909078 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx6q2\" (UniqueName: \"kubernetes.io/projected/50e2c4b5-f884-480d-a6ae-8f215ec9d392-kube-api-access-qx6q2\") pod \"maas-api-revoke-keys-e2e-ipp-b-9e9ce7-v789d\" (UID: \"50e2c4b5-f884-480d-a6ae-8f215ec9d392\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9e9ce7-v789d" Oct 30 17:34:31.909192 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:31.909119 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50e2c4b5-f884-480d-a6ae-8f215ec9d392-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-9e9ce7-v789d\" (UID: \"50e2c4b5-f884-480d-a6ae-8f215ec9d392\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9e9ce7-v789d" Oct 30 17:34:32.011057 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:32.011021 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50e2c4b5-f884-480d-a6ae-8f215ec9d392-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-9e9ce7-v789d\" (UID: \"50e2c4b5-f884-480d-a6ae-8f215ec9d392\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9e9ce7-v789d" Oct 30 17:34:32.011220 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:32.011134 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx6q2\" (UniqueName: \"kubernetes.io/projected/50e2c4b5-f884-480d-a6ae-8f215ec9d392-kube-api-access-qx6q2\") pod \"maas-api-revoke-keys-e2e-ipp-b-9e9ce7-v789d\" (UID: \"50e2c4b5-f884-480d-a6ae-8f215ec9d392\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9e9ce7-v789d" Oct 30 17:34:32.011220 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:32.011156 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50e2c4b5-f884-480d-a6ae-8f215ec9d392-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-9e9ce7-v789d\" (UID: \"50e2c4b5-f884-480d-a6ae-8f215ec9d392\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9e9ce7-v789d" Oct 30 17:34:32.011752 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:32.011724 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50e2c4b5-f884-480d-a6ae-8f215ec9d392-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-9e9ce7-v789d\" (UID: \"50e2c4b5-f884-480d-a6ae-8f215ec9d392\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9e9ce7-v789d" Oct 30 17:34:32.013647 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:32.013624 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50e2c4b5-f884-480d-a6ae-8f215ec9d392-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-9e9ce7-v789d\" (UID: \"50e2c4b5-f884-480d-a6ae-8f215ec9d392\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9e9ce7-v789d" Oct 30 17:34:32.036716 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:32.036685 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx6q2\" (UniqueName: \"kubernetes.io/projected/50e2c4b5-f884-480d-a6ae-8f215ec9d392-kube-api-access-qx6q2\") pod \"maas-api-revoke-keys-e2e-ipp-b-9e9ce7-v789d\" (UID: \"50e2c4b5-f884-480d-a6ae-8f215ec9d392\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9e9ce7-v789d" Oct 30 17:34:32.113775 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:32.113682 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9e9ce7-v789d" Oct 30 17:34:32.426297 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:34:32.426265 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50e2c4b5_f884_480d_a6ae_8f215ec9d392.slice/crio-3fc7cdd117e4411c8a5f66235f82ca7f604a428b963ad2aee5173f195ecba827 WatchSource:0}: Error finding container 3fc7cdd117e4411c8a5f66235f82ca7f604a428b963ad2aee5173f195ecba827: Status 404 returned error can't find the container with id 3fc7cdd117e4411c8a5f66235f82ca7f604a428b963ad2aee5173f195ecba827 Oct 30 17:34:32.429483 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:32.429445 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9e9ce7-v789d"] Oct 30 17:34:32.541271 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:32.541235 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9e9ce7-v789d" event={"ID":"50e2c4b5-f884-480d-a6ae-8f215ec9d392","Type":"ContainerStarted","Data":"3fc7cdd117e4411c8a5f66235f82ca7f604a428b963ad2aee5173f195ecba827"} Oct 30 17:34:33.551767 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:33.551675 4522 generic.go:356] "Generic (PLEG): container finished" podID="50e2c4b5-f884-480d-a6ae-8f215ec9d392" containerID="4a28b6d54658fe1829bec3f964c180b1566a416c6b3ea2862d1a10cc1fa1bb89" exitCode=0 Oct 30 17:34:33.552192 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:33.551757 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9e9ce7-v789d" event={"ID":"50e2c4b5-f884-480d-a6ae-8f215ec9d392","Type":"ContainerDied","Data":"4a28b6d54658fe1829bec3f964c180b1566a416c6b3ea2862d1a10cc1fa1bb89"} Oct 30 17:34:34.734149 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:34.734121 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9e9ce7-v789d" Oct 30 17:34:34.856030 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:34.855946 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50e2c4b5-f884-480d-a6ae-8f215ec9d392-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50e2c4b5-f884-480d-a6ae-8f215ec9d392-maas-api-xks-ca\") pod \"50e2c4b5-f884-480d-a6ae-8f215ec9d392\" (UID: \"50e2c4b5-f884-480d-a6ae-8f215ec9d392\") " Oct 30 17:34:34.856186 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:34.856046 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50e2c4b5-f884-480d-a6ae-8f215ec9d392-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50e2c4b5-f884-480d-a6ae-8f215ec9d392-maas-api-openshift-ca\") pod \"50e2c4b5-f884-480d-a6ae-8f215ec9d392\" (UID: \"50e2c4b5-f884-480d-a6ae-8f215ec9d392\") " Oct 30 17:34:34.856186 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:34.856145 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50e2c4b5-f884-480d-a6ae-8f215ec9d392-kube-api-access-qx6q2\" (UniqueName: \"kubernetes.io/projected/50e2c4b5-f884-480d-a6ae-8f215ec9d392-kube-api-access-qx6q2\") pod \"50e2c4b5-f884-480d-a6ae-8f215ec9d392\" (UID: \"50e2c4b5-f884-480d-a6ae-8f215ec9d392\") " Oct 30 17:34:34.856443 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:34.856410 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50e2c4b5-f884-480d-a6ae-8f215ec9d392-maas-api-openshift-ca" pod "50e2c4b5-f884-480d-a6ae-8f215ec9d392" (UID: "50e2c4b5-f884-480d-a6ae-8f215ec9d392"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:34:34.856777 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:34.856759 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50e2c4b5-f884-480d-a6ae-8f215ec9d392-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:34:34.858289 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:34.858262 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50e2c4b5-f884-480d-a6ae-8f215ec9d392-maas-api-xks-ca" pod "50e2c4b5-f884-480d-a6ae-8f215ec9d392" (UID: "50e2c4b5-f884-480d-a6ae-8f215ec9d392"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:34:34.858417 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:34.858338 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50e2c4b5-f884-480d-a6ae-8f215ec9d392-kube-api-access-qx6q2" pod "50e2c4b5-f884-480d-a6ae-8f215ec9d392" (UID: "50e2c4b5-f884-480d-a6ae-8f215ec9d392"). InnerVolumeSpecName "kube-api-access-qx6q2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:34:34.958708 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:34.958672 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx6q2\" (UniqueName: \"kubernetes.io/projected/50e2c4b5-f884-480d-a6ae-8f215ec9d392-kube-api-access-qx6q2\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:34:34.958708 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:34.958705 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50e2c4b5-f884-480d-a6ae-8f215ec9d392-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:34:35.479448 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:35.479415 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-4fb821-77ff87d6fd-pq2qr" Oct 30 17:34:35.569429 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:35.569404 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9e9ce7-v789d" Oct 30 17:34:35.569630 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:35.569445 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9e9ce7-v789d" event={"ID":"50e2c4b5-f884-480d-a6ae-8f215ec9d392","Type":"ContainerDied","Data":"3fc7cdd117e4411c8a5f66235f82ca7f604a428b963ad2aee5173f195ecba827"} Oct 30 17:34:35.569630 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:35.569493 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3fc7cdd117e4411c8a5f66235f82ca7f604a428b963ad2aee5173f195ecba827" Oct 30 17:34:36.578524 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:36.578498 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd" Oct 30 17:34:36.578942 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:36.578914 4522 generic.go:356] "Generic (PLEG): container finished" podID="1e03b6f6-d0a5-45ac-af7f-3f1c50faf173" containerID="ada327d8a66a878d51ad3cfb2ee3c115b90383176a43c6db6f271bb03f62220d" exitCode=0 Oct 30 17:34:36.579026 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:36.578981 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd" event={"ID":"1e03b6f6-d0a5-45ac-af7f-3f1c50faf173","Type":"ContainerDied","Data":"ada327d8a66a878d51ad3cfb2ee3c115b90383176a43c6db6f271bb03f62220d"} Oct 30 17:34:36.579077 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:36.579054 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd" event={"ID":"1e03b6f6-d0a5-45ac-af7f-3f1c50faf173","Type":"ContainerDied","Data":"d2cbc10460aab23ba64c799e8ab243e026c48f30acb4f25b78cf621a347a01b1"} Oct 30 17:34:36.579125 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:36.579077 4522 scope.go:122] "RemoveContainer" containerID="ada327d8a66a878d51ad3cfb2ee3c115b90383176a43c6db6f271bb03f62220d" Oct 30 17:34:36.589920 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:36.589898 4522 scope.go:122] "RemoveContainer" containerID="2c5da7495b1854a538ccac343f5ce39d06254be0b95fa5557e4f0a1bdb5c6ec5" Oct 30 17:34:36.600069 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:36.600044 4522 scope.go:122] "RemoveContainer" containerID="ada327d8a66a878d51ad3cfb2ee3c115b90383176a43c6db6f271bb03f62220d" Oct 30 17:34:36.600360 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:34:36.600340 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ada327d8a66a878d51ad3cfb2ee3c115b90383176a43c6db6f271bb03f62220d\": container with ID starting with ada327d8a66a878d51ad3cfb2ee3c115b90383176a43c6db6f271bb03f62220d not found: ID does not exist" containerID="ada327d8a66a878d51ad3cfb2ee3c115b90383176a43c6db6f271bb03f62220d" Oct 30 17:34:36.600438 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:36.600373 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ada327d8a66a878d51ad3cfb2ee3c115b90383176a43c6db6f271bb03f62220d"} err="failed to get container status \"ada327d8a66a878d51ad3cfb2ee3c115b90383176a43c6db6f271bb03f62220d\": rpc error: code = NotFound desc = could not find container \"ada327d8a66a878d51ad3cfb2ee3c115b90383176a43c6db6f271bb03f62220d\": container with ID starting with ada327d8a66a878d51ad3cfb2ee3c115b90383176a43c6db6f271bb03f62220d not found: ID does not exist" Oct 30 17:34:36.600438 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:36.600393 4522 scope.go:122] "RemoveContainer" containerID="2c5da7495b1854a538ccac343f5ce39d06254be0b95fa5557e4f0a1bdb5c6ec5" Oct 30 17:34:36.600647 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:34:36.600633 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c5da7495b1854a538ccac343f5ce39d06254be0b95fa5557e4f0a1bdb5c6ec5\": container with ID starting with 2c5da7495b1854a538ccac343f5ce39d06254be0b95fa5557e4f0a1bdb5c6ec5 not found: ID does not exist" containerID="2c5da7495b1854a538ccac343f5ce39d06254be0b95fa5557e4f0a1bdb5c6ec5" Oct 30 17:34:36.600704 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:36.600649 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c5da7495b1854a538ccac343f5ce39d06254be0b95fa5557e4f0a1bdb5c6ec5"} err="failed to get container status \"2c5da7495b1854a538ccac343f5ce39d06254be0b95fa5557e4f0a1bdb5c6ec5\": rpc error: code = NotFound desc = could not find container \"2c5da7495b1854a538ccac343f5ce39d06254be0b95fa5557e4f0a1bdb5c6ec5\": container with ID starting with 2c5da7495b1854a538ccac343f5ce39d06254be0b95fa5557e4f0a1bdb5c6ec5 not found: ID does not exist" Oct 30 17:34:36.688736 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:36.688686 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173-istiod-ca-cert\") pod \"1e03b6f6-d0a5-45ac-af7f-3f1c50faf173\" (UID: \"1e03b6f6-d0a5-45ac-af7f-3f1c50faf173\") " Oct 30 17:34:36.688950 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:36.688785 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173-istio-envoy\") pod \"1e03b6f6-d0a5-45ac-af7f-3f1c50faf173\" (UID: \"1e03b6f6-d0a5-45ac-af7f-3f1c50faf173\") " Oct 30 17:34:36.688950 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:36.688914 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173-istio-data\") pod \"1e03b6f6-d0a5-45ac-af7f-3f1c50faf173\" (UID: \"1e03b6f6-d0a5-45ac-af7f-3f1c50faf173\") " Oct 30 17:34:36.689090 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:36.688985 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173-credential-socket\") pod \"1e03b6f6-d0a5-45ac-af7f-3f1c50faf173\" (UID: \"1e03b6f6-d0a5-45ac-af7f-3f1c50faf173\") " Oct 30 17:34:36.689090 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:36.689023 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173-workload-socket\") pod \"1e03b6f6-d0a5-45ac-af7f-3f1c50faf173\" (UID: \"1e03b6f6-d0a5-45ac-af7f-3f1c50faf173\") " Oct 30 17:34:36.689090 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:36.689057 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173-kube-api-access-4p46z\" (UniqueName: \"kubernetes.io/projected/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173-kube-api-access-4p46z\") pod \"1e03b6f6-d0a5-45ac-af7f-3f1c50faf173\" (UID: \"1e03b6f6-d0a5-45ac-af7f-3f1c50faf173\") " Oct 30 17:34:36.689264 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:36.689142 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173-istiod-ca-cert" pod "1e03b6f6-d0a5-45ac-af7f-3f1c50faf173" (UID: "1e03b6f6-d0a5-45ac-af7f-3f1c50faf173"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:34:36.689264 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:36.689155 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173-istio-podinfo\") pod \"1e03b6f6-d0a5-45ac-af7f-3f1c50faf173\" (UID: \"1e03b6f6-d0a5-45ac-af7f-3f1c50faf173\") " Oct 30 17:34:36.689496 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:36.689351 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173-workload-certs\") pod \"1e03b6f6-d0a5-45ac-af7f-3f1c50faf173\" (UID: \"1e03b6f6-d0a5-45ac-af7f-3f1c50faf173\") " Oct 30 17:34:36.689496 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:36.689456 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173-istio-data" pod "1e03b6f6-d0a5-45ac-af7f-3f1c50faf173" (UID: "1e03b6f6-d0a5-45ac-af7f-3f1c50faf173"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:34:36.689686 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:36.689562 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173-workload-socket" pod "1e03b6f6-d0a5-45ac-af7f-3f1c50faf173" (UID: "1e03b6f6-d0a5-45ac-af7f-3f1c50faf173"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:34:36.689686 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:36.689587 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173-istio-token\" (UniqueName: \"kubernetes.io/projected/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173-istio-token\") pod \"1e03b6f6-d0a5-45ac-af7f-3f1c50faf173\" (UID: \"1e03b6f6-d0a5-45ac-af7f-3f1c50faf173\") " Oct 30 17:34:36.689804 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:36.689699 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173-workload-certs" pod "1e03b6f6-d0a5-45ac-af7f-3f1c50faf173" (UID: "1e03b6f6-d0a5-45ac-af7f-3f1c50faf173"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:34:36.690381 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:36.690357 4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173-workload-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:34:36.690495 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:36.690385 4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173-istiod-ca-cert\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:34:36.690495 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:36.690399 4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173-istio-data\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:34:36.690495 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:36.690411 4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173-workload-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:34:36.690495 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:36.690459 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173-credential-socket" pod "1e03b6f6-d0a5-45ac-af7f-3f1c50faf173" (UID: "1e03b6f6-d0a5-45ac-af7f-3f1c50faf173"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:34:36.692485 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:36.692433 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173-istio-token" pod "1e03b6f6-d0a5-45ac-af7f-3f1c50faf173" (UID: "1e03b6f6-d0a5-45ac-af7f-3f1c50faf173"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:34:36.693162 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:36.693022 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173-kube-api-access-4p46z" pod "1e03b6f6-d0a5-45ac-af7f-3f1c50faf173" (UID: "1e03b6f6-d0a5-45ac-af7f-3f1c50faf173"). InnerVolumeSpecName "kube-api-access-4p46z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:34:36.693775 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:36.693749 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173-istio-podinfo" pod "1e03b6f6-d0a5-45ac-af7f-3f1c50faf173" (UID: "1e03b6f6-d0a5-45ac-af7f-3f1c50faf173"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:34:36.693872 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:36.693751 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173-istio-envoy" pod "1e03b6f6-d0a5-45ac-af7f-3f1c50faf173" (UID: "1e03b6f6-d0a5-45ac-af7f-3f1c50faf173"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:34:36.792073 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:36.792041 4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173-istio-envoy\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:34:36.792073 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:36.792073 4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173-credential-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:34:36.792073 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:36.792083 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4p46z\" (UniqueName: \"kubernetes.io/projected/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173-kube-api-access-4p46z\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:34:36.792326 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:36.792092 4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173-istio-podinfo\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:34:36.792326 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:36.792101 4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173-istio-token\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:34:37.588595 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:37.588560 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd" Oct 30 17:34:37.662936 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:37.662898 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd"] Oct 30 17:34:37.673522 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:37.673489 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-c17e8c-openshift-default-5459f58b4f-xwzgd"] Oct 30 17:34:38.146892 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:38.146857 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e03b6f6-d0a5-45ac-af7f-3f1c50faf173" path="/var/lib/kubelet/pods/1e03b6f6-d0a5-45ac-af7f-3f1c50faf173/volumes" Oct 30 17:34:38.718126 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:38.718087 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8a7e4a-8qxbg"] Oct 30 17:34:38.719053 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:38.719028 4522 cpu_manager.go:414] "containerMap: removing container" podUID="1e03b6f6-d0a5-45ac-af7f-3f1c50faf173" containerName="istio-proxy" Oct 30 17:34:38.719053 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:38.719053 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e03b6f6-d0a5-45ac-af7f-3f1c50faf173" containerName="istio-proxy" Oct 30 17:34:38.719228 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:38.719137 4522 cpu_manager.go:414] "containerMap: removing container" podUID="50e2c4b5-f884-480d-a6ae-8f215ec9d392" containerName="revoke-keys" Oct 30 17:34:38.719228 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:38.719145 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50e2c4b5-f884-480d-a6ae-8f215ec9d392" containerName="revoke-keys" Oct 30 17:34:38.719366 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:38.719351 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e03b6f6-d0a5-45ac-af7f-3f1c50faf173" containerName="istio-proxy" Oct 30 17:34:38.719436 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:38.719376 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e03b6f6-d0a5-45ac-af7f-3f1c50faf173" containerName="istio-proxy" Oct 30 17:34:38.719436 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:38.719396 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="50e2c4b5-f884-480d-a6ae-8f215ec9d392" containerName="revoke-keys" Oct 30 17:34:38.724945 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:38.724914 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8a7e4a-8qxbg" Oct 30 17:34:38.743163 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:38.743135 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8a7e4a-8qxbg"] Oct 30 17:34:38.750913 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:38.750880 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:34:38.751143 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:38.751124 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:34:38.818940 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:38.818900 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3be4416f-abcb-4621-b1b7-b00c4181e0ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8a7e4a-8qxbg\" (UID: \"3be4416f-abcb-4621-b1b7-b00c4181e0ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8a7e4a-8qxbg" Oct 30 17:34:38.819153 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:38.819064 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdvhz\" (UniqueName: \"kubernetes.io/projected/3be4416f-abcb-4621-b1b7-b00c4181e0ce-kube-api-access-rdvhz\") pod \"maas-api-revoke-keys-e2e-ipp-a-8a7e4a-8qxbg\" (UID: \"3be4416f-abcb-4621-b1b7-b00c4181e0ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8a7e4a-8qxbg" Oct 30 17:34:38.819317 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:38.819284 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3be4416f-abcb-4621-b1b7-b00c4181e0ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8a7e4a-8qxbg\" (UID: \"3be4416f-abcb-4621-b1b7-b00c4181e0ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8a7e4a-8qxbg" Oct 30 17:34:38.921065 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:38.921025 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdvhz\" (UniqueName: \"kubernetes.io/projected/3be4416f-abcb-4621-b1b7-b00c4181e0ce-kube-api-access-rdvhz\") pod \"maas-api-revoke-keys-e2e-ipp-a-8a7e4a-8qxbg\" (UID: \"3be4416f-abcb-4621-b1b7-b00c4181e0ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8a7e4a-8qxbg" Oct 30 17:34:38.921251 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:38.921190 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3be4416f-abcb-4621-b1b7-b00c4181e0ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8a7e4a-8qxbg\" (UID: \"3be4416f-abcb-4621-b1b7-b00c4181e0ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8a7e4a-8qxbg" Oct 30 17:34:38.921251 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:38.921225 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3be4416f-abcb-4621-b1b7-b00c4181e0ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8a7e4a-8qxbg\" (UID: \"3be4416f-abcb-4621-b1b7-b00c4181e0ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8a7e4a-8qxbg" Oct 30 17:34:38.921953 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:38.921928 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3be4416f-abcb-4621-b1b7-b00c4181e0ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8a7e4a-8qxbg\" (UID: \"3be4416f-abcb-4621-b1b7-b00c4181e0ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8a7e4a-8qxbg" Oct 30 17:34:38.923569 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:38.923548 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3be4416f-abcb-4621-b1b7-b00c4181e0ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8a7e4a-8qxbg\" (UID: \"3be4416f-abcb-4621-b1b7-b00c4181e0ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8a7e4a-8qxbg" Oct 30 17:34:38.948784 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:38.948755 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdvhz\" (UniqueName: \"kubernetes.io/projected/3be4416f-abcb-4621-b1b7-b00c4181e0ce-kube-api-access-rdvhz\") pod \"maas-api-revoke-keys-e2e-ipp-a-8a7e4a-8qxbg\" (UID: \"3be4416f-abcb-4621-b1b7-b00c4181e0ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8a7e4a-8qxbg" Oct 30 17:34:39.039853 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:39.039769 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8a7e4a-8qxbg" Oct 30 17:34:39.349335 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:34:39.349283 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3be4416f_abcb_4621_b1b7_b00c4181e0ce.slice/crio-cb25d5f0a0122da9b7dae0975753ad4fb477f213b43cb713e44c6927480ca5f9 WatchSource:0}: Error finding container cb25d5f0a0122da9b7dae0975753ad4fb477f213b43cb713e44c6927480ca5f9: Status 404 returned error can't find the container with id cb25d5f0a0122da9b7dae0975753ad4fb477f213b43cb713e44c6927480ca5f9 Oct 30 17:34:39.350430 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:39.350409 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8a7e4a-8qxbg"] Oct 30 17:34:39.609947 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:39.609918 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8a7e4a-8qxbg" event={"ID":"3be4416f-abcb-4621-b1b7-b00c4181e0ce","Type":"ContainerStarted","Data":"cb25d5f0a0122da9b7dae0975753ad4fb477f213b43cb713e44c6927480ca5f9"} Oct 30 17:34:40.620245 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:40.620208 4522 generic.go:356] "Generic (PLEG): container finished" podID="3be4416f-abcb-4621-b1b7-b00c4181e0ce" containerID="4593c955277d68961d927e44192f9f7414aba43357ed2fe6dbd7b096e8d40d8a" exitCode=0 Oct 30 17:34:40.620689 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:40.620247 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8a7e4a-8qxbg" event={"ID":"3be4416f-abcb-4621-b1b7-b00c4181e0ce","Type":"ContainerDied","Data":"4593c955277d68961d927e44192f9f7414aba43357ed2fe6dbd7b096e8d40d8a"} Oct 30 17:34:41.805127 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:41.805102 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8a7e4a-8qxbg" Oct 30 17:34:41.859831 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:41.859801 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3be4416f-abcb-4621-b1b7-b00c4181e0ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3be4416f-abcb-4621-b1b7-b00c4181e0ce-maas-api-xks-ca\") pod \"3be4416f-abcb-4621-b1b7-b00c4181e0ce\" (UID: \"3be4416f-abcb-4621-b1b7-b00c4181e0ce\") " Oct 30 17:34:41.860045 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:41.859851 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3be4416f-abcb-4621-b1b7-b00c4181e0ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3be4416f-abcb-4621-b1b7-b00c4181e0ce-maas-api-openshift-ca\") pod \"3be4416f-abcb-4621-b1b7-b00c4181e0ce\" (UID: \"3be4416f-abcb-4621-b1b7-b00c4181e0ce\") " Oct 30 17:34:41.860045 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:41.860003 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3be4416f-abcb-4621-b1b7-b00c4181e0ce-kube-api-access-rdvhz\" (UniqueName: \"kubernetes.io/projected/3be4416f-abcb-4621-b1b7-b00c4181e0ce-kube-api-access-rdvhz\") pod \"3be4416f-abcb-4621-b1b7-b00c4181e0ce\" (UID: \"3be4416f-abcb-4621-b1b7-b00c4181e0ce\") " Oct 30 17:34:41.860655 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:41.860626 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3be4416f-abcb-4621-b1b7-b00c4181e0ce-maas-api-openshift-ca" pod "3be4416f-abcb-4621-b1b7-b00c4181e0ce" (UID: "3be4416f-abcb-4621-b1b7-b00c4181e0ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:34:41.863104 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:41.863044 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3be4416f-abcb-4621-b1b7-b00c4181e0ce-maas-api-xks-ca" pod "3be4416f-abcb-4621-b1b7-b00c4181e0ce" (UID: "3be4416f-abcb-4621-b1b7-b00c4181e0ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:34:41.869530 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:41.869495 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3be4416f-abcb-4621-b1b7-b00c4181e0ce-kube-api-access-rdvhz" pod "3be4416f-abcb-4621-b1b7-b00c4181e0ce" (UID: "3be4416f-abcb-4621-b1b7-b00c4181e0ce"). InnerVolumeSpecName "kube-api-access-rdvhz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:34:41.962528 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:41.962497 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3be4416f-abcb-4621-b1b7-b00c4181e0ce-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:34:41.962528 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:41.962525 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3be4416f-abcb-4621-b1b7-b00c4181e0ce-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:34:41.962734 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:41.962536 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdvhz\" (UniqueName: \"kubernetes.io/projected/3be4416f-abcb-4621-b1b7-b00c4181e0ce-kube-api-access-rdvhz\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:34:42.637777 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:42.637745 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8a7e4a-8qxbg" Oct 30 17:34:42.637988 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:42.637744 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8a7e4a-8qxbg" event={"ID":"3be4416f-abcb-4621-b1b7-b00c4181e0ce","Type":"ContainerDied","Data":"cb25d5f0a0122da9b7dae0975753ad4fb477f213b43cb713e44c6927480ca5f9"} Oct 30 17:34:42.637988 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:42.637874 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb25d5f0a0122da9b7dae0975753ad4fb477f213b43cb713e44c6927480ca5f9" Oct 30 17:34:42.950652 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:34:42.950569 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:34:44.150500 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:44.150470 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7b9df-59f556c645-8f7kp"] Oct 30 17:34:44.150941 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:44.150658 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7b9df-59f556c645-8f7kp" podUID="6f5a1cdd-9232-496b-8838-a21ec0cc78d3" containerName="maas-api" containerID="cri-o://73109e2519e55e28124cafb358ca74d3945a6bb45bbdf44481920314cf2ee263" gracePeriod=30 Oct 30 17:34:44.485961 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:44.485933 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7b9df-59f556c645-8f7kp" Oct 30 17:34:44.506801 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:44.506770 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f5a1cdd-9232-496b-8838-a21ec0cc78d3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f5a1cdd-9232-496b-8838-a21ec0cc78d3-maas-api-tls\") pod \"6f5a1cdd-9232-496b-8838-a21ec0cc78d3\" (UID: \"6f5a1cdd-9232-496b-8838-a21ec0cc78d3\") " Oct 30 17:34:44.506953 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:44.506808 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f5a1cdd-9232-496b-8838-a21ec0cc78d3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f5a1cdd-9232-496b-8838-a21ec0cc78d3-openshift-service-ca\") pod \"6f5a1cdd-9232-496b-8838-a21ec0cc78d3\" (UID: \"6f5a1cdd-9232-496b-8838-a21ec0cc78d3\") " Oct 30 17:34:44.506953 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:44.506893 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f5a1cdd-9232-496b-8838-a21ec0cc78d3-kube-api-access-x4xd2\" (UniqueName: \"kubernetes.io/projected/6f5a1cdd-9232-496b-8838-a21ec0cc78d3-kube-api-access-x4xd2\") pod \"6f5a1cdd-9232-496b-8838-a21ec0cc78d3\" (UID: \"6f5a1cdd-9232-496b-8838-a21ec0cc78d3\") " Oct 30 17:34:44.507043 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:44.506957 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f5a1cdd-9232-496b-8838-a21ec0cc78d3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f5a1cdd-9232-496b-8838-a21ec0cc78d3-metrics-certs\") pod \"6f5a1cdd-9232-496b-8838-a21ec0cc78d3\" (UID: \"6f5a1cdd-9232-496b-8838-a21ec0cc78d3\") " Oct 30 17:34:44.507231 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:44.507203 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f5a1cdd-9232-496b-8838-a21ec0cc78d3-openshift-service-ca" pod "6f5a1cdd-9232-496b-8838-a21ec0cc78d3" (UID: "6f5a1cdd-9232-496b-8838-a21ec0cc78d3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:34:44.507447 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:44.507428 4522 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f5a1cdd-9232-496b-8838-a21ec0cc78d3-openshift-service-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:34:44.508979 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:44.508960 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f5a1cdd-9232-496b-8838-a21ec0cc78d3-maas-api-tls" pod "6f5a1cdd-9232-496b-8838-a21ec0cc78d3" (UID: "6f5a1cdd-9232-496b-8838-a21ec0cc78d3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:34:44.509493 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:44.509473 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f5a1cdd-9232-496b-8838-a21ec0cc78d3-metrics-certs" pod "6f5a1cdd-9232-496b-8838-a21ec0cc78d3" (UID: "6f5a1cdd-9232-496b-8838-a21ec0cc78d3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:34:44.509573 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:44.509530 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f5a1cdd-9232-496b-8838-a21ec0cc78d3-kube-api-access-x4xd2" pod "6f5a1cdd-9232-496b-8838-a21ec0cc78d3" (UID: "6f5a1cdd-9232-496b-8838-a21ec0cc78d3"). InnerVolumeSpecName "kube-api-access-x4xd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:34:44.626340 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:44.624510 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f5a1cdd-9232-496b-8838-a21ec0cc78d3-maas-api-tls\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:34:44.626340 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:44.624551 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4xd2\" (UniqueName: \"kubernetes.io/projected/6f5a1cdd-9232-496b-8838-a21ec0cc78d3-kube-api-access-x4xd2\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:34:44.626340 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:44.624566 4522 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f5a1cdd-9232-496b-8838-a21ec0cc78d3-metrics-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:34:44.665328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:44.658300 4522 generic.go:356] "Generic (PLEG): container finished" podID="6f5a1cdd-9232-496b-8838-a21ec0cc78d3" containerID="73109e2519e55e28124cafb358ca74d3945a6bb45bbdf44481920314cf2ee263" exitCode=0 Oct 30 17:34:44.665328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:44.658373 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7b9df-59f556c645-8f7kp" Oct 30 17:34:44.665328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:44.658368 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7b9df-59f556c645-8f7kp" event={"ID":"6f5a1cdd-9232-496b-8838-a21ec0cc78d3","Type":"ContainerDied","Data":"73109e2519e55e28124cafb358ca74d3945a6bb45bbdf44481920314cf2ee263"} Oct 30 17:34:44.665328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:44.658747 4522 scope.go:122] "RemoveContainer" containerID="73109e2519e55e28124cafb358ca74d3945a6bb45bbdf44481920314cf2ee263" Oct 30 17:34:44.665328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:44.658834 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7b9df-59f556c645-8f7kp" event={"ID":"6f5a1cdd-9232-496b-8838-a21ec0cc78d3","Type":"ContainerDied","Data":"1caf7508178b34c2925dd7f7536ba16739159f30243618b63f13d4e6d856d4dd"} Oct 30 17:34:44.675561 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:44.675538 4522 scope.go:122] "RemoveContainer" containerID="73109e2519e55e28124cafb358ca74d3945a6bb45bbdf44481920314cf2ee263" Oct 30 17:34:44.675898 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:34:44.675879 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73109e2519e55e28124cafb358ca74d3945a6bb45bbdf44481920314cf2ee263\": container with ID starting with 73109e2519e55e28124cafb358ca74d3945a6bb45bbdf44481920314cf2ee263 not found: ID does not exist" containerID="73109e2519e55e28124cafb358ca74d3945a6bb45bbdf44481920314cf2ee263" Oct 30 17:34:44.675956 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:44.675908 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73109e2519e55e28124cafb358ca74d3945a6bb45bbdf44481920314cf2ee263"} err="failed to get container status \"73109e2519e55e28124cafb358ca74d3945a6bb45bbdf44481920314cf2ee263\": rpc error: code = NotFound desc = could not find container \"73109e2519e55e28124cafb358ca74d3945a6bb45bbdf44481920314cf2ee263\": container with ID starting with 73109e2519e55e28124cafb358ca74d3945a6bb45bbdf44481920314cf2ee263 not found: ID does not exist" Oct 30 17:34:44.717932 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:44.717849 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7b9df-59f556c645-8f7kp"] Oct 30 17:34:44.721759 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:44.721737 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7b9df-59f556c645-8f7kp"] Oct 30 17:34:46.146450 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:46.146412 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f5a1cdd-9232-496b-8838-a21ec0cc78d3" path="/var/lib/kubelet/pods/6f5a1cdd-9232-496b-8838-a21ec0cc78d3/volumes" Oct 30 17:34:50.673079 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:50.673042 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-11f480-f4b24"] Oct 30 17:34:50.675884 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:50.673798 4522 cpu_manager.go:414] "containerMap: removing container" podUID="3be4416f-abcb-4621-b1b7-b00c4181e0ce" containerName="revoke-keys" Oct 30 17:34:50.675884 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:50.673811 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3be4416f-abcb-4621-b1b7-b00c4181e0ce" containerName="revoke-keys" Oct 30 17:34:50.675884 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:50.673826 4522 cpu_manager.go:414] "containerMap: removing container" podUID="6f5a1cdd-9232-496b-8838-a21ec0cc78d3" containerName="maas-api" Oct 30 17:34:50.675884 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:50.673831 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f5a1cdd-9232-496b-8838-a21ec0cc78d3" containerName="maas-api" Oct 30 17:34:50.675884 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:50.673856 4522 cpu_manager.go:414] "containerMap: removing container" podUID="1e03b6f6-d0a5-45ac-af7f-3f1c50faf173" containerName="istio-proxy" Oct 30 17:34:50.675884 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:50.673861 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e03b6f6-d0a5-45ac-af7f-3f1c50faf173" containerName="istio-proxy" Oct 30 17:34:50.675884 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:50.673977 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="3be4416f-abcb-4621-b1b7-b00c4181e0ce" containerName="revoke-keys" Oct 30 17:34:50.675884 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:50.673987 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f5a1cdd-9232-496b-8838-a21ec0cc78d3" containerName="maas-api" Oct 30 17:34:50.677608 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:50.677591 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-11f480-f4b24" Oct 30 17:34:50.699317 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:50.699277 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-11f480-f4b24"] Oct 30 17:34:50.715980 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:50.715947 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:34:50.716277 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:50.716260 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:34:50.719202 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:50.719176 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cfef672-8038-43d3-bb96-6ee9793abd6d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-11f480-f4b24\" (UID: \"4cfef672-8038-43d3-bb96-6ee9793abd6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-11f480-f4b24" Oct 30 17:34:50.719510 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:50.719479 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cfef672-8038-43d3-bb96-6ee9793abd6d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-11f480-f4b24\" (UID: \"4cfef672-8038-43d3-bb96-6ee9793abd6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-11f480-f4b24" Oct 30 17:34:50.719641 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:50.719577 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sclg\" (UniqueName: \"kubernetes.io/projected/4cfef672-8038-43d3-bb96-6ee9793abd6d-kube-api-access-8sclg\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-11f480-f4b24\" (UID: \"4cfef672-8038-43d3-bb96-6ee9793abd6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-11f480-f4b24" Oct 30 17:34:50.820845 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:50.820811 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cfef672-8038-43d3-bb96-6ee9793abd6d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-11f480-f4b24\" (UID: \"4cfef672-8038-43d3-bb96-6ee9793abd6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-11f480-f4b24" Oct 30 17:34:50.821056 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:50.820883 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sclg\" (UniqueName: \"kubernetes.io/projected/4cfef672-8038-43d3-bb96-6ee9793abd6d-kube-api-access-8sclg\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-11f480-f4b24\" (UID: \"4cfef672-8038-43d3-bb96-6ee9793abd6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-11f480-f4b24" Oct 30 17:34:50.821056 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:50.821030 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cfef672-8038-43d3-bb96-6ee9793abd6d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-11f480-f4b24\" (UID: \"4cfef672-8038-43d3-bb96-6ee9793abd6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-11f480-f4b24" Oct 30 17:34:50.821733 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:50.821705 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cfef672-8038-43d3-bb96-6ee9793abd6d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-11f480-f4b24\" (UID: \"4cfef672-8038-43d3-bb96-6ee9793abd6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-11f480-f4b24" Oct 30 17:34:50.823318 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:50.823286 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cfef672-8038-43d3-bb96-6ee9793abd6d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-11f480-f4b24\" (UID: \"4cfef672-8038-43d3-bb96-6ee9793abd6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-11f480-f4b24" Oct 30 17:34:50.846454 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:50.846425 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sclg\" (UniqueName: \"kubernetes.io/projected/4cfef672-8038-43d3-bb96-6ee9793abd6d-kube-api-access-8sclg\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-11f480-f4b24\" (UID: \"4cfef672-8038-43d3-bb96-6ee9793abd6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-11f480-f4b24" Oct 30 17:34:50.992383 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:50.992352 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-11f480-f4b24" Oct 30 17:34:51.308710 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:34:51.308678 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cfef672_8038_43d3_bb96_6ee9793abd6d.slice/crio-bbe52f95aba7c259351e268e3e879a3a55025e7f72c226ee7697cd966bb812ea WatchSource:0}: Error finding container bbe52f95aba7c259351e268e3e879a3a55025e7f72c226ee7697cd966bb812ea: Status 404 returned error can't find the container with id bbe52f95aba7c259351e268e3e879a3a55025e7f72c226ee7697cd966bb812ea Oct 30 17:34:51.310980 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:51.310957 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-11f480-f4b24"] Oct 30 17:34:51.343429 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:51.343394 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a7e4a-77547b685c-pm9ms"] Oct 30 17:34:51.343645 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:51.343619 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a7e4a-77547b685c-pm9ms" podUID="c633f97f-ac84-44f1-a029-75ead2b39058" containerName="maas-api" containerID="cri-o://7ec8a8f4fc63cebc2ceee27c90565f761a7550d0d1f63f0a8e8dfcfac278044a" gracePeriod=30 Oct 30 17:34:51.516804 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:51.516769 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8a7e4a-66cd879994-drkhg"] Oct 30 17:34:51.516992 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:51.516971 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-8a7e4a-66cd879994-drkhg" podUID="2abedac3-fc80-4dcf-9f92-c6aefa1a2cf7" containerName="payload-processing" containerID="cri-o://bc4f2034dccfe0f060278b751d23e73ba8968d4c63e1a2aea11c6ddc24c438a6" gracePeriod=30 Oct 30 17:34:51.550918 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:51.550875 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8a7e4a-5d55c87fcf-qf6bc"] Oct 30 17:34:51.551139 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:51.551114 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8a7e4a-5d55c87fcf-qf6bc" podUID="fcfedbe8-98e7-479a-8f7b-f8e6d1be8b1a" containerName="payload-pre-processing" containerID="cri-o://adfaa78121d678486c4a401fadaf11a38f6616f0258f7a83f623b13d8bf37292" gracePeriod=30 Oct 30 17:34:51.701688 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:51.701389 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a7e4a-77547b685c-pm9ms" Oct 30 17:34:51.721074 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:51.721032 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-11f480-f4b24" event={"ID":"4cfef672-8038-43d3-bb96-6ee9793abd6d","Type":"ContainerStarted","Data":"bbe52f95aba7c259351e268e3e879a3a55025e7f72c226ee7697cd966bb812ea"} Oct 30 17:34:51.722739 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:51.722707 4522 generic.go:356] "Generic (PLEG): container finished" podID="c633f97f-ac84-44f1-a029-75ead2b39058" containerID="7ec8a8f4fc63cebc2ceee27c90565f761a7550d0d1f63f0a8e8dfcfac278044a" exitCode=0 Oct 30 17:34:51.722888 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:51.722785 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a7e4a-77547b685c-pm9ms" event={"ID":"c633f97f-ac84-44f1-a029-75ead2b39058","Type":"ContainerDied","Data":"7ec8a8f4fc63cebc2ceee27c90565f761a7550d0d1f63f0a8e8dfcfac278044a"} Oct 30 17:34:51.722888 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:51.722829 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a7e4a-77547b685c-pm9ms" event={"ID":"c633f97f-ac84-44f1-a029-75ead2b39058","Type":"ContainerDied","Data":"cb914cf7752111d263cd5d47d0c9cbf9f53c38039edfadbb7680e582fc47a984"} Oct 30 17:34:51.722888 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:51.722860 4522 scope.go:122] "RemoveContainer" containerID="7ec8a8f4fc63cebc2ceee27c90565f761a7550d0d1f63f0a8e8dfcfac278044a" Oct 30 17:34:51.723038 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:51.722957 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a7e4a-77547b685c-pm9ms" Oct 30 17:34:51.725174 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:51.725152 4522 generic.go:356] "Generic (PLEG): container finished" podID="2abedac3-fc80-4dcf-9f92-c6aefa1a2cf7" containerID="bc4f2034dccfe0f060278b751d23e73ba8968d4c63e1a2aea11c6ddc24c438a6" exitCode=0 Oct 30 17:34:51.726114 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:51.725333 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8a7e4a-66cd879994-drkhg" event={"ID":"2abedac3-fc80-4dcf-9f92-c6aefa1a2cf7","Type":"ContainerDied","Data":"bc4f2034dccfe0f060278b751d23e73ba8968d4c63e1a2aea11c6ddc24c438a6"} Oct 30 17:34:51.727856 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:51.727825 4522 generic.go:356] "Generic (PLEG): container finished" podID="fcfedbe8-98e7-479a-8f7b-f8e6d1be8b1a" containerID="adfaa78121d678486c4a401fadaf11a38f6616f0258f7a83f623b13d8bf37292" exitCode=0 Oct 30 17:34:51.727977 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:51.727848 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8a7e4a-5d55c87fcf-qf6bc" event={"ID":"fcfedbe8-98e7-479a-8f7b-f8e6d1be8b1a","Type":"ContainerDied","Data":"adfaa78121d678486c4a401fadaf11a38f6616f0258f7a83f623b13d8bf37292"} Oct 30 17:34:51.735080 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:51.735052 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c633f97f-ac84-44f1-a029-75ead2b39058-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c633f97f-ac84-44f1-a029-75ead2b39058-metrics-certs\") pod \"c633f97f-ac84-44f1-a029-75ead2b39058\" (UID: \"c633f97f-ac84-44f1-a029-75ead2b39058\") " Oct 30 17:34:51.735194 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:51.735124 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c633f97f-ac84-44f1-a029-75ead2b39058-kube-api-access-cf78z\" (UniqueName: \"kubernetes.io/projected/c633f97f-ac84-44f1-a029-75ead2b39058-kube-api-access-cf78z\") pod \"c633f97f-ac84-44f1-a029-75ead2b39058\" (UID: \"c633f97f-ac84-44f1-a029-75ead2b39058\") " Oct 30 17:34:51.735569 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:51.735544 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c633f97f-ac84-44f1-a029-75ead2b39058-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c633f97f-ac84-44f1-a029-75ead2b39058-openshift-service-ca\") pod \"c633f97f-ac84-44f1-a029-75ead2b39058\" (UID: \"c633f97f-ac84-44f1-a029-75ead2b39058\") " Oct 30 17:34:51.735671 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:51.735596 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c633f97f-ac84-44f1-a029-75ead2b39058-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c633f97f-ac84-44f1-a029-75ead2b39058-maas-api-tls\") pod \"c633f97f-ac84-44f1-a029-75ead2b39058\" (UID: \"c633f97f-ac84-44f1-a029-75ead2b39058\") " Oct 30 17:34:51.737251 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:51.737223 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c633f97f-ac84-44f1-a029-75ead2b39058-openshift-service-ca" pod "c633f97f-ac84-44f1-a029-75ead2b39058" (UID: "c633f97f-ac84-44f1-a029-75ead2b39058"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:34:51.743816 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:51.743795 4522 scope.go:122] "RemoveContainer" containerID="7ec8a8f4fc63cebc2ceee27c90565f761a7550d0d1f63f0a8e8dfcfac278044a" Oct 30 17:34:51.744170 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:34:51.744150 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ec8a8f4fc63cebc2ceee27c90565f761a7550d0d1f63f0a8e8dfcfac278044a\": container with ID starting with 7ec8a8f4fc63cebc2ceee27c90565f761a7550d0d1f63f0a8e8dfcfac278044a not found: ID does not exist" containerID="7ec8a8f4fc63cebc2ceee27c90565f761a7550d0d1f63f0a8e8dfcfac278044a" Oct 30 17:34:51.744218 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:51.744179 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ec8a8f4fc63cebc2ceee27c90565f761a7550d0d1f63f0a8e8dfcfac278044a"} err="failed to get container status \"7ec8a8f4fc63cebc2ceee27c90565f761a7550d0d1f63f0a8e8dfcfac278044a\": rpc error: code = NotFound desc = could not find container \"7ec8a8f4fc63cebc2ceee27c90565f761a7550d0d1f63f0a8e8dfcfac278044a\": container with ID starting with 7ec8a8f4fc63cebc2ceee27c90565f761a7550d0d1f63f0a8e8dfcfac278044a not found: ID does not exist" Oct 30 17:34:51.792221 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:51.792133 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c633f97f-ac84-44f1-a029-75ead2b39058-metrics-certs" pod "c633f97f-ac84-44f1-a029-75ead2b39058" (UID: "c633f97f-ac84-44f1-a029-75ead2b39058"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:34:51.792712 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:51.792672 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c633f97f-ac84-44f1-a029-75ead2b39058-maas-api-tls" pod "c633f97f-ac84-44f1-a029-75ead2b39058" (UID: "c633f97f-ac84-44f1-a029-75ead2b39058"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:34:51.793301 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:51.793209 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c633f97f-ac84-44f1-a029-75ead2b39058-kube-api-access-cf78z" pod "c633f97f-ac84-44f1-a029-75ead2b39058" (UID: "c633f97f-ac84-44f1-a029-75ead2b39058"). InnerVolumeSpecName "kube-api-access-cf78z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:34:51.837461 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:51.837431 4522 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c633f97f-ac84-44f1-a029-75ead2b39058-openshift-service-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:34:51.837461 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:51.837459 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c633f97f-ac84-44f1-a029-75ead2b39058-maas-api-tls\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:34:51.837461 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:51.837468 4522 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c633f97f-ac84-44f1-a029-75ead2b39058-metrics-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:34:51.837693 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:51.837476 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cf78z\" (UniqueName: \"kubernetes.io/projected/c633f97f-ac84-44f1-a029-75ead2b39058-kube-api-access-cf78z\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:34:52.039365 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:52.014272 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8a7e4a-66cd879994-drkhg" Oct 30 17:34:52.071360 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:52.071219 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2abedac3-fc80-4dcf-9f92-c6aefa1a2cf7-kube-api-access-rfwlq\" (UniqueName: \"kubernetes.io/projected/2abedac3-fc80-4dcf-9f92-c6aefa1a2cf7-kube-api-access-rfwlq\") pod \"2abedac3-fc80-4dcf-9f92-c6aefa1a2cf7\" (UID: \"2abedac3-fc80-4dcf-9f92-c6aefa1a2cf7\") " Oct 30 17:34:52.071658 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:52.071630 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2abedac3-fc80-4dcf-9f92-c6aefa1a2cf7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2abedac3-fc80-4dcf-9f92-c6aefa1a2cf7-plugins-config-volume\") pod \"2abedac3-fc80-4dcf-9f92-c6aefa1a2cf7\" (UID: \"2abedac3-fc80-4dcf-9f92-c6aefa1a2cf7\") " Oct 30 17:34:52.071928 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:52.071906 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2abedac3-fc80-4dcf-9f92-c6aefa1a2cf7-plugins-config-volume" pod "2abedac3-fc80-4dcf-9f92-c6aefa1a2cf7" (UID: "2abedac3-fc80-4dcf-9f92-c6aefa1a2cf7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:34:52.072343 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:52.072323 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2abedac3-fc80-4dcf-9f92-c6aefa1a2cf7-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:34:52.073733 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:52.073714 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2abedac3-fc80-4dcf-9f92-c6aefa1a2cf7-kube-api-access-rfwlq" pod "2abedac3-fc80-4dcf-9f92-c6aefa1a2cf7" (UID: "2abedac3-fc80-4dcf-9f92-c6aefa1a2cf7"). InnerVolumeSpecName "kube-api-access-rfwlq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:34:52.094553 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:52.094525 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8a7e4a-5d55c87fcf-qf6bc" Oct 30 17:34:52.149776 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:52.149752 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a7e4a-77547b685c-pm9ms"] Oct 30 17:34:52.149963 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:52.149786 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a7e4a-77547b685c-pm9ms"] Oct 30 17:34:52.173785 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:52.173754 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcfedbe8-98e7-479a-8f7b-f8e6d1be8b1a-kube-api-access-g5rkg\" (UniqueName: \"kubernetes.io/projected/fcfedbe8-98e7-479a-8f7b-f8e6d1be8b1a-kube-api-access-g5rkg\") pod \"fcfedbe8-98e7-479a-8f7b-f8e6d1be8b1a\" (UID: \"fcfedbe8-98e7-479a-8f7b-f8e6d1be8b1a\") " Oct 30 17:34:52.173936 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:52.173798 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcfedbe8-98e7-479a-8f7b-f8e6d1be8b1a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcfedbe8-98e7-479a-8f7b-f8e6d1be8b1a-plugins-config-volume\") pod \"fcfedbe8-98e7-479a-8f7b-f8e6d1be8b1a\" (UID: \"fcfedbe8-98e7-479a-8f7b-f8e6d1be8b1a\") " Oct 30 17:34:52.174200 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:52.174180 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfwlq\" (UniqueName: \"kubernetes.io/projected/2abedac3-fc80-4dcf-9f92-c6aefa1a2cf7-kube-api-access-rfwlq\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:34:52.174340 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:52.174205 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcfedbe8-98e7-479a-8f7b-f8e6d1be8b1a-plugins-config-volume" pod "fcfedbe8-98e7-479a-8f7b-f8e6d1be8b1a" (UID: "fcfedbe8-98e7-479a-8f7b-f8e6d1be8b1a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:34:52.176038 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:52.176016 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcfedbe8-98e7-479a-8f7b-f8e6d1be8b1a-kube-api-access-g5rkg" pod "fcfedbe8-98e7-479a-8f7b-f8e6d1be8b1a" (UID: "fcfedbe8-98e7-479a-8f7b-f8e6d1be8b1a"). InnerVolumeSpecName "kube-api-access-g5rkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:34:52.285373 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:52.284629 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5rkg\" (UniqueName: \"kubernetes.io/projected/fcfedbe8-98e7-479a-8f7b-f8e6d1be8b1a-kube-api-access-g5rkg\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:34:52.285373 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:52.284666 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcfedbe8-98e7-479a-8f7b-f8e6d1be8b1a-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:34:52.451432 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:52.451392 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8a7e4a-8qxbg"] Oct 30 17:34:52.455126 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:52.455101 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8a7e4a-8qxbg"] Oct 30 17:34:52.738964 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:52.738928 4522 generic.go:356] "Generic (PLEG): container finished" podID="4cfef672-8038-43d3-bb96-6ee9793abd6d" containerID="d68ecf395852633b0f0efe526aec7bdb11348fc5b3016510484b195a6d78ecdd" exitCode=0 Oct 30 17:34:52.739483 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:52.739006 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-11f480-f4b24" event={"ID":"4cfef672-8038-43d3-bb96-6ee9793abd6d","Type":"ContainerDied","Data":"d68ecf395852633b0f0efe526aec7bdb11348fc5b3016510484b195a6d78ecdd"} Oct 30 17:34:52.741676 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:52.741657 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8a7e4a-66cd879994-drkhg" Oct 30 17:34:52.741770 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:52.741675 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8a7e4a-66cd879994-drkhg" event={"ID":"2abedac3-fc80-4dcf-9f92-c6aefa1a2cf7","Type":"ContainerDied","Data":"b3e03a058d23fd4063ca377b5c42499ffa3143dd1d2c4b7d4ede6f035534955a"} Oct 30 17:34:52.741770 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:52.741732 4522 scope.go:122] "RemoveContainer" containerID="bc4f2034dccfe0f060278b751d23e73ba8968d4c63e1a2aea11c6ddc24c438a6" Oct 30 17:34:52.743236 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:52.743209 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8a7e4a-5d55c87fcf-qf6bc" Oct 30 17:34:52.743236 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:52.743227 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8a7e4a-5d55c87fcf-qf6bc" event={"ID":"fcfedbe8-98e7-479a-8f7b-f8e6d1be8b1a","Type":"ContainerDied","Data":"a851f33e2ba6cab8c84b1e3c0f847608aa0de0f3956b115c89ec6986accc7757"} Oct 30 17:34:52.755713 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:52.755694 4522 scope.go:122] "RemoveContainer" containerID="adfaa78121d678486c4a401fadaf11a38f6616f0258f7a83f623b13d8bf37292" Oct 30 17:34:52.827101 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:52.827066 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8a7e4a-5d55c87fcf-qf6bc"] Oct 30 17:34:52.831934 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:52.831910 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8a7e4a-5d55c87fcf-qf6bc"] Oct 30 17:34:52.880537 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:52.880503 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8a7e4a-66cd879994-drkhg"] Oct 30 17:34:52.884621 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:52.884592 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8a7e4a-66cd879994-drkhg"] Oct 30 17:34:53.915011 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:53.914982 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-11f480-f4b24" Oct 30 17:34:54.034488 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:54.034403 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cfef672-8038-43d3-bb96-6ee9793abd6d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cfef672-8038-43d3-bb96-6ee9793abd6d-maas-api-openshift-ca\") pod \"4cfef672-8038-43d3-bb96-6ee9793abd6d\" (UID: \"4cfef672-8038-43d3-bb96-6ee9793abd6d\") " Oct 30 17:34:54.034635 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:54.034544 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cfef672-8038-43d3-bb96-6ee9793abd6d-kube-api-access-8sclg\" (UniqueName: \"kubernetes.io/projected/4cfef672-8038-43d3-bb96-6ee9793abd6d-kube-api-access-8sclg\") pod \"4cfef672-8038-43d3-bb96-6ee9793abd6d\" (UID: \"4cfef672-8038-43d3-bb96-6ee9793abd6d\") " Oct 30 17:34:54.034741 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:54.034722 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cfef672-8038-43d3-bb96-6ee9793abd6d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cfef672-8038-43d3-bb96-6ee9793abd6d-maas-api-xks-ca\") pod \"4cfef672-8038-43d3-bb96-6ee9793abd6d\" (UID: \"4cfef672-8038-43d3-bb96-6ee9793abd6d\") " Oct 30 17:34:54.034838 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:54.034819 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cfef672-8038-43d3-bb96-6ee9793abd6d-maas-api-openshift-ca" pod "4cfef672-8038-43d3-bb96-6ee9793abd6d" (UID: "4cfef672-8038-43d3-bb96-6ee9793abd6d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:34:54.035267 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:54.035243 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cfef672-8038-43d3-bb96-6ee9793abd6d-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:34:54.036713 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:54.036681 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cfef672-8038-43d3-bb96-6ee9793abd6d-kube-api-access-8sclg" pod "4cfef672-8038-43d3-bb96-6ee9793abd6d" (UID: "4cfef672-8038-43d3-bb96-6ee9793abd6d"). InnerVolumeSpecName "kube-api-access-8sclg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:34:54.036825 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:54.036777 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cfef672-8038-43d3-bb96-6ee9793abd6d-maas-api-xks-ca" pod "4cfef672-8038-43d3-bb96-6ee9793abd6d" (UID: "4cfef672-8038-43d3-bb96-6ee9793abd6d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:34:54.136990 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:54.136957 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cfef672-8038-43d3-bb96-6ee9793abd6d-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:34:54.136990 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:54.136988 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sclg\" (UniqueName: \"kubernetes.io/projected/4cfef672-8038-43d3-bb96-6ee9793abd6d-kube-api-access-8sclg\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:34:54.146191 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:54.146161 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2abedac3-fc80-4dcf-9f92-c6aefa1a2cf7" path="/var/lib/kubelet/pods/2abedac3-fc80-4dcf-9f92-c6aefa1a2cf7/volumes" Oct 30 17:34:54.146667 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:54.146653 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3be4416f-abcb-4621-b1b7-b00c4181e0ce" path="/var/lib/kubelet/pods/3be4416f-abcb-4621-b1b7-b00c4181e0ce/volumes" Oct 30 17:34:54.147071 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:54.147057 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c633f97f-ac84-44f1-a029-75ead2b39058" path="/var/lib/kubelet/pods/c633f97f-ac84-44f1-a029-75ead2b39058/volumes" Oct 30 17:34:54.147771 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:54.147754 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcfedbe8-98e7-479a-8f7b-f8e6d1be8b1a" path="/var/lib/kubelet/pods/fcfedbe8-98e7-479a-8f7b-f8e6d1be8b1a/volumes" Oct 30 17:34:54.761503 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:54.761472 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-11f480-f4b24" Oct 30 17:34:54.761503 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:54.761494 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-11f480-f4b24" event={"ID":"4cfef672-8038-43d3-bb96-6ee9793abd6d","Type":"ContainerDied","Data":"bbe52f95aba7c259351e268e3e879a3a55025e7f72c226ee7697cd966bb812ea"} Oct 30 17:34:54.761733 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:34:54.761542 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbe52f95aba7c259351e268e3e879a3a55025e7f72c226ee7697cd966bb812ea" Oct 30 17:35:00.678428 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:00.678384 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4fb821-mqfs9"] Oct 30 17:35:00.679108 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:00.679093 4522 cpu_manager.go:414] "containerMap: removing container" podUID="c633f97f-ac84-44f1-a029-75ead2b39058" containerName="maas-api" Oct 30 17:35:00.679159 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:00.679110 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c633f97f-ac84-44f1-a029-75ead2b39058" containerName="maas-api" Oct 30 17:35:00.679159 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:00.679141 4522 cpu_manager.go:414] "containerMap: removing container" podUID="2abedac3-fc80-4dcf-9f92-c6aefa1a2cf7" containerName="payload-processing" Oct 30 17:35:00.679159 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:00.679147 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2abedac3-fc80-4dcf-9f92-c6aefa1a2cf7" containerName="payload-processing" Oct 30 17:35:00.679267 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:00.679161 4522 cpu_manager.go:414] "containerMap: removing container" podUID="4cfef672-8038-43d3-bb96-6ee9793abd6d" containerName="revoke-keys" Oct 30 17:35:00.679267 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:00.679167 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cfef672-8038-43d3-bb96-6ee9793abd6d" containerName="revoke-keys" Oct 30 17:35:00.679267 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:00.679174 4522 cpu_manager.go:414] "containerMap: removing container" podUID="fcfedbe8-98e7-479a-8f7b-f8e6d1be8b1a" containerName="payload-pre-processing" Oct 30 17:35:00.679267 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:00.679179 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcfedbe8-98e7-479a-8f7b-f8e6d1be8b1a" containerName="payload-pre-processing" Oct 30 17:35:00.679427 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:00.679286 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="2abedac3-fc80-4dcf-9f92-c6aefa1a2cf7" containerName="payload-processing" Oct 30 17:35:00.679427 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:00.679297 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cfef672-8038-43d3-bb96-6ee9793abd6d" containerName="revoke-keys" Oct 30 17:35:00.679427 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:00.679318 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcfedbe8-98e7-479a-8f7b-f8e6d1be8b1a" containerName="payload-pre-processing" Oct 30 17:35:00.679427 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:00.679331 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="c633f97f-ac84-44f1-a029-75ead2b39058" containerName="maas-api" Oct 30 17:35:00.684558 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:00.684539 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4fb821-mqfs9" Oct 30 17:35:00.703252 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:00.703225 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4fb821-mqfs9"] Oct 30 17:35:00.732829 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:00.732794 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55blp\" (UniqueName: \"kubernetes.io/projected/329e13d7-a002-45f3-96f9-dfde687ce97e-kube-api-access-55blp\") pod \"maas-api-revoke-keys-e2e-ipp-b-4fb821-mqfs9\" (UID: \"329e13d7-a002-45f3-96f9-dfde687ce97e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4fb821-mqfs9" Oct 30 17:35:00.732978 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:00.732858 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/329e13d7-a002-45f3-96f9-dfde687ce97e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4fb821-mqfs9\" (UID: \"329e13d7-a002-45f3-96f9-dfde687ce97e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4fb821-mqfs9" Oct 30 17:35:00.733063 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:00.733037 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/329e13d7-a002-45f3-96f9-dfde687ce97e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4fb821-mqfs9\" (UID: \"329e13d7-a002-45f3-96f9-dfde687ce97e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4fb821-mqfs9" Oct 30 17:35:00.778122 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:00.778086 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-11f480-59c5945bbd-fjf2v"] Oct 30 17:35:00.789063 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:00.789033 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:35:00.789237 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:00.789221 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:35:00.815008 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:00.814980 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-11f480-5d57fc56dc-w48fx"] Oct 30 17:35:00.845828 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:00.845791 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55blp\" (UniqueName: \"kubernetes.io/projected/329e13d7-a002-45f3-96f9-dfde687ce97e-kube-api-access-55blp\") pod \"maas-api-revoke-keys-e2e-ipp-b-4fb821-mqfs9\" (UID: \"329e13d7-a002-45f3-96f9-dfde687ce97e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4fb821-mqfs9" Oct 30 17:35:00.846001 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:00.845834 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/329e13d7-a002-45f3-96f9-dfde687ce97e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4fb821-mqfs9\" (UID: \"329e13d7-a002-45f3-96f9-dfde687ce97e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4fb821-mqfs9" Oct 30 17:35:00.846001 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:00.845920 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/329e13d7-a002-45f3-96f9-dfde687ce97e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4fb821-mqfs9\" (UID: \"329e13d7-a002-45f3-96f9-dfde687ce97e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4fb821-mqfs9" Oct 30 17:35:00.846649 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:00.846619 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/329e13d7-a002-45f3-96f9-dfde687ce97e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4fb821-mqfs9\" (UID: \"329e13d7-a002-45f3-96f9-dfde687ce97e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4fb821-mqfs9" Oct 30 17:35:00.848927 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:00.848902 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/329e13d7-a002-45f3-96f9-dfde687ce97e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4fb821-mqfs9\" (UID: \"329e13d7-a002-45f3-96f9-dfde687ce97e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4fb821-mqfs9" Oct 30 17:35:00.871676 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:00.871647 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55blp\" (UniqueName: \"kubernetes.io/projected/329e13d7-a002-45f3-96f9-dfde687ce97e-kube-api-access-55blp\") pod \"maas-api-revoke-keys-e2e-ipp-b-4fb821-mqfs9\" (UID: \"329e13d7-a002-45f3-96f9-dfde687ce97e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4fb821-mqfs9" Oct 30 17:35:01.006997 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:01.006956 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4fb821-mqfs9" Oct 30 17:35:01.321202 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:35:01.321158 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod329e13d7_a002_45f3_96f9_dfde687ce97e.slice/crio-7ac06a473fb5ce915d3c041ea49b9df0efabc1e72c5942de2ac1fc37852bb61f WatchSource:0}: Error finding container 7ac06a473fb5ce915d3c041ea49b9df0efabc1e72c5942de2ac1fc37852bb61f: Status 404 returned error can't find the container with id 7ac06a473fb5ce915d3c041ea49b9df0efabc1e72c5942de2ac1fc37852bb61f Oct 30 17:35:01.322826 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:01.322765 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4fb821-mqfs9"] Oct 30 17:35:01.818206 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:01.818171 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4fb821-mqfs9" event={"ID":"329e13d7-a002-45f3-96f9-dfde687ce97e","Type":"ContainerStarted","Data":"707999ee8800ff92209c079b94eebf7134160cb768a4f398ad37dcda39ca04d8"} Oct 30 17:35:01.818602 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:01.818228 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4fb821-mqfs9" event={"ID":"329e13d7-a002-45f3-96f9-dfde687ce97e","Type":"ContainerStarted","Data":"7ac06a473fb5ce915d3c041ea49b9df0efabc1e72c5942de2ac1fc37852bb61f"} Oct 30 17:35:01.870658 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:01.870593 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4fb821-mqfs9" podStartSLOduration=1.870577703 podStartE2EDuration="1.870577703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:35:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:35:01.863634614 +0000 UTC m=+2875.864389516" watchObservedRunningTime="2026-10-30 17:35:01.870577703 +0000 UTC m=+2875.871332656" Oct 30 17:35:02.027272 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:02.027236 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5d8947-mnjf2"] Oct 30 17:35:02.031767 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:02.031740 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5d8947-mnjf2"] Oct 30 17:35:02.146445 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:02.146365 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac1d35a2-1aee-4407-8d21-7f1bdfdbc3dc" path="/var/lib/kubelet/pods/ac1d35a2-1aee-4407-8d21-7f1bdfdbc3dc/volumes" Oct 30 17:35:02.828495 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:02.828454 4522 generic.go:356] "Generic (PLEG): container finished" podID="329e13d7-a002-45f3-96f9-dfde687ce97e" containerID="707999ee8800ff92209c079b94eebf7134160cb768a4f398ad37dcda39ca04d8" exitCode=0 Oct 30 17:35:02.828912 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:02.828541 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4fb821-mqfs9" event={"ID":"329e13d7-a002-45f3-96f9-dfde687ce97e","Type":"ContainerDied","Data":"707999ee8800ff92209c079b94eebf7134160cb768a4f398ad37dcda39ca04d8"} Oct 30 17:35:04.029779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:04.029751 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4fb821-mqfs9" Oct 30 17:35:04.103656 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:04.103560 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/329e13d7-a002-45f3-96f9-dfde687ce97e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/329e13d7-a002-45f3-96f9-dfde687ce97e-maas-api-xks-ca\") pod \"329e13d7-a002-45f3-96f9-dfde687ce97e\" (UID: \"329e13d7-a002-45f3-96f9-dfde687ce97e\") " Oct 30 17:35:04.103846 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:04.103822 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/329e13d7-a002-45f3-96f9-dfde687ce97e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/329e13d7-a002-45f3-96f9-dfde687ce97e-maas-api-openshift-ca\") pod \"329e13d7-a002-45f3-96f9-dfde687ce97e\" (UID: \"329e13d7-a002-45f3-96f9-dfde687ce97e\") " Oct 30 17:35:04.103919 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:04.103897 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/329e13d7-a002-45f3-96f9-dfde687ce97e-kube-api-access-55blp\" (UniqueName: \"kubernetes.io/projected/329e13d7-a002-45f3-96f9-dfde687ce97e-kube-api-access-55blp\") pod \"329e13d7-a002-45f3-96f9-dfde687ce97e\" (UID: \"329e13d7-a002-45f3-96f9-dfde687ce97e\") " Oct 30 17:35:04.104283 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:04.104255 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/329e13d7-a002-45f3-96f9-dfde687ce97e-maas-api-openshift-ca" pod "329e13d7-a002-45f3-96f9-dfde687ce97e" (UID: "329e13d7-a002-45f3-96f9-dfde687ce97e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:35:04.104592 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:04.104569 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/329e13d7-a002-45f3-96f9-dfde687ce97e-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:35:04.105968 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:04.105926 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/329e13d7-a002-45f3-96f9-dfde687ce97e-maas-api-xks-ca" pod "329e13d7-a002-45f3-96f9-dfde687ce97e" (UID: "329e13d7-a002-45f3-96f9-dfde687ce97e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:35:04.106630 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:04.106606 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/329e13d7-a002-45f3-96f9-dfde687ce97e-kube-api-access-55blp" pod "329e13d7-a002-45f3-96f9-dfde687ce97e" (UID: "329e13d7-a002-45f3-96f9-dfde687ce97e"). InnerVolumeSpecName "kube-api-access-55blp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:35:04.205735 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:04.205705 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/329e13d7-a002-45f3-96f9-dfde687ce97e-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:35:04.205735 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:04.205734 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55blp\" (UniqueName: \"kubernetes.io/projected/329e13d7-a002-45f3-96f9-dfde687ce97e-kube-api-access-55blp\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:35:04.856477 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:04.856442 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4fb821-mqfs9" Oct 30 17:35:04.856677 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:04.856483 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4fb821-mqfs9" event={"ID":"329e13d7-a002-45f3-96f9-dfde687ce97e","Type":"ContainerDied","Data":"7ac06a473fb5ce915d3c041ea49b9df0efabc1e72c5942de2ac1fc37852bb61f"} Oct 30 17:35:04.856677 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:04.856556 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ac06a473fb5ce915d3c041ea49b9df0efabc1e72c5942de2ac1fc37852bb61f" Oct 30 17:35:05.778664 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:05.778598 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-11f480-59c5945bbd-fjf2v" podUID="b4c4560b-846b-40b9-b143-bab12720112a" containerName="payload-processing" containerID="cri-o://c7be5c1c043be5aa62343ed71512a0259d409caef31a9a6b2bf1a7c3d7d7e015" gracePeriod=25 Oct 30 17:35:05.815948 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:05.815885 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-11f480-5d57fc56dc-w48fx" podUID="e520acad-555a-429f-be9a-928ef8749774" containerName="payload-pre-processing" containerID="cri-o://fd8ee145cc9b2eaf530cdb66f081a5a655fab467b846bf95d651d07b85033bf1" gracePeriod=25 Oct 30 17:35:09.556956 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:09.556920 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4"] Oct 30 17:35:09.557595 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:09.557220 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4" podUID="ef9c4eec-8676-4a8c-9913-da321bb96baf" containerName="istio-proxy" containerID="cri-o://de6f92a0ac08af360214a01412eea84e8ea5fa6c0c56e2880e309b1ea89643e9" gracePeriod=30 Oct 30 17:35:10.106430 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:10.106397 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a43843-bbmq5"] Oct 30 17:35:10.107140 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:10.107126 4522 cpu_manager.go:414] "containerMap: removing container" podUID="329e13d7-a002-45f3-96f9-dfde687ce97e" containerName="revoke-keys" Oct 30 17:35:10.107194 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:10.107141 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="329e13d7-a002-45f3-96f9-dfde687ce97e" containerName="revoke-keys" Oct 30 17:35:10.107250 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:10.107240 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="329e13d7-a002-45f3-96f9-dfde687ce97e" containerName="revoke-keys" Oct 30 17:35:10.115504 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:10.115478 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a43843-bbmq5" Oct 30 17:35:10.131650 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:10.131623 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a43843-bbmq5"] Oct 30 17:35:10.217241 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:10.217206 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:35:10.217515 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:10.217330 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:35:10.310142 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:10.310098 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e38a4dd-06dd-4088-b857-b40962e8908b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-a43843-bbmq5\" (UID: \"1e38a4dd-06dd-4088-b857-b40962e8908b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a43843-bbmq5" Oct 30 17:35:10.310384 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:10.310269 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7jhf\" (UniqueName: \"kubernetes.io/projected/1e38a4dd-06dd-4088-b857-b40962e8908b-kube-api-access-s7jhf\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-a43843-bbmq5\" (UID: \"1e38a4dd-06dd-4088-b857-b40962e8908b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a43843-bbmq5" Oct 30 17:35:10.310465 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:10.310436 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e38a4dd-06dd-4088-b857-b40962e8908b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-a43843-bbmq5\" (UID: \"1e38a4dd-06dd-4088-b857-b40962e8908b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a43843-bbmq5" Oct 30 17:35:10.412433 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:10.412297 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e38a4dd-06dd-4088-b857-b40962e8908b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-a43843-bbmq5\" (UID: \"1e38a4dd-06dd-4088-b857-b40962e8908b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a43843-bbmq5" Oct 30 17:35:10.412619 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:10.412498 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7jhf\" (UniqueName: \"kubernetes.io/projected/1e38a4dd-06dd-4088-b857-b40962e8908b-kube-api-access-s7jhf\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-a43843-bbmq5\" (UID: \"1e38a4dd-06dd-4088-b857-b40962e8908b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a43843-bbmq5" Oct 30 17:35:10.412619 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:10.412587 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e38a4dd-06dd-4088-b857-b40962e8908b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-a43843-bbmq5\" (UID: \"1e38a4dd-06dd-4088-b857-b40962e8908b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a43843-bbmq5" Oct 30 17:35:10.413384 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:10.413365 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e38a4dd-06dd-4088-b857-b40962e8908b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-a43843-bbmq5\" (UID: \"1e38a4dd-06dd-4088-b857-b40962e8908b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a43843-bbmq5" Oct 30 17:35:10.414798 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:10.414778 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e38a4dd-06dd-4088-b857-b40962e8908b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-a43843-bbmq5\" (UID: \"1e38a4dd-06dd-4088-b857-b40962e8908b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a43843-bbmq5" Oct 30 17:35:10.438648 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:10.438624 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7jhf\" (UniqueName: \"kubernetes.io/projected/1e38a4dd-06dd-4088-b857-b40962e8908b-kube-api-access-s7jhf\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-a43843-bbmq5\" (UID: \"1e38a4dd-06dd-4088-b857-b40962e8908b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a43843-bbmq5" Oct 30 17:35:10.730649 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:10.730628 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a43843-bbmq5" Oct 30 17:35:11.049606 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:35:11.049573 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e38a4dd_06dd_4088_b857_b40962e8908b.slice/crio-c84d7de6c5072601ed7d2569b4af634b5f2d792a27186697605a8ce15fcc87df WatchSource:0}: Error finding container c84d7de6c5072601ed7d2569b4af634b5f2d792a27186697605a8ce15fcc87df: Status 404 returned error can't find the container with id c84d7de6c5072601ed7d2569b4af634b5f2d792a27186697605a8ce15fcc87df Oct 30 17:35:11.050852 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:11.050825 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a43843-bbmq5"] Oct 30 17:35:11.369176 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:11.369142 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-11f480-59c5945bbd-fjf2v" podUID="b4c4560b-846b-40b9-b143-bab12720112a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:35:11.369396 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:11.369144 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-11f480-5d57fc56dc-w48fx" podUID="e520acad-555a-429f-be9a-928ef8749774" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:35:11.953840 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:11.953795 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a43843-bbmq5" event={"ID":"1e38a4dd-06dd-4088-b857-b40962e8908b","Type":"ContainerStarted","Data":"22a310e48f7ba0a5e0a894f9974d8aaa48aaa3ddf1ed6ea53890940718d6b019"} Oct 30 17:35:11.954382 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:11.953864 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a43843-bbmq5" event={"ID":"1e38a4dd-06dd-4088-b857-b40962e8908b","Type":"ContainerStarted","Data":"c84d7de6c5072601ed7d2569b4af634b5f2d792a27186697605a8ce15fcc87df"} Oct 30 17:35:12.001476 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:12.001418 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a43843-bbmq5" podStartSLOduration=2.001404043 podStartE2EDuration="2.001404043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:35:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:35:11.998371133 +0000 UTC m=+2885.999126051" watchObservedRunningTime="2026-10-30 17:35:12.001404043 +0000 UTC m=+2886.002158955" Oct 30 17:35:12.952044 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:35:12.952004 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:35:12.962379 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:12.962345 4522 generic.go:356] "Generic (PLEG): container finished" podID="1e38a4dd-06dd-4088-b857-b40962e8908b" containerID="22a310e48f7ba0a5e0a894f9974d8aaa48aaa3ddf1ed6ea53890940718d6b019" exitCode=0 Oct 30 17:35:12.962752 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:12.962420 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a43843-bbmq5" event={"ID":"1e38a4dd-06dd-4088-b857-b40962e8908b","Type":"ContainerDied","Data":"22a310e48f7ba0a5e0a894f9974d8aaa48aaa3ddf1ed6ea53890940718d6b019"} Oct 30 17:35:14.139034 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:14.139007 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a43843-bbmq5" Oct 30 17:35:14.189205 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:14.189154 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e38a4dd-06dd-4088-b857-b40962e8908b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e38a4dd-06dd-4088-b857-b40962e8908b-maas-api-xks-ca\") pod \"1e38a4dd-06dd-4088-b857-b40962e8908b\" (UID: \"1e38a4dd-06dd-4088-b857-b40962e8908b\") " Oct 30 17:35:14.189416 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:14.189396 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e38a4dd-06dd-4088-b857-b40962e8908b-kube-api-access-s7jhf\" (UniqueName: \"kubernetes.io/projected/1e38a4dd-06dd-4088-b857-b40962e8908b-kube-api-access-s7jhf\") pod \"1e38a4dd-06dd-4088-b857-b40962e8908b\" (UID: \"1e38a4dd-06dd-4088-b857-b40962e8908b\") " Oct 30 17:35:14.189515 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:14.189496 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e38a4dd-06dd-4088-b857-b40962e8908b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e38a4dd-06dd-4088-b857-b40962e8908b-maas-api-openshift-ca\") pod \"1e38a4dd-06dd-4088-b857-b40962e8908b\" (UID: \"1e38a4dd-06dd-4088-b857-b40962e8908b\") " Oct 30 17:35:14.190173 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:14.190147 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e38a4dd-06dd-4088-b857-b40962e8908b-maas-api-openshift-ca" pod "1e38a4dd-06dd-4088-b857-b40962e8908b" (UID: "1e38a4dd-06dd-4088-b857-b40962e8908b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:35:14.191997 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:14.191961 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e38a4dd-06dd-4088-b857-b40962e8908b-maas-api-xks-ca" pod "1e38a4dd-06dd-4088-b857-b40962e8908b" (UID: "1e38a4dd-06dd-4088-b857-b40962e8908b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:35:14.192863 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:14.192824 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e38a4dd-06dd-4088-b857-b40962e8908b-kube-api-access-s7jhf" pod "1e38a4dd-06dd-4088-b857-b40962e8908b" (UID: "1e38a4dd-06dd-4088-b857-b40962e8908b"). InnerVolumeSpecName "kube-api-access-s7jhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:35:14.292057 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:14.291976 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7jhf\" (UniqueName: \"kubernetes.io/projected/1e38a4dd-06dd-4088-b857-b40962e8908b-kube-api-access-s7jhf\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:35:14.292057 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:14.292006 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e38a4dd-06dd-4088-b857-b40962e8908b-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:35:14.292057 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:14.292021 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e38a4dd-06dd-4088-b857-b40962e8908b-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:35:14.833291 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:14.833227 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4" Oct 30 17:35:14.900634 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:14.900601 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef9c4eec-8676-4a8c-9913-da321bb96baf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef9c4eec-8676-4a8c-9913-da321bb96baf-credential-socket\") pod \"ef9c4eec-8676-4a8c-9913-da321bb96baf\" (UID: \"ef9c4eec-8676-4a8c-9913-da321bb96baf\") " Oct 30 17:35:14.900835 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:14.900649 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef9c4eec-8676-4a8c-9913-da321bb96baf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef9c4eec-8676-4a8c-9913-da321bb96baf-workload-socket\") pod \"ef9c4eec-8676-4a8c-9913-da321bb96baf\" (UID: \"ef9c4eec-8676-4a8c-9913-da321bb96baf\") " Oct 30 17:35:14.900835 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:14.900807 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef9c4eec-8676-4a8c-9913-da321bb96baf-istio-token\" (UniqueName: \"kubernetes.io/projected/ef9c4eec-8676-4a8c-9913-da321bb96baf-istio-token\") pod \"ef9c4eec-8676-4a8c-9913-da321bb96baf\" (UID: \"ef9c4eec-8676-4a8c-9913-da321bb96baf\") " Oct 30 17:35:14.900970 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:14.900836 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef9c4eec-8676-4a8c-9913-da321bb96baf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef9c4eec-8676-4a8c-9913-da321bb96baf-istio-envoy\") pod \"ef9c4eec-8676-4a8c-9913-da321bb96baf\" (UID: \"ef9c4eec-8676-4a8c-9913-da321bb96baf\") " Oct 30 17:35:14.900970 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:14.900866 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ef9c4eec-8676-4a8c-9913-da321bb96baf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef9c4eec-8676-4a8c-9913-da321bb96baf-istio-podinfo\") pod \"ef9c4eec-8676-4a8c-9913-da321bb96baf\" (UID: \"ef9c4eec-8676-4a8c-9913-da321bb96baf\") " Oct 30 17:35:14.900970 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:14.900933 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef9c4eec-8676-4a8c-9913-da321bb96baf-credential-socket" pod "ef9c4eec-8676-4a8c-9913-da321bb96baf" (UID: "ef9c4eec-8676-4a8c-9913-da321bb96baf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:35:14.901151 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:14.901004 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef9c4eec-8676-4a8c-9913-da321bb96baf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef9c4eec-8676-4a8c-9913-da321bb96baf-workload-certs\") pod \"ef9c4eec-8676-4a8c-9913-da321bb96baf\" (UID: \"ef9c4eec-8676-4a8c-9913-da321bb96baf\") " Oct 30 17:35:14.901151 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:14.901010 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef9c4eec-8676-4a8c-9913-da321bb96baf-workload-socket" pod "ef9c4eec-8676-4a8c-9913-da321bb96baf" (UID: "ef9c4eec-8676-4a8c-9913-da321bb96baf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:35:14.901151 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:14.901095 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef9c4eec-8676-4a8c-9913-da321bb96baf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef9c4eec-8676-4a8c-9913-da321bb96baf-istiod-ca-cert\") pod \"ef9c4eec-8676-4a8c-9913-da321bb96baf\" (UID: \"ef9c4eec-8676-4a8c-9913-da321bb96baf\") " Oct 30 17:35:14.901380 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:14.901179 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef9c4eec-8676-4a8c-9913-da321bb96baf-workload-certs" pod "ef9c4eec-8676-4a8c-9913-da321bb96baf" (UID: "ef9c4eec-8676-4a8c-9913-da321bb96baf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:35:14.901380 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:14.901212 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef9c4eec-8676-4a8c-9913-da321bb96baf-kube-api-access-5d5qs\" (UniqueName: \"kubernetes.io/projected/ef9c4eec-8676-4a8c-9913-da321bb96baf-kube-api-access-5d5qs\") pod \"ef9c4eec-8676-4a8c-9913-da321bb96baf\" (UID: \"ef9c4eec-8676-4a8c-9913-da321bb96baf\") " Oct 30 17:35:14.901380 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:14.901329 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef9c4eec-8676-4a8c-9913-da321bb96baf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef9c4eec-8676-4a8c-9913-da321bb96baf-istio-data\") pod \"ef9c4eec-8676-4a8c-9913-da321bb96baf\" (UID: \"ef9c4eec-8676-4a8c-9913-da321bb96baf\") " Oct 30 17:35:14.901547 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:14.901453 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef9c4eec-8676-4a8c-9913-da321bb96baf-istiod-ca-cert" pod "ef9c4eec-8676-4a8c-9913-da321bb96baf" (UID: "ef9c4eec-8676-4a8c-9913-da321bb96baf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:35:14.901824 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:14.901800 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef9c4eec-8676-4a8c-9913-da321bb96baf-istio-data" pod "ef9c4eec-8676-4a8c-9913-da321bb96baf" (UID: "ef9c4eec-8676-4a8c-9913-da321bb96baf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:35:14.902051 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:14.902030 4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef9c4eec-8676-4a8c-9913-da321bb96baf-workload-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:35:14.902128 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:14.902060 4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef9c4eec-8676-4a8c-9913-da321bb96baf-istiod-ca-cert\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:35:14.902128 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:14.902074 4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef9c4eec-8676-4a8c-9913-da321bb96baf-istio-data\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:35:14.902128 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:14.902088 4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef9c4eec-8676-4a8c-9913-da321bb96baf-credential-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:35:14.902128 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:14.902102 4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef9c4eec-8676-4a8c-9913-da321bb96baf-workload-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:35:14.903888 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:14.903866 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef9c4eec-8676-4a8c-9913-da321bb96baf-istio-envoy" pod "ef9c4eec-8676-4a8c-9913-da321bb96baf" (UID: "ef9c4eec-8676-4a8c-9913-da321bb96baf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:35:14.903989 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:14.903893 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef9c4eec-8676-4a8c-9913-da321bb96baf-kube-api-access-5d5qs" pod "ef9c4eec-8676-4a8c-9913-da321bb96baf" (UID: "ef9c4eec-8676-4a8c-9913-da321bb96baf"). InnerVolumeSpecName "kube-api-access-5d5qs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:35:14.903989 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:14.903923 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef9c4eec-8676-4a8c-9913-da321bb96baf-istio-token" pod "ef9c4eec-8676-4a8c-9913-da321bb96baf" (UID: "ef9c4eec-8676-4a8c-9913-da321bb96baf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:35:14.904068 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:14.903997 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef9c4eec-8676-4a8c-9913-da321bb96baf-istio-podinfo" pod "ef9c4eec-8676-4a8c-9913-da321bb96baf" (UID: "ef9c4eec-8676-4a8c-9913-da321bb96baf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:35:14.979207 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:14.979176 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a43843-bbmq5" Oct 30 17:35:14.979207 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:14.979189 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a43843-bbmq5" event={"ID":"1e38a4dd-06dd-4088-b857-b40962e8908b","Type":"ContainerDied","Data":"c84d7de6c5072601ed7d2569b4af634b5f2d792a27186697605a8ce15fcc87df"} Oct 30 17:35:14.979522 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:14.979232 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c84d7de6c5072601ed7d2569b4af634b5f2d792a27186697605a8ce15fcc87df" Oct 30 17:35:14.981365 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:14.981340 4522 generic.go:356] "Generic (PLEG): container finished" podID="ef9c4eec-8676-4a8c-9913-da321bb96baf" containerID="de6f92a0ac08af360214a01412eea84e8ea5fa6c0c56e2880e309b1ea89643e9" exitCode=0 Oct 30 17:35:14.981484 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:14.981378 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4" Oct 30 17:35:14.981484 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:14.981400 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4" event={"ID":"ef9c4eec-8676-4a8c-9913-da321bb96baf","Type":"ContainerDied","Data":"de6f92a0ac08af360214a01412eea84e8ea5fa6c0c56e2880e309b1ea89643e9"} Oct 30 17:35:14.981484 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:14.981450 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4" event={"ID":"ef9c4eec-8676-4a8c-9913-da321bb96baf","Type":"ContainerDied","Data":"0d83b7929ebd4e864989a5da912380ea87ec62337ad7c69b20c6927d19b4e04e"} Oct 30 17:35:14.981614 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:14.981476 4522 scope.go:122] "RemoveContainer" containerID="de6f92a0ac08af360214a01412eea84e8ea5fa6c0c56e2880e309b1ea89643e9" Oct 30 17:35:14.992861 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:14.992845 4522 scope.go:122] "RemoveContainer" containerID="adf01b0a01c87f846fad0f93969483a10b706c4c156414f8d5a068c9d364f732" Oct 30 17:35:15.002749 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:15.002725 4522 scope.go:122] "RemoveContainer" containerID="de6f92a0ac08af360214a01412eea84e8ea5fa6c0c56e2880e309b1ea89643e9" Oct 30 17:35:15.003038 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:35:15.003013 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de6f92a0ac08af360214a01412eea84e8ea5fa6c0c56e2880e309b1ea89643e9\": container with ID starting with de6f92a0ac08af360214a01412eea84e8ea5fa6c0c56e2880e309b1ea89643e9 not found: ID does not exist" containerID="de6f92a0ac08af360214a01412eea84e8ea5fa6c0c56e2880e309b1ea89643e9" Oct 30 17:35:15.003122 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:15.003051 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de6f92a0ac08af360214a01412eea84e8ea5fa6c0c56e2880e309b1ea89643e9"} err="failed to get container status \"de6f92a0ac08af360214a01412eea84e8ea5fa6c0c56e2880e309b1ea89643e9\": rpc error: code = NotFound desc = could not find container \"de6f92a0ac08af360214a01412eea84e8ea5fa6c0c56e2880e309b1ea89643e9\": container with ID starting with de6f92a0ac08af360214a01412eea84e8ea5fa6c0c56e2880e309b1ea89643e9 not found: ID does not exist" Oct 30 17:35:15.003122 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:15.003077 4522 scope.go:122] "RemoveContainer" containerID="adf01b0a01c87f846fad0f93969483a10b706c4c156414f8d5a068c9d364f732" Oct 30 17:35:15.003367 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:35:15.003348 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adf01b0a01c87f846fad0f93969483a10b706c4c156414f8d5a068c9d364f732\": container with ID starting with adf01b0a01c87f846fad0f93969483a10b706c4c156414f8d5a068c9d364f732 not found: ID does not exist" containerID="adf01b0a01c87f846fad0f93969483a10b706c4c156414f8d5a068c9d364f732" Oct 30 17:35:15.003478 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:15.003375 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adf01b0a01c87f846fad0f93969483a10b706c4c156414f8d5a068c9d364f732"} err="failed to get container status \"adf01b0a01c87f846fad0f93969483a10b706c4c156414f8d5a068c9d364f732\": rpc error: code = NotFound desc = could not find container \"adf01b0a01c87f846fad0f93969483a10b706c4c156414f8d5a068c9d364f732\": container with ID starting with adf01b0a01c87f846fad0f93969483a10b706c4c156414f8d5a068c9d364f732 not found: ID does not exist" Oct 30 17:35:15.003653 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:15.003636 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d5qs\" (UniqueName: \"kubernetes.io/projected/ef9c4eec-8676-4a8c-9913-da321bb96baf-kube-api-access-5d5qs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:35:15.003702 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:15.003657 4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef9c4eec-8676-4a8c-9913-da321bb96baf-istio-token\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:35:15.003702 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:15.003667 4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef9c4eec-8676-4a8c-9913-da321bb96baf-istio-envoy\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:35:15.003702 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:15.003674 4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef9c4eec-8676-4a8c-9913-da321bb96baf-istio-podinfo\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:35:15.058453 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:15.058423 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4"] Oct 30 17:35:15.075780 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:15.075747 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-wcjt4"] Oct 30 17:35:16.149405 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:16.149377 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef9c4eec-8676-4a8c-9913-da321bb96baf" path="/var/lib/kubelet/pods/ef9c4eec-8676-4a8c-9913-da321bb96baf/volumes" Oct 30 17:35:17.114969 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:17.114935 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-11f480-f4b24"] Oct 30 17:35:17.128178 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:17.128152 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-11f480-f4b24"] Oct 30 17:35:18.146451 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:18.146414 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cfef672-8038-43d3-bb96-6ee9793abd6d" path="/var/lib/kubelet/pods/4cfef672-8038-43d3-bb96-6ee9793abd6d/volumes" Oct 30 17:35:18.859408 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:18.859375 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9e9ce7-5dd848f984-zhxlv"] Oct 30 17:35:18.859629 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:18.859601 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9e9ce7-5dd848f984-zhxlv" podUID="3f335c19-096c-4904-ae7f-68883ae53596" containerName="maas-api" containerID="cri-o://627f4775570cf128ceee713f2ad9215ef95b6ca3f48e5cadfd8777b02b2898bd" gracePeriod=30 Oct 30 17:35:19.016655 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:19.016623 4522 generic.go:356] "Generic (PLEG): container finished" podID="3f335c19-096c-4904-ae7f-68883ae53596" containerID="627f4775570cf128ceee713f2ad9215ef95b6ca3f48e5cadfd8777b02b2898bd" exitCode=0 Oct 30 17:35:19.016821 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:19.016704 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9e9ce7-5dd848f984-zhxlv" event={"ID":"3f335c19-096c-4904-ae7f-68883ae53596","Type":"ContainerDied","Data":"627f4775570cf128ceee713f2ad9215ef95b6ca3f48e5cadfd8777b02b2898bd"} Oct 30 17:35:19.025768 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:19.025738 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9e9ce7-779c78d4b4-2997b"] Oct 30 17:35:19.025940 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:19.025919 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-9e9ce7-779c78d4b4-2997b" podUID="a3ece1ba-f9cb-4d06-93ad-ab7b4165b1cd" containerName="payload-processing" containerID="cri-o://e3dccd30adadd4db62e923c4898ce9f7e3b2ff6375be7c8a2c09851c5263d6cb" gracePeriod=30 Oct 30 17:35:19.061056 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:19.061024 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9e9ce7-fc6b84c5f-qwmnv"] Oct 30 17:35:19.061290 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:19.061264 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9e9ce7-fc6b84c5f-qwmnv" podUID="3c4810b7-b48e-41a5-9366-b9bcfec990ea" containerName="payload-pre-processing" containerID="cri-o://a975c055bec194df6583f72e98db67558e66dfdd97d9fda65bb68a6a8ea66584" gracePeriod=30 Oct 30 17:35:19.235841 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:19.235813 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9e9ce7-5dd848f984-zhxlv" Oct 30 17:35:19.363985 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:19.363954 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9e9ce7-779c78d4b4-2997b" Oct 30 17:35:19.391205 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:19.391114 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f335c19-096c-4904-ae7f-68883ae53596-kube-api-access-2fvxs\" (UniqueName: \"kubernetes.io/projected/3f335c19-096c-4904-ae7f-68883ae53596-kube-api-access-2fvxs\") pod \"3f335c19-096c-4904-ae7f-68883ae53596\" (UID: \"3f335c19-096c-4904-ae7f-68883ae53596\") " Oct 30 17:35:19.391411 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:19.391211 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f335c19-096c-4904-ae7f-68883ae53596-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f335c19-096c-4904-ae7f-68883ae53596-openshift-service-ca\") pod \"3f335c19-096c-4904-ae7f-68883ae53596\" (UID: \"3f335c19-096c-4904-ae7f-68883ae53596\") " Oct 30 17:35:19.391531 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:19.391509 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f335c19-096c-4904-ae7f-68883ae53596-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f335c19-096c-4904-ae7f-68883ae53596-metrics-certs\") pod \"3f335c19-096c-4904-ae7f-68883ae53596\" (UID: \"3f335c19-096c-4904-ae7f-68883ae53596\") " Oct 30 17:35:19.391630 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:19.391610 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f335c19-096c-4904-ae7f-68883ae53596-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f335c19-096c-4904-ae7f-68883ae53596-maas-api-tls\") pod \"3f335c19-096c-4904-ae7f-68883ae53596\" (UID: \"3f335c19-096c-4904-ae7f-68883ae53596\") " Oct 30 17:35:19.391691 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:19.391644 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f335c19-096c-4904-ae7f-68883ae53596-openshift-service-ca" pod "3f335c19-096c-4904-ae7f-68883ae53596" (UID: "3f335c19-096c-4904-ae7f-68883ae53596"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:35:19.392260 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:19.392232 4522 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f335c19-096c-4904-ae7f-68883ae53596-openshift-service-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:35:19.394558 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:19.394528 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f335c19-096c-4904-ae7f-68883ae53596-metrics-certs" pod "3f335c19-096c-4904-ae7f-68883ae53596" (UID: "3f335c19-096c-4904-ae7f-68883ae53596"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:35:19.395101 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:19.395075 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f335c19-096c-4904-ae7f-68883ae53596-maas-api-tls" pod "3f335c19-096c-4904-ae7f-68883ae53596" (UID: "3f335c19-096c-4904-ae7f-68883ae53596"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:35:19.395211 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:19.395120 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f335c19-096c-4904-ae7f-68883ae53596-kube-api-access-2fvxs" pod "3f335c19-096c-4904-ae7f-68883ae53596" (UID: "3f335c19-096c-4904-ae7f-68883ae53596"). InnerVolumeSpecName "kube-api-access-2fvxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:35:19.522332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:19.501755 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3ece1ba-f9cb-4d06-93ad-ab7b4165b1cd-kube-api-access-snt4g\" (UniqueName: \"kubernetes.io/projected/a3ece1ba-f9cb-4d06-93ad-ab7b4165b1cd-kube-api-access-snt4g\") pod \"a3ece1ba-f9cb-4d06-93ad-ab7b4165b1cd\" (UID: \"a3ece1ba-f9cb-4d06-93ad-ab7b4165b1cd\") " Oct 30 17:35:19.522332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:19.501974 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3ece1ba-f9cb-4d06-93ad-ab7b4165b1cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3ece1ba-f9cb-4d06-93ad-ab7b4165b1cd-plugins-config-volume\") pod \"a3ece1ba-f9cb-4d06-93ad-ab7b4165b1cd\" (UID: \"a3ece1ba-f9cb-4d06-93ad-ab7b4165b1cd\") " Oct 30 17:35:19.522332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:19.502620 4522 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f335c19-096c-4904-ae7f-68883ae53596-metrics-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:35:19.522332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:19.502641 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f335c19-096c-4904-ae7f-68883ae53596-maas-api-tls\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:35:19.522332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:19.502654 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fvxs\" (UniqueName: \"kubernetes.io/projected/3f335c19-096c-4904-ae7f-68883ae53596-kube-api-access-2fvxs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:35:19.522332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:19.510777 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3ece1ba-f9cb-4d06-93ad-ab7b4165b1cd-kube-api-access-snt4g" pod "a3ece1ba-f9cb-4d06-93ad-ab7b4165b1cd" (UID: "a3ece1ba-f9cb-4d06-93ad-ab7b4165b1cd"). InnerVolumeSpecName "kube-api-access-snt4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:35:19.522332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:19.512651 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3ece1ba-f9cb-4d06-93ad-ab7b4165b1cd-plugins-config-volume" pod "a3ece1ba-f9cb-4d06-93ad-ab7b4165b1cd" (UID: "a3ece1ba-f9cb-4d06-93ad-ab7b4165b1cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:35:19.533013 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:19.532982 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9e9ce7-fc6b84c5f-qwmnv" Oct 30 17:35:19.603974 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:19.603934 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c4810b7-b48e-41a5-9366-b9bcfec990ea-kube-api-access-89ddq\" (UniqueName: \"kubernetes.io/projected/3c4810b7-b48e-41a5-9366-b9bcfec990ea-kube-api-access-89ddq\") pod \"3c4810b7-b48e-41a5-9366-b9bcfec990ea\" (UID: \"3c4810b7-b48e-41a5-9366-b9bcfec990ea\") " Oct 30 17:35:19.604285 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:19.604259 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c4810b7-b48e-41a5-9366-b9bcfec990ea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c4810b7-b48e-41a5-9366-b9bcfec990ea-plugins-config-volume\") pod \"3c4810b7-b48e-41a5-9366-b9bcfec990ea\" (UID: \"3c4810b7-b48e-41a5-9366-b9bcfec990ea\") " Oct 30 17:35:19.604727 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:19.604700 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c4810b7-b48e-41a5-9366-b9bcfec990ea-plugins-config-volume" pod "3c4810b7-b48e-41a5-9366-b9bcfec990ea" (UID: "3c4810b7-b48e-41a5-9366-b9bcfec990ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:35:19.604995 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:19.604973 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3ece1ba-f9cb-4d06-93ad-ab7b4165b1cd-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:35:19.605089 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:19.605000 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snt4g\" (UniqueName: \"kubernetes.io/projected/a3ece1ba-f9cb-4d06-93ad-ab7b4165b1cd-kube-api-access-snt4g\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:35:19.605089 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:19.605017 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c4810b7-b48e-41a5-9366-b9bcfec990ea-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:35:19.606410 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:19.606384 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c4810b7-b48e-41a5-9366-b9bcfec990ea-kube-api-access-89ddq" pod "3c4810b7-b48e-41a5-9366-b9bcfec990ea" (UID: "3c4810b7-b48e-41a5-9366-b9bcfec990ea"). InnerVolumeSpecName "kube-api-access-89ddq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:35:19.706131 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:19.706101 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89ddq\" (UniqueName: \"kubernetes.io/projected/3c4810b7-b48e-41a5-9366-b9bcfec990ea-kube-api-access-89ddq\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:35:19.819713 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:19.819675 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9e9ce7-v789d"] Oct 30 17:35:19.823446 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:19.823417 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9e9ce7-v789d"] Oct 30 17:35:20.027012 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:20.026933 4522 generic.go:356] "Generic (PLEG): container finished" podID="3c4810b7-b48e-41a5-9366-b9bcfec990ea" containerID="a975c055bec194df6583f72e98db67558e66dfdd97d9fda65bb68a6a8ea66584" exitCode=0 Oct 30 17:35:20.027012 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:20.026985 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9e9ce7-fc6b84c5f-qwmnv" Oct 30 17:35:20.027255 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:20.027023 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9e9ce7-fc6b84c5f-qwmnv" event={"ID":"3c4810b7-b48e-41a5-9366-b9bcfec990ea","Type":"ContainerDied","Data":"a975c055bec194df6583f72e98db67558e66dfdd97d9fda65bb68a6a8ea66584"} Oct 30 17:35:20.027255 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:20.027081 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9e9ce7-fc6b84c5f-qwmnv" event={"ID":"3c4810b7-b48e-41a5-9366-b9bcfec990ea","Type":"ContainerDied","Data":"a2aa7ef03ac93a263701ac772c8a4c72075fa1c1ccbc63e8be5d62d2d758bab6"} Oct 30 17:35:20.027255 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:20.027109 4522 scope.go:122] "RemoveContainer" containerID="a975c055bec194df6583f72e98db67558e66dfdd97d9fda65bb68a6a8ea66584" Oct 30 17:35:20.029243 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:20.029223 4522 generic.go:356] "Generic (PLEG): container finished" podID="a3ece1ba-f9cb-4d06-93ad-ab7b4165b1cd" containerID="e3dccd30adadd4db62e923c4898ce9f7e3b2ff6375be7c8a2c09851c5263d6cb" exitCode=0 Oct 30 17:35:20.029505 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:20.029263 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9e9ce7-779c78d4b4-2997b" Oct 30 17:35:20.029505 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:20.029368 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9e9ce7-779c78d4b4-2997b" event={"ID":"a3ece1ba-f9cb-4d06-93ad-ab7b4165b1cd","Type":"ContainerDied","Data":"e3dccd30adadd4db62e923c4898ce9f7e3b2ff6375be7c8a2c09851c5263d6cb"} Oct 30 17:35:20.029505 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:20.029419 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9e9ce7-779c78d4b4-2997b" event={"ID":"a3ece1ba-f9cb-4d06-93ad-ab7b4165b1cd","Type":"ContainerDied","Data":"c33d6b2db83e449e75efa173bf7726cef75e39d14d74385a9578a6c1860d6191"} Oct 30 17:35:20.031020 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:20.030999 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9e9ce7-5dd848f984-zhxlv" Oct 30 17:35:20.031113 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:20.031023 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9e9ce7-5dd848f984-zhxlv" event={"ID":"3f335c19-096c-4904-ae7f-68883ae53596","Type":"ContainerDied","Data":"84627766ce32c401c25ff2356acb5a41b6ebe5972633627768016b5a301807c2"} Oct 30 17:35:20.043120 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:20.042934 4522 scope.go:122] "RemoveContainer" containerID="a975c055bec194df6583f72e98db67558e66dfdd97d9fda65bb68a6a8ea66584" Oct 30 17:35:20.043371 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:35:20.043321 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a975c055bec194df6583f72e98db67558e66dfdd97d9fda65bb68a6a8ea66584\": container with ID starting with a975c055bec194df6583f72e98db67558e66dfdd97d9fda65bb68a6a8ea66584 not found: ID does not exist" containerID="a975c055bec194df6583f72e98db67558e66dfdd97d9fda65bb68a6a8ea66584" Oct 30 17:35:20.043466 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:20.043362 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a975c055bec194df6583f72e98db67558e66dfdd97d9fda65bb68a6a8ea66584"} err="failed to get container status \"a975c055bec194df6583f72e98db67558e66dfdd97d9fda65bb68a6a8ea66584\": rpc error: code = NotFound desc = could not find container \"a975c055bec194df6583f72e98db67558e66dfdd97d9fda65bb68a6a8ea66584\": container with ID starting with a975c055bec194df6583f72e98db67558e66dfdd97d9fda65bb68a6a8ea66584 not found: ID does not exist" Oct 30 17:35:20.043466 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:20.043401 4522 scope.go:122] "RemoveContainer" containerID="e3dccd30adadd4db62e923c4898ce9f7e3b2ff6375be7c8a2c09851c5263d6cb" Oct 30 17:35:20.060438 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:20.060415 4522 scope.go:122] "RemoveContainer" containerID="e3dccd30adadd4db62e923c4898ce9f7e3b2ff6375be7c8a2c09851c5263d6cb" Oct 30 17:35:20.060782 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:35:20.060760 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3dccd30adadd4db62e923c4898ce9f7e3b2ff6375be7c8a2c09851c5263d6cb\": container with ID starting with e3dccd30adadd4db62e923c4898ce9f7e3b2ff6375be7c8a2c09851c5263d6cb not found: ID does not exist" containerID="e3dccd30adadd4db62e923c4898ce9f7e3b2ff6375be7c8a2c09851c5263d6cb" Oct 30 17:35:20.060857 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:20.060794 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3dccd30adadd4db62e923c4898ce9f7e3b2ff6375be7c8a2c09851c5263d6cb"} err="failed to get container status \"e3dccd30adadd4db62e923c4898ce9f7e3b2ff6375be7c8a2c09851c5263d6cb\": rpc error: code = NotFound desc = could not find container \"e3dccd30adadd4db62e923c4898ce9f7e3b2ff6375be7c8a2c09851c5263d6cb\": container with ID starting with e3dccd30adadd4db62e923c4898ce9f7e3b2ff6375be7c8a2c09851c5263d6cb not found: ID does not exist" Oct 30 17:35:20.060857 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:20.060820 4522 scope.go:122] "RemoveContainer" containerID="627f4775570cf128ceee713f2ad9215ef95b6ca3f48e5cadfd8777b02b2898bd" Oct 30 17:35:20.146512 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:20.146472 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50e2c4b5-f884-480d-a6ae-8f215ec9d392" path="/var/lib/kubelet/pods/50e2c4b5-f884-480d-a6ae-8f215ec9d392/volumes" Oct 30 17:35:20.151653 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:20.151629 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9e9ce7-fc6b84c5f-qwmnv"] Oct 30 17:35:20.165781 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:20.165730 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9e9ce7-fc6b84c5f-qwmnv"] Oct 30 17:35:20.223490 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:20.223449 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9e9ce7-779c78d4b4-2997b"] Oct 30 17:35:20.227116 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:20.227090 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9e9ce7-779c78d4b4-2997b"] Oct 30 17:35:20.283476 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:20.283386 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9e9ce7-5dd848f984-zhxlv"] Oct 30 17:35:20.301245 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:20.301213 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9e9ce7-5dd848f984-zhxlv"] Oct 30 17:35:20.386388 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:20.386349 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a43843-86c7c5b7-dxhbc"] Oct 30 17:35:21.369412 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:21.369378 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-11f480-5d57fc56dc-w48fx" podUID="e520acad-555a-429f-be9a-928ef8749774" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:35:21.369841 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:21.369415 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-11f480-59c5945bbd-fjf2v" podUID="b4c4560b-846b-40b9-b143-bab12720112a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:35:22.146343 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:22.146297 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c4810b7-b48e-41a5-9366-b9bcfec990ea" path="/var/lib/kubelet/pods/3c4810b7-b48e-41a5-9366-b9bcfec990ea/volumes" Oct 30 17:35:22.146757 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:22.146739 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f335c19-096c-4904-ae7f-68883ae53596" path="/var/lib/kubelet/pods/3f335c19-096c-4904-ae7f-68883ae53596/volumes" Oct 30 17:35:22.147124 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:22.147112 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3ece1ba-f9cb-4d06-93ad-ab7b4165b1cd" path="/var/lib/kubelet/pods/a3ece1ba-f9cb-4d06-93ad-ab7b4165b1cd/volumes" Oct 30 17:35:25.387372 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:25.387291 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a43843-86c7c5b7-dxhbc" podUID="f396ca5e-93c9-4dd4-a03e-f9150e784873" containerName="payload-pre-processing" containerID="cri-o://2ea2424f211d1167717197695a1bb59f56f24a1a98c0efb798094c67585c15c2" gracePeriod=25 Oct 30 17:35:25.585345 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:25.584741 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6"] Oct 30 17:35:25.585345 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:25.585043 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6" podUID="d534a90d-d839-4d06-b709-83d44c667b86" containerName="istio-proxy" containerID="cri-o://2c27612d534dc3ed63ca17ee6308591836c5dfdbd5e3da315c91c0b7b043f34e" gracePeriod=30 Oct 30 17:35:25.600204 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:25.600175 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595"] Oct 30 17:35:25.600427 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:25.600398 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595" podUID="1a5d5676-2fbf-4892-8b12-862c483b2d17" containerName="istio-proxy" containerID="cri-o://4bfd3354e7a42dae397dd62b3824c2129b5c1b1bf65841352b13313c5eed0a4c" gracePeriod=30 Oct 30 17:35:25.698209 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:25.698179 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a43843-86c7c5b7-dxhbc" Oct 30 17:35:25.813481 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:25.793272 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f396ca5e-93c9-4dd4-a03e-f9150e784873-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f396ca5e-93c9-4dd4-a03e-f9150e784873-plugins-config-volume\") pod \"f396ca5e-93c9-4dd4-a03e-f9150e784873\" (UID: \"f396ca5e-93c9-4dd4-a03e-f9150e784873\") " Oct 30 17:35:25.813481 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:25.793532 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f396ca5e-93c9-4dd4-a03e-f9150e784873-kube-api-access-xbt7h\" (UniqueName: \"kubernetes.io/projected/f396ca5e-93c9-4dd4-a03e-f9150e784873-kube-api-access-xbt7h\") pod \"f396ca5e-93c9-4dd4-a03e-f9150e784873\" (UID: \"f396ca5e-93c9-4dd4-a03e-f9150e784873\") " Oct 30 17:35:25.813481 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:25.811498 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f396ca5e-93c9-4dd4-a03e-f9150e784873-plugins-config-volume" pod "f396ca5e-93c9-4dd4-a03e-f9150e784873" (UID: "f396ca5e-93c9-4dd4-a03e-f9150e784873"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:35:25.813481 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:25.812332 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f396ca5e-93c9-4dd4-a03e-f9150e784873-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:35:25.864995 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:25.864957 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f396ca5e-93c9-4dd4-a03e-f9150e784873-kube-api-access-xbt7h" pod "f396ca5e-93c9-4dd4-a03e-f9150e784873" (UID: "f396ca5e-93c9-4dd4-a03e-f9150e784873"). InnerVolumeSpecName "kube-api-access-xbt7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:35:25.913632 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:25.913561 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbt7h\" (UniqueName: \"kubernetes.io/projected/f396ca5e-93c9-4dd4-a03e-f9150e784873-kube-api-access-xbt7h\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:35:26.078824 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.078793 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-11f480-59c5945bbd-fjf2v" Oct 30 17:35:26.115549 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.115521 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4c4560b-846b-40b9-b143-bab12720112a-kube-api-access-7vf8m\" (UniqueName: \"kubernetes.io/projected/b4c4560b-846b-40b9-b143-bab12720112a-kube-api-access-7vf8m\") pod \"b4c4560b-846b-40b9-b143-bab12720112a\" (UID: \"b4c4560b-846b-40b9-b143-bab12720112a\") " Oct 30 17:35:26.115714 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.115561 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4c4560b-846b-40b9-b143-bab12720112a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4c4560b-846b-40b9-b143-bab12720112a-plugins-config-volume\") pod \"b4c4560b-846b-40b9-b143-bab12720112a\" (UID: \"b4c4560b-846b-40b9-b143-bab12720112a\") " Oct 30 17:35:26.115960 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.115931 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4c4560b-846b-40b9-b143-bab12720112a-plugins-config-volume" pod "b4c4560b-846b-40b9-b143-bab12720112a" (UID: "b4c4560b-846b-40b9-b143-bab12720112a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:35:26.116120 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.116105 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4c4560b-846b-40b9-b143-bab12720112a-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:35:26.117562 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.117543 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4c4560b-846b-40b9-b143-bab12720112a-kube-api-access-7vf8m" pod "b4c4560b-846b-40b9-b143-bab12720112a" (UID: "b4c4560b-846b-40b9-b143-bab12720112a"). InnerVolumeSpecName "kube-api-access-7vf8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:35:26.124521 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.124496 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-11f480-5d57fc56dc-w48fx" Oct 30 17:35:26.126460 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.126436 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6d4f99-rwdd8"] Oct 30 17:35:26.127354 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.127330 4522 cpu_manager.go:414] "containerMap: removing container" podUID="ef9c4eec-8676-4a8c-9913-da321bb96baf" containerName="istio-proxy" Oct 30 17:35:26.127354 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.127353 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef9c4eec-8676-4a8c-9913-da321bb96baf" containerName="istio-proxy" Oct 30 17:35:26.127510 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.127379 4522 cpu_manager.go:414] "containerMap: removing container" podUID="e520acad-555a-429f-be9a-928ef8749774" containerName="payload-pre-processing" Oct 30 17:35:26.127510 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.127384 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e520acad-555a-429f-be9a-928ef8749774" containerName="payload-pre-processing" Oct 30 17:35:26.127510 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.127395 4522 cpu_manager.go:414] "containerMap: removing container" podUID="a3ece1ba-f9cb-4d06-93ad-ab7b4165b1cd" containerName="payload-processing" Oct 30 17:35:26.127510 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.127401 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3ece1ba-f9cb-4d06-93ad-ab7b4165b1cd" containerName="payload-processing" Oct 30 17:35:26.127510 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.127409 4522 cpu_manager.go:414] "containerMap: removing container" podUID="3f335c19-096c-4904-ae7f-68883ae53596" containerName="maas-api" Oct 30 17:35:26.127510 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.127414 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f335c19-096c-4904-ae7f-68883ae53596" containerName="maas-api" Oct 30 17:35:26.127510 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.127429 4522 cpu_manager.go:414] "containerMap: removing container" podUID="1e38a4dd-06dd-4088-b857-b40962e8908b" containerName="revoke-keys" Oct 30 17:35:26.127510 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.127434 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e38a4dd-06dd-4088-b857-b40962e8908b" containerName="revoke-keys" Oct 30 17:35:26.127510 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.127449 4522 cpu_manager.go:414] "containerMap: removing container" podUID="f396ca5e-93c9-4dd4-a03e-f9150e784873" containerName="payload-pre-processing" Oct 30 17:35:26.127510 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.127454 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f396ca5e-93c9-4dd4-a03e-f9150e784873" containerName="payload-pre-processing" Oct 30 17:35:26.127510 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.127464 4522 cpu_manager.go:414] "containerMap: removing container" podUID="3c4810b7-b48e-41a5-9366-b9bcfec990ea" containerName="payload-pre-processing" Oct 30 17:35:26.127510 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.127469 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c4810b7-b48e-41a5-9366-b9bcfec990ea" containerName="payload-pre-processing" Oct 30 17:35:26.127510 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.127485 4522 cpu_manager.go:414] "containerMap: removing container" podUID="b4c4560b-846b-40b9-b143-bab12720112a" containerName="payload-processing" Oct 30 17:35:26.127510 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.127489 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4c4560b-846b-40b9-b143-bab12720112a" containerName="payload-processing" Oct 30 17:35:26.128178 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.127603 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="f396ca5e-93c9-4dd4-a03e-f9150e784873" containerName="payload-pre-processing" Oct 30 17:35:26.128178 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.127612 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c4810b7-b48e-41a5-9366-b9bcfec990ea" containerName="payload-pre-processing" Oct 30 17:35:26.128178 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.127622 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4c4560b-846b-40b9-b143-bab12720112a" containerName="payload-processing" Oct 30 17:35:26.128178 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.127629 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef9c4eec-8676-4a8c-9913-da321bb96baf" containerName="istio-proxy" Oct 30 17:35:26.128178 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.127635 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef9c4eec-8676-4a8c-9913-da321bb96baf" containerName="istio-proxy" Oct 30 17:35:26.128178 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.127642 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="e520acad-555a-429f-be9a-928ef8749774" containerName="payload-pre-processing" Oct 30 17:35:26.128178 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.127648 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3ece1ba-f9cb-4d06-93ad-ab7b4165b1cd" containerName="payload-processing" Oct 30 17:35:26.128178 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.127653 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f335c19-096c-4904-ae7f-68883ae53596" containerName="maas-api" Oct 30 17:35:26.128178 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.127660 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e38a4dd-06dd-4088-b857-b40962e8908b" containerName="revoke-keys" Oct 30 17:35:26.134227 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.131701 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6d4f99-rwdd8" Oct 30 17:35:26.143874 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.143812 4522 generic.go:356] "Generic (PLEG): container finished" podID="e520acad-555a-429f-be9a-928ef8749774" containerID="fd8ee145cc9b2eaf530cdb66f081a5a655fab467b846bf95d651d07b85033bf1" exitCode=0 Oct 30 17:35:26.144014 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.143992 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-11f480-5d57fc56dc-w48fx" Oct 30 17:35:26.145821 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.145792 4522 generic.go:356] "Generic (PLEG): container finished" podID="b4c4560b-846b-40b9-b143-bab12720112a" containerID="c7be5c1c043be5aa62343ed71512a0259d409caef31a9a6b2bf1a7c3d7d7e015" exitCode=0 Oct 30 17:35:26.145957 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.145941 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-11f480-59c5945bbd-fjf2v" Oct 30 17:35:26.157734 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.153401 4522 generic.go:356] "Generic (PLEG): container finished" podID="f396ca5e-93c9-4dd4-a03e-f9150e784873" containerID="2ea2424f211d1167717197695a1bb59f56f24a1a98c0efb798094c67585c15c2" exitCode=0 Oct 30 17:35:26.157734 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.153686 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a43843-86c7c5b7-dxhbc" Oct 30 17:35:26.157734 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.154886 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:35:26.157734 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.154940 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:35:26.158939 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.158854 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-11f480-5d57fc56dc-w48fx" event={"ID":"e520acad-555a-429f-be9a-928ef8749774","Type":"ContainerDied","Data":"fd8ee145cc9b2eaf530cdb66f081a5a655fab467b846bf95d651d07b85033bf1"} Oct 30 17:35:26.159021 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.158999 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6d4f99-rwdd8"] Oct 30 17:35:26.159083 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.159035 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-11f480-5d57fc56dc-w48fx" event={"ID":"e520acad-555a-429f-be9a-928ef8749774","Type":"ContainerDied","Data":"007d3176417e7216a2485ef7c2c9591e8cdec4dc3ad07d050b6b6326efdd1118"} Oct 30 17:35:26.159083 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.159061 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-11f480-59c5945bbd-fjf2v" event={"ID":"b4c4560b-846b-40b9-b143-bab12720112a","Type":"ContainerDied","Data":"c7be5c1c043be5aa62343ed71512a0259d409caef31a9a6b2bf1a7c3d7d7e015"} Oct 30 17:35:26.159083 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.159081 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-11f480-59c5945bbd-fjf2v" event={"ID":"b4c4560b-846b-40b9-b143-bab12720112a","Type":"ContainerDied","Data":"da1ba77907466f7ccd5df0b6d542085eabf5c57fd1787ef177a112362a96b229"} Oct 30 17:35:26.159260 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.159101 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a43843-86c7c5b7-dxhbc" event={"ID":"f396ca5e-93c9-4dd4-a03e-f9150e784873","Type":"ContainerDied","Data":"2ea2424f211d1167717197695a1bb59f56f24a1a98c0efb798094c67585c15c2"} Oct 30 17:35:26.160123 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.159769 4522 scope.go:122] "RemoveContainer" containerID="fd8ee145cc9b2eaf530cdb66f081a5a655fab467b846bf95d651d07b85033bf1" Oct 30 17:35:26.163474 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.162586 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a43843-86c7c5b7-dxhbc" event={"ID":"f396ca5e-93c9-4dd4-a03e-f9150e784873","Type":"ContainerDied","Data":"c31757f918cf270e392802503bfa327f85befb74b8743bb31870c7e05a2c4bf2"} Oct 30 17:35:26.180535 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.180507 4522 scope.go:122] "RemoveContainer" containerID="fd8ee145cc9b2eaf530cdb66f081a5a655fab467b846bf95d651d07b85033bf1" Oct 30 17:35:26.180972 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:35:26.180936 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd8ee145cc9b2eaf530cdb66f081a5a655fab467b846bf95d651d07b85033bf1\": container with ID starting with fd8ee145cc9b2eaf530cdb66f081a5a655fab467b846bf95d651d07b85033bf1 not found: ID does not exist" containerID="fd8ee145cc9b2eaf530cdb66f081a5a655fab467b846bf95d651d07b85033bf1" Oct 30 17:35:26.181875 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.180990 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd8ee145cc9b2eaf530cdb66f081a5a655fab467b846bf95d651d07b85033bf1"} err="failed to get container status \"fd8ee145cc9b2eaf530cdb66f081a5a655fab467b846bf95d651d07b85033bf1\": rpc error: code = NotFound desc = could not find container \"fd8ee145cc9b2eaf530cdb66f081a5a655fab467b846bf95d651d07b85033bf1\": container with ID starting with fd8ee145cc9b2eaf530cdb66f081a5a655fab467b846bf95d651d07b85033bf1 not found: ID does not exist" Oct 30 17:35:26.181875 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.181014 4522 scope.go:122] "RemoveContainer" containerID="c7be5c1c043be5aa62343ed71512a0259d409caef31a9a6b2bf1a7c3d7d7e015" Oct 30 17:35:26.195670 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.195647 4522 scope.go:122] "RemoveContainer" containerID="c7be5c1c043be5aa62343ed71512a0259d409caef31a9a6b2bf1a7c3d7d7e015" Oct 30 17:35:26.196014 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:35:26.195985 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7be5c1c043be5aa62343ed71512a0259d409caef31a9a6b2bf1a7c3d7d7e015\": container with ID starting with c7be5c1c043be5aa62343ed71512a0259d409caef31a9a6b2bf1a7c3d7d7e015 not found: ID does not exist" containerID="c7be5c1c043be5aa62343ed71512a0259d409caef31a9a6b2bf1a7c3d7d7e015" Oct 30 17:35:26.196100 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.196024 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7be5c1c043be5aa62343ed71512a0259d409caef31a9a6b2bf1a7c3d7d7e015"} err="failed to get container status \"c7be5c1c043be5aa62343ed71512a0259d409caef31a9a6b2bf1a7c3d7d7e015\": rpc error: code = NotFound desc = could not find container \"c7be5c1c043be5aa62343ed71512a0259d409caef31a9a6b2bf1a7c3d7d7e015\": container with ID starting with c7be5c1c043be5aa62343ed71512a0259d409caef31a9a6b2bf1a7c3d7d7e015 not found: ID does not exist" Oct 30 17:35:26.196100 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.196052 4522 scope.go:122] "RemoveContainer" containerID="2ea2424f211d1167717197695a1bb59f56f24a1a98c0efb798094c67585c15c2" Oct 30 17:35:26.207089 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.207061 4522 scope.go:122] "RemoveContainer" containerID="2ea2424f211d1167717197695a1bb59f56f24a1a98c0efb798094c67585c15c2" Oct 30 17:35:26.207433 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:35:26.207401 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ea2424f211d1167717197695a1bb59f56f24a1a98c0efb798094c67585c15c2\": container with ID starting with 2ea2424f211d1167717197695a1bb59f56f24a1a98c0efb798094c67585c15c2 not found: ID does not exist" containerID="2ea2424f211d1167717197695a1bb59f56f24a1a98c0efb798094c67585c15c2" Oct 30 17:35:26.207547 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.207437 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ea2424f211d1167717197695a1bb59f56f24a1a98c0efb798094c67585c15c2"} err="failed to get container status \"2ea2424f211d1167717197695a1bb59f56f24a1a98c0efb798094c67585c15c2\": rpc error: code = NotFound desc = could not find container \"2ea2424f211d1167717197695a1bb59f56f24a1a98c0efb798094c67585c15c2\": container with ID starting with 2ea2424f211d1167717197695a1bb59f56f24a1a98c0efb798094c67585c15c2 not found: ID does not exist" Oct 30 17:35:26.217036 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.217000 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e520acad-555a-429f-be9a-928ef8749774-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e520acad-555a-429f-be9a-928ef8749774-plugins-config-volume\") pod \"e520acad-555a-429f-be9a-928ef8749774\" (UID: \"e520acad-555a-429f-be9a-928ef8749774\") " Oct 30 17:35:26.217146 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.217129 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e520acad-555a-429f-be9a-928ef8749774-kube-api-access-74p2n\" (UniqueName: \"kubernetes.io/projected/e520acad-555a-429f-be9a-928ef8749774-kube-api-access-74p2n\") pod \"e520acad-555a-429f-be9a-928ef8749774\" (UID: \"e520acad-555a-429f-be9a-928ef8749774\") " Oct 30 17:35:26.217331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.217300 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zz6g\" (UniqueName: \"kubernetes.io/projected/b88fde22-4b6e-4a63-a0ba-aa3e812472da-kube-api-access-5zz6g\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6d4f99-rwdd8\" (UID: \"b88fde22-4b6e-4a63-a0ba-aa3e812472da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6d4f99-rwdd8" Oct 30 17:35:26.217461 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.217419 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e520acad-555a-429f-be9a-928ef8749774-plugins-config-volume" pod "e520acad-555a-429f-be9a-928ef8749774" (UID: "e520acad-555a-429f-be9a-928ef8749774"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:35:26.217592 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.217570 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b88fde22-4b6e-4a63-a0ba-aa3e812472da-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6d4f99-rwdd8\" (UID: \"b88fde22-4b6e-4a63-a0ba-aa3e812472da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6d4f99-rwdd8" Oct 30 17:35:26.217707 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.217691 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b88fde22-4b6e-4a63-a0ba-aa3e812472da-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6d4f99-rwdd8\" (UID: \"b88fde22-4b6e-4a63-a0ba-aa3e812472da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6d4f99-rwdd8" Oct 30 17:35:26.218000 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.217979 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e520acad-555a-429f-be9a-928ef8749774-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:35:26.218087 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.218002 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vf8m\" (UniqueName: \"kubernetes.io/projected/b4c4560b-846b-40b9-b143-bab12720112a-kube-api-access-7vf8m\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:35:26.220102 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.220081 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e520acad-555a-429f-be9a-928ef8749774-kube-api-access-74p2n" pod "e520acad-555a-429f-be9a-928ef8749774" (UID: "e520acad-555a-429f-be9a-928ef8749774"). InnerVolumeSpecName "kube-api-access-74p2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:35:26.310091 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.310055 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-11f480-59c5945bbd-fjf2v"] Oct 30 17:35:26.318870 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.318841 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b88fde22-4b6e-4a63-a0ba-aa3e812472da-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6d4f99-rwdd8\" (UID: \"b88fde22-4b6e-4a63-a0ba-aa3e812472da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6d4f99-rwdd8" Oct 30 17:35:26.319003 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.318906 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b88fde22-4b6e-4a63-a0ba-aa3e812472da-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6d4f99-rwdd8\" (UID: \"b88fde22-4b6e-4a63-a0ba-aa3e812472da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6d4f99-rwdd8" Oct 30 17:35:26.319003 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.318977 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zz6g\" (UniqueName: \"kubernetes.io/projected/b88fde22-4b6e-4a63-a0ba-aa3e812472da-kube-api-access-5zz6g\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6d4f99-rwdd8\" (UID: \"b88fde22-4b6e-4a63-a0ba-aa3e812472da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6d4f99-rwdd8" Oct 30 17:35:26.319102 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.319025 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74p2n\" (UniqueName: \"kubernetes.io/projected/e520acad-555a-429f-be9a-928ef8749774-kube-api-access-74p2n\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:35:26.319637 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.319618 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b88fde22-4b6e-4a63-a0ba-aa3e812472da-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6d4f99-rwdd8\" (UID: \"b88fde22-4b6e-4a63-a0ba-aa3e812472da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6d4f99-rwdd8" Oct 30 17:35:26.321241 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.321220 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b88fde22-4b6e-4a63-a0ba-aa3e812472da-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6d4f99-rwdd8\" (UID: \"b88fde22-4b6e-4a63-a0ba-aa3e812472da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6d4f99-rwdd8" Oct 30 17:35:26.322153 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.322132 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-11f480-59c5945bbd-fjf2v"] Oct 30 17:35:26.354157 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.354121 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zz6g\" (UniqueName: \"kubernetes.io/projected/b88fde22-4b6e-4a63-a0ba-aa3e812472da-kube-api-access-5zz6g\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6d4f99-rwdd8\" (UID: \"b88fde22-4b6e-4a63-a0ba-aa3e812472da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6d4f99-rwdd8" Oct 30 17:35:26.381877 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.381844 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a43843-86c7c5b7-dxhbc"] Oct 30 17:35:26.401335 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.401294 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a43843-86c7c5b7-dxhbc"] Oct 30 17:35:26.459502 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.459477 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6d4f99-rwdd8" Oct 30 17:35:26.497915 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.497881 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-11f480-5d57fc56dc-w48fx"] Oct 30 17:35:26.512802 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.512753 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-11f480-5d57fc56dc-w48fx"] Oct 30 17:35:26.776775 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:35:26.776733 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb88fde22_4b6e_4a63_a0ba_aa3e812472da.slice/crio-40bad0f5f648c94e2a7e03857adc61c98fc016896bf05efa2acc90e174e5df57 WatchSource:0}: Error finding container 40bad0f5f648c94e2a7e03857adc61c98fc016896bf05efa2acc90e174e5df57: Status 404 returned error can't find the container with id 40bad0f5f648c94e2a7e03857adc61c98fc016896bf05efa2acc90e174e5df57 Oct 30 17:35:26.777778 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:26.777749 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6d4f99-rwdd8"] Oct 30 17:35:27.164509 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:27.164461 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6d4f99-rwdd8" event={"ID":"b88fde22-4b6e-4a63-a0ba-aa3e812472da","Type":"ContainerStarted","Data":"83c02e96ba2d051cfe1c4ecc0e0c23803afca77d0ce59b2f5a7b9902ba4870b7"} Oct 30 17:35:27.164888 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:27.164526 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6d4f99-rwdd8" event={"ID":"b88fde22-4b6e-4a63-a0ba-aa3e812472da","Type":"ContainerStarted","Data":"40bad0f5f648c94e2a7e03857adc61c98fc016896bf05efa2acc90e174e5df57"} Oct 30 17:35:27.212549 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:27.212495 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6d4f99-rwdd8" podStartSLOduration=1.212471727 podStartE2EDuration="1.212471727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:35:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:35:27.211620843 +0000 UTC m=+2901.212375756" watchObservedRunningTime="2026-10-30 17:35:27.212471727 +0000 UTC m=+2901.213226629" Oct 30 17:35:28.147365 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:28.147269 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4c4560b-846b-40b9-b143-bab12720112a" path="/var/lib/kubelet/pods/b4c4560b-846b-40b9-b143-bab12720112a/volumes" Oct 30 17:35:28.147765 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:28.147747 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e520acad-555a-429f-be9a-928ef8749774" path="/var/lib/kubelet/pods/e520acad-555a-429f-be9a-928ef8749774/volumes" Oct 30 17:35:28.148120 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:28.148108 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f396ca5e-93c9-4dd4-a03e-f9150e784873" path="/var/lib/kubelet/pods/f396ca5e-93c9-4dd4-a03e-f9150e784873/volumes" Oct 30 17:35:28.173983 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:28.173953 4522 generic.go:356] "Generic (PLEG): container finished" podID="b88fde22-4b6e-4a63-a0ba-aa3e812472da" containerID="83c02e96ba2d051cfe1c4ecc0e0c23803afca77d0ce59b2f5a7b9902ba4870b7" exitCode=0 Oct 30 17:35:28.174137 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:28.174038 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6d4f99-rwdd8" event={"ID":"b88fde22-4b6e-4a63-a0ba-aa3e812472da","Type":"ContainerDied","Data":"83c02e96ba2d051cfe1c4ecc0e0c23803afca77d0ce59b2f5a7b9902ba4870b7"} Oct 30 17:35:28.851006 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:28.850970 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8681b5-t4zkh"] Oct 30 17:35:28.851939 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:28.851910 4522 cpu_manager.go:414] "containerMap: removing container" podUID="ef9c4eec-8676-4a8c-9913-da321bb96baf" containerName="istio-proxy" Oct 30 17:35:28.851939 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:28.851933 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef9c4eec-8676-4a8c-9913-da321bb96baf" containerName="istio-proxy" Oct 30 17:35:28.857231 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:28.857208 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8681b5-t4zkh" Oct 30 17:35:28.875880 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:28.875851 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8681b5-t4zkh"] Oct 30 17:35:28.894036 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:28.894004 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh5tq\" (UniqueName: \"kubernetes.io/projected/7b57591a-1a9e-463c-aca5-45a2492c8689-kube-api-access-jh5tq\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-8681b5-t4zkh\" (UID: \"7b57591a-1a9e-463c-aca5-45a2492c8689\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8681b5-t4zkh" Oct 30 17:35:28.894172 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:28.894155 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b57591a-1a9e-463c-aca5-45a2492c8689-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-8681b5-t4zkh\" (UID: \"7b57591a-1a9e-463c-aca5-45a2492c8689\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8681b5-t4zkh" Oct 30 17:35:28.894212 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:28.894192 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b57591a-1a9e-463c-aca5-45a2492c8689-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-8681b5-t4zkh\" (UID: \"7b57591a-1a9e-463c-aca5-45a2492c8689\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8681b5-t4zkh" Oct 30 17:35:28.995562 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:28.995527 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh5tq\" (UniqueName: \"kubernetes.io/projected/7b57591a-1a9e-463c-aca5-45a2492c8689-kube-api-access-jh5tq\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-8681b5-t4zkh\" (UID: \"7b57591a-1a9e-463c-aca5-45a2492c8689\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8681b5-t4zkh" Oct 30 17:35:28.995746 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:28.995632 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b57591a-1a9e-463c-aca5-45a2492c8689-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-8681b5-t4zkh\" (UID: \"7b57591a-1a9e-463c-aca5-45a2492c8689\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8681b5-t4zkh" Oct 30 17:35:28.995746 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:28.995660 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b57591a-1a9e-463c-aca5-45a2492c8689-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-8681b5-t4zkh\" (UID: \"7b57591a-1a9e-463c-aca5-45a2492c8689\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8681b5-t4zkh" Oct 30 17:35:28.996340 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:28.996302 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b57591a-1a9e-463c-aca5-45a2492c8689-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-8681b5-t4zkh\" (UID: \"7b57591a-1a9e-463c-aca5-45a2492c8689\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8681b5-t4zkh" Oct 30 17:35:28.998029 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:28.998009 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b57591a-1a9e-463c-aca5-45a2492c8689-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-8681b5-t4zkh\" (UID: \"7b57591a-1a9e-463c-aca5-45a2492c8689\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8681b5-t4zkh" Oct 30 17:35:29.020848 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:29.020815 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh5tq\" (UniqueName: \"kubernetes.io/projected/7b57591a-1a9e-463c-aca5-45a2492c8689-kube-api-access-jh5tq\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-8681b5-t4zkh\" (UID: \"7b57591a-1a9e-463c-aca5-45a2492c8689\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8681b5-t4zkh" Oct 30 17:35:29.172830 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:29.172749 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8681b5-t4zkh" Oct 30 17:35:29.375960 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:29.375925 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6d4f99-rwdd8" Oct 30 17:35:29.399982 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:29.399952 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b88fde22-4b6e-4a63-a0ba-aa3e812472da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b88fde22-4b6e-4a63-a0ba-aa3e812472da-maas-api-openshift-ca\") pod \"b88fde22-4b6e-4a63-a0ba-aa3e812472da\" (UID: \"b88fde22-4b6e-4a63-a0ba-aa3e812472da\") " Oct 30 17:35:29.400173 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:29.399996 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b88fde22-4b6e-4a63-a0ba-aa3e812472da-kube-api-access-5zz6g\" (UniqueName: \"kubernetes.io/projected/b88fde22-4b6e-4a63-a0ba-aa3e812472da-kube-api-access-5zz6g\") pod \"b88fde22-4b6e-4a63-a0ba-aa3e812472da\" (UID: \"b88fde22-4b6e-4a63-a0ba-aa3e812472da\") " Oct 30 17:35:29.400173 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:29.400144 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b88fde22-4b6e-4a63-a0ba-aa3e812472da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b88fde22-4b6e-4a63-a0ba-aa3e812472da-maas-api-xks-ca\") pod \"b88fde22-4b6e-4a63-a0ba-aa3e812472da\" (UID: \"b88fde22-4b6e-4a63-a0ba-aa3e812472da\") " Oct 30 17:35:29.400302 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:29.400279 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b88fde22-4b6e-4a63-a0ba-aa3e812472da-maas-api-openshift-ca" pod "b88fde22-4b6e-4a63-a0ba-aa3e812472da" (UID: "b88fde22-4b6e-4a63-a0ba-aa3e812472da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:35:29.400644 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:29.400627 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b88fde22-4b6e-4a63-a0ba-aa3e812472da-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:35:29.402249 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:29.402225 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b88fde22-4b6e-4a63-a0ba-aa3e812472da-kube-api-access-5zz6g" pod "b88fde22-4b6e-4a63-a0ba-aa3e812472da" (UID: "b88fde22-4b6e-4a63-a0ba-aa3e812472da"). InnerVolumeSpecName "kube-api-access-5zz6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:35:29.402378 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:29.402246 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b88fde22-4b6e-4a63-a0ba-aa3e812472da-maas-api-xks-ca" pod "b88fde22-4b6e-4a63-a0ba-aa3e812472da" (UID: "b88fde22-4b6e-4a63-a0ba-aa3e812472da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:35:29.485499 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:35:29.485461 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b57591a_1a9e_463c_aca5_45a2492c8689.slice/crio-c46156249c6546dfc88d8dc37425a19cd27844b4272c1b2db9e324059d1a72ce WatchSource:0}: Error finding container c46156249c6546dfc88d8dc37425a19cd27844b4272c1b2db9e324059d1a72ce: Status 404 returned error can't find the container with id c46156249c6546dfc88d8dc37425a19cd27844b4272c1b2db9e324059d1a72ce Oct 30 17:35:29.486644 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:29.486618 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8681b5-t4zkh"] Oct 30 17:35:29.502279 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:29.502245 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b88fde22-4b6e-4a63-a0ba-aa3e812472da-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:35:29.502279 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:29.502276 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zz6g\" (UniqueName: \"kubernetes.io/projected/b88fde22-4b6e-4a63-a0ba-aa3e812472da-kube-api-access-5zz6g\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:35:30.191699 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:30.191659 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8681b5-t4zkh" event={"ID":"7b57591a-1a9e-463c-aca5-45a2492c8689","Type":"ContainerStarted","Data":"3d5820a1611860355bf8cd7f388dbe3ad6e6045cd47d56d5041b78c74cf9c9d3"} Oct 30 17:35:30.192155 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:30.191731 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8681b5-t4zkh" event={"ID":"7b57591a-1a9e-463c-aca5-45a2492c8689","Type":"ContainerStarted","Data":"c46156249c6546dfc88d8dc37425a19cd27844b4272c1b2db9e324059d1a72ce"} Oct 30 17:35:30.193153 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:30.193127 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6d4f99-rwdd8" Oct 30 17:35:30.193256 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:30.193167 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6d4f99-rwdd8" event={"ID":"b88fde22-4b6e-4a63-a0ba-aa3e812472da","Type":"ContainerDied","Data":"40bad0f5f648c94e2a7e03857adc61c98fc016896bf05efa2acc90e174e5df57"} Oct 30 17:35:30.193256 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:30.193207 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40bad0f5f648c94e2a7e03857adc61c98fc016896bf05efa2acc90e174e5df57" Oct 30 17:35:30.236624 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:30.236574 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8681b5-t4zkh" podStartSLOduration=2.236556532 podStartE2EDuration="2.236556532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:35:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:35:30.2355025 +0000 UTC m=+2904.236257412" watchObservedRunningTime="2026-10-30 17:35:30.236556532 +0000 UTC m=+2904.237311502" Oct 30 17:35:30.882014 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:30.881985 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6" Oct 30 17:35:30.903583 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:30.903558 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595" Oct 30 17:35:30.921370 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:30.921344 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d534a90d-d839-4d06-b709-83d44c667b86-kube-api-access-fmnk2\" (UniqueName: \"kubernetes.io/projected/d534a90d-d839-4d06-b709-83d44c667b86-kube-api-access-fmnk2\") pod \"d534a90d-d839-4d06-b709-83d44c667b86\" (UID: \"d534a90d-d839-4d06-b709-83d44c667b86\") " Oct 30 17:35:30.921516 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:30.921445 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d534a90d-d839-4d06-b709-83d44c667b86-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d534a90d-d839-4d06-b709-83d44c667b86-istio-podinfo\") pod \"d534a90d-d839-4d06-b709-83d44c667b86\" (UID: \"d534a90d-d839-4d06-b709-83d44c667b86\") " Oct 30 17:35:30.921516 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:30.921471 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a5d5676-2fbf-4892-8b12-862c483b2d17-kube-api-access-4gczm\" (UniqueName: \"kubernetes.io/projected/1a5d5676-2fbf-4892-8b12-862c483b2d17-kube-api-access-4gczm\") pod \"1a5d5676-2fbf-4892-8b12-862c483b2d17\" (UID: \"1a5d5676-2fbf-4892-8b12-862c483b2d17\") " Oct 30 17:35:30.921602 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:30.921529 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d534a90d-d839-4d06-b709-83d44c667b86-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d534a90d-d839-4d06-b709-83d44c667b86-istiod-ca-cert\") pod \"d534a90d-d839-4d06-b709-83d44c667b86\" (UID: \"d534a90d-d839-4d06-b709-83d44c667b86\") " Oct 30 17:35:30.921602 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:30.921563 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d534a90d-d839-4d06-b709-83d44c667b86-istio-token\" (UniqueName: \"kubernetes.io/projected/d534a90d-d839-4d06-b709-83d44c667b86-istio-token\") pod \"d534a90d-d839-4d06-b709-83d44c667b86\" (UID: \"d534a90d-d839-4d06-b709-83d44c667b86\") " Oct 30 17:35:30.921707 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:30.921605 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d534a90d-d839-4d06-b709-83d44c667b86-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d534a90d-d839-4d06-b709-83d44c667b86-istio-envoy\") pod \"d534a90d-d839-4d06-b709-83d44c667b86\" (UID: \"d534a90d-d839-4d06-b709-83d44c667b86\") " Oct 30 17:35:30.921707 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:30.921635 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a5d5676-2fbf-4892-8b12-862c483b2d17-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a5d5676-2fbf-4892-8b12-862c483b2d17-credential-socket\") pod \"1a5d5676-2fbf-4892-8b12-862c483b2d17\" (UID: \"1a5d5676-2fbf-4892-8b12-862c483b2d17\") " Oct 30 17:35:30.921707 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:30.921673 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a5d5676-2fbf-4892-8b12-862c483b2d17-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a5d5676-2fbf-4892-8b12-862c483b2d17-workload-socket\") pod \"1a5d5676-2fbf-4892-8b12-862c483b2d17\" (UID: \"1a5d5676-2fbf-4892-8b12-862c483b2d17\") " Oct 30 17:35:30.921868 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:30.921759 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d534a90d-d839-4d06-b709-83d44c667b86-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d534a90d-d839-4d06-b709-83d44c667b86-istio-data\") pod \"d534a90d-d839-4d06-b709-83d44c667b86\" (UID: \"d534a90d-d839-4d06-b709-83d44c667b86\") " Oct 30 17:35:30.921868 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:30.921824 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a5d5676-2fbf-4892-8b12-862c483b2d17-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a5d5676-2fbf-4892-8b12-862c483b2d17-istio-envoy\") pod \"1a5d5676-2fbf-4892-8b12-862c483b2d17\" (UID: \"1a5d5676-2fbf-4892-8b12-862c483b2d17\") " Oct 30 17:35:30.921978 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:30.921870 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a5d5676-2fbf-4892-8b12-862c483b2d17-istio-token\" (UniqueName: \"kubernetes.io/projected/1a5d5676-2fbf-4892-8b12-862c483b2d17-istio-token\") pod \"1a5d5676-2fbf-4892-8b12-862c483b2d17\" (UID: \"1a5d5676-2fbf-4892-8b12-862c483b2d17\") " Oct 30 17:35:30.921978 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:30.921898 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a5d5676-2fbf-4892-8b12-862c483b2d17-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a5d5676-2fbf-4892-8b12-862c483b2d17-workload-certs\") pod \"1a5d5676-2fbf-4892-8b12-862c483b2d17\" (UID: \"1a5d5676-2fbf-4892-8b12-862c483b2d17\") " Oct 30 17:35:30.921978 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:30.921944 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a5d5676-2fbf-4892-8b12-862c483b2d17-credential-socket" pod "1a5d5676-2fbf-4892-8b12-862c483b2d17" (UID: "1a5d5676-2fbf-4892-8b12-862c483b2d17"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:35:30.921978 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:30.921952 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a5d5676-2fbf-4892-8b12-862c483b2d17-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a5d5676-2fbf-4892-8b12-862c483b2d17-istiod-ca-cert\") pod \"1a5d5676-2fbf-4892-8b12-862c483b2d17\" (UID: \"1a5d5676-2fbf-4892-8b12-862c483b2d17\") " Oct 30 17:35:30.922200 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:30.921995 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d534a90d-d839-4d06-b709-83d44c667b86-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d534a90d-d839-4d06-b709-83d44c667b86-workload-socket\") pod \"d534a90d-d839-4d06-b709-83d44c667b86\" (UID: \"d534a90d-d839-4d06-b709-83d44c667b86\") " Oct 30 17:35:30.922200 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:30.922009 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d534a90d-d839-4d06-b709-83d44c667b86-istiod-ca-cert" pod "d534a90d-d839-4d06-b709-83d44c667b86" (UID: "d534a90d-d839-4d06-b709-83d44c667b86"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:35:30.922200 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:30.922046 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d534a90d-d839-4d06-b709-83d44c667b86-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d534a90d-d839-4d06-b709-83d44c667b86-credential-socket\") pod \"d534a90d-d839-4d06-b709-83d44c667b86\" (UID: \"d534a90d-d839-4d06-b709-83d44c667b86\") " Oct 30 17:35:30.922200 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:30.922089 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a5d5676-2fbf-4892-8b12-862c483b2d17-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a5d5676-2fbf-4892-8b12-862c483b2d17-istio-data\") pod \"1a5d5676-2fbf-4892-8b12-862c483b2d17\" (UID: \"1a5d5676-2fbf-4892-8b12-862c483b2d17\") " Oct 30 17:35:30.922200 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:30.922155 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a5d5676-2fbf-4892-8b12-862c483b2d17-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a5d5676-2fbf-4892-8b12-862c483b2d17-istio-podinfo\") pod \"1a5d5676-2fbf-4892-8b12-862c483b2d17\" (UID: \"1a5d5676-2fbf-4892-8b12-862c483b2d17\") " Oct 30 17:35:30.922200 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:30.922200 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d534a90d-d839-4d06-b709-83d44c667b86-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d534a90d-d839-4d06-b709-83d44c667b86-workload-certs\") pod \"d534a90d-d839-4d06-b709-83d44c667b86\" (UID: \"d534a90d-d839-4d06-b709-83d44c667b86\") " Oct 30 17:35:30.922581 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:30.922284 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a5d5676-2fbf-4892-8b12-862c483b2d17-workload-certs" pod "1a5d5676-2fbf-4892-8b12-862c483b2d17" (UID: "1a5d5676-2fbf-4892-8b12-862c483b2d17"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:35:30.922642 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:30.922583 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d534a90d-d839-4d06-b709-83d44c667b86-credential-socket" pod "d534a90d-d839-4d06-b709-83d44c667b86" (UID: "d534a90d-d839-4d06-b709-83d44c667b86"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:35:30.922696 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:30.922666 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a5d5676-2fbf-4892-8b12-862c483b2d17-istio-data" pod "1a5d5676-2fbf-4892-8b12-862c483b2d17" (UID: "1a5d5676-2fbf-4892-8b12-862c483b2d17"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:35:30.922696 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:30.922671 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d534a90d-d839-4d06-b709-83d44c667b86-workload-certs" pod "d534a90d-d839-4d06-b709-83d44c667b86" (UID: "d534a90d-d839-4d06-b709-83d44c667b86"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:35:30.923190 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:30.922875 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d534a90d-d839-4d06-b709-83d44c667b86-workload-socket" pod "d534a90d-d839-4d06-b709-83d44c667b86" (UID: "d534a90d-d839-4d06-b709-83d44c667b86"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:35:30.923190 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:30.923020 4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d534a90d-d839-4d06-b709-83d44c667b86-workload-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:35:30.923190 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:30.923032 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a5d5676-2fbf-4892-8b12-862c483b2d17-workload-socket" pod "1a5d5676-2fbf-4892-8b12-862c483b2d17" (UID: "1a5d5676-2fbf-4892-8b12-862c483b2d17"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:35:30.923190 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:30.923035 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a5d5676-2fbf-4892-8b12-862c483b2d17-istiod-ca-cert" pod "1a5d5676-2fbf-4892-8b12-862c483b2d17" (UID: "1a5d5676-2fbf-4892-8b12-862c483b2d17"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:35:30.923190 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:30.923040 4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d534a90d-d839-4d06-b709-83d44c667b86-istiod-ca-cert\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:35:30.923190 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:30.923071 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d534a90d-d839-4d06-b709-83d44c667b86-istio-data" pod "d534a90d-d839-4d06-b709-83d44c667b86" (UID: "d534a90d-d839-4d06-b709-83d44c667b86"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:35:30.923190 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:30.923090 4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a5d5676-2fbf-4892-8b12-862c483b2d17-credential-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:35:30.923190 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:30.923112 4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a5d5676-2fbf-4892-8b12-862c483b2d17-workload-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:35:30.923190 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:30.923132 4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d534a90d-d839-4d06-b709-83d44c667b86-workload-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:35:30.923190 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:30.923150 4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d534a90d-d839-4d06-b709-83d44c667b86-credential-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:35:30.923190 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:30.923164 4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a5d5676-2fbf-4892-8b12-862c483b2d17-istio-data\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:35:30.924269 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:30.924233 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d534a90d-d839-4d06-b709-83d44c667b86-istio-podinfo" pod "d534a90d-d839-4d06-b709-83d44c667b86" (UID: "d534a90d-d839-4d06-b709-83d44c667b86"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:35:30.924903 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:30.924561 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a5d5676-2fbf-4892-8b12-862c483b2d17-istio-envoy" pod "1a5d5676-2fbf-4892-8b12-862c483b2d17" (UID: "1a5d5676-2fbf-4892-8b12-862c483b2d17"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:35:30.924903 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:30.924726 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a5d5676-2fbf-4892-8b12-862c483b2d17-istio-token" pod "1a5d5676-2fbf-4892-8b12-862c483b2d17" (UID: "1a5d5676-2fbf-4892-8b12-862c483b2d17"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:35:30.924903 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:30.924813 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a5d5676-2fbf-4892-8b12-862c483b2d17-kube-api-access-4gczm" pod "1a5d5676-2fbf-4892-8b12-862c483b2d17" (UID: "1a5d5676-2fbf-4892-8b12-862c483b2d17"). InnerVolumeSpecName "kube-api-access-4gczm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:35:30.924903 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:30.924847 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d534a90d-d839-4d06-b709-83d44c667b86-kube-api-access-fmnk2" pod "d534a90d-d839-4d06-b709-83d44c667b86" (UID: "d534a90d-d839-4d06-b709-83d44c667b86"). InnerVolumeSpecName "kube-api-access-fmnk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:35:30.925204 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:30.925149 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a5d5676-2fbf-4892-8b12-862c483b2d17-istio-podinfo" pod "1a5d5676-2fbf-4892-8b12-862c483b2d17" (UID: "1a5d5676-2fbf-4892-8b12-862c483b2d17"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:35:30.925676 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:30.925659 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d534a90d-d839-4d06-b709-83d44c667b86-istio-token" pod "d534a90d-d839-4d06-b709-83d44c667b86" (UID: "d534a90d-d839-4d06-b709-83d44c667b86"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:35:30.925676 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:30.925664 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d534a90d-d839-4d06-b709-83d44c667b86-istio-envoy" pod "d534a90d-d839-4d06-b709-83d44c667b86" (UID: "d534a90d-d839-4d06-b709-83d44c667b86"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:35:31.026169 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:31.025989 4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d534a90d-d839-4d06-b709-83d44c667b86-istio-podinfo\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:35:31.026169 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:31.026026 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gczm\" (UniqueName: \"kubernetes.io/projected/1a5d5676-2fbf-4892-8b12-862c483b2d17-kube-api-access-4gczm\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:35:31.026169 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:31.026042 4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d534a90d-d839-4d06-b709-83d44c667b86-istio-token\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:35:31.026169 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:31.026055 4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d534a90d-d839-4d06-b709-83d44c667b86-istio-envoy\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:35:31.026169 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:31.026068 4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a5d5676-2fbf-4892-8b12-862c483b2d17-workload-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:35:31.026169 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:31.026080 4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d534a90d-d839-4d06-b709-83d44c667b86-istio-data\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:35:31.026169 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:31.026094 4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a5d5676-2fbf-4892-8b12-862c483b2d17-istio-envoy\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:35:31.026169 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:31.026108 4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a5d5676-2fbf-4892-8b12-862c483b2d17-istio-token\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:35:31.026169 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:31.026121 4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a5d5676-2fbf-4892-8b12-862c483b2d17-istiod-ca-cert\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:35:31.026169 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:31.026132 4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a5d5676-2fbf-4892-8b12-862c483b2d17-istio-podinfo\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:35:31.026169 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:31.026144 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmnk2\" (UniqueName: \"kubernetes.io/projected/d534a90d-d839-4d06-b709-83d44c667b86-kube-api-access-fmnk2\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:35:31.202038 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:31.202000 4522 generic.go:356] "Generic (PLEG): container finished" podID="d534a90d-d839-4d06-b709-83d44c667b86" containerID="2c27612d534dc3ed63ca17ee6308591836c5dfdbd5e3da315c91c0b7b043f34e" exitCode=0 Oct 30 17:35:31.202518 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:31.202052 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6" Oct 30 17:35:31.202518 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:31.202092 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6" event={"ID":"d534a90d-d839-4d06-b709-83d44c667b86","Type":"ContainerDied","Data":"2c27612d534dc3ed63ca17ee6308591836c5dfdbd5e3da315c91c0b7b043f34e"} Oct 30 17:35:31.202518 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:31.202149 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6" event={"ID":"d534a90d-d839-4d06-b709-83d44c667b86","Type":"ContainerDied","Data":"e14af1cff3d036e4c86d37d3403deadcecf4d7f02aa64fd0b51f950ced60b6ee"} Oct 30 17:35:31.202518 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:31.202178 4522 scope.go:122] "RemoveContainer" containerID="2c27612d534dc3ed63ca17ee6308591836c5dfdbd5e3da315c91c0b7b043f34e" Oct 30 17:35:31.203572 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:31.203551 4522 generic.go:356] "Generic (PLEG): container finished" podID="1a5d5676-2fbf-4892-8b12-862c483b2d17" containerID="4bfd3354e7a42dae397dd62b3824c2129b5c1b1bf65841352b13313c5eed0a4c" exitCode=0 Oct 30 17:35:31.203660 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:31.203580 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595" event={"ID":"1a5d5676-2fbf-4892-8b12-862c483b2d17","Type":"ContainerDied","Data":"4bfd3354e7a42dae397dd62b3824c2129b5c1b1bf65841352b13313c5eed0a4c"} Oct 30 17:35:31.203660 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:31.203608 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595" Oct 30 17:35:31.203660 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:31.203620 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595" event={"ID":"1a5d5676-2fbf-4892-8b12-862c483b2d17","Type":"ContainerDied","Data":"d24e113a8041fdbde7528723ed7a807a4a5208036d7a0b3e89bfa86277999443"} Oct 30 17:35:31.205504 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:31.205479 4522 generic.go:356] "Generic (PLEG): container finished" podID="7b57591a-1a9e-463c-aca5-45a2492c8689" containerID="3d5820a1611860355bf8cd7f388dbe3ad6e6045cd47d56d5041b78c74cf9c9d3" exitCode=0 Oct 30 17:35:31.205595 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:31.205583 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8681b5-t4zkh" event={"ID":"7b57591a-1a9e-463c-aca5-45a2492c8689","Type":"ContainerDied","Data":"3d5820a1611860355bf8cd7f388dbe3ad6e6045cd47d56d5041b78c74cf9c9d3"} Oct 30 17:35:31.214983 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:31.214968 4522 scope.go:122] "RemoveContainer" containerID="2c27612d534dc3ed63ca17ee6308591836c5dfdbd5e3da315c91c0b7b043f34e" Oct 30 17:35:31.215255 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:35:31.215231 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c27612d534dc3ed63ca17ee6308591836c5dfdbd5e3da315c91c0b7b043f34e\": container with ID starting with 2c27612d534dc3ed63ca17ee6308591836c5dfdbd5e3da315c91c0b7b043f34e not found: ID does not exist" containerID="2c27612d534dc3ed63ca17ee6308591836c5dfdbd5e3da315c91c0b7b043f34e" Oct 30 17:35:31.215378 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:31.215264 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c27612d534dc3ed63ca17ee6308591836c5dfdbd5e3da315c91c0b7b043f34e"} err="failed to get container status \"2c27612d534dc3ed63ca17ee6308591836c5dfdbd5e3da315c91c0b7b043f34e\": rpc error: code = NotFound desc = could not find container \"2c27612d534dc3ed63ca17ee6308591836c5dfdbd5e3da315c91c0b7b043f34e\": container with ID starting with 2c27612d534dc3ed63ca17ee6308591836c5dfdbd5e3da315c91c0b7b043f34e not found: ID does not exist" Oct 30 17:35:31.215378 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:31.215280 4522 scope.go:122] "RemoveContainer" containerID="4bfd3354e7a42dae397dd62b3824c2129b5c1b1bf65841352b13313c5eed0a4c" Oct 30 17:35:31.226943 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:31.226926 4522 scope.go:122] "RemoveContainer" containerID="4bfd3354e7a42dae397dd62b3824c2129b5c1b1bf65841352b13313c5eed0a4c" Oct 30 17:35:31.227233 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:35:31.227209 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bfd3354e7a42dae397dd62b3824c2129b5c1b1bf65841352b13313c5eed0a4c\": container with ID starting with 4bfd3354e7a42dae397dd62b3824c2129b5c1b1bf65841352b13313c5eed0a4c not found: ID does not exist" containerID="4bfd3354e7a42dae397dd62b3824c2129b5c1b1bf65841352b13313c5eed0a4c" Oct 30 17:35:31.227361 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:31.227240 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bfd3354e7a42dae397dd62b3824c2129b5c1b1bf65841352b13313c5eed0a4c"} err="failed to get container status \"4bfd3354e7a42dae397dd62b3824c2129b5c1b1bf65841352b13313c5eed0a4c\": rpc error: code = NotFound desc = could not find container \"4bfd3354e7a42dae397dd62b3824c2129b5c1b1bf65841352b13313c5eed0a4c\": container with ID starting with 4bfd3354e7a42dae397dd62b3824c2129b5c1b1bf65841352b13313c5eed0a4c not found: ID does not exist" Oct 30 17:35:31.310504 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:31.310423 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6"] Oct 30 17:35:31.313792 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:31.313769 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-hn5b6"] Oct 30 17:35:31.352171 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:31.352138 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595"] Oct 30 17:35:31.358631 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:31.358600 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-v6595"] Oct 30 17:35:32.146531 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:32.146501 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a5d5676-2fbf-4892-8b12-862c483b2d17" path="/var/lib/kubelet/pods/1a5d5676-2fbf-4892-8b12-862c483b2d17/volumes" Oct 30 17:35:32.147062 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:32.147045 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d534a90d-d839-4d06-b709-83d44c667b86" path="/var/lib/kubelet/pods/d534a90d-d839-4d06-b709-83d44c667b86/volumes" Oct 30 17:35:32.382181 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:32.382146 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8681b5-t4zkh" Oct 30 17:35:32.444840 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:32.444758 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b57591a-1a9e-463c-aca5-45a2492c8689-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b57591a-1a9e-463c-aca5-45a2492c8689-maas-api-openshift-ca\") pod \"7b57591a-1a9e-463c-aca5-45a2492c8689\" (UID: \"7b57591a-1a9e-463c-aca5-45a2492c8689\") " Oct 30 17:35:32.445009 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:32.444854 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b57591a-1a9e-463c-aca5-45a2492c8689-kube-api-access-jh5tq\" (UniqueName: \"kubernetes.io/projected/7b57591a-1a9e-463c-aca5-45a2492c8689-kube-api-access-jh5tq\") pod \"7b57591a-1a9e-463c-aca5-45a2492c8689\" (UID: \"7b57591a-1a9e-463c-aca5-45a2492c8689\") " Oct 30 17:35:32.445009 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:32.444890 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b57591a-1a9e-463c-aca5-45a2492c8689-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b57591a-1a9e-463c-aca5-45a2492c8689-maas-api-xks-ca\") pod \"7b57591a-1a9e-463c-aca5-45a2492c8689\" (UID: \"7b57591a-1a9e-463c-aca5-45a2492c8689\") " Oct 30 17:35:32.445199 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:32.445172 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b57591a-1a9e-463c-aca5-45a2492c8689-maas-api-openshift-ca" pod "7b57591a-1a9e-463c-aca5-45a2492c8689" (UID: "7b57591a-1a9e-463c-aca5-45a2492c8689"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:35:32.445523 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:32.445498 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b57591a-1a9e-463c-aca5-45a2492c8689-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:35:32.447090 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:32.447060 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b57591a-1a9e-463c-aca5-45a2492c8689-maas-api-xks-ca" pod "7b57591a-1a9e-463c-aca5-45a2492c8689" (UID: "7b57591a-1a9e-463c-aca5-45a2492c8689"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:35:32.447199 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:32.447103 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b57591a-1a9e-463c-aca5-45a2492c8689-kube-api-access-jh5tq" pod "7b57591a-1a9e-463c-aca5-45a2492c8689" (UID: "7b57591a-1a9e-463c-aca5-45a2492c8689"). InnerVolumeSpecName "kube-api-access-jh5tq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:35:32.547506 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:32.547469 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh5tq\" (UniqueName: \"kubernetes.io/projected/7b57591a-1a9e-463c-aca5-45a2492c8689-kube-api-access-jh5tq\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:35:32.547506 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:32.547507 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b57591a-1a9e-463c-aca5-45a2492c8689-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:35:33.231045 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:33.231006 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8681b5-t4zkh" event={"ID":"7b57591a-1a9e-463c-aca5-45a2492c8689","Type":"ContainerDied","Data":"c46156249c6546dfc88d8dc37425a19cd27844b4272c1b2db9e324059d1a72ce"} Oct 30 17:35:33.231237 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:33.231064 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c46156249c6546dfc88d8dc37425a19cd27844b4272c1b2db9e324059d1a72ce" Oct 30 17:35:33.231237 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:33.231023 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8681b5-t4zkh" Oct 30 17:35:36.202463 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:36.202428 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-6d4f99-59dc8b84b6-z9qh6"] Oct 30 17:35:38.950113 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:38.950067 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-8681b5-7d9676c6d6-khtxf"] Oct 30 17:35:38.984648 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:38.984615 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8681b5-d7469f799-jsz5b"] Oct 30 17:35:41.203241 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:41.203195 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6d4f99-59dc8b84b6-z9qh6" podUID="49b34c5e-95a9-43e7-8eb4-0034619e1733" containerName="payload-processing" containerID="cri-o://475908fe8b9912a214bf22fe965d27ced77dc9aacc8f21d968bf92781fb34081" gracePeriod=25 Oct 30 17:35:42.558398 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:42.558358 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6d4f99-59dc8b84b6-z9qh6" podUID="49b34c5e-95a9-43e7-8eb4-0034619e1733" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:35:42.948641 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:35:42.948555 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:35:43.950648 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:43.950602 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8681b5-7d9676c6d6-khtxf" podUID="ac7e71fc-8c27-4f6f-b9f2-bf5fbd115267" containerName="payload-processing" containerID="cri-o://a81c11b95b85566b3fa66e25ef6d1cbd67a8cdce505f8cd739ecaebd95d43108" gracePeriod=25 Oct 30 17:35:43.984932 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:43.984888 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8681b5-d7469f799-jsz5b" podUID="af8374c3-9f76-4183-9eee-d781479514a0" containerName="payload-pre-processing" containerID="cri-o://cf41024ad80e3b478243d0c04b796986fe8976a6568f6c7ea4f1b1df9eb68ecd" gracePeriod=25 Oct 30 17:35:45.729966 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:45.729924 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4fb821-77ff87d6fd-pq2qr"] Oct 30 17:35:45.730592 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:45.730194 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-4fb821-77ff87d6fd-pq2qr" podUID="b9e25d51-dd1c-419c-b938-656d13abb61a" containerName="payload-processing" containerID="cri-o://79535d25881a691d062a665025a4587551844b7d014f86b0f516ba680d641fa1" gracePeriod=30 Oct 30 17:35:46.019161 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:46.019120 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4fb821-77ff87d6fd-pq2qr" Oct 30 17:35:46.049663 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:46.049629 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9e25d51-dd1c-419c-b938-656d13abb61a-kube-api-access-xnzgm\" (UniqueName: \"kubernetes.io/projected/b9e25d51-dd1c-419c-b938-656d13abb61a-kube-api-access-xnzgm\") pod \"b9e25d51-dd1c-419c-b938-656d13abb61a\" (UID: \"b9e25d51-dd1c-419c-b938-656d13abb61a\") " Oct 30 17:35:46.049895 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:46.049876 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9e25d51-dd1c-419c-b938-656d13abb61a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9e25d51-dd1c-419c-b938-656d13abb61a-plugins-config-volume\") pod \"b9e25d51-dd1c-419c-b938-656d13abb61a\" (UID: \"b9e25d51-dd1c-419c-b938-656d13abb61a\") " Oct 30 17:35:46.050379 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:46.050347 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9e25d51-dd1c-419c-b938-656d13abb61a-plugins-config-volume" pod "b9e25d51-dd1c-419c-b938-656d13abb61a" (UID: "b9e25d51-dd1c-419c-b938-656d13abb61a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:35:46.052039 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:46.052007 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9e25d51-dd1c-419c-b938-656d13abb61a-kube-api-access-xnzgm" pod "b9e25d51-dd1c-419c-b938-656d13abb61a" (UID: "b9e25d51-dd1c-419c-b938-656d13abb61a"). InnerVolumeSpecName "kube-api-access-xnzgm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:35:46.154588 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:46.154542 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9e25d51-dd1c-419c-b938-656d13abb61a-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:35:46.154588 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:46.154575 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnzgm\" (UniqueName: \"kubernetes.io/projected/b9e25d51-dd1c-419c-b938-656d13abb61a-kube-api-access-xnzgm\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:35:46.344853 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:46.344750 4522 generic.go:356] "Generic (PLEG): container finished" podID="b9e25d51-dd1c-419c-b938-656d13abb61a" containerID="79535d25881a691d062a665025a4587551844b7d014f86b0f516ba680d641fa1" exitCode=0 Oct 30 17:35:46.344853 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:46.344800 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4fb821-77ff87d6fd-pq2qr" Oct 30 17:35:46.345125 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:46.344798 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4fb821-77ff87d6fd-pq2qr" event={"ID":"b9e25d51-dd1c-419c-b938-656d13abb61a","Type":"ContainerDied","Data":"79535d25881a691d062a665025a4587551844b7d014f86b0f516ba680d641fa1"} Oct 30 17:35:46.345125 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:46.344922 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4fb821-77ff87d6fd-pq2qr" event={"ID":"b9e25d51-dd1c-419c-b938-656d13abb61a","Type":"ContainerDied","Data":"7cf6491a3cc97d0628648ed23438dbb8557cbf8a94502d33e9429ff41a05b89e"} Oct 30 17:35:46.345125 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:46.344944 4522 scope.go:122] "RemoveContainer" containerID="79535d25881a691d062a665025a4587551844b7d014f86b0f516ba680d641fa1" Oct 30 17:35:46.355809 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:46.355784 4522 scope.go:122] "RemoveContainer" containerID="79535d25881a691d062a665025a4587551844b7d014f86b0f516ba680d641fa1" Oct 30 17:35:46.356082 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:35:46.356059 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79535d25881a691d062a665025a4587551844b7d014f86b0f516ba680d641fa1\": container with ID starting with 79535d25881a691d062a665025a4587551844b7d014f86b0f516ba680d641fa1 not found: ID does not exist" containerID="79535d25881a691d062a665025a4587551844b7d014f86b0f516ba680d641fa1" Oct 30 17:35:46.356163 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:46.356094 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79535d25881a691d062a665025a4587551844b7d014f86b0f516ba680d641fa1"} err="failed to get container status \"79535d25881a691d062a665025a4587551844b7d014f86b0f516ba680d641fa1\": rpc error: code = NotFound desc = could not find container \"79535d25881a691d062a665025a4587551844b7d014f86b0f516ba680d641fa1\": container with ID starting with 79535d25881a691d062a665025a4587551844b7d014f86b0f516ba680d641fa1 not found: ID does not exist" Oct 30 17:35:46.407429 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:46.407394 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4fb821-77ff87d6fd-pq2qr"] Oct 30 17:35:46.414811 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:46.414779 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4fb821-77ff87d6fd-pq2qr"] Oct 30 17:35:46.565893 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:46.565852 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4fb821-mqfs9"] Oct 30 17:35:46.571021 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:46.570996 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4fb821-mqfs9"] Oct 30 17:35:47.043800 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:47.043765 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-031223-79c4b96cc9-4twf6"] Oct 30 17:35:47.044443 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:47.044428 4522 cpu_manager.go:414] "containerMap: removing container" podUID="b9e25d51-dd1c-419c-b938-656d13abb61a" containerName="payload-processing" Oct 30 17:35:47.044518 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:47.044444 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9e25d51-dd1c-419c-b938-656d13abb61a" containerName="payload-processing" Oct 30 17:35:47.044518 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:47.044458 4522 cpu_manager.go:414] "containerMap: removing container" podUID="b88fde22-4b6e-4a63-a0ba-aa3e812472da" containerName="revoke-keys" Oct 30 17:35:47.044518 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:47.044467 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b88fde22-4b6e-4a63-a0ba-aa3e812472da" containerName="revoke-keys" Oct 30 17:35:47.044518 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:47.044489 4522 cpu_manager.go:414] "containerMap: removing container" podUID="d534a90d-d839-4d06-b709-83d44c667b86" containerName="istio-proxy" Oct 30 17:35:47.044518 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:47.044496 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d534a90d-d839-4d06-b709-83d44c667b86" containerName="istio-proxy" Oct 30 17:35:47.044518 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:47.044517 4522 cpu_manager.go:414] "containerMap: removing container" podUID="1a5d5676-2fbf-4892-8b12-862c483b2d17" containerName="istio-proxy" Oct 30 17:35:47.044726 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:47.044522 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a5d5676-2fbf-4892-8b12-862c483b2d17" containerName="istio-proxy" Oct 30 17:35:47.044726 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:47.044532 4522 cpu_manager.go:414] "containerMap: removing container" podUID="7b57591a-1a9e-463c-aca5-45a2492c8689" containerName="revoke-keys" Oct 30 17:35:47.044726 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:47.044537 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b57591a-1a9e-463c-aca5-45a2492c8689" containerName="revoke-keys" Oct 30 17:35:47.044726 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:47.044643 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9e25d51-dd1c-419c-b938-656d13abb61a" containerName="payload-processing" Oct 30 17:35:47.044726 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:47.044651 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="b88fde22-4b6e-4a63-a0ba-aa3e812472da" containerName="revoke-keys" Oct 30 17:35:47.044726 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:47.044658 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="d534a90d-d839-4d06-b709-83d44c667b86" containerName="istio-proxy" Oct 30 17:35:47.044726 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:47.044667 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a5d5676-2fbf-4892-8b12-862c483b2d17" containerName="istio-proxy" Oct 30 17:35:47.044726 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:47.044676 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b57591a-1a9e-463c-aca5-45a2492c8689" containerName="revoke-keys" Oct 30 17:35:47.049709 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:47.049671 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-031223-79c4b96cc9-4twf6" Oct 30 17:35:47.066721 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:47.066690 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-031223-serving-cert\"" Oct 30 17:35:47.067432 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:47.067386 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xkc5\" (UniqueName: \"kubernetes.io/projected/9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4-kube-api-access-4xkc5\") pod \"maas-api-e2e-ipp-b-031223-79c4b96cc9-4twf6\" (UID: \"9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-031223-79c4b96cc9-4twf6" Oct 30 17:35:47.067563 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:47.067533 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-031223-79c4b96cc9-4twf6\" (UID: \"9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-031223-79c4b96cc9-4twf6" Oct 30 17:35:47.067638 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:47.067592 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4-metrics-certs\") pod \"maas-api-e2e-ipp-b-031223-79c4b96cc9-4twf6\" (UID: \"9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-031223-79c4b96cc9-4twf6" Oct 30 17:35:47.067716 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:47.067695 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4-maas-api-tls\") pod \"maas-api-e2e-ipp-b-031223-79c4b96cc9-4twf6\" (UID: \"9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-031223-79c4b96cc9-4twf6" Oct 30 17:35:47.079696 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:47.079667 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-031223-79c4b96cc9-4twf6"] Oct 30 17:35:47.168654 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:47.168622 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xkc5\" (UniqueName: \"kubernetes.io/projected/9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4-kube-api-access-4xkc5\") pod \"maas-api-e2e-ipp-b-031223-79c4b96cc9-4twf6\" (UID: \"9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-031223-79c4b96cc9-4twf6" Oct 30 17:35:47.168851 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:47.168743 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-031223-79c4b96cc9-4twf6\" (UID: \"9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-031223-79c4b96cc9-4twf6" Oct 30 17:35:47.168851 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:47.168763 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4-metrics-certs\") pod \"maas-api-e2e-ipp-b-031223-79c4b96cc9-4twf6\" (UID: \"9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-031223-79c4b96cc9-4twf6" Oct 30 17:35:47.168851 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:47.168808 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4-maas-api-tls\") pod \"maas-api-e2e-ipp-b-031223-79c4b96cc9-4twf6\" (UID: \"9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-031223-79c4b96cc9-4twf6" Oct 30 17:35:47.169058 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:35:47.168978 4522 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-031223-serving-cert: secret "maas-api-e2e-ipp-b-031223-serving-cert" not found Oct 30 17:35:47.169058 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:35:47.169050 4522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4-maas-api-tls podName:9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4 nodeName:}" failed. No retries permitted until 2026-10-30 17:35:47.66902922 +0000 UTC m=+2921.669784121 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4-maas-api-tls") pod "maas-api-e2e-ipp-b-031223-79c4b96cc9-4twf6" (UID: "9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4") : secret "maas-api-e2e-ipp-b-031223-serving-cert" not found Oct 30 17:35:47.169559 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:47.169525 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-031223-79c4b96cc9-4twf6\" (UID: \"9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-031223-79c4b96cc9-4twf6" Oct 30 17:35:47.171368 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:47.171340 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4-metrics-certs\") pod \"maas-api-e2e-ipp-b-031223-79c4b96cc9-4twf6\" (UID: \"9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-031223-79c4b96cc9-4twf6" Oct 30 17:35:47.194958 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:47.194924 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xkc5\" (UniqueName: \"kubernetes.io/projected/9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4-kube-api-access-4xkc5\") pod \"maas-api-e2e-ipp-b-031223-79c4b96cc9-4twf6\" (UID: \"9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-031223-79c4b96cc9-4twf6" Oct 30 17:35:47.686724 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:47.686674 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8681b5-t4zkh"] Oct 30 17:35:47.690187 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:47.690158 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8681b5-t4zkh"] Oct 30 17:35:47.714763 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:47.714729 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4-maas-api-tls\") pod \"maas-api-e2e-ipp-b-031223-79c4b96cc9-4twf6\" (UID: \"9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-031223-79c4b96cc9-4twf6" Oct 30 17:35:47.717566 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:47.717541 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4-maas-api-tls\") pod \"maas-api-e2e-ipp-b-031223-79c4b96cc9-4twf6\" (UID: \"9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-031223-79c4b96cc9-4twf6" Oct 30 17:35:47.964139 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:47.964107 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-031223-79c4b96cc9-4twf6" Oct 30 17:35:48.146217 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:48.146184 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="329e13d7-a002-45f3-96f9-dfde687ce97e" path="/var/lib/kubelet/pods/329e13d7-a002-45f3-96f9-dfde687ce97e/volumes" Oct 30 17:35:48.146700 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:48.146683 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b57591a-1a9e-463c-aca5-45a2492c8689" path="/var/lib/kubelet/pods/7b57591a-1a9e-463c-aca5-45a2492c8689/volumes" Oct 30 17:35:48.147081 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:48.147068 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9e25d51-dd1c-419c-b938-656d13abb61a" path="/var/lib/kubelet/pods/b9e25d51-dd1c-419c-b938-656d13abb61a/volumes" Oct 30 17:35:48.287219 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:48.287190 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-031223-79c4b96cc9-4twf6"] Oct 30 17:35:48.289687 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:35:48.289662 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f24bd0b_f62f_4634_91f3_5f4ae16c9fb4.slice/crio-2dc69ed45b8734caf315f58b6ebb60c2d9b2c24d700d99be0b72b7206f5e302d WatchSource:0}: Error finding container 2dc69ed45b8734caf315f58b6ebb60c2d9b2c24d700d99be0b72b7206f5e302d: Status 404 returned error can't find the container with id 2dc69ed45b8734caf315f58b6ebb60c2d9b2c24d700d99be0b72b7206f5e302d Oct 30 17:35:48.291674 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:48.291652 4522 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:35:48.456896 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:48.456857 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-031223-79c4b96cc9-4twf6" event={"ID":"9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4","Type":"ContainerStarted","Data":"2dc69ed45b8734caf315f58b6ebb60c2d9b2c24d700d99be0b72b7206f5e302d"} Oct 30 17:35:48.979104 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:48.979068 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8681b5-d7469f799-jsz5b" podUID="af8374c3-9f76-4183-9eee-d781479514a0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:35:48.979326 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:48.979074 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8681b5-7d9676c6d6-khtxf" podUID="ac7e71fc-8c27-4f6f-b9f2-bf5fbd115267" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:35:49.466903 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:49.466867 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-031223-79c4b96cc9-4twf6" event={"ID":"9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4","Type":"ContainerStarted","Data":"0bef0c6853ef220fb2e888c67d5e6a4a2f65f1643b2a8e3755a4570864d77973"} Oct 30 17:35:49.467319 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:49.466955 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-031223-79c4b96cc9-4twf6" Oct 30 17:35:49.515264 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:49.515204 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-031223-79c4b96cc9-4twf6" podStartSLOduration=2.187306235 podStartE2EDuration="2.515191092s" totalImagesPullingTime="327.884857ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:35:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:35:49.513649583 +0000 UTC m=+2923.514404506" watchObservedRunningTime="2026-10-30 17:35:49.515191092 +0000 UTC m=+2923.515945993" Oct 30 17:35:52.558793 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:52.558754 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6d4f99-59dc8b84b6-z9qh6" podUID="49b34c5e-95a9-43e7-8eb4-0034619e1733" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:35:54.237569 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:54.237534 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-78c9c2-w9btc"] Oct 30 17:35:54.242468 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:54.242448 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-78c9c2-w9btc" Oct 30 17:35:54.263079 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:54.263052 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-78c9c2-w9btc"] Oct 30 17:35:54.305640 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:54.305602 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/642f81c8-be98-4b6f-a34d-5bc3fe9d9579-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-78c9c2-w9btc\" (UID: \"642f81c8-be98-4b6f-a34d-5bc3fe9d9579\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-78c9c2-w9btc" Oct 30 17:35:54.305891 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:54.305870 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/642f81c8-be98-4b6f-a34d-5bc3fe9d9579-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-78c9c2-w9btc\" (UID: \"642f81c8-be98-4b6f-a34d-5bc3fe9d9579\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-78c9c2-w9btc" Oct 30 17:35:54.305960 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:54.305915 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6q6w\" (UniqueName: \"kubernetes.io/projected/642f81c8-be98-4b6f-a34d-5bc3fe9d9579-kube-api-access-k6q6w\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-78c9c2-w9btc\" (UID: \"642f81c8-be98-4b6f-a34d-5bc3fe9d9579\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-78c9c2-w9btc" Oct 30 17:35:54.309591 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:54.309565 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:35:54.309672 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:54.309607 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:35:54.407242 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:54.407205 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/642f81c8-be98-4b6f-a34d-5bc3fe9d9579-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-78c9c2-w9btc\" (UID: \"642f81c8-be98-4b6f-a34d-5bc3fe9d9579\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-78c9c2-w9btc" Oct 30 17:35:54.407495 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:54.407410 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/642f81c8-be98-4b6f-a34d-5bc3fe9d9579-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-78c9c2-w9btc\" (UID: \"642f81c8-be98-4b6f-a34d-5bc3fe9d9579\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-78c9c2-w9btc" Oct 30 17:35:54.407495 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:54.407442 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6q6w\" (UniqueName: \"kubernetes.io/projected/642f81c8-be98-4b6f-a34d-5bc3fe9d9579-kube-api-access-k6q6w\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-78c9c2-w9btc\" (UID: \"642f81c8-be98-4b6f-a34d-5bc3fe9d9579\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-78c9c2-w9btc" Oct 30 17:35:54.408136 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:54.408116 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/642f81c8-be98-4b6f-a34d-5bc3fe9d9579-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-78c9c2-w9btc\" (UID: \"642f81c8-be98-4b6f-a34d-5bc3fe9d9579\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-78c9c2-w9btc" Oct 30 17:35:54.409715 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:54.409694 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/642f81c8-be98-4b6f-a34d-5bc3fe9d9579-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-78c9c2-w9btc\" (UID: \"642f81c8-be98-4b6f-a34d-5bc3fe9d9579\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-78c9c2-w9btc" Oct 30 17:35:54.433966 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:54.433776 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6q6w\" (UniqueName: \"kubernetes.io/projected/642f81c8-be98-4b6f-a34d-5bc3fe9d9579-kube-api-access-k6q6w\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-78c9c2-w9btc\" (UID: \"642f81c8-be98-4b6f-a34d-5bc3fe9d9579\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-78c9c2-w9btc" Oct 30 17:35:54.557377 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:54.557267 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-78c9c2-w9btc" Oct 30 17:35:54.864719 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:35:54.864666 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod642f81c8_be98_4b6f_a34d_5bc3fe9d9579.slice/crio-14addc493ad0444b15179cfa4547926286e24ef9409ed914989c71759912120b WatchSource:0}: Error finding container 14addc493ad0444b15179cfa4547926286e24ef9409ed914989c71759912120b: Status 404 returned error can't find the container with id 14addc493ad0444b15179cfa4547926286e24ef9409ed914989c71759912120b Oct 30 17:35:54.921221 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:54.921129 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-78c9c2-w9btc"] Oct 30 17:35:55.480656 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:55.480626 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-031223-79c4b96cc9-4twf6" Oct 30 17:35:55.514632 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:55.514597 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-78c9c2-w9btc" event={"ID":"642f81c8-be98-4b6f-a34d-5bc3fe9d9579","Type":"ContainerStarted","Data":"84c9e63948375892cfd014cd46bef80e01f2ea2054bebebb746eeae3a738b597"} Oct 30 17:35:55.514796 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:55.514647 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-78c9c2-w9btc" event={"ID":"642f81c8-be98-4b6f-a34d-5bc3fe9d9579","Type":"ContainerStarted","Data":"14addc493ad0444b15179cfa4547926286e24ef9409ed914989c71759912120b"} Oct 30 17:35:55.625346 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:55.625075 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-78c9c2-w9btc" podStartSLOduration=1.625058225 podStartE2EDuration="1.625058225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:35:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:35:55.624005314 +0000 UTC m=+2929.624760240" watchObservedRunningTime="2026-10-30 17:35:55.625058225 +0000 UTC m=+2929.625813127" Oct 30 17:35:55.650876 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:55.650844 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql"] Oct 30 17:35:55.651052 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:55.651023 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql" podUID="827251e7-f88d-4ed1-81bb-ce4960304aed" containerName="istio-proxy" containerID="cri-o://b5d6f8e901f3808466f8a618688cb7b2a63835c6ba874496af0c889333c51f2f" gracePeriod=30 Oct 30 17:35:55.798402 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:55.798359 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a43843-bbmq5"] Oct 30 17:35:55.804323 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:55.802829 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a43843-bbmq5"] Oct 30 17:35:56.145547 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:56.145507 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e38a4dd-06dd-4088-b857-b40962e8908b" path="/var/lib/kubelet/pods/1e38a4dd-06dd-4088-b857-b40962e8908b/volumes" Oct 30 17:35:56.178407 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:56.178368 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-db8417-nskzf"] Oct 30 17:35:56.182887 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:56.182869 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-db8417-nskzf" Oct 30 17:35:56.203433 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:56.203361 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-db8417-nskzf"] Oct 30 17:35:56.233992 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:56.233951 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/847b63e9-4a1c-4c2f-85f2-1295c356fb57-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-db8417-nskzf\" (UID: \"847b63e9-4a1c-4c2f-85f2-1295c356fb57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-db8417-nskzf" Oct 30 17:35:56.234200 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:56.234020 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/847b63e9-4a1c-4c2f-85f2-1295c356fb57-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-db8417-nskzf\" (UID: \"847b63e9-4a1c-4c2f-85f2-1295c356fb57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-db8417-nskzf" Oct 30 17:35:56.234200 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:56.234124 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46zhg\" (UniqueName: \"kubernetes.io/projected/847b63e9-4a1c-4c2f-85f2-1295c356fb57-kube-api-access-46zhg\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-db8417-nskzf\" (UID: \"847b63e9-4a1c-4c2f-85f2-1295c356fb57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-db8417-nskzf" Oct 30 17:35:56.336039 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:56.335995 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/847b63e9-4a1c-4c2f-85f2-1295c356fb57-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-db8417-nskzf\" (UID: \"847b63e9-4a1c-4c2f-85f2-1295c356fb57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-db8417-nskzf" Oct 30 17:35:56.336249 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:56.336060 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/847b63e9-4a1c-4c2f-85f2-1295c356fb57-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-db8417-nskzf\" (UID: \"847b63e9-4a1c-4c2f-85f2-1295c356fb57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-db8417-nskzf" Oct 30 17:35:56.336249 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:56.336109 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46zhg\" (UniqueName: \"kubernetes.io/projected/847b63e9-4a1c-4c2f-85f2-1295c356fb57-kube-api-access-46zhg\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-db8417-nskzf\" (UID: \"847b63e9-4a1c-4c2f-85f2-1295c356fb57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-db8417-nskzf" Oct 30 17:35:56.336903 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:56.336858 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/847b63e9-4a1c-4c2f-85f2-1295c356fb57-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-db8417-nskzf\" (UID: \"847b63e9-4a1c-4c2f-85f2-1295c356fb57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-db8417-nskzf" Oct 30 17:35:56.338509 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:56.338487 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/847b63e9-4a1c-4c2f-85f2-1295c356fb57-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-db8417-nskzf\" (UID: \"847b63e9-4a1c-4c2f-85f2-1295c356fb57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-db8417-nskzf" Oct 30 17:35:56.361042 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:56.361008 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46zhg\" (UniqueName: \"kubernetes.io/projected/847b63e9-4a1c-4c2f-85f2-1295c356fb57-kube-api-access-46zhg\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-db8417-nskzf\" (UID: \"847b63e9-4a1c-4c2f-85f2-1295c356fb57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-db8417-nskzf" Oct 30 17:35:56.497061 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:56.496981 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-db8417-nskzf" Oct 30 17:35:56.524149 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:56.524121 4522 generic.go:356] "Generic (PLEG): container finished" podID="642f81c8-be98-4b6f-a34d-5bc3fe9d9579" containerID="84c9e63948375892cfd014cd46bef80e01f2ea2054bebebb746eeae3a738b597" exitCode=0 Oct 30 17:35:56.524298 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:56.524198 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-78c9c2-w9btc" event={"ID":"642f81c8-be98-4b6f-a34d-5bc3fe9d9579","Type":"ContainerDied","Data":"84c9e63948375892cfd014cd46bef80e01f2ea2054bebebb746eeae3a738b597"} Oct 30 17:35:56.809497 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:35:56.809462 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod847b63e9_4a1c_4c2f_85f2_1295c356fb57.slice/crio-c2e9a9c1ac11ffa8e367cad86f427c2a763a8ea384760e4ef402eb0c1848215e WatchSource:0}: Error finding container c2e9a9c1ac11ffa8e367cad86f427c2a763a8ea384760e4ef402eb0c1848215e: Status 404 returned error can't find the container with id c2e9a9c1ac11ffa8e367cad86f427c2a763a8ea384760e4ef402eb0c1848215e Oct 30 17:35:56.810078 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:56.809978 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-db8417-nskzf"] Oct 30 17:35:57.532974 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:57.532930 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-db8417-nskzf" event={"ID":"847b63e9-4a1c-4c2f-85f2-1295c356fb57","Type":"ContainerStarted","Data":"44e48c57f6dd1bf710e820062e3994d0cfc0257f972d23653b44ab3fc6071f27"} Oct 30 17:35:57.533415 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:57.532990 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-db8417-nskzf" event={"ID":"847b63e9-4a1c-4c2f-85f2-1295c356fb57","Type":"ContainerStarted","Data":"c2e9a9c1ac11ffa8e367cad86f427c2a763a8ea384760e4ef402eb0c1848215e"} Oct 30 17:35:57.579068 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:57.579014 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-db8417-nskzf" podStartSLOduration=1.57899836 podStartE2EDuration="1.57899836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:35:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:35:57.576026196 +0000 UTC m=+2931.576781101" watchObservedRunningTime="2026-10-30 17:35:57.57899836 +0000 UTC m=+2931.579753261" Oct 30 17:35:57.696748 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:57.696720 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-78c9c2-w9btc" Oct 30 17:35:57.757676 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:57.757642 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/642f81c8-be98-4b6f-a34d-5bc3fe9d9579-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/642f81c8-be98-4b6f-a34d-5bc3fe9d9579-maas-api-openshift-ca\") pod \"642f81c8-be98-4b6f-a34d-5bc3fe9d9579\" (UID: \"642f81c8-be98-4b6f-a34d-5bc3fe9d9579\") " Oct 30 17:35:57.757952 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:57.757932 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/642f81c8-be98-4b6f-a34d-5bc3fe9d9579-kube-api-access-k6q6w\" (UniqueName: \"kubernetes.io/projected/642f81c8-be98-4b6f-a34d-5bc3fe9d9579-kube-api-access-k6q6w\") pod \"642f81c8-be98-4b6f-a34d-5bc3fe9d9579\" (UID: \"642f81c8-be98-4b6f-a34d-5bc3fe9d9579\") " Oct 30 17:35:57.758021 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:57.757987 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/642f81c8-be98-4b6f-a34d-5bc3fe9d9579-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/642f81c8-be98-4b6f-a34d-5bc3fe9d9579-maas-api-xks-ca\") pod \"642f81c8-be98-4b6f-a34d-5bc3fe9d9579\" (UID: \"642f81c8-be98-4b6f-a34d-5bc3fe9d9579\") " Oct 30 17:35:57.758072 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:57.758048 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/642f81c8-be98-4b6f-a34d-5bc3fe9d9579-maas-api-openshift-ca" pod "642f81c8-be98-4b6f-a34d-5bc3fe9d9579" (UID: "642f81c8-be98-4b6f-a34d-5bc3fe9d9579"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:35:57.758656 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:57.758630 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/642f81c8-be98-4b6f-a34d-5bc3fe9d9579-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:35:57.760341 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:57.760293 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/642f81c8-be98-4b6f-a34d-5bc3fe9d9579-maas-api-xks-ca" pod "642f81c8-be98-4b6f-a34d-5bc3fe9d9579" (UID: "642f81c8-be98-4b6f-a34d-5bc3fe9d9579"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:35:57.760438 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:57.760356 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/642f81c8-be98-4b6f-a34d-5bc3fe9d9579-kube-api-access-k6q6w" pod "642f81c8-be98-4b6f-a34d-5bc3fe9d9579" (UID: "642f81c8-be98-4b6f-a34d-5bc3fe9d9579"). InnerVolumeSpecName "kube-api-access-k6q6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:35:57.859998 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:57.859961 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6q6w\" (UniqueName: \"kubernetes.io/projected/642f81c8-be98-4b6f-a34d-5bc3fe9d9579-kube-api-access-k6q6w\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:35:57.859998 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:57.859990 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/642f81c8-be98-4b6f-a34d-5bc3fe9d9579-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:35:58.543669 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:58.543632 4522 generic.go:356] "Generic (PLEG): container finished" podID="847b63e9-4a1c-4c2f-85f2-1295c356fb57" containerID="44e48c57f6dd1bf710e820062e3994d0cfc0257f972d23653b44ab3fc6071f27" exitCode=0 Oct 30 17:35:58.544109 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:58.543708 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-db8417-nskzf" event={"ID":"847b63e9-4a1c-4c2f-85f2-1295c356fb57","Type":"ContainerDied","Data":"44e48c57f6dd1bf710e820062e3994d0cfc0257f972d23653b44ab3fc6071f27"} Oct 30 17:35:58.545360 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:58.545336 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-78c9c2-w9btc" Oct 30 17:35:58.545360 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:58.545347 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-78c9c2-w9btc" event={"ID":"642f81c8-be98-4b6f-a34d-5bc3fe9d9579","Type":"ContainerDied","Data":"14addc493ad0444b15179cfa4547926286e24ef9409ed914989c71759912120b"} Oct 30 17:35:58.545533 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:58.545394 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14addc493ad0444b15179cfa4547926286e24ef9409ed914989c71759912120b" Oct 30 17:35:58.978982 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:58.978941 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8681b5-d7469f799-jsz5b" podUID="af8374c3-9f76-4183-9eee-d781479514a0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:35:58.979167 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:58.978955 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8681b5-7d9676c6d6-khtxf" podUID="ac7e71fc-8c27-4f6f-b9f2-bf5fbd115267" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:35:59.705866 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:59.705840 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-db8417-nskzf" Oct 30 17:35:59.789426 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:59.789393 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/847b63e9-4a1c-4c2f-85f2-1295c356fb57-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/847b63e9-4a1c-4c2f-85f2-1295c356fb57-maas-api-openshift-ca\") pod \"847b63e9-4a1c-4c2f-85f2-1295c356fb57\" (UID: \"847b63e9-4a1c-4c2f-85f2-1295c356fb57\") " Oct 30 17:35:59.789633 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:59.789465 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/847b63e9-4a1c-4c2f-85f2-1295c356fb57-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/847b63e9-4a1c-4c2f-85f2-1295c356fb57-maas-api-xks-ca\") pod \"847b63e9-4a1c-4c2f-85f2-1295c356fb57\" (UID: \"847b63e9-4a1c-4c2f-85f2-1295c356fb57\") " Oct 30 17:35:59.789704 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:59.789655 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/847b63e9-4a1c-4c2f-85f2-1295c356fb57-kube-api-access-46zhg\" (UniqueName: \"kubernetes.io/projected/847b63e9-4a1c-4c2f-85f2-1295c356fb57-kube-api-access-46zhg\") pod \"847b63e9-4a1c-4c2f-85f2-1295c356fb57\" (UID: \"847b63e9-4a1c-4c2f-85f2-1295c356fb57\") " Oct 30 17:35:59.789852 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:59.789821 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/847b63e9-4a1c-4c2f-85f2-1295c356fb57-maas-api-openshift-ca" pod "847b63e9-4a1c-4c2f-85f2-1295c356fb57" (UID: "847b63e9-4a1c-4c2f-85f2-1295c356fb57"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:35:59.790212 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:59.790190 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/847b63e9-4a1c-4c2f-85f2-1295c356fb57-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:35:59.791721 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:59.791688 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/847b63e9-4a1c-4c2f-85f2-1295c356fb57-maas-api-xks-ca" pod "847b63e9-4a1c-4c2f-85f2-1295c356fb57" (UID: "847b63e9-4a1c-4c2f-85f2-1295c356fb57"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:35:59.791855 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:59.791829 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/847b63e9-4a1c-4c2f-85f2-1295c356fb57-kube-api-access-46zhg" pod "847b63e9-4a1c-4c2f-85f2-1295c356fb57" (UID: "847b63e9-4a1c-4c2f-85f2-1295c356fb57"). InnerVolumeSpecName "kube-api-access-46zhg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:35:59.891800 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:59.891765 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46zhg\" (UniqueName: \"kubernetes.io/projected/847b63e9-4a1c-4c2f-85f2-1295c356fb57-kube-api-access-46zhg\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:35:59.891800 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:35:59.891797 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/847b63e9-4a1c-4c2f-85f2-1295c356fb57-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:00.563028 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:00.562995 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-db8417-nskzf" Oct 30 17:36:00.563196 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:00.563027 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-db8417-nskzf" event={"ID":"847b63e9-4a1c-4c2f-85f2-1295c356fb57","Type":"ContainerDied","Data":"c2e9a9c1ac11ffa8e367cad86f427c2a763a8ea384760e4ef402eb0c1848215e"} Oct 30 17:36:00.563196 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:00.563093 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2e9a9c1ac11ffa8e367cad86f427c2a763a8ea384760e4ef402eb0c1848215e" Oct 30 17:36:00.922756 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:00.922732 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql" Oct 30 17:36:01.007176 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:01.007135 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/827251e7-f88d-4ed1-81bb-ce4960304aed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/827251e7-f88d-4ed1-81bb-ce4960304aed-workload-certs\") pod \"827251e7-f88d-4ed1-81bb-ce4960304aed\" (UID: \"827251e7-f88d-4ed1-81bb-ce4960304aed\") " Oct 30 17:36:01.007406 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:01.007221 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/827251e7-f88d-4ed1-81bb-ce4960304aed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/827251e7-f88d-4ed1-81bb-ce4960304aed-istiod-ca-cert\") pod \"827251e7-f88d-4ed1-81bb-ce4960304aed\" (UID: \"827251e7-f88d-4ed1-81bb-ce4960304aed\") " Oct 30 17:36:01.007406 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:01.007285 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/827251e7-f88d-4ed1-81bb-ce4960304aed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/827251e7-f88d-4ed1-81bb-ce4960304aed-workload-socket\") pod \"827251e7-f88d-4ed1-81bb-ce4960304aed\" (UID: \"827251e7-f88d-4ed1-81bb-ce4960304aed\") " Oct 30 17:36:01.007406 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:01.007327 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/827251e7-f88d-4ed1-81bb-ce4960304aed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/827251e7-f88d-4ed1-81bb-ce4960304aed-istio-data\") pod \"827251e7-f88d-4ed1-81bb-ce4960304aed\" (UID: \"827251e7-f88d-4ed1-81bb-ce4960304aed\") " Oct 30 17:36:01.007693 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:01.007426 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/827251e7-f88d-4ed1-81bb-ce4960304aed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/827251e7-f88d-4ed1-81bb-ce4960304aed-istio-envoy\") pod \"827251e7-f88d-4ed1-81bb-ce4960304aed\" (UID: \"827251e7-f88d-4ed1-81bb-ce4960304aed\") " Oct 30 17:36:01.007693 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:01.007452 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/827251e7-f88d-4ed1-81bb-ce4960304aed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/827251e7-f88d-4ed1-81bb-ce4960304aed-istio-podinfo\") pod \"827251e7-f88d-4ed1-81bb-ce4960304aed\" (UID: \"827251e7-f88d-4ed1-81bb-ce4960304aed\") " Oct 30 17:36:01.007693 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:01.007467 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/827251e7-f88d-4ed1-81bb-ce4960304aed-workload-certs" pod "827251e7-f88d-4ed1-81bb-ce4960304aed" (UID: "827251e7-f88d-4ed1-81bb-ce4960304aed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:01.007693 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:01.007530 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/827251e7-f88d-4ed1-81bb-ce4960304aed-kube-api-access-24qrb\" (UniqueName: \"kubernetes.io/projected/827251e7-f88d-4ed1-81bb-ce4960304aed-kube-api-access-24qrb\") pod \"827251e7-f88d-4ed1-81bb-ce4960304aed\" (UID: \"827251e7-f88d-4ed1-81bb-ce4960304aed\") " Oct 30 17:36:01.007693 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:01.007590 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/827251e7-f88d-4ed1-81bb-ce4960304aed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/827251e7-f88d-4ed1-81bb-ce4960304aed-credential-socket\") pod \"827251e7-f88d-4ed1-81bb-ce4960304aed\" (UID: \"827251e7-f88d-4ed1-81bb-ce4960304aed\") " Oct 30 17:36:01.007693 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:01.007617 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/827251e7-f88d-4ed1-81bb-ce4960304aed-workload-socket" pod "827251e7-f88d-4ed1-81bb-ce4960304aed" (UID: "827251e7-f88d-4ed1-81bb-ce4960304aed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:01.008041 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:01.007841 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/827251e7-f88d-4ed1-81bb-ce4960304aed-istio-token\" (UniqueName: \"kubernetes.io/projected/827251e7-f88d-4ed1-81bb-ce4960304aed-istio-token\") pod \"827251e7-f88d-4ed1-81bb-ce4960304aed\" (UID: \"827251e7-f88d-4ed1-81bb-ce4960304aed\") " Oct 30 17:36:01.008041 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:01.007928 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/827251e7-f88d-4ed1-81bb-ce4960304aed-istio-data" pod "827251e7-f88d-4ed1-81bb-ce4960304aed" (UID: "827251e7-f88d-4ed1-81bb-ce4960304aed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:01.008189 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:01.008164 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/827251e7-f88d-4ed1-81bb-ce4960304aed-credential-socket" pod "827251e7-f88d-4ed1-81bb-ce4960304aed" (UID: "827251e7-f88d-4ed1-81bb-ce4960304aed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:01.008604 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:01.008577 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/827251e7-f88d-4ed1-81bb-ce4960304aed-istiod-ca-cert" pod "827251e7-f88d-4ed1-81bb-ce4960304aed" (UID: "827251e7-f88d-4ed1-81bb-ce4960304aed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:36:01.008721 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:01.008615 4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/827251e7-f88d-4ed1-81bb-ce4960304aed-workload-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:01.008721 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:01.008637 4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/827251e7-f88d-4ed1-81bb-ce4960304aed-workload-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:01.008721 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:01.008653 4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/827251e7-f88d-4ed1-81bb-ce4960304aed-istio-data\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:01.008721 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:01.008666 4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/827251e7-f88d-4ed1-81bb-ce4960304aed-credential-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:01.010254 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:01.010225 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/827251e7-f88d-4ed1-81bb-ce4960304aed-kube-api-access-24qrb" pod "827251e7-f88d-4ed1-81bb-ce4960304aed" (UID: "827251e7-f88d-4ed1-81bb-ce4960304aed"). InnerVolumeSpecName "kube-api-access-24qrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:36:01.010665 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:01.010643 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/827251e7-f88d-4ed1-81bb-ce4960304aed-istio-envoy" pod "827251e7-f88d-4ed1-81bb-ce4960304aed" (UID: "827251e7-f88d-4ed1-81bb-ce4960304aed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:01.010834 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:01.010804 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/827251e7-f88d-4ed1-81bb-ce4960304aed-istio-podinfo" pod "827251e7-f88d-4ed1-81bb-ce4960304aed" (UID: "827251e7-f88d-4ed1-81bb-ce4960304aed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:36:01.010905 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:01.010829 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/827251e7-f88d-4ed1-81bb-ce4960304aed-istio-token" pod "827251e7-f88d-4ed1-81bb-ce4960304aed" (UID: "827251e7-f88d-4ed1-81bb-ce4960304aed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:36:01.109622 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:01.109535 4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/827251e7-f88d-4ed1-81bb-ce4960304aed-istio-token\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:01.109622 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:01.109569 4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/827251e7-f88d-4ed1-81bb-ce4960304aed-istiod-ca-cert\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:01.109622 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:01.109577 4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/827251e7-f88d-4ed1-81bb-ce4960304aed-istio-envoy\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:01.109622 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:01.109586 4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/827251e7-f88d-4ed1-81bb-ce4960304aed-istio-podinfo\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:01.109622 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:01.109594 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24qrb\" (UniqueName: \"kubernetes.io/projected/827251e7-f88d-4ed1-81bb-ce4960304aed-kube-api-access-24qrb\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:01.477466 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:01.477435 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6d4f99-59dc8b84b6-z9qh6" Oct 30 17:36:01.573821 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:01.573790 4522 generic.go:356] "Generic (PLEG): container finished" podID="49b34c5e-95a9-43e7-8eb4-0034619e1733" containerID="475908fe8b9912a214bf22fe965d27ced77dc9aacc8f21d968bf92781fb34081" exitCode=0 Oct 30 17:36:01.574032 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:01.573838 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6d4f99-59dc8b84b6-z9qh6" Oct 30 17:36:01.574032 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:01.573875 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6d4f99-59dc8b84b6-z9qh6" event={"ID":"49b34c5e-95a9-43e7-8eb4-0034619e1733","Type":"ContainerDied","Data":"475908fe8b9912a214bf22fe965d27ced77dc9aacc8f21d968bf92781fb34081"} Oct 30 17:36:01.574032 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:01.573939 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6d4f99-59dc8b84b6-z9qh6" event={"ID":"49b34c5e-95a9-43e7-8eb4-0034619e1733","Type":"ContainerDied","Data":"aa2b6b4ce4ea389cd23264626ca86470fb90791baaa3000bd799893bb0caa1c4"} Oct 30 17:36:01.574032 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:01.573970 4522 scope.go:122] "RemoveContainer" containerID="475908fe8b9912a214bf22fe965d27ced77dc9aacc8f21d968bf92781fb34081" Oct 30 17:36:01.575747 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:01.575728 4522 generic.go:356] "Generic (PLEG): container finished" podID="827251e7-f88d-4ed1-81bb-ce4960304aed" containerID="b5d6f8e901f3808466f8a618688cb7b2a63835c6ba874496af0c889333c51f2f" exitCode=0 Oct 30 17:36:01.575873 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:01.575780 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql" Oct 30 17:36:01.575873 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:01.575807 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql" event={"ID":"827251e7-f88d-4ed1-81bb-ce4960304aed","Type":"ContainerDied","Data":"b5d6f8e901f3808466f8a618688cb7b2a63835c6ba874496af0c889333c51f2f"} Oct 30 17:36:01.575873 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:01.575863 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql" event={"ID":"827251e7-f88d-4ed1-81bb-ce4960304aed","Type":"ContainerDied","Data":"7f0d7cc1a6cb2fafa04bfa2a265e64d43512492f4db50f38f5a461fb1a50c740"} Oct 30 17:36:01.584921 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:01.584905 4522 scope.go:122] "RemoveContainer" containerID="475908fe8b9912a214bf22fe965d27ced77dc9aacc8f21d968bf92781fb34081" Oct 30 17:36:01.585199 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:36:01.585181 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"475908fe8b9912a214bf22fe965d27ced77dc9aacc8f21d968bf92781fb34081\": container with ID starting with 475908fe8b9912a214bf22fe965d27ced77dc9aacc8f21d968bf92781fb34081 not found: ID does not exist" containerID="475908fe8b9912a214bf22fe965d27ced77dc9aacc8f21d968bf92781fb34081" Oct 30 17:36:01.585256 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:01.585209 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"475908fe8b9912a214bf22fe965d27ced77dc9aacc8f21d968bf92781fb34081"} err="failed to get container status \"475908fe8b9912a214bf22fe965d27ced77dc9aacc8f21d968bf92781fb34081\": rpc error: code = NotFound desc = could not find container \"475908fe8b9912a214bf22fe965d27ced77dc9aacc8f21d968bf92781fb34081\": container with ID starting with 475908fe8b9912a214bf22fe965d27ced77dc9aacc8f21d968bf92781fb34081 not found: ID does not exist" Oct 30 17:36:01.585256 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:01.585226 4522 scope.go:122] "RemoveContainer" containerID="b5d6f8e901f3808466f8a618688cb7b2a63835c6ba874496af0c889333c51f2f" Oct 30 17:36:01.594892 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:01.594872 4522 scope.go:122] "RemoveContainer" containerID="b686e611a6d354837289bab219e846e56d5bfc25c70979e7d7e2df9e7faf90a1" Oct 30 17:36:01.605924 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:01.605901 4522 scope.go:122] "RemoveContainer" containerID="b5d6f8e901f3808466f8a618688cb7b2a63835c6ba874496af0c889333c51f2f" Oct 30 17:36:01.606420 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:36:01.606389 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5d6f8e901f3808466f8a618688cb7b2a63835c6ba874496af0c889333c51f2f\": container with ID starting with b5d6f8e901f3808466f8a618688cb7b2a63835c6ba874496af0c889333c51f2f not found: ID does not exist" containerID="b5d6f8e901f3808466f8a618688cb7b2a63835c6ba874496af0c889333c51f2f" Oct 30 17:36:01.606540 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:01.606431 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5d6f8e901f3808466f8a618688cb7b2a63835c6ba874496af0c889333c51f2f"} err="failed to get container status \"b5d6f8e901f3808466f8a618688cb7b2a63835c6ba874496af0c889333c51f2f\": rpc error: code = NotFound desc = could not find container \"b5d6f8e901f3808466f8a618688cb7b2a63835c6ba874496af0c889333c51f2f\": container with ID starting with b5d6f8e901f3808466f8a618688cb7b2a63835c6ba874496af0c889333c51f2f not found: ID does not exist" Oct 30 17:36:01.606540 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:01.606455 4522 scope.go:122] "RemoveContainer" containerID="b686e611a6d354837289bab219e846e56d5bfc25c70979e7d7e2df9e7faf90a1" Oct 30 17:36:01.606849 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:36:01.606772 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b686e611a6d354837289bab219e846e56d5bfc25c70979e7d7e2df9e7faf90a1\": container with ID starting with b686e611a6d354837289bab219e846e56d5bfc25c70979e7d7e2df9e7faf90a1 not found: ID does not exist" containerID="b686e611a6d354837289bab219e846e56d5bfc25c70979e7d7e2df9e7faf90a1" Oct 30 17:36:01.606849 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:01.606808 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b686e611a6d354837289bab219e846e56d5bfc25c70979e7d7e2df9e7faf90a1"} err="failed to get container status \"b686e611a6d354837289bab219e846e56d5bfc25c70979e7d7e2df9e7faf90a1\": rpc error: code = NotFound desc = could not find container \"b686e611a6d354837289bab219e846e56d5bfc25c70979e7d7e2df9e7faf90a1\": container with ID starting with b686e611a6d354837289bab219e846e56d5bfc25c70979e7d7e2df9e7faf90a1 not found: ID does not exist" Oct 30 17:36:01.631372 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:01.631277 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49b34c5e-95a9-43e7-8eb4-0034619e1733-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49b34c5e-95a9-43e7-8eb4-0034619e1733-plugins-config-volume\") pod \"49b34c5e-95a9-43e7-8eb4-0034619e1733\" (UID: \"49b34c5e-95a9-43e7-8eb4-0034619e1733\") " Oct 30 17:36:01.631524 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:01.631474 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49b34c5e-95a9-43e7-8eb4-0034619e1733-kube-api-access-d9895\" (UniqueName: \"kubernetes.io/projected/49b34c5e-95a9-43e7-8eb4-0034619e1733-kube-api-access-d9895\") pod \"49b34c5e-95a9-43e7-8eb4-0034619e1733\" (UID: \"49b34c5e-95a9-43e7-8eb4-0034619e1733\") " Oct 30 17:36:01.631785 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:01.631762 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49b34c5e-95a9-43e7-8eb4-0034619e1733-plugins-config-volume" pod "49b34c5e-95a9-43e7-8eb4-0034619e1733" (UID: "49b34c5e-95a9-43e7-8eb4-0034619e1733"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:36:01.632065 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:01.632047 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49b34c5e-95a9-43e7-8eb4-0034619e1733-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:01.634529 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:01.634504 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql"] Oct 30 17:36:01.634980 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:01.634960 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49b34c5e-95a9-43e7-8eb4-0034619e1733-kube-api-access-d9895" pod "49b34c5e-95a9-43e7-8eb4-0034619e1733" (UID: "49b34c5e-95a9-43e7-8eb4-0034619e1733"). InnerVolumeSpecName "kube-api-access-d9895". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:36:01.640232 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:01.640206 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-8npql"] Oct 30 17:36:01.733496 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:01.733463 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9895\" (UniqueName: \"kubernetes.io/projected/49b34c5e-95a9-43e7-8eb4-0034619e1733-kube-api-access-d9895\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:01.925101 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:01.925063 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-6d4f99-59dc8b84b6-z9qh6"] Oct 30 17:36:01.930607 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:01.930579 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-6d4f99-59dc8b84b6-z9qh6"] Oct 30 17:36:02.146358 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:02.146267 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49b34c5e-95a9-43e7-8eb4-0034619e1733" path="/var/lib/kubelet/pods/49b34c5e-95a9-43e7-8eb4-0034619e1733/volumes" Oct 30 17:36:02.146817 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:02.146800 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="827251e7-f88d-4ed1-81bb-ce4960304aed" path="/var/lib/kubelet/pods/827251e7-f88d-4ed1-81bb-ce4960304aed/volumes" Oct 30 17:36:04.270880 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.270849 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8681b5-7d9676c6d6-khtxf" Oct 30 17:36:04.318385 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.318357 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8681b5-d7469f799-jsz5b" Oct 30 17:36:04.349341 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.349286 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-78c9c2-6b9c4447c4-p5z99"] Oct 30 17:36:04.382637 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.382605 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac7e71fc-8c27-4f6f-b9f2-bf5fbd115267-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac7e71fc-8c27-4f6f-b9f2-bf5fbd115267-plugins-config-volume\") pod \"ac7e71fc-8c27-4f6f-b9f2-bf5fbd115267\" (UID: \"ac7e71fc-8c27-4f6f-b9f2-bf5fbd115267\") " Oct 30 17:36:04.382803 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.382725 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac7e71fc-8c27-4f6f-b9f2-bf5fbd115267-kube-api-access-ptvk6\" (UniqueName: \"kubernetes.io/projected/ac7e71fc-8c27-4f6f-b9f2-bf5fbd115267-kube-api-access-ptvk6\") pod \"ac7e71fc-8c27-4f6f-b9f2-bf5fbd115267\" (UID: \"ac7e71fc-8c27-4f6f-b9f2-bf5fbd115267\") " Oct 30 17:36:04.382873 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.382809 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af8374c3-9f76-4183-9eee-d781479514a0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af8374c3-9f76-4183-9eee-d781479514a0-plugins-config-volume\") pod \"af8374c3-9f76-4183-9eee-d781479514a0\" (UID: \"af8374c3-9f76-4183-9eee-d781479514a0\") " Oct 30 17:36:04.383208 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.383175 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac7e71fc-8c27-4f6f-b9f2-bf5fbd115267-plugins-config-volume" pod "ac7e71fc-8c27-4f6f-b9f2-bf5fbd115267" (UID: "ac7e71fc-8c27-4f6f-b9f2-bf5fbd115267"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:36:04.383362 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.383237 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af8374c3-9f76-4183-9eee-d781479514a0-plugins-config-volume" pod "af8374c3-9f76-4183-9eee-d781479514a0" (UID: "af8374c3-9f76-4183-9eee-d781479514a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:36:04.383431 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.383415 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af8374c3-9f76-4183-9eee-d781479514a0-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:04.383473 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.383438 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac7e71fc-8c27-4f6f-b9f2-bf5fbd115267-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:04.385750 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.385726 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac7e71fc-8c27-4f6f-b9f2-bf5fbd115267-kube-api-access-ptvk6" pod "ac7e71fc-8c27-4f6f-b9f2-bf5fbd115267" (UID: "ac7e71fc-8c27-4f6f-b9f2-bf5fbd115267"). InnerVolumeSpecName "kube-api-access-ptvk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:36:04.388405 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.388378 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-78c9c2-777cdc8975-7qp6m"] Oct 30 17:36:04.420504 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.420459 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-04248e-6xkgh"] Oct 30 17:36:04.421056 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.421043 4522 cpu_manager.go:414] "containerMap: removing container" podUID="827251e7-f88d-4ed1-81bb-ce4960304aed" containerName="istio-proxy" Oct 30 17:36:04.421108 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.421058 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="827251e7-f88d-4ed1-81bb-ce4960304aed" containerName="istio-proxy" Oct 30 17:36:04.421108 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.421073 4522 cpu_manager.go:414] "containerMap: removing container" podUID="49b34c5e-95a9-43e7-8eb4-0034619e1733" containerName="payload-processing" Oct 30 17:36:04.421108 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.421079 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49b34c5e-95a9-43e7-8eb4-0034619e1733" containerName="payload-processing" Oct 30 17:36:04.421108 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.421106 4522 cpu_manager.go:414] "containerMap: removing container" podUID="847b63e9-4a1c-4c2f-85f2-1295c356fb57" containerName="revoke-keys" Oct 30 17:36:04.421238 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.421111 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="847b63e9-4a1c-4c2f-85f2-1295c356fb57" containerName="revoke-keys" Oct 30 17:36:04.421238 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.421120 4522 cpu_manager.go:414] "containerMap: removing container" podUID="af8374c3-9f76-4183-9eee-d781479514a0" containerName="payload-pre-processing" Oct 30 17:36:04.421238 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.421125 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af8374c3-9f76-4183-9eee-d781479514a0" containerName="payload-pre-processing" Oct 30 17:36:04.421238 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.421138 4522 cpu_manager.go:414] "containerMap: removing container" podUID="827251e7-f88d-4ed1-81bb-ce4960304aed" containerName="istio-proxy" Oct 30 17:36:04.421238 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.421142 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="827251e7-f88d-4ed1-81bb-ce4960304aed" containerName="istio-proxy" Oct 30 17:36:04.421238 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.421152 4522 cpu_manager.go:414] "containerMap: removing container" podUID="ac7e71fc-8c27-4f6f-b9f2-bf5fbd115267" containerName="payload-processing" Oct 30 17:36:04.421238 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.421156 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac7e71fc-8c27-4f6f-b9f2-bf5fbd115267" containerName="payload-processing" Oct 30 17:36:04.421238 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.421170 4522 cpu_manager.go:414] "containerMap: removing container" podUID="642f81c8-be98-4b6f-a34d-5bc3fe9d9579" containerName="revoke-keys" Oct 30 17:36:04.421238 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.421175 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="642f81c8-be98-4b6f-a34d-5bc3fe9d9579" containerName="revoke-keys" Oct 30 17:36:04.421558 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.421256 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="847b63e9-4a1c-4c2f-85f2-1295c356fb57" containerName="revoke-keys" Oct 30 17:36:04.421558 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.421265 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="af8374c3-9f76-4183-9eee-d781479514a0" containerName="payload-pre-processing" Oct 30 17:36:04.421558 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.421272 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="827251e7-f88d-4ed1-81bb-ce4960304aed" containerName="istio-proxy" Oct 30 17:36:04.421558 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.421278 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac7e71fc-8c27-4f6f-b9f2-bf5fbd115267" containerName="payload-processing" Oct 30 17:36:04.421558 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.421285 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="642f81c8-be98-4b6f-a34d-5bc3fe9d9579" containerName="revoke-keys" Oct 30 17:36:04.421558 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.421292 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="49b34c5e-95a9-43e7-8eb4-0034619e1733" containerName="payload-processing" Oct 30 17:36:04.424895 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.424877 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-04248e-6xkgh" Oct 30 17:36:04.447190 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.447138 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-04248e-6xkgh"] Oct 30 17:36:04.457359 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.454258 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:36:04.457359 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.454391 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:36:04.488100 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.487635 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af8374c3-9f76-4183-9eee-d781479514a0-kube-api-access-bptb4\" (UniqueName: \"kubernetes.io/projected/af8374c3-9f76-4183-9eee-d781479514a0-kube-api-access-bptb4\") pod \"af8374c3-9f76-4183-9eee-d781479514a0\" (UID: \"af8374c3-9f76-4183-9eee-d781479514a0\") " Oct 30 17:36:04.489749 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.489640 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b9ebedf-4812-4ebf-82cd-c80d06d1886a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-04248e-6xkgh\" (UID: \"6b9ebedf-4812-4ebf-82cd-c80d06d1886a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-04248e-6xkgh" Oct 30 17:36:04.490060 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.489999 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw84f\" (UniqueName: \"kubernetes.io/projected/6b9ebedf-4812-4ebf-82cd-c80d06d1886a-kube-api-access-lw84f\") pod \"maas-api-revoke-keys-e2e-worker-w1-04248e-6xkgh\" (UID: \"6b9ebedf-4812-4ebf-82cd-c80d06d1886a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-04248e-6xkgh" Oct 30 17:36:04.490154 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.490094 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b9ebedf-4812-4ebf-82cd-c80d06d1886a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-04248e-6xkgh\" (UID: \"6b9ebedf-4812-4ebf-82cd-c80d06d1886a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-04248e-6xkgh" Oct 30 17:36:04.490376 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.490292 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptvk6\" (UniqueName: \"kubernetes.io/projected/ac7e71fc-8c27-4f6f-b9f2-bf5fbd115267-kube-api-access-ptvk6\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:04.493346 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.493301 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af8374c3-9f76-4183-9eee-d781479514a0-kube-api-access-bptb4" pod "af8374c3-9f76-4183-9eee-d781479514a0" (UID: "af8374c3-9f76-4183-9eee-d781479514a0"). InnerVolumeSpecName "kube-api-access-bptb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:36:04.592387 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.592347 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw84f\" (UniqueName: \"kubernetes.io/projected/6b9ebedf-4812-4ebf-82cd-c80d06d1886a-kube-api-access-lw84f\") pod \"maas-api-revoke-keys-e2e-worker-w1-04248e-6xkgh\" (UID: \"6b9ebedf-4812-4ebf-82cd-c80d06d1886a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-04248e-6xkgh" Oct 30 17:36:04.592665 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.592413 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b9ebedf-4812-4ebf-82cd-c80d06d1886a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-04248e-6xkgh\" (UID: \"6b9ebedf-4812-4ebf-82cd-c80d06d1886a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-04248e-6xkgh" Oct 30 17:36:04.592665 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.592513 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b9ebedf-4812-4ebf-82cd-c80d06d1886a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-04248e-6xkgh\" (UID: \"6b9ebedf-4812-4ebf-82cd-c80d06d1886a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-04248e-6xkgh" Oct 30 17:36:04.592806 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.592755 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bptb4\" (UniqueName: \"kubernetes.io/projected/af8374c3-9f76-4183-9eee-d781479514a0-kube-api-access-bptb4\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:04.594166 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.594134 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b9ebedf-4812-4ebf-82cd-c80d06d1886a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-04248e-6xkgh\" (UID: \"6b9ebedf-4812-4ebf-82cd-c80d06d1886a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-04248e-6xkgh" Oct 30 17:36:04.604007 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.603968 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b9ebedf-4812-4ebf-82cd-c80d06d1886a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-04248e-6xkgh\" (UID: \"6b9ebedf-4812-4ebf-82cd-c80d06d1886a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-04248e-6xkgh" Oct 30 17:36:04.608783 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.608740 4522 generic.go:356] "Generic (PLEG): container finished" podID="ac7e71fc-8c27-4f6f-b9f2-bf5fbd115267" containerID="a81c11b95b85566b3fa66e25ef6d1cbd67a8cdce505f8cd739ecaebd95d43108" exitCode=0 Oct 30 17:36:04.608990 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.608972 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8681b5-7d9676c6d6-khtxf" Oct 30 17:36:04.609660 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.609625 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8681b5-7d9676c6d6-khtxf" event={"ID":"ac7e71fc-8c27-4f6f-b9f2-bf5fbd115267","Type":"ContainerDied","Data":"a81c11b95b85566b3fa66e25ef6d1cbd67a8cdce505f8cd739ecaebd95d43108"} Oct 30 17:36:04.609821 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.609687 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8681b5-7d9676c6d6-khtxf" event={"ID":"ac7e71fc-8c27-4f6f-b9f2-bf5fbd115267","Type":"ContainerDied","Data":"d743e7dc83295ff7e4e83da602efb2550e241a2750a35fb5a4dbbd78e8985892"} Oct 30 17:36:04.609821 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.609716 4522 scope.go:122] "RemoveContainer" containerID="a81c11b95b85566b3fa66e25ef6d1cbd67a8cdce505f8cd739ecaebd95d43108" Oct 30 17:36:04.611967 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.611940 4522 generic.go:356] "Generic (PLEG): container finished" podID="af8374c3-9f76-4183-9eee-d781479514a0" containerID="cf41024ad80e3b478243d0c04b796986fe8976a6568f6c7ea4f1b1df9eb68ecd" exitCode=0 Oct 30 17:36:04.612086 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.611991 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8681b5-d7469f799-jsz5b" Oct 30 17:36:04.612086 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.612037 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8681b5-d7469f799-jsz5b" event={"ID":"af8374c3-9f76-4183-9eee-d781479514a0","Type":"ContainerDied","Data":"cf41024ad80e3b478243d0c04b796986fe8976a6568f6c7ea4f1b1df9eb68ecd"} Oct 30 17:36:04.612203 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.612096 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8681b5-d7469f799-jsz5b" event={"ID":"af8374c3-9f76-4183-9eee-d781479514a0","Type":"ContainerDied","Data":"82f488457990c0fd284f63f9b8ca792b4d3cb8a7165dca2d13f8c95831a3fa1f"} Oct 30 17:36:04.619019 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.618757 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw84f\" (UniqueName: \"kubernetes.io/projected/6b9ebedf-4812-4ebf-82cd-c80d06d1886a-kube-api-access-lw84f\") pod \"maas-api-revoke-keys-e2e-worker-w1-04248e-6xkgh\" (UID: \"6b9ebedf-4812-4ebf-82cd-c80d06d1886a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-04248e-6xkgh" Oct 30 17:36:04.623758 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.623734 4522 scope.go:122] "RemoveContainer" containerID="a81c11b95b85566b3fa66e25ef6d1cbd67a8cdce505f8cd739ecaebd95d43108" Oct 30 17:36:04.624083 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:36:04.624062 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a81c11b95b85566b3fa66e25ef6d1cbd67a8cdce505f8cd739ecaebd95d43108\": container with ID starting with a81c11b95b85566b3fa66e25ef6d1cbd67a8cdce505f8cd739ecaebd95d43108 not found: ID does not exist" containerID="a81c11b95b85566b3fa66e25ef6d1cbd67a8cdce505f8cd739ecaebd95d43108" Oct 30 17:36:04.624182 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.624090 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a81c11b95b85566b3fa66e25ef6d1cbd67a8cdce505f8cd739ecaebd95d43108"} err="failed to get container status \"a81c11b95b85566b3fa66e25ef6d1cbd67a8cdce505f8cd739ecaebd95d43108\": rpc error: code = NotFound desc = could not find container \"a81c11b95b85566b3fa66e25ef6d1cbd67a8cdce505f8cd739ecaebd95d43108\": container with ID starting with a81c11b95b85566b3fa66e25ef6d1cbd67a8cdce505f8cd739ecaebd95d43108 not found: ID does not exist" Oct 30 17:36:04.624182 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.624109 4522 scope.go:122] "RemoveContainer" containerID="cf41024ad80e3b478243d0c04b796986fe8976a6568f6c7ea4f1b1df9eb68ecd" Oct 30 17:36:04.638766 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.638746 4522 scope.go:122] "RemoveContainer" containerID="cf41024ad80e3b478243d0c04b796986fe8976a6568f6c7ea4f1b1df9eb68ecd" Oct 30 17:36:04.639089 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:36:04.639068 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf41024ad80e3b478243d0c04b796986fe8976a6568f6c7ea4f1b1df9eb68ecd\": container with ID starting with cf41024ad80e3b478243d0c04b796986fe8976a6568f6c7ea4f1b1df9eb68ecd not found: ID does not exist" containerID="cf41024ad80e3b478243d0c04b796986fe8976a6568f6c7ea4f1b1df9eb68ecd" Oct 30 17:36:04.639140 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.639098 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf41024ad80e3b478243d0c04b796986fe8976a6568f6c7ea4f1b1df9eb68ecd"} err="failed to get container status \"cf41024ad80e3b478243d0c04b796986fe8976a6568f6c7ea4f1b1df9eb68ecd\": rpc error: code = NotFound desc = could not find container \"cf41024ad80e3b478243d0c04b796986fe8976a6568f6c7ea4f1b1df9eb68ecd\": container with ID starting with cf41024ad80e3b478243d0c04b796986fe8976a6568f6c7ea4f1b1df9eb68ecd not found: ID does not exist" Oct 30 17:36:04.666921 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.666886 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-8681b5-7d9676c6d6-khtxf"] Oct 30 17:36:04.675146 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.675115 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-8681b5-7d9676c6d6-khtxf"] Oct 30 17:36:04.715940 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.715906 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8681b5-d7469f799-jsz5b"] Oct 30 17:36:04.720139 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.720109 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8681b5-d7469f799-jsz5b"] Oct 30 17:36:04.740197 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:04.740118 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-04248e-6xkgh" Oct 30 17:36:05.081043 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:36:05.081012 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b9ebedf_4812_4ebf_82cd_c80d06d1886a.slice/crio-e054891cd0cdd3fcb8ea995e9ceca53aed2055e25611ce10957864c1b15eca07 WatchSource:0}: Error finding container e054891cd0cdd3fcb8ea995e9ceca53aed2055e25611ce10957864c1b15eca07: Status 404 returned error can't find the container with id e054891cd0cdd3fcb8ea995e9ceca53aed2055e25611ce10957864c1b15eca07 Oct 30 17:36:05.081917 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:05.081884 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-04248e-6xkgh"] Oct 30 17:36:05.622914 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:05.622881 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-04248e-6xkgh" event={"ID":"6b9ebedf-4812-4ebf-82cd-c80d06d1886a","Type":"ContainerStarted","Data":"5dea86fde6ced17f0ff315d1655a40987d2209a106ce22d20ce4dc244e84ff3f"} Oct 30 17:36:05.623440 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:05.622931 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-04248e-6xkgh" event={"ID":"6b9ebedf-4812-4ebf-82cd-c80d06d1886a","Type":"ContainerStarted","Data":"e054891cd0cdd3fcb8ea995e9ceca53aed2055e25611ce10957864c1b15eca07"} Oct 30 17:36:05.669381 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:05.669236 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-04248e-6xkgh" podStartSLOduration=1.669223629 podStartE2EDuration="1.669223629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:36:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:36:05.66620477 +0000 UTC m=+2939.666959672" watchObservedRunningTime="2026-10-30 17:36:05.669223629 +0000 UTC m=+2939.669978554" Oct 30 17:36:06.145716 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:06.145681 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac7e71fc-8c27-4f6f-b9f2-bf5fbd115267" path="/var/lib/kubelet/pods/ac7e71fc-8c27-4f6f-b9f2-bf5fbd115267/volumes" Oct 30 17:36:06.146184 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:06.146162 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af8374c3-9f76-4183-9eee-d781479514a0" path="/var/lib/kubelet/pods/af8374c3-9f76-4183-9eee-d781479514a0/volumes" Oct 30 17:36:06.276835 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:06.276749 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-db8417-bfdb94d4d-frgtn"] Oct 30 17:36:06.311847 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:06.311818 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-db8417-74d9d84dc9-ddws5"] Oct 30 17:36:06.632968 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:06.632885 4522 generic.go:356] "Generic (PLEG): container finished" podID="6b9ebedf-4812-4ebf-82cd-c80d06d1886a" containerID="5dea86fde6ced17f0ff315d1655a40987d2209a106ce22d20ce4dc244e84ff3f" exitCode=0 Oct 30 17:36:06.633377 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:06.632980 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-04248e-6xkgh" event={"ID":"6b9ebedf-4812-4ebf-82cd-c80d06d1886a","Type":"ContainerDied","Data":"5dea86fde6ced17f0ff315d1655a40987d2209a106ce22d20ce4dc244e84ff3f"} Oct 30 17:36:07.791243 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:07.791216 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-04248e-6xkgh" Oct 30 17:36:07.840640 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:07.840608 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b9ebedf-4812-4ebf-82cd-c80d06d1886a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b9ebedf-4812-4ebf-82cd-c80d06d1886a-maas-api-xks-ca\") pod \"6b9ebedf-4812-4ebf-82cd-c80d06d1886a\" (UID: \"6b9ebedf-4812-4ebf-82cd-c80d06d1886a\") " Oct 30 17:36:07.840845 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:07.840665 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b9ebedf-4812-4ebf-82cd-c80d06d1886a-kube-api-access-lw84f\" (UniqueName: \"kubernetes.io/projected/6b9ebedf-4812-4ebf-82cd-c80d06d1886a-kube-api-access-lw84f\") pod \"6b9ebedf-4812-4ebf-82cd-c80d06d1886a\" (UID: \"6b9ebedf-4812-4ebf-82cd-c80d06d1886a\") " Oct 30 17:36:07.840845 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:07.840806 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b9ebedf-4812-4ebf-82cd-c80d06d1886a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b9ebedf-4812-4ebf-82cd-c80d06d1886a-maas-api-openshift-ca\") pod \"6b9ebedf-4812-4ebf-82cd-c80d06d1886a\" (UID: \"6b9ebedf-4812-4ebf-82cd-c80d06d1886a\") " Oct 30 17:36:07.841170 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:07.841140 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b9ebedf-4812-4ebf-82cd-c80d06d1886a-maas-api-openshift-ca" pod "6b9ebedf-4812-4ebf-82cd-c80d06d1886a" (UID: "6b9ebedf-4812-4ebf-82cd-c80d06d1886a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:36:07.841411 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:07.841394 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b9ebedf-4812-4ebf-82cd-c80d06d1886a-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:07.842918 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:07.842891 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b9ebedf-4812-4ebf-82cd-c80d06d1886a-kube-api-access-lw84f" pod "6b9ebedf-4812-4ebf-82cd-c80d06d1886a" (UID: "6b9ebedf-4812-4ebf-82cd-c80d06d1886a"). InnerVolumeSpecName "kube-api-access-lw84f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:36:07.843164 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:07.843141 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b9ebedf-4812-4ebf-82cd-c80d06d1886a-maas-api-xks-ca" pod "6b9ebedf-4812-4ebf-82cd-c80d06d1886a" (UID: "6b9ebedf-4812-4ebf-82cd-c80d06d1886a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:36:07.942726 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:07.942638 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b9ebedf-4812-4ebf-82cd-c80d06d1886a-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:07.942726 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:07.942671 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lw84f\" (UniqueName: \"kubernetes.io/projected/6b9ebedf-4812-4ebf-82cd-c80d06d1886a-kube-api-access-lw84f\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:08.649072 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:08.649036 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-04248e-6xkgh" Oct 30 17:36:08.649254 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:08.649073 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-04248e-6xkgh" event={"ID":"6b9ebedf-4812-4ebf-82cd-c80d06d1886a","Type":"ContainerDied","Data":"e054891cd0cdd3fcb8ea995e9ceca53aed2055e25611ce10957864c1b15eca07"} Oct 30 17:36:08.649254 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:08.649124 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e054891cd0cdd3fcb8ea995e9ceca53aed2055e25611ce10957864c1b15eca07" Oct 30 17:36:09.087850 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:09.087814 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-04248e/distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq"] Oct 30 17:36:09.103283 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:09.103247 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-04248e/distinct-w1-04248e-kserve-5859d49965-kgq46"] Oct 30 17:36:09.349761 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:09.349667 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-78c9c2-6b9c4447c4-p5z99" podUID="63ee8702-b07d-4e00-81d4-bffadca6608d" containerName="payload-processing" containerID="cri-o://fe602c2e074fc9de9203620088c8a6c36983991f0872c2359eba459749d5edc3" gracePeriod=25 Oct 30 17:36:09.389515 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:09.389447 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-78c9c2-777cdc8975-7qp6m" podUID="90cb278c-f9d6-4948-9843-0330b688521a" containerName="payload-pre-processing" containerID="cri-o://2cd513a1c4dbe4618ea797e265570f43d628568fd7b64a7816c5368b256d12a2" gracePeriod=25 Oct 30 17:36:11.283326 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:11.279773 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-db8417-bfdb94d4d-frgtn" podUID="47fa8756-130e-48fc-bcf9-008ab31f95cf" containerName="payload-processing" containerID="cri-o://eaad7e2d0590e24706f9d1084db25ae7b57647af207968ccc4f4d99782071560" gracePeriod=25 Oct 30 17:36:11.312665 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:11.312434 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-db8417-74d9d84dc9-ddws5" podUID="e6e99dad-79f2-4b63-87f9-1c761080cada" containerName="payload-pre-processing" containerID="cri-o://a0df871312b2d488be253a98a94a9998a29369b3f162066331873f625d033746" gracePeriod=25 Oct 30 17:36:11.673037 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:11.672952 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6d4f99-rwdd8"] Oct 30 17:36:11.685924 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:11.685894 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6d4f99-rwdd8"] Oct 30 17:36:12.146345 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:12.146289 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b88fde22-4b6e-4a63-a0ba-aa3e812472da" path="/var/lib/kubelet/pods/b88fde22-4b6e-4a63-a0ba-aa3e812472da/volumes" Oct 30 17:36:12.157887 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:12.157851 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-db8417-bfdb94d4d-frgtn" podUID="47fa8756-130e-48fc-bcf9-008ab31f95cf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:36:12.158126 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:12.157864 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-db8417-74d9d84dc9-ddws5" podUID="e6e99dad-79f2-4b63-87f9-1c761080cada" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:36:12.949880 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:36:12.949843 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:36:14.660639 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:14.660603 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-04248e-6dfd587cc9-n8brs"] Oct 30 17:36:15.708850 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:15.708811 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-78c9c2-w9btc"] Oct 30 17:36:15.729673 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:15.729637 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-78c9c2-w9btc"] Oct 30 17:36:16.146138 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:16.146060 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="642f81c8-be98-4b6f-a34d-5bc3fe9d9579" path="/var/lib/kubelet/pods/642f81c8-be98-4b6f-a34d-5bc3fe9d9579/volumes" Oct 30 17:36:16.609453 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:16.609408 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-78c9c2-6b9c4447c4-p5z99" podUID="63ee8702-b07d-4e00-81d4-bffadca6608d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:36:16.609453 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:16.609435 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-78c9c2-777cdc8975-7qp6m" podUID="90cb278c-f9d6-4948-9843-0330b688521a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:36:17.674577 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:17.674499 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8f92ef-5bd8786b7-52rzg"] Oct 30 17:36:17.675225 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:17.675209 4522 cpu_manager.go:414] "containerMap: removing container" podUID="6b9ebedf-4812-4ebf-82cd-c80d06d1886a" containerName="revoke-keys" Oct 30 17:36:17.675285 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:17.675228 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b9ebedf-4812-4ebf-82cd-c80d06d1886a" containerName="revoke-keys" Oct 30 17:36:17.675367 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:17.675355 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="827251e7-f88d-4ed1-81bb-ce4960304aed" containerName="istio-proxy" Oct 30 17:36:17.675415 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:17.675376 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b9ebedf-4812-4ebf-82cd-c80d06d1886a" containerName="revoke-keys" Oct 30 17:36:17.681267 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:17.681225 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f92ef-5bd8786b7-52rzg" Oct 30 17:36:17.698850 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:17.698817 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8f92ef-5bd8786b7-52rzg"] Oct 30 17:36:17.707829 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:17.707798 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-8f92ef\"" Oct 30 17:36:17.707972 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:17.707911 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-8f92ef-dockercfg-gbw2r\"" Oct 30 17:36:17.790292 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:17.790256 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4klbj\" (UniqueName: \"kubernetes.io/projected/0e442fef-3f96-4523-8cae-814507c01bab-kube-api-access-4klbj\") pod \"payload-pre-processing-e2e-ipp-a-8f92ef-5bd8786b7-52rzg\" (UID: \"0e442fef-3f96-4523-8cae-814507c01bab\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f92ef-5bd8786b7-52rzg" Oct 30 17:36:17.790604 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:17.790579 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e442fef-3f96-4523-8cae-814507c01bab-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8f92ef-5bd8786b7-52rzg\" (UID: \"0e442fef-3f96-4523-8cae-814507c01bab\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f92ef-5bd8786b7-52rzg" Oct 30 17:36:17.832708 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:17.832676 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8f92ef-85f6b78466-9mbs9"] Oct 30 17:36:17.837071 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:17.837051 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8f92ef-85f6b78466-9mbs9" Oct 30 17:36:17.857986 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:17.857960 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8f92ef-85f6b78466-9mbs9"] Oct 30 17:36:17.891686 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:17.891654 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e442fef-3f96-4523-8cae-814507c01bab-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8f92ef-5bd8786b7-52rzg\" (UID: \"0e442fef-3f96-4523-8cae-814507c01bab\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f92ef-5bd8786b7-52rzg" Oct 30 17:36:17.891858 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:17.891751 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4klbj\" (UniqueName: \"kubernetes.io/projected/0e442fef-3f96-4523-8cae-814507c01bab-kube-api-access-4klbj\") pod \"payload-pre-processing-e2e-ipp-a-8f92ef-5bd8786b7-52rzg\" (UID: \"0e442fef-3f96-4523-8cae-814507c01bab\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f92ef-5bd8786b7-52rzg" Oct 30 17:36:17.892664 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:17.892640 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e442fef-3f96-4523-8cae-814507c01bab-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8f92ef-5bd8786b7-52rzg\" (UID: \"0e442fef-3f96-4523-8cae-814507c01bab\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f92ef-5bd8786b7-52rzg" Oct 30 17:36:17.916721 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:17.916689 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4klbj\" (UniqueName: \"kubernetes.io/projected/0e442fef-3f96-4523-8cae-814507c01bab-kube-api-access-4klbj\") pod \"payload-pre-processing-e2e-ipp-a-8f92ef-5bd8786b7-52rzg\" (UID: \"0e442fef-3f96-4523-8cae-814507c01bab\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f92ef-5bd8786b7-52rzg" Oct 30 17:36:17.992842 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:17.992811 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmcmc\" (UniqueName: \"kubernetes.io/projected/bca92d15-cd50-43f0-bf26-e5603842657c-kube-api-access-fmcmc\") pod \"payload-processing-e2e-ipp-a-8f92ef-85f6b78466-9mbs9\" (UID: \"bca92d15-cd50-43f0-bf26-e5603842657c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8f92ef-85f6b78466-9mbs9" Oct 30 17:36:17.993013 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:17.992872 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bca92d15-cd50-43f0-bf26-e5603842657c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8f92ef-85f6b78466-9mbs9\" (UID: \"bca92d15-cd50-43f0-bf26-e5603842657c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8f92ef-85f6b78466-9mbs9" Oct 30 17:36:18.004454 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:18.004434 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f92ef-5bd8786b7-52rzg" Oct 30 17:36:18.094468 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:18.094434 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bca92d15-cd50-43f0-bf26-e5603842657c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8f92ef-85f6b78466-9mbs9\" (UID: \"bca92d15-cd50-43f0-bf26-e5603842657c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8f92ef-85f6b78466-9mbs9" Oct 30 17:36:18.095617 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:18.094632 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmcmc\" (UniqueName: \"kubernetes.io/projected/bca92d15-cd50-43f0-bf26-e5603842657c-kube-api-access-fmcmc\") pod \"payload-processing-e2e-ipp-a-8f92ef-85f6b78466-9mbs9\" (UID: \"bca92d15-cd50-43f0-bf26-e5603842657c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8f92ef-85f6b78466-9mbs9" Oct 30 17:36:18.095617 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:18.095567 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bca92d15-cd50-43f0-bf26-e5603842657c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8f92ef-85f6b78466-9mbs9\" (UID: \"bca92d15-cd50-43f0-bf26-e5603842657c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8f92ef-85f6b78466-9mbs9" Oct 30 17:36:18.120097 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:18.120069 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmcmc\" (UniqueName: \"kubernetes.io/projected/bca92d15-cd50-43f0-bf26-e5603842657c-kube-api-access-fmcmc\") pod \"payload-processing-e2e-ipp-a-8f92ef-85f6b78466-9mbs9\" (UID: \"bca92d15-cd50-43f0-bf26-e5603842657c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8f92ef-85f6b78466-9mbs9" Oct 30 17:36:18.150736 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:18.150703 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8f92ef-85f6b78466-9mbs9" Oct 30 17:36:18.332046 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:36:18.332015 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e442fef_3f96_4523_8cae_814507c01bab.slice/crio-6091bf82d790a0026aa8f8d7262c0a49d0d314b1b2c261408ef35ef1c298f217 WatchSource:0}: Error finding container 6091bf82d790a0026aa8f8d7262c0a49d0d314b1b2c261408ef35ef1c298f217: Status 404 returned error can't find the container with id 6091bf82d790a0026aa8f8d7262c0a49d0d314b1b2c261408ef35ef1c298f217 Oct 30 17:36:18.333511 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:18.333486 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8f92ef-5bd8786b7-52rzg"] Oct 30 17:36:18.458783 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:36:18.458752 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbca92d15_cd50_43f0_bf26_e5603842657c.slice/crio-784944fff68fe43bff6ac285bf918203d2125f8b59237fa0edcc430aaa25d778 WatchSource:0}: Error finding container 784944fff68fe43bff6ac285bf918203d2125f8b59237fa0edcc430aaa25d778: Status 404 returned error can't find the container with id 784944fff68fe43bff6ac285bf918203d2125f8b59237fa0edcc430aaa25d778 Oct 30 17:36:18.460267 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:18.460243 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8f92ef-85f6b78466-9mbs9"] Oct 30 17:36:18.755814 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:18.755777 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8f92ef-85f6b78466-9mbs9" event={"ID":"bca92d15-cd50-43f0-bf26-e5603842657c","Type":"ContainerStarted","Data":"74144b8c162b483de434cbdb732535e0ae24d78d39bd3b8175d1c263f81147df"} Oct 30 17:36:18.756269 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:18.755843 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8f92ef-85f6b78466-9mbs9" event={"ID":"bca92d15-cd50-43f0-bf26-e5603842657c","Type":"ContainerStarted","Data":"784944fff68fe43bff6ac285bf918203d2125f8b59237fa0edcc430aaa25d778"} Oct 30 17:36:18.756269 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:18.755888 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8f92ef-85f6b78466-9mbs9" Oct 30 17:36:18.757660 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:18.757632 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f92ef-5bd8786b7-52rzg" event={"ID":"0e442fef-3f96-4523-8cae-814507c01bab","Type":"ContainerStarted","Data":"914bd156d6dc284fe2405f8f1d4d7e14691d8401a2d385659c3678d7dce2ec97"} Oct 30 17:36:18.757772 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:18.757683 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f92ef-5bd8786b7-52rzg" event={"ID":"0e442fef-3f96-4523-8cae-814507c01bab","Type":"ContainerStarted","Data":"6091bf82d790a0026aa8f8d7262c0a49d0d314b1b2c261408ef35ef1c298f217"} Oct 30 17:36:18.757772 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:18.757713 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f92ef-5bd8786b7-52rzg" Oct 30 17:36:18.804279 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:18.804223 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-8f92ef-85f6b78466-9mbs9" podStartSLOduration=1.804209033 podStartE2EDuration="1.804209033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:36:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:36:18.801443354 +0000 UTC m=+2952.802198274" watchObservedRunningTime="2026-10-30 17:36:18.804209033 +0000 UTC m=+2952.804963933" Oct 30 17:36:18.855015 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:18.854966 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f92ef-5bd8786b7-52rzg" podStartSLOduration=1.854948935 podStartE2EDuration="1.854948935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:36:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:36:18.850100584 +0000 UTC m=+2952.850855485" watchObservedRunningTime="2026-10-30 17:36:18.854948935 +0000 UTC m=+2952.855703837" Oct 30 17:36:19.611294 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:19.611258 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc"] Oct 30 17:36:19.611668 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:19.611563 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc" podUID="a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7" containerName="istio-proxy" containerID="cri-o://f8e750719d86c8219dd9e97846d47e2a982963c63adda49e8d9b17a09e3df519" gracePeriod=30 Oct 30 17:36:19.661193 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:19.661156 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-04248e-6dfd587cc9-n8brs" podUID="80f8edc4-4e9f-435e-8518-e5f53d262030" containerName="payload-pre-processing" containerID="cri-o://ee44950c663dabf7409634f43dd79fe985e5ec6c00f06f95918d4042c0d9a567" gracePeriod=25 Oct 30 17:36:20.033030 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:20.032993 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9722dad0-hrmt2"] Oct 30 17:36:20.038605 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:20.038578 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9722dad0-hrmt2"] Oct 30 17:36:20.146105 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:20.146073 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52db46a9-175b-4cad-bc70-1345d8ddb498" path="/var/lib/kubelet/pods/52db46a9-175b-4cad-bc70-1345d8ddb498/volumes" Oct 30 17:36:20.387713 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:20.387635 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b4f406-7cw59"] Oct 30 17:36:20.400648 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:20.400610 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b4f406-7cw59" Oct 30 17:36:20.414402 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:20.414365 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b4f406-7cw59"] Oct 30 17:36:20.446353 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:20.446321 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:36:20.446353 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:20.446321 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:36:20.528269 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:20.528233 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgl4n\" (UniqueName: \"kubernetes.io/projected/450fab43-5d99-4c34-97e6-59a6d64de032-kube-api-access-hgl4n\") pod \"maas-api-revoke-keys-e2e-worker-w2-b4f406-7cw59\" (UID: \"450fab43-5d99-4c34-97e6-59a6d64de032\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b4f406-7cw59" Oct 30 17:36:20.528510 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:20.528387 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/450fab43-5d99-4c34-97e6-59a6d64de032-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-b4f406-7cw59\" (UID: \"450fab43-5d99-4c34-97e6-59a6d64de032\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b4f406-7cw59" Oct 30 17:36:20.528510 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:20.528465 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/450fab43-5d99-4c34-97e6-59a6d64de032-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-b4f406-7cw59\" (UID: \"450fab43-5d99-4c34-97e6-59a6d64de032\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b4f406-7cw59" Oct 30 17:36:20.629781 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:20.629748 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/450fab43-5d99-4c34-97e6-59a6d64de032-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-b4f406-7cw59\" (UID: \"450fab43-5d99-4c34-97e6-59a6d64de032\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b4f406-7cw59" Oct 30 17:36:20.629964 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:20.629816 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/450fab43-5d99-4c34-97e6-59a6d64de032-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-b4f406-7cw59\" (UID: \"450fab43-5d99-4c34-97e6-59a6d64de032\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b4f406-7cw59" Oct 30 17:36:20.629964 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:20.629956 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgl4n\" (UniqueName: \"kubernetes.io/projected/450fab43-5d99-4c34-97e6-59a6d64de032-kube-api-access-hgl4n\") pod \"maas-api-revoke-keys-e2e-worker-w2-b4f406-7cw59\" (UID: \"450fab43-5d99-4c34-97e6-59a6d64de032\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b4f406-7cw59" Oct 30 17:36:20.630518 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:20.630480 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/450fab43-5d99-4c34-97e6-59a6d64de032-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-b4f406-7cw59\" (UID: \"450fab43-5d99-4c34-97e6-59a6d64de032\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b4f406-7cw59" Oct 30 17:36:20.632206 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:20.632185 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/450fab43-5d99-4c34-97e6-59a6d64de032-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-b4f406-7cw59\" (UID: \"450fab43-5d99-4c34-97e6-59a6d64de032\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b4f406-7cw59" Oct 30 17:36:20.654943 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:20.654877 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgl4n\" (UniqueName: \"kubernetes.io/projected/450fab43-5d99-4c34-97e6-59a6d64de032-kube-api-access-hgl4n\") pod \"maas-api-revoke-keys-e2e-worker-w2-b4f406-7cw59\" (UID: \"450fab43-5d99-4c34-97e6-59a6d64de032\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b4f406-7cw59" Oct 30 17:36:20.717378 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:20.717348 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b4f406-7cw59" Oct 30 17:36:21.029747 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:36:21.029713 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod450fab43_5d99_4c34_97e6_59a6d64de032.slice/crio-9f8ed7845712e7b6fea28fa7d0ec645be0a591d81e96eaf81b3b16bfe8ce23f4 WatchSource:0}: Error finding container 9f8ed7845712e7b6fea28fa7d0ec645be0a591d81e96eaf81b3b16bfe8ce23f4: Status 404 returned error can't find the container with id 9f8ed7845712e7b6fea28fa7d0ec645be0a591d81e96eaf81b3b16bfe8ce23f4 Oct 30 17:36:21.030961 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:21.030932 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b4f406-7cw59"] Oct 30 17:36:21.792878 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:21.792844 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b4f406-7cw59" event={"ID":"450fab43-5d99-4c34-97e6-59a6d64de032","Type":"ContainerStarted","Data":"72b286b72405fbbfeb40ffad83bf89f6f248c0e19c2429a4ab99a646a2a15957"} Oct 30 17:36:21.793288 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:21.792904 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b4f406-7cw59" event={"ID":"450fab43-5d99-4c34-97e6-59a6d64de032","Type":"ContainerStarted","Data":"9f8ed7845712e7b6fea28fa7d0ec645be0a591d81e96eaf81b3b16bfe8ce23f4"} Oct 30 17:36:21.875171 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:21.875118 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b4f406-7cw59" podStartSLOduration=1.875104324 podStartE2EDuration="1.875104324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:36:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:36:21.871356924 +0000 UTC m=+2955.872111836" watchObservedRunningTime="2026-10-30 17:36:21.875104324 +0000 UTC m=+2955.875859219" Oct 30 17:36:22.175830 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:22.175404 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-db8417-bfdb94d4d-frgtn" podUID="47fa8756-130e-48fc-bcf9-008ab31f95cf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:36:22.176330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:22.176242 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-db8417-74d9d84dc9-ddws5" podUID="e6e99dad-79f2-4b63-87f9-1c761080cada" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:36:22.801409 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:22.801375 4522 generic.go:356] "Generic (PLEG): container finished" podID="450fab43-5d99-4c34-97e6-59a6d64de032" containerID="72b286b72405fbbfeb40ffad83bf89f6f248c0e19c2429a4ab99a646a2a15957" exitCode=0 Oct 30 17:36:22.801845 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:22.801418 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b4f406-7cw59" event={"ID":"450fab43-5d99-4c34-97e6-59a6d64de032","Type":"ContainerDied","Data":"72b286b72405fbbfeb40ffad83bf89f6f248c0e19c2429a4ab99a646a2a15957"} Oct 30 17:36:23.903635 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:23.903577 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv"] Oct 30 17:36:23.904738 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:23.904671 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv" podUID="ba5dbb57-7fdc-4509-a870-6c95778f650b" containerName="istio-proxy" containerID="cri-o://dccf5ac7282e6ac94a97385dc7ef4b1ce5c239819a40858e0350c069f8e9356b" gracePeriod=30 Oct 30 17:36:24.017649 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.017289 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b4f406-7cw59" Oct 30 17:36:24.111966 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.111856 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-04248e/distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq" podUID="f18303fb-6d27-4ff1-9cc9-ed032e43d098" containerName="main" containerID="cri-o://fe4e8732b1fc8a2bc0dddc27106aa9615f6269fa006fec9fcea562011fa00f1c" gracePeriod=45 Oct 30 17:36:24.124777 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.124722 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-04248e/distinct-w1-04248e-kserve-5859d49965-kgq46" podUID="8dda5ac4-30dd-432e-81eb-cb6d39800628" containerName="main" containerID="cri-o://26b7d27f48006df06e7014bee645a72ebe5ba485469fb4b64fe85d7fe30c94d9" gracePeriod=45 Oct 30 17:36:24.180165 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.180130 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/450fab43-5d99-4c34-97e6-59a6d64de032-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/450fab43-5d99-4c34-97e6-59a6d64de032-maas-api-xks-ca\") pod \"450fab43-5d99-4c34-97e6-59a6d64de032\" (UID: \"450fab43-5d99-4c34-97e6-59a6d64de032\") " Oct 30 17:36:24.180302 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.180234 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/450fab43-5d99-4c34-97e6-59a6d64de032-kube-api-access-hgl4n\" (UniqueName: \"kubernetes.io/projected/450fab43-5d99-4c34-97e6-59a6d64de032-kube-api-access-hgl4n\") pod \"450fab43-5d99-4c34-97e6-59a6d64de032\" (UID: \"450fab43-5d99-4c34-97e6-59a6d64de032\") " Oct 30 17:36:24.180302 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.180266 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/450fab43-5d99-4c34-97e6-59a6d64de032-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/450fab43-5d99-4c34-97e6-59a6d64de032-maas-api-openshift-ca\") pod \"450fab43-5d99-4c34-97e6-59a6d64de032\" (UID: \"450fab43-5d99-4c34-97e6-59a6d64de032\") " Oct 30 17:36:24.180672 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.180649 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/450fab43-5d99-4c34-97e6-59a6d64de032-maas-api-openshift-ca" pod "450fab43-5d99-4c34-97e6-59a6d64de032" (UID: "450fab43-5d99-4c34-97e6-59a6d64de032"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:36:24.180781 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.180732 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/450fab43-5d99-4c34-97e6-59a6d64de032-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:24.182276 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.182252 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/450fab43-5d99-4c34-97e6-59a6d64de032-maas-api-xks-ca" pod "450fab43-5d99-4c34-97e6-59a6d64de032" (UID: "450fab43-5d99-4c34-97e6-59a6d64de032"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:36:24.182401 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.182386 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/450fab43-5d99-4c34-97e6-59a6d64de032-kube-api-access-hgl4n" pod "450fab43-5d99-4c34-97e6-59a6d64de032" (UID: "450fab43-5d99-4c34-97e6-59a6d64de032"). InnerVolumeSpecName "kube-api-access-hgl4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:36:24.210520 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.210485 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/distinct-w1-04248e-kserve-5859d49965-kgq46" podUID="8dda5ac4-30dd-432e-81eb-cb6d39800628" containerName="main" probeResult="failure" output="Get \"https://10.134.0.108:8000/ready\": dial tcp 10.134.0.108:8000: connect: connection refused" Oct 30 17:36:24.256369 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.256329 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq" podUID="f18303fb-6d27-4ff1-9cc9-ed032e43d098" containerName="main" probeResult="failure" output="Get \"https://10.134.0.120:8000/ready\": dial tcp 10.134.0.120:8000: connect: connection refused" Oct 30 17:36:24.286324 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.282230 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/450fab43-5d99-4c34-97e6-59a6d64de032-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:24.286324 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.282267 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgl4n\" (UniqueName: \"kubernetes.io/projected/450fab43-5d99-4c34-97e6-59a6d64de032-kube-api-access-hgl4n\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:24.399101 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.399072 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-04248e/distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq" Oct 30 17:36:24.414042 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.414015 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-04248e/distinct-w1-04248e-kserve-5859d49965-kgq46" Oct 30 17:36:24.456824 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.456793 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-82243b-nrh8z"] Oct 30 17:36:24.457717 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.457690 4522 cpu_manager.go:414] "containerMap: removing container" podUID="450fab43-5d99-4c34-97e6-59a6d64de032" containerName="revoke-keys" Oct 30 17:36:24.457717 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.457714 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="450fab43-5d99-4c34-97e6-59a6d64de032" containerName="revoke-keys" Oct 30 17:36:24.457938 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.457739 4522 cpu_manager.go:414] "containerMap: removing container" podUID="8dda5ac4-30dd-432e-81eb-cb6d39800628" containerName="main" Oct 30 17:36:24.457938 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.457749 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dda5ac4-30dd-432e-81eb-cb6d39800628" containerName="main" Oct 30 17:36:24.457938 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.457775 4522 cpu_manager.go:414] "containerMap: removing container" podUID="f18303fb-6d27-4ff1-9cc9-ed032e43d098" containerName="main" Oct 30 17:36:24.457938 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.457782 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f18303fb-6d27-4ff1-9cc9-ed032e43d098" containerName="main" Oct 30 17:36:24.458158 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.457950 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="450fab43-5d99-4c34-97e6-59a6d64de032" containerName="revoke-keys" Oct 30 17:36:24.458158 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.457970 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dda5ac4-30dd-432e-81eb-cb6d39800628" containerName="main" Oct 30 17:36:24.458158 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.457984 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="f18303fb-6d27-4ff1-9cc9-ed032e43d098" containerName="main" Oct 30 17:36:24.463146 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.463119 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-82243b-nrh8z" Oct 30 17:36:24.487885 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.487853 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-82243b-nrh8z"] Oct 30 17:36:24.584919 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.584877 4522 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:36:24.585127 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.584933 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc" podUID="a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:36:24.590471 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.590447 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dda5ac4-30dd-432e-81eb-cb6d39800628-home\" (UniqueName: \"kubernetes.io/empty-dir/8dda5ac4-30dd-432e-81eb-cb6d39800628-home\") pod \"8dda5ac4-30dd-432e-81eb-cb6d39800628\" (UID: \"8dda5ac4-30dd-432e-81eb-cb6d39800628\") " Oct 30 17:36:24.590628 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.590491 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dda5ac4-30dd-432e-81eb-cb6d39800628-kube-api-access-vrkfp\" (UniqueName: \"kubernetes.io/projected/8dda5ac4-30dd-432e-81eb-cb6d39800628-kube-api-access-vrkfp\") pod \"8dda5ac4-30dd-432e-81eb-cb6d39800628\" (UID: \"8dda5ac4-30dd-432e-81eb-cb6d39800628\") " Oct 30 17:36:24.590628 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.590552 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dda5ac4-30dd-432e-81eb-cb6d39800628-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dda5ac4-30dd-432e-81eb-cb6d39800628-tmp-dir\") pod \"8dda5ac4-30dd-432e-81eb-cb6d39800628\" (UID: \"8dda5ac4-30dd-432e-81eb-cb6d39800628\") " Oct 30 17:36:24.590628 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.590572 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8dda5ac4-30dd-432e-81eb-cb6d39800628-tls-certs\" (UniqueName: \"kubernetes.io/secret/8dda5ac4-30dd-432e-81eb-cb6d39800628-tls-certs\") pod \"8dda5ac4-30dd-432e-81eb-cb6d39800628\" (UID: \"8dda5ac4-30dd-432e-81eb-cb6d39800628\") " Oct 30 17:36:24.590628 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.590595 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f18303fb-6d27-4ff1-9cc9-ed032e43d098-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f18303fb-6d27-4ff1-9cc9-ed032e43d098-tmp-dir\") pod \"f18303fb-6d27-4ff1-9cc9-ed032e43d098\" (UID: \"f18303fb-6d27-4ff1-9cc9-ed032e43d098\") " Oct 30 17:36:24.590865 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.590727 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f18303fb-6d27-4ff1-9cc9-ed032e43d098-home\" (UniqueName: \"kubernetes.io/empty-dir/f18303fb-6d27-4ff1-9cc9-ed032e43d098-home\") pod \"f18303fb-6d27-4ff1-9cc9-ed032e43d098\" (UID: \"f18303fb-6d27-4ff1-9cc9-ed032e43d098\") " Oct 30 17:36:24.590865 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.590731 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dda5ac4-30dd-432e-81eb-cb6d39800628-home" pod "8dda5ac4-30dd-432e-81eb-cb6d39800628" (UID: "8dda5ac4-30dd-432e-81eb-cb6d39800628"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:24.590865 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.590767 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f18303fb-6d27-4ff1-9cc9-ed032e43d098-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f18303fb-6d27-4ff1-9cc9-ed032e43d098-dshm\") pod \"f18303fb-6d27-4ff1-9cc9-ed032e43d098\" (UID: \"f18303fb-6d27-4ff1-9cc9-ed032e43d098\") " Oct 30 17:36:24.590865 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.590837 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f18303fb-6d27-4ff1-9cc9-ed032e43d098-kube-api-access-br48l\" (UniqueName: \"kubernetes.io/projected/f18303fb-6d27-4ff1-9cc9-ed032e43d098-kube-api-access-br48l\") pod \"f18303fb-6d27-4ff1-9cc9-ed032e43d098\" (UID: \"f18303fb-6d27-4ff1-9cc9-ed032e43d098\") " Oct 30 17:36:24.590865 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.590864 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f18303fb-6d27-4ff1-9cc9-ed032e43d098-tls-certs\" (UniqueName: \"kubernetes.io/secret/f18303fb-6d27-4ff1-9cc9-ed032e43d098-tls-certs\") pod \"f18303fb-6d27-4ff1-9cc9-ed032e43d098\" (UID: \"f18303fb-6d27-4ff1-9cc9-ed032e43d098\") " Oct 30 17:36:24.591155 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.590878 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dda5ac4-30dd-432e-81eb-cb6d39800628-tmp-dir" pod "8dda5ac4-30dd-432e-81eb-cb6d39800628" (UID: "8dda5ac4-30dd-432e-81eb-cb6d39800628"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:24.591155 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.590894 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dda5ac4-30dd-432e-81eb-cb6d39800628-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dda5ac4-30dd-432e-81eb-cb6d39800628-dshm\") pod \"8dda5ac4-30dd-432e-81eb-cb6d39800628\" (UID: \"8dda5ac4-30dd-432e-81eb-cb6d39800628\") " Oct 30 17:36:24.591155 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.590925 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f18303fb-6d27-4ff1-9cc9-ed032e43d098-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f18303fb-6d27-4ff1-9cc9-ed032e43d098-model-cache\") pod \"f18303fb-6d27-4ff1-9cc9-ed032e43d098\" (UID: \"f18303fb-6d27-4ff1-9cc9-ed032e43d098\") " Oct 30 17:36:24.591155 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.590968 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f18303fb-6d27-4ff1-9cc9-ed032e43d098-home" pod "f18303fb-6d27-4ff1-9cc9-ed032e43d098" (UID: "f18303fb-6d27-4ff1-9cc9-ed032e43d098"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:24.591155 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.590987 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f18303fb-6d27-4ff1-9cc9-ed032e43d098-tmp-dir" pod "f18303fb-6d27-4ff1-9cc9-ed032e43d098" (UID: "f18303fb-6d27-4ff1-9cc9-ed032e43d098"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:24.591155 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.591049 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dda5ac4-30dd-432e-81eb-cb6d39800628-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dda5ac4-30dd-432e-81eb-cb6d39800628-model-cache\") pod \"8dda5ac4-30dd-432e-81eb-cb6d39800628\" (UID: \"8dda5ac4-30dd-432e-81eb-cb6d39800628\") " Oct 30 17:36:24.591571 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.591280 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f18303fb-6d27-4ff1-9cc9-ed032e43d098-model-cache" pod "f18303fb-6d27-4ff1-9cc9-ed032e43d098" (UID: "f18303fb-6d27-4ff1-9cc9-ed032e43d098"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:24.591571 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.591509 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dda5ac4-30dd-432e-81eb-cb6d39800628-model-cache" pod "8dda5ac4-30dd-432e-81eb-cb6d39800628" (UID: "8dda5ac4-30dd-432e-81eb-cb6d39800628"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:24.591695 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.591622 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/043dfb0d-b8be-47b0-bd85-5842628a50f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-82243b-nrh8z\" (UID: \"043dfb0d-b8be-47b0-bd85-5842628a50f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-82243b-nrh8z" Oct 30 17:36:24.591695 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.591675 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsxqr\" (UniqueName: \"kubernetes.io/projected/043dfb0d-b8be-47b0-bd85-5842628a50f9-kube-api-access-hsxqr\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-82243b-nrh8z\" (UID: \"043dfb0d-b8be-47b0-bd85-5842628a50f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-82243b-nrh8z" Oct 30 17:36:24.591911 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.591859 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/043dfb0d-b8be-47b0-bd85-5842628a50f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-82243b-nrh8z\" (UID: \"043dfb0d-b8be-47b0-bd85-5842628a50f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-82243b-nrh8z" Oct 30 17:36:24.592109 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.592091 4522 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f18303fb-6d27-4ff1-9cc9-ed032e43d098-home\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:24.592178 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.592114 4522 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f18303fb-6d27-4ff1-9cc9-ed032e43d098-model-cache\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:24.592178 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.592130 4522 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dda5ac4-30dd-432e-81eb-cb6d39800628-model-cache\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:24.592178 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.592145 4522 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dda5ac4-30dd-432e-81eb-cb6d39800628-home\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:24.592178 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.592159 4522 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dda5ac4-30dd-432e-81eb-cb6d39800628-tmp-dir\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:24.592178 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.592171 4522 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f18303fb-6d27-4ff1-9cc9-ed032e43d098-tmp-dir\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:24.593667 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.593640 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dda5ac4-30dd-432e-81eb-cb6d39800628-dshm" pod "8dda5ac4-30dd-432e-81eb-cb6d39800628" (UID: "8dda5ac4-30dd-432e-81eb-cb6d39800628"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:24.593667 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.593645 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dda5ac4-30dd-432e-81eb-cb6d39800628-tls-certs" pod "8dda5ac4-30dd-432e-81eb-cb6d39800628" (UID: "8dda5ac4-30dd-432e-81eb-cb6d39800628"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:36:24.593817 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.593655 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f18303fb-6d27-4ff1-9cc9-ed032e43d098-dshm" pod "f18303fb-6d27-4ff1-9cc9-ed032e43d098" (UID: "f18303fb-6d27-4ff1-9cc9-ed032e43d098"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:24.593817 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.593670 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dda5ac4-30dd-432e-81eb-cb6d39800628-kube-api-access-vrkfp" pod "8dda5ac4-30dd-432e-81eb-cb6d39800628" (UID: "8dda5ac4-30dd-432e-81eb-cb6d39800628"). InnerVolumeSpecName "kube-api-access-vrkfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:36:24.593817 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.593675 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f18303fb-6d27-4ff1-9cc9-ed032e43d098-tls-certs" pod "f18303fb-6d27-4ff1-9cc9-ed032e43d098" (UID: "f18303fb-6d27-4ff1-9cc9-ed032e43d098"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:36:24.594846 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.594825 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f18303fb-6d27-4ff1-9cc9-ed032e43d098-kube-api-access-br48l" pod "f18303fb-6d27-4ff1-9cc9-ed032e43d098" (UID: "f18303fb-6d27-4ff1-9cc9-ed032e43d098"). InnerVolumeSpecName "kube-api-access-br48l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:36:24.693055 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.692964 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/043dfb0d-b8be-47b0-bd85-5842628a50f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-82243b-nrh8z\" (UID: \"043dfb0d-b8be-47b0-bd85-5842628a50f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-82243b-nrh8z" Oct 30 17:36:24.693055 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.693007 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsxqr\" (UniqueName: \"kubernetes.io/projected/043dfb0d-b8be-47b0-bd85-5842628a50f9-kube-api-access-hsxqr\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-82243b-nrh8z\" (UID: \"043dfb0d-b8be-47b0-bd85-5842628a50f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-82243b-nrh8z" Oct 30 17:36:24.693055 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.693040 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/043dfb0d-b8be-47b0-bd85-5842628a50f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-82243b-nrh8z\" (UID: \"043dfb0d-b8be-47b0-bd85-5842628a50f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-82243b-nrh8z" Oct 30 17:36:24.693285 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.693190 4522 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f18303fb-6d27-4ff1-9cc9-ed032e43d098-dshm\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:24.693285 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.693202 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-br48l\" (UniqueName: \"kubernetes.io/projected/f18303fb-6d27-4ff1-9cc9-ed032e43d098-kube-api-access-br48l\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:24.693285 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.693212 4522 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f18303fb-6d27-4ff1-9cc9-ed032e43d098-tls-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:24.693285 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.693221 4522 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dda5ac4-30dd-432e-81eb-cb6d39800628-dshm\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:24.693285 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.693230 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrkfp\" (UniqueName: \"kubernetes.io/projected/8dda5ac4-30dd-432e-81eb-cb6d39800628-kube-api-access-vrkfp\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:24.693285 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.693242 4522 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dda5ac4-30dd-432e-81eb-cb6d39800628-tls-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:24.693771 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.693746 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/043dfb0d-b8be-47b0-bd85-5842628a50f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-82243b-nrh8z\" (UID: \"043dfb0d-b8be-47b0-bd85-5842628a50f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-82243b-nrh8z" Oct 30 17:36:24.695443 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.695425 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/043dfb0d-b8be-47b0-bd85-5842628a50f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-82243b-nrh8z\" (UID: \"043dfb0d-b8be-47b0-bd85-5842628a50f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-82243b-nrh8z" Oct 30 17:36:24.718944 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.718918 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsxqr\" (UniqueName: \"kubernetes.io/projected/043dfb0d-b8be-47b0-bd85-5842628a50f9-kube-api-access-hsxqr\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-82243b-nrh8z\" (UID: \"043dfb0d-b8be-47b0-bd85-5842628a50f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-82243b-nrh8z" Oct 30 17:36:24.778923 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.778888 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-82243b-nrh8z" Oct 30 17:36:24.825134 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.825102 4522 generic.go:356] "Generic (PLEG): container finished" podID="a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7" containerID="f8e750719d86c8219dd9e97846d47e2a982963c63adda49e8d9b17a09e3df519" exitCode=0 Oct 30 17:36:24.825339 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.825275 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc" event={"ID":"a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7","Type":"ContainerDied","Data":"f8e750719d86c8219dd9e97846d47e2a982963c63adda49e8d9b17a09e3df519"} Oct 30 17:36:24.826725 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.826702 4522 generic.go:356] "Generic (PLEG): container finished" podID="f18303fb-6d27-4ff1-9cc9-ed032e43d098" containerID="fe4e8732b1fc8a2bc0dddc27106aa9615f6269fa006fec9fcea562011fa00f1c" exitCode=0 Oct 30 17:36:24.826922 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.826907 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-04248e/distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq" Oct 30 17:36:24.827962 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.827686 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-04248e/distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq" event={"ID":"f18303fb-6d27-4ff1-9cc9-ed032e43d098","Type":"ContainerDied","Data":"fe4e8732b1fc8a2bc0dddc27106aa9615f6269fa006fec9fcea562011fa00f1c"} Oct 30 17:36:24.827962 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.827726 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-04248e/distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq" event={"ID":"f18303fb-6d27-4ff1-9cc9-ed032e43d098","Type":"ContainerDied","Data":"e000f0143fe2a1164f158a948e00aadbcb465d982540c55cebdeda53b6717584"} Oct 30 17:36:24.827962 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.827749 4522 scope.go:122] "RemoveContainer" containerID="fe4e8732b1fc8a2bc0dddc27106aa9615f6269fa006fec9fcea562011fa00f1c" Oct 30 17:36:24.835775 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.835747 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b4f406-7cw59" event={"ID":"450fab43-5d99-4c34-97e6-59a6d64de032","Type":"ContainerDied","Data":"9f8ed7845712e7b6fea28fa7d0ec645be0a591d81e96eaf81b3b16bfe8ce23f4"} Oct 30 17:36:24.835886 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.835797 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f8ed7845712e7b6fea28fa7d0ec645be0a591d81e96eaf81b3b16bfe8ce23f4" Oct 30 17:36:24.835886 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.835881 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b4f406-7cw59" Oct 30 17:36:24.848664 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.843147 4522 generic.go:356] "Generic (PLEG): container finished" podID="8dda5ac4-30dd-432e-81eb-cb6d39800628" containerID="26b7d27f48006df06e7014bee645a72ebe5ba485469fb4b64fe85d7fe30c94d9" exitCode=0 Oct 30 17:36:24.848664 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.843239 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-04248e/distinct-w1-04248e-kserve-5859d49965-kgq46" event={"ID":"8dda5ac4-30dd-432e-81eb-cb6d39800628","Type":"ContainerDied","Data":"26b7d27f48006df06e7014bee645a72ebe5ba485469fb4b64fe85d7fe30c94d9"} Oct 30 17:36:24.848664 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.843268 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-04248e/distinct-w1-04248e-kserve-5859d49965-kgq46" event={"ID":"8dda5ac4-30dd-432e-81eb-cb6d39800628","Type":"ContainerDied","Data":"6230ec148b5348c7196d8a846d8baaec77975bb33548d680f33794ec680846ec"} Oct 30 17:36:24.848664 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.843366 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-04248e/distinct-w1-04248e-kserve-5859d49965-kgq46" Oct 30 17:36:24.861143 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.861119 4522 scope.go:122] "RemoveContainer" containerID="fe4e8732b1fc8a2bc0dddc27106aa9615f6269fa006fec9fcea562011fa00f1c" Oct 30 17:36:24.862491 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:36:24.862160 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe4e8732b1fc8a2bc0dddc27106aa9615f6269fa006fec9fcea562011fa00f1c\": container with ID starting with fe4e8732b1fc8a2bc0dddc27106aa9615f6269fa006fec9fcea562011fa00f1c not found: ID does not exist" containerID="fe4e8732b1fc8a2bc0dddc27106aa9615f6269fa006fec9fcea562011fa00f1c" Oct 30 17:36:24.862491 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.862203 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe4e8732b1fc8a2bc0dddc27106aa9615f6269fa006fec9fcea562011fa00f1c"} err="failed to get container status \"fe4e8732b1fc8a2bc0dddc27106aa9615f6269fa006fec9fcea562011fa00f1c\": rpc error: code = NotFound desc = could not find container \"fe4e8732b1fc8a2bc0dddc27106aa9615f6269fa006fec9fcea562011fa00f1c\": container with ID starting with fe4e8732b1fc8a2bc0dddc27106aa9615f6269fa006fec9fcea562011fa00f1c not found: ID does not exist" Oct 30 17:36:24.862491 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.862227 4522 scope.go:122] "RemoveContainer" containerID="26b7d27f48006df06e7014bee645a72ebe5ba485469fb4b64fe85d7fe30c94d9" Oct 30 17:36:24.881511 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.881246 4522 scope.go:122] "RemoveContainer" containerID="26b7d27f48006df06e7014bee645a72ebe5ba485469fb4b64fe85d7fe30c94d9" Oct 30 17:36:24.882641 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:36:24.882251 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26b7d27f48006df06e7014bee645a72ebe5ba485469fb4b64fe85d7fe30c94d9\": container with ID starting with 26b7d27f48006df06e7014bee645a72ebe5ba485469fb4b64fe85d7fe30c94d9 not found: ID does not exist" containerID="26b7d27f48006df06e7014bee645a72ebe5ba485469fb4b64fe85d7fe30c94d9" Oct 30 17:36:24.882757 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.882292 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26b7d27f48006df06e7014bee645a72ebe5ba485469fb4b64fe85d7fe30c94d9"} err="failed to get container status \"26b7d27f48006df06e7014bee645a72ebe5ba485469fb4b64fe85d7fe30c94d9\": rpc error: code = NotFound desc = could not find container \"26b7d27f48006df06e7014bee645a72ebe5ba485469fb4b64fe85d7fe30c94d9\": container with ID starting with 26b7d27f48006df06e7014bee645a72ebe5ba485469fb4b64fe85d7fe30c94d9 not found: ID does not exist" Oct 30 17:36:24.904189 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.902721 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc" Oct 30 17:36:24.909394 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.909363 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-04248e/distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq"] Oct 30 17:36:24.915224 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.915196 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-04248e/distinct-2-w1-04248e-kserve-9bfb7c898-rrzbq"] Oct 30 17:36:24.963596 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.963568 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-04248e/distinct-w1-04248e-kserve-5859d49965-kgq46"] Oct 30 17:36:24.970063 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:24.970039 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-04248e/distinct-w1-04248e-kserve-5859d49965-kgq46"] Oct 30 17:36:25.097673 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:25.097642 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7-kube-api-access-q827x\" (UniqueName: \"kubernetes.io/projected/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7-kube-api-access-q827x\") pod \"a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7\" (UID: \"a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7\") " Oct 30 17:36:25.097877 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:25.097706 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7-istiod-ca-cert\") pod \"a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7\" (UID: \"a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7\") " Oct 30 17:36:25.097877 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:25.097758 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7-istio-envoy\") pod \"a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7\" (UID: \"a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7\") " Oct 30 17:36:25.097877 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:25.097783 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7-workload-socket\") pod \"a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7\" (UID: \"a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7\") " Oct 30 17:36:25.097877 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:25.097829 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7-credential-socket\") pod \"a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7\" (UID: \"a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7\") " Oct 30 17:36:25.097877 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:25.097874 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7-istio-data\") pod \"a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7\" (UID: \"a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7\") " Oct 30 17:36:25.098183 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:25.098083 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7-istio-podinfo\") pod \"a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7\" (UID: \"a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7\") " Oct 30 17:36:25.098183 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:25.098156 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7-credential-socket" pod "a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7" (UID: "a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:25.098290 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:25.098163 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7-workload-socket" pod "a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7" (UID: "a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:25.098290 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:25.098213 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7-istio-data" pod "a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7" (UID: "a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:25.098290 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:25.098239 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7-istiod-ca-cert" pod "a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7" (UID: "a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:36:25.098478 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:25.098353 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7-istio-token\" (UniqueName: \"kubernetes.io/projected/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7-istio-token\") pod \"a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7\" (UID: \"a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7\") " Oct 30 17:36:25.098478 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:25.098403 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7-workload-certs\") pod \"a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7\" (UID: \"a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7\") " Oct 30 17:36:25.098725 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:25.098702 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7-workload-certs" pod "a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7" (UID: "a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:25.098945 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:25.098930 4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7-istio-data\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:25.099021 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:25.098947 4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7-workload-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:25.099021 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:25.098957 4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7-istiod-ca-cert\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:25.099021 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:25.098969 4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7-workload-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:25.099021 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:25.098983 4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7-credential-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:25.100149 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:25.100122 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7-kube-api-access-q827x" pod "a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7" (UID: "a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7"). InnerVolumeSpecName "kube-api-access-q827x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:36:25.100256 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:25.100157 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7-istio-podinfo" pod "a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7" (UID: "a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:36:25.100472 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:25.100451 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7-istio-envoy" pod "a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7" (UID: "a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:25.100630 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:25.100615 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7-istio-token" pod "a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7" (UID: "a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:36:25.110629 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:36:25.110593 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod043dfb0d_b8be_47b0_bd85_5842628a50f9.slice/crio-d5f4549fa58cc85e6edcfd8ba5e7d82102ef84f8db43e690def8d76f254f4651 WatchSource:0}: Error finding container d5f4549fa58cc85e6edcfd8ba5e7d82102ef84f8db43e690def8d76f254f4651: Status 404 returned error can't find the container with id d5f4549fa58cc85e6edcfd8ba5e7d82102ef84f8db43e690def8d76f254f4651 Oct 30 17:36:25.110940 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:25.110886 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-82243b-nrh8z"] Oct 30 17:36:25.138622 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:25.138598 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-04248e-6dfd587cc9-n8brs" podUID="80f8edc4-4e9f-435e-8518-e5f53d262030" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:36:25.158111 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:25.158087 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-b4f406/distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb"] Oct 30 17:36:25.166126 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:25.166103 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-b4f406/distinct-w2-b4f406-kserve-5db4cc6969-7n846"] Oct 30 17:36:25.200596 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:25.200560 4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7-istio-podinfo\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:25.200596 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:25.200594 4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7-istio-token\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:25.200840 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:25.200606 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q827x\" (UniqueName: \"kubernetes.io/projected/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7-kube-api-access-q827x\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:25.200840 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:25.200620 4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7-istio-envoy\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:25.851740 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:25.851709 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc" Oct 30 17:36:25.851951 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:25.851741 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc" event={"ID":"a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7","Type":"ContainerDied","Data":"5ccd40da17c1de11e972b06de2e3ad98335e25bf69def48367ba50ab37779357"} Oct 30 17:36:25.851951 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:25.851805 4522 scope.go:122] "RemoveContainer" containerID="f8e750719d86c8219dd9e97846d47e2a982963c63adda49e8d9b17a09e3df519" Oct 30 17:36:25.854961 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:25.854928 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-82243b-nrh8z" event={"ID":"043dfb0d-b8be-47b0-bd85-5842628a50f9","Type":"ContainerStarted","Data":"1a711f43fd4fb0e883aa107f076c4164cd79a410b4888159eb8c32820d067121"} Oct 30 17:36:25.855132 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:25.854990 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-82243b-nrh8z" event={"ID":"043dfb0d-b8be-47b0-bd85-5842628a50f9","Type":"ContainerStarted","Data":"d5f4549fa58cc85e6edcfd8ba5e7d82102ef84f8db43e690def8d76f254f4651"} Oct 30 17:36:25.915409 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:25.915350 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-82243b-nrh8z" podStartSLOduration=1.9153348970000001 podStartE2EDuration="1.915334897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:36:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:36:25.912475166 +0000 UTC m=+2959.913230084" watchObservedRunningTime="2026-10-30 17:36:25.915334897 +0000 UTC m=+2959.916089815" Oct 30 17:36:25.954105 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:25.954080 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc"] Oct 30 17:36:25.962654 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:25.962632 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-mxkhc"] Oct 30 17:36:26.155597 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:26.155519 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dda5ac4-30dd-432e-81eb-cb6d39800628" path="/var/lib/kubelet/pods/8dda5ac4-30dd-432e-81eb-cb6d39800628/volumes" Oct 30 17:36:26.156010 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:26.155996 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7" path="/var/lib/kubelet/pods/a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7/volumes" Oct 30 17:36:26.156417 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:26.156404 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f18303fb-6d27-4ff1-9cc9-ed032e43d098" path="/var/lib/kubelet/pods/f18303fb-6d27-4ff1-9cc9-ed032e43d098/volumes" Oct 30 17:36:26.609751 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:26.609705 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-78c9c2-777cdc8975-7qp6m" podUID="90cb278c-f9d6-4948-9843-0330b688521a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:36:26.609941 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:26.609718 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-78c9c2-6b9c4447c4-p5z99" podUID="63ee8702-b07d-4e00-81d4-bffadca6608d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:36:26.866378 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:26.866257 4522 generic.go:356] "Generic (PLEG): container finished" podID="043dfb0d-b8be-47b0-bd85-5842628a50f9" containerID="1a711f43fd4fb0e883aa107f076c4164cd79a410b4888159eb8c32820d067121" exitCode=0 Oct 30 17:36:26.866378 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:26.866345 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-82243b-nrh8z" event={"ID":"043dfb0d-b8be-47b0-bd85-5842628a50f9","Type":"ContainerDied","Data":"1a711f43fd4fb0e883aa107f076c4164cd79a410b4888159eb8c32820d067121"} Oct 30 17:36:27.263831 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:27.263789 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-db8417-nskzf"] Oct 30 17:36:27.270456 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:27.270427 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-db8417-nskzf"] Oct 30 17:36:28.030127 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:28.030101 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-82243b-nrh8z" Oct 30 17:36:28.037809 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:28.037780 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/043dfb0d-b8be-47b0-bd85-5842628a50f9-kube-api-access-hsxqr\" (UniqueName: \"kubernetes.io/projected/043dfb0d-b8be-47b0-bd85-5842628a50f9-kube-api-access-hsxqr\") pod \"043dfb0d-b8be-47b0-bd85-5842628a50f9\" (UID: \"043dfb0d-b8be-47b0-bd85-5842628a50f9\") " Oct 30 17:36:28.038026 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:28.038010 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/043dfb0d-b8be-47b0-bd85-5842628a50f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/043dfb0d-b8be-47b0-bd85-5842628a50f9-maas-api-openshift-ca\") pod \"043dfb0d-b8be-47b0-bd85-5842628a50f9\" (UID: \"043dfb0d-b8be-47b0-bd85-5842628a50f9\") " Oct 30 17:36:28.038095 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:28.038079 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/043dfb0d-b8be-47b0-bd85-5842628a50f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/043dfb0d-b8be-47b0-bd85-5842628a50f9-maas-api-xks-ca\") pod \"043dfb0d-b8be-47b0-bd85-5842628a50f9\" (UID: \"043dfb0d-b8be-47b0-bd85-5842628a50f9\") " Oct 30 17:36:28.038407 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:28.038384 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/043dfb0d-b8be-47b0-bd85-5842628a50f9-maas-api-openshift-ca" pod "043dfb0d-b8be-47b0-bd85-5842628a50f9" (UID: "043dfb0d-b8be-47b0-bd85-5842628a50f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:36:28.038570 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:28.038548 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/043dfb0d-b8be-47b0-bd85-5842628a50f9-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:28.039821 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:28.039800 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/043dfb0d-b8be-47b0-bd85-5842628a50f9-kube-api-access-hsxqr" pod "043dfb0d-b8be-47b0-bd85-5842628a50f9" (UID: "043dfb0d-b8be-47b0-bd85-5842628a50f9"). InnerVolumeSpecName "kube-api-access-hsxqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:36:28.040135 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:28.040120 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/043dfb0d-b8be-47b0-bd85-5842628a50f9-maas-api-xks-ca" pod "043dfb0d-b8be-47b0-bd85-5842628a50f9" (UID: "043dfb0d-b8be-47b0-bd85-5842628a50f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:36:28.139928 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:28.139845 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsxqr\" (UniqueName: \"kubernetes.io/projected/043dfb0d-b8be-47b0-bd85-5842628a50f9-kube-api-access-hsxqr\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:28.139928 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:28.139876 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/043dfb0d-b8be-47b0-bd85-5842628a50f9-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:28.145001 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:28.144975 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="847b63e9-4a1c-4c2f-85f2-1295c356fb57" path="/var/lib/kubelet/pods/847b63e9-4a1c-4c2f-85f2-1295c356fb57/volumes" Oct 30 17:36:28.676930 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:28.676896 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cb7367c-m8xfm"] Oct 30 17:36:28.677596 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:28.677579 4522 cpu_manager.go:414] "containerMap: removing container" podUID="043dfb0d-b8be-47b0-bd85-5842628a50f9" containerName="revoke-keys" Oct 30 17:36:28.677673 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:28.677597 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="043dfb0d-b8be-47b0-bd85-5842628a50f9" containerName="revoke-keys" Oct 30 17:36:28.677673 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:28.677619 4522 cpu_manager.go:414] "containerMap: removing container" podUID="a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7" containerName="istio-proxy" Oct 30 17:36:28.677673 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:28.677624 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7" containerName="istio-proxy" Oct 30 17:36:28.677847 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:28.677727 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="043dfb0d-b8be-47b0-bd85-5842628a50f9" containerName="revoke-keys" Oct 30 17:36:28.677847 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:28.677739 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2f5cba4-9318-4bd1-ba06-f77f17ca1ae7" containerName="istio-proxy" Oct 30 17:36:28.681626 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:28.681603 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cb7367c-m8xfm" Oct 30 17:36:28.719338 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:28.719278 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cb7367c-m8xfm"] Oct 30 17:36:28.745890 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:28.745849 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j4gq\" (UniqueName: \"kubernetes.io/projected/444cd488-a651-413f-afc9-907bf73a1879-kube-api-access-9j4gq\") pod \"maas-api-revoke-sub-1cb7367c-m8xfm\" (UID: \"444cd488-a651-413f-afc9-907bf73a1879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cb7367c-m8xfm" Oct 30 17:36:28.746083 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:28.746067 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/444cd488-a651-413f-afc9-907bf73a1879-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cb7367c-m8xfm\" (UID: \"444cd488-a651-413f-afc9-907bf73a1879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cb7367c-m8xfm" Oct 30 17:36:28.746176 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:28.746116 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/444cd488-a651-413f-afc9-907bf73a1879-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cb7367c-m8xfm\" (UID: \"444cd488-a651-413f-afc9-907bf73a1879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cb7367c-m8xfm" Oct 30 17:36:28.847609 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:28.847567 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j4gq\" (UniqueName: \"kubernetes.io/projected/444cd488-a651-413f-afc9-907bf73a1879-kube-api-access-9j4gq\") pod \"maas-api-revoke-sub-1cb7367c-m8xfm\" (UID: \"444cd488-a651-413f-afc9-907bf73a1879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cb7367c-m8xfm" Oct 30 17:36:28.847810 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:28.847763 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/444cd488-a651-413f-afc9-907bf73a1879-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cb7367c-m8xfm\" (UID: \"444cd488-a651-413f-afc9-907bf73a1879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cb7367c-m8xfm" Oct 30 17:36:28.847903 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:28.847812 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/444cd488-a651-413f-afc9-907bf73a1879-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cb7367c-m8xfm\" (UID: \"444cd488-a651-413f-afc9-907bf73a1879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cb7367c-m8xfm" Oct 30 17:36:28.848441 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:28.848421 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/444cd488-a651-413f-afc9-907bf73a1879-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cb7367c-m8xfm\" (UID: \"444cd488-a651-413f-afc9-907bf73a1879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cb7367c-m8xfm" Oct 30 17:36:28.850169 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:28.850140 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/444cd488-a651-413f-afc9-907bf73a1879-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cb7367c-m8xfm\" (UID: \"444cd488-a651-413f-afc9-907bf73a1879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cb7367c-m8xfm" Oct 30 17:36:28.873063 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:28.873040 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j4gq\" (UniqueName: \"kubernetes.io/projected/444cd488-a651-413f-afc9-907bf73a1879-kube-api-access-9j4gq\") pod \"maas-api-revoke-sub-1cb7367c-m8xfm\" (UID: \"444cd488-a651-413f-afc9-907bf73a1879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cb7367c-m8xfm" Oct 30 17:36:28.883079 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:28.883058 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-82243b-nrh8z" Oct 30 17:36:28.883218 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:28.883102 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-82243b-nrh8z" event={"ID":"043dfb0d-b8be-47b0-bd85-5842628a50f9","Type":"ContainerDied","Data":"d5f4549fa58cc85e6edcfd8ba5e7d82102ef84f8db43e690def8d76f254f4651"} Oct 30 17:36:28.883218 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:28.883160 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5f4549fa58cc85e6edcfd8ba5e7d82102ef84f8db43e690def8d76f254f4651" Oct 30 17:36:28.997901 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:28.997865 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cb7367c-m8xfm" Oct 30 17:36:29.228659 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.228635 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv" Oct 30 17:36:29.253330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.253279 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba5dbb57-7fdc-4509-a870-6c95778f650b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba5dbb57-7fdc-4509-a870-6c95778f650b-istio-data\") pod \"ba5dbb57-7fdc-4509-a870-6c95778f650b\" (UID: \"ba5dbb57-7fdc-4509-a870-6c95778f650b\") " Oct 30 17:36:29.253481 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.253400 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba5dbb57-7fdc-4509-a870-6c95778f650b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba5dbb57-7fdc-4509-a870-6c95778f650b-istiod-ca-cert\") pod \"ba5dbb57-7fdc-4509-a870-6c95778f650b\" (UID: \"ba5dbb57-7fdc-4509-a870-6c95778f650b\") " Oct 30 17:36:29.253481 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.253445 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba5dbb57-7fdc-4509-a870-6c95778f650b-istio-token\" (UniqueName: \"kubernetes.io/projected/ba5dbb57-7fdc-4509-a870-6c95778f650b-istio-token\") pod \"ba5dbb57-7fdc-4509-a870-6c95778f650b\" (UID: \"ba5dbb57-7fdc-4509-a870-6c95778f650b\") " Oct 30 17:36:29.253585 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.253503 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba5dbb57-7fdc-4509-a870-6c95778f650b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba5dbb57-7fdc-4509-a870-6c95778f650b-workload-certs\") pod \"ba5dbb57-7fdc-4509-a870-6c95778f650b\" (UID: \"ba5dbb57-7fdc-4509-a870-6c95778f650b\") " Oct 30 17:36:29.253585 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.253548 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba5dbb57-7fdc-4509-a870-6c95778f650b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba5dbb57-7fdc-4509-a870-6c95778f650b-istio-podinfo\") pod \"ba5dbb57-7fdc-4509-a870-6c95778f650b\" (UID: \"ba5dbb57-7fdc-4509-a870-6c95778f650b\") " Oct 30 17:36:29.253585 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.253543 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba5dbb57-7fdc-4509-a870-6c95778f650b-istio-data" pod "ba5dbb57-7fdc-4509-a870-6c95778f650b" (UID: "ba5dbb57-7fdc-4509-a870-6c95778f650b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:29.253753 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.253587 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba5dbb57-7fdc-4509-a870-6c95778f650b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5dbb57-7fdc-4509-a870-6c95778f650b-workload-socket\") pod \"ba5dbb57-7fdc-4509-a870-6c95778f650b\" (UID: \"ba5dbb57-7fdc-4509-a870-6c95778f650b\") " Oct 30 17:36:29.253753 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.253614 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba5dbb57-7fdc-4509-a870-6c95778f650b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba5dbb57-7fdc-4509-a870-6c95778f650b-istio-envoy\") pod \"ba5dbb57-7fdc-4509-a870-6c95778f650b\" (UID: \"ba5dbb57-7fdc-4509-a870-6c95778f650b\") " Oct 30 17:36:29.253753 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.253691 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba5dbb57-7fdc-4509-a870-6c95778f650b-kube-api-access-7kbjq\" (UniqueName: \"kubernetes.io/projected/ba5dbb57-7fdc-4509-a870-6c95778f650b-kube-api-access-7kbjq\") pod \"ba5dbb57-7fdc-4509-a870-6c95778f650b\" (UID: \"ba5dbb57-7fdc-4509-a870-6c95778f650b\") " Oct 30 17:36:29.253753 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.253727 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba5dbb57-7fdc-4509-a870-6c95778f650b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5dbb57-7fdc-4509-a870-6c95778f650b-credential-socket\") pod \"ba5dbb57-7fdc-4509-a870-6c95778f650b\" (UID: \"ba5dbb57-7fdc-4509-a870-6c95778f650b\") " Oct 30 17:36:29.253996 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.253830 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba5dbb57-7fdc-4509-a870-6c95778f650b-workload-certs" pod "ba5dbb57-7fdc-4509-a870-6c95778f650b" (UID: "ba5dbb57-7fdc-4509-a870-6c95778f650b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:29.253996 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.253880 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba5dbb57-7fdc-4509-a870-6c95778f650b-workload-socket" pod "ba5dbb57-7fdc-4509-a870-6c95778f650b" (UID: "ba5dbb57-7fdc-4509-a870-6c95778f650b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:29.253996 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.253894 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba5dbb57-7fdc-4509-a870-6c95778f650b-istiod-ca-cert" pod "ba5dbb57-7fdc-4509-a870-6c95778f650b" (UID: "ba5dbb57-7fdc-4509-a870-6c95778f650b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:36:29.254192 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.254166 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba5dbb57-7fdc-4509-a870-6c95778f650b-credential-socket" pod "ba5dbb57-7fdc-4509-a870-6c95778f650b" (UID: "ba5dbb57-7fdc-4509-a870-6c95778f650b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:29.254331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.254290 4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba5dbb57-7fdc-4509-a870-6c95778f650b-istio-data\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:29.254389 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.254341 4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba5dbb57-7fdc-4509-a870-6c95778f650b-istiod-ca-cert\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:29.254389 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.254356 4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba5dbb57-7fdc-4509-a870-6c95778f650b-workload-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:29.254389 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.254369 4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5dbb57-7fdc-4509-a870-6c95778f650b-workload-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:29.254389 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.254383 4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5dbb57-7fdc-4509-a870-6c95778f650b-credential-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:29.255805 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.255776 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba5dbb57-7fdc-4509-a870-6c95778f650b-istio-podinfo" pod "ba5dbb57-7fdc-4509-a870-6c95778f650b" (UID: "ba5dbb57-7fdc-4509-a870-6c95778f650b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:36:29.255916 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.255901 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba5dbb57-7fdc-4509-a870-6c95778f650b-kube-api-access-7kbjq" pod "ba5dbb57-7fdc-4509-a870-6c95778f650b" (UID: "ba5dbb57-7fdc-4509-a870-6c95778f650b"). InnerVolumeSpecName "kube-api-access-7kbjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:36:29.256180 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.256160 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba5dbb57-7fdc-4509-a870-6c95778f650b-istio-envoy" pod "ba5dbb57-7fdc-4509-a870-6c95778f650b" (UID: "ba5dbb57-7fdc-4509-a870-6c95778f650b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:29.256235 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.256180 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba5dbb57-7fdc-4509-a870-6c95778f650b-istio-token" pod "ba5dbb57-7fdc-4509-a870-6c95778f650b" (UID: "ba5dbb57-7fdc-4509-a870-6c95778f650b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:36:29.325582 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:36:29.325551 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod444cd488_a651_413f_afc9_907bf73a1879.slice/crio-65052c24983560a503db8cee440c23c61f31b6b0e051f7fc6309b4c3578d606f WatchSource:0}: Error finding container 65052c24983560a503db8cee440c23c61f31b6b0e051f7fc6309b4c3578d606f: Status 404 returned error can't find the container with id 65052c24983560a503db8cee440c23c61f31b6b0e051f7fc6309b4c3578d606f Oct 30 17:36:29.327697 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.327672 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cb7367c-m8xfm"] Oct 30 17:36:29.355319 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.355284 4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba5dbb57-7fdc-4509-a870-6c95778f650b-istio-token\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:29.355429 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.355325 4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba5dbb57-7fdc-4509-a870-6c95778f650b-istio-podinfo\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:29.355429 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.355339 4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba5dbb57-7fdc-4509-a870-6c95778f650b-istio-envoy\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:29.355429 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.355352 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kbjq\" (UniqueName: \"kubernetes.io/projected/ba5dbb57-7fdc-4509-a870-6c95778f650b-kube-api-access-7kbjq\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:29.640422 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.640393 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-78c9c2-6b9c4447c4-p5z99" Oct 30 17:36:29.659802 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.659774 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63ee8702-b07d-4e00-81d4-bffadca6608d-kube-api-access-qbzk5\" (UniqueName: \"kubernetes.io/projected/63ee8702-b07d-4e00-81d4-bffadca6608d-kube-api-access-qbzk5\") pod \"63ee8702-b07d-4e00-81d4-bffadca6608d\" (UID: \"63ee8702-b07d-4e00-81d4-bffadca6608d\") " Oct 30 17:36:29.659970 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.659941 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63ee8702-b07d-4e00-81d4-bffadca6608d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63ee8702-b07d-4e00-81d4-bffadca6608d-plugins-config-volume\") pod \"63ee8702-b07d-4e00-81d4-bffadca6608d\" (UID: \"63ee8702-b07d-4e00-81d4-bffadca6608d\") " Oct 30 17:36:29.660323 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.660286 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63ee8702-b07d-4e00-81d4-bffadca6608d-plugins-config-volume" pod "63ee8702-b07d-4e00-81d4-bffadca6608d" (UID: "63ee8702-b07d-4e00-81d4-bffadca6608d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:36:29.662849 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.662829 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63ee8702-b07d-4e00-81d4-bffadca6608d-kube-api-access-qbzk5" pod "63ee8702-b07d-4e00-81d4-bffadca6608d" (UID: "63ee8702-b07d-4e00-81d4-bffadca6608d"). InnerVolumeSpecName "kube-api-access-qbzk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:36:29.700506 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.700470 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-78c9c2-777cdc8975-7qp6m" Oct 30 17:36:29.761046 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.760952 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90cb278c-f9d6-4948-9843-0330b688521a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90cb278c-f9d6-4948-9843-0330b688521a-plugins-config-volume\") pod \"90cb278c-f9d6-4948-9843-0330b688521a\" (UID: \"90cb278c-f9d6-4948-9843-0330b688521a\") " Oct 30 17:36:29.761232 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.761093 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90cb278c-f9d6-4948-9843-0330b688521a-kube-api-access-6kpsq\" (UniqueName: \"kubernetes.io/projected/90cb278c-f9d6-4948-9843-0330b688521a-kube-api-access-6kpsq\") pod \"90cb278c-f9d6-4948-9843-0330b688521a\" (UID: \"90cb278c-f9d6-4948-9843-0330b688521a\") " Oct 30 17:36:29.761377 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.761347 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90cb278c-f9d6-4948-9843-0330b688521a-plugins-config-volume" pod "90cb278c-f9d6-4948-9843-0330b688521a" (UID: "90cb278c-f9d6-4948-9843-0330b688521a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:36:29.761583 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.761565 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63ee8702-b07d-4e00-81d4-bffadca6608d-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:29.761686 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.761586 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90cb278c-f9d6-4948-9843-0330b688521a-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:29.761686 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.761598 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbzk5\" (UniqueName: \"kubernetes.io/projected/63ee8702-b07d-4e00-81d4-bffadca6608d-kube-api-access-qbzk5\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:29.763339 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.763279 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90cb278c-f9d6-4948-9843-0330b688521a-kube-api-access-6kpsq" pod "90cb278c-f9d6-4948-9843-0330b688521a" (UID: "90cb278c-f9d6-4948-9843-0330b688521a"). InnerVolumeSpecName "kube-api-access-6kpsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:36:29.774089 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.774065 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8f92ef-85f6b78466-9mbs9" Oct 30 17:36:29.774216 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.774138 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f92ef-5bd8786b7-52rzg" Oct 30 17:36:29.863276 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.863238 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kpsq\" (UniqueName: \"kubernetes.io/projected/90cb278c-f9d6-4948-9843-0330b688521a-kube-api-access-6kpsq\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:29.893087 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.893048 4522 generic.go:356] "Generic (PLEG): container finished" podID="90cb278c-f9d6-4948-9843-0330b688521a" containerID="2cd513a1c4dbe4618ea797e265570f43d628568fd7b64a7816c5368b256d12a2" exitCode=0 Oct 30 17:36:29.893286 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.893113 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-78c9c2-777cdc8975-7qp6m" Oct 30 17:36:29.893286 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.893125 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-78c9c2-777cdc8975-7qp6m" event={"ID":"90cb278c-f9d6-4948-9843-0330b688521a","Type":"ContainerDied","Data":"2cd513a1c4dbe4618ea797e265570f43d628568fd7b64a7816c5368b256d12a2"} Oct 30 17:36:29.893286 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.893188 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-78c9c2-777cdc8975-7qp6m" event={"ID":"90cb278c-f9d6-4948-9843-0330b688521a","Type":"ContainerDied","Data":"b075e29656963aa8384f69b24c261b1ec8ede8bc885d27535d126472e7421c38"} Oct 30 17:36:29.893286 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.893219 4522 scope.go:122] "RemoveContainer" containerID="2cd513a1c4dbe4618ea797e265570f43d628568fd7b64a7816c5368b256d12a2" Oct 30 17:36:29.896140 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.895773 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cb7367c-m8xfm" event={"ID":"444cd488-a651-413f-afc9-907bf73a1879","Type":"ContainerStarted","Data":"257e7f139feea5598bad5b77784789b47f9b1789737cd077734e6e422e36e960"} Oct 30 17:36:29.896140 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.895857 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cb7367c-m8xfm" event={"ID":"444cd488-a651-413f-afc9-907bf73a1879","Type":"ContainerStarted","Data":"65052c24983560a503db8cee440c23c61f31b6b0e051f7fc6309b4c3578d606f"} Oct 30 17:36:29.897942 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.897918 4522 generic.go:356] "Generic (PLEG): container finished" podID="63ee8702-b07d-4e00-81d4-bffadca6608d" containerID="fe602c2e074fc9de9203620088c8a6c36983991f0872c2359eba459749d5edc3" exitCode=0 Oct 30 17:36:29.898196 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.897973 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-78c9c2-6b9c4447c4-p5z99" Oct 30 17:36:29.898196 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.897990 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-78c9c2-6b9c4447c4-p5z99" event={"ID":"63ee8702-b07d-4e00-81d4-bffadca6608d","Type":"ContainerDied","Data":"fe602c2e074fc9de9203620088c8a6c36983991f0872c2359eba459749d5edc3"} Oct 30 17:36:29.898196 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.898054 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-78c9c2-6b9c4447c4-p5z99" event={"ID":"63ee8702-b07d-4e00-81d4-bffadca6608d","Type":"ContainerDied","Data":"802f14b19138c3bd2017ebf0fac71c5db3831bcc3dfeaa0d2155e9797da4225d"} Oct 30 17:36:29.901346 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.899802 4522 generic.go:356] "Generic (PLEG): container finished" podID="ba5dbb57-7fdc-4509-a870-6c95778f650b" containerID="dccf5ac7282e6ac94a97385dc7ef4b1ce5c239819a40858e0350c069f8e9356b" exitCode=0 Oct 30 17:36:29.901346 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.899844 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv" Oct 30 17:36:29.901346 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.899987 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv" event={"ID":"ba5dbb57-7fdc-4509-a870-6c95778f650b","Type":"ContainerDied","Data":"dccf5ac7282e6ac94a97385dc7ef4b1ce5c239819a40858e0350c069f8e9356b"} Oct 30 17:36:29.901346 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.900026 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv" event={"ID":"ba5dbb57-7fdc-4509-a870-6c95778f650b","Type":"ContainerDied","Data":"eb51f51118247985525b5901c88dfd64c02d8402e5777759bfa3ac1155f71fcf"} Oct 30 17:36:29.912058 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.909557 4522 scope.go:122] "RemoveContainer" containerID="2cd513a1c4dbe4618ea797e265570f43d628568fd7b64a7816c5368b256d12a2" Oct 30 17:36:29.912058 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:36:29.910092 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cd513a1c4dbe4618ea797e265570f43d628568fd7b64a7816c5368b256d12a2\": container with ID starting with 2cd513a1c4dbe4618ea797e265570f43d628568fd7b64a7816c5368b256d12a2 not found: ID does not exist" containerID="2cd513a1c4dbe4618ea797e265570f43d628568fd7b64a7816c5368b256d12a2" Oct 30 17:36:29.912058 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.910131 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cd513a1c4dbe4618ea797e265570f43d628568fd7b64a7816c5368b256d12a2"} err="failed to get container status \"2cd513a1c4dbe4618ea797e265570f43d628568fd7b64a7816c5368b256d12a2\": rpc error: code = NotFound desc = could not find container \"2cd513a1c4dbe4618ea797e265570f43d628568fd7b64a7816c5368b256d12a2\": container with ID starting with 2cd513a1c4dbe4618ea797e265570f43d628568fd7b64a7816c5368b256d12a2 not found: ID does not exist" Oct 30 17:36:29.912058 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.910169 4522 scope.go:122] "RemoveContainer" containerID="fe602c2e074fc9de9203620088c8a6c36983991f0872c2359eba459749d5edc3" Oct 30 17:36:29.923370 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.923350 4522 scope.go:122] "RemoveContainer" containerID="fe602c2e074fc9de9203620088c8a6c36983991f0872c2359eba459749d5edc3" Oct 30 17:36:29.923667 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:36:29.923646 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe602c2e074fc9de9203620088c8a6c36983991f0872c2359eba459749d5edc3\": container with ID starting with fe602c2e074fc9de9203620088c8a6c36983991f0872c2359eba459749d5edc3 not found: ID does not exist" containerID="fe602c2e074fc9de9203620088c8a6c36983991f0872c2359eba459749d5edc3" Oct 30 17:36:29.923831 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.923673 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe602c2e074fc9de9203620088c8a6c36983991f0872c2359eba459749d5edc3"} err="failed to get container status \"fe602c2e074fc9de9203620088c8a6c36983991f0872c2359eba459749d5edc3\": rpc error: code = NotFound desc = could not find container \"fe602c2e074fc9de9203620088c8a6c36983991f0872c2359eba459749d5edc3\": container with ID starting with fe602c2e074fc9de9203620088c8a6c36983991f0872c2359eba459749d5edc3 not found: ID does not exist" Oct 30 17:36:29.923831 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.923691 4522 scope.go:122] "RemoveContainer" containerID="dccf5ac7282e6ac94a97385dc7ef4b1ce5c239819a40858e0350c069f8e9356b" Oct 30 17:36:29.935403 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.935383 4522 scope.go:122] "RemoveContainer" containerID="dccf5ac7282e6ac94a97385dc7ef4b1ce5c239819a40858e0350c069f8e9356b" Oct 30 17:36:29.935726 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:36:29.935701 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dccf5ac7282e6ac94a97385dc7ef4b1ce5c239819a40858e0350c069f8e9356b\": container with ID starting with dccf5ac7282e6ac94a97385dc7ef4b1ce5c239819a40858e0350c069f8e9356b not found: ID does not exist" containerID="dccf5ac7282e6ac94a97385dc7ef4b1ce5c239819a40858e0350c069f8e9356b" Oct 30 17:36:29.935823 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.935735 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dccf5ac7282e6ac94a97385dc7ef4b1ce5c239819a40858e0350c069f8e9356b"} err="failed to get container status \"dccf5ac7282e6ac94a97385dc7ef4b1ce5c239819a40858e0350c069f8e9356b\": rpc error: code = NotFound desc = could not find container \"dccf5ac7282e6ac94a97385dc7ef4b1ce5c239819a40858e0350c069f8e9356b\": container with ID starting with dccf5ac7282e6ac94a97385dc7ef4b1ce5c239819a40858e0350c069f8e9356b not found: ID does not exist" Oct 30 17:36:29.953065 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.953007 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-78c9c2-777cdc8975-7qp6m"] Oct 30 17:36:29.956665 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:29.956643 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-78c9c2-777cdc8975-7qp6m"] Oct 30 17:36:30.000431 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:30.000374 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cb7367c-m8xfm" podStartSLOduration=2.00035728 podStartE2EDuration="2.00035728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:36:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:36:29.995990675 +0000 UTC m=+2963.996745577" watchObservedRunningTime="2026-10-30 17:36:30.00035728 +0000 UTC m=+2964.001112197" Oct 30 17:36:30.031773 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:30.031648 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a884078c-88sll"] Oct 30 17:36:30.035808 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:30.035784 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a884078c-88sll"] Oct 30 17:36:30.067984 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:30.067950 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-78c9c2-6b9c4447c4-p5z99"] Oct 30 17:36:30.072550 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:30.072526 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-78c9c2-6b9c4447c4-p5z99"] Oct 30 17:36:30.109059 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:30.109028 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv"] Oct 30 17:36:30.115595 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:30.115564 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-x9rsv"] Oct 30 17:36:30.145466 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:30.145435 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63ee8702-b07d-4e00-81d4-bffadca6608d" path="/var/lib/kubelet/pods/63ee8702-b07d-4e00-81d4-bffadca6608d/volumes" Oct 30 17:36:30.145951 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:30.145930 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90cb278c-f9d6-4948-9843-0330b688521a" path="/var/lib/kubelet/pods/90cb278c-f9d6-4948-9843-0330b688521a/volumes" Oct 30 17:36:30.146465 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:30.146446 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba5dbb57-7fdc-4509-a870-6c95778f650b" path="/var/lib/kubelet/pods/ba5dbb57-7fdc-4509-a870-6c95778f650b/volumes" Oct 30 17:36:30.147209 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:30.147189 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c01c53da-a670-4554-9aac-8948be829acf" path="/var/lib/kubelet/pods/c01c53da-a670-4554-9aac-8948be829acf/volumes" Oct 30 17:36:30.483027 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:30.482995 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-b4f406-6d66f86b4b-bzcnr"] Oct 30 17:36:30.521188 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:30.521148 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-b4f406-786855ccd7-ns4j8"] Oct 30 17:36:30.910011 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:30.909923 4522 generic.go:356] "Generic (PLEG): container finished" podID="444cd488-a651-413f-afc9-907bf73a1879" containerID="257e7f139feea5598bad5b77784789b47f9b1789737cd077734e6e422e36e960" exitCode=0 Oct 30 17:36:30.910453 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:30.910014 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cb7367c-m8xfm" event={"ID":"444cd488-a651-413f-afc9-907bf73a1879","Type":"ContainerDied","Data":"257e7f139feea5598bad5b77784789b47f9b1789737cd077734e6e422e36e960"} Oct 30 17:36:31.485096 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:31.485071 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-db8417-bfdb94d4d-frgtn" Oct 30 17:36:31.517620 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:31.517594 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-db8417-74d9d84dc9-ddws5" Oct 30 17:36:31.582235 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:31.582147 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47fa8756-130e-48fc-bcf9-008ab31f95cf-kube-api-access-mt5qx\" (UniqueName: \"kubernetes.io/projected/47fa8756-130e-48fc-bcf9-008ab31f95cf-kube-api-access-mt5qx\") pod \"47fa8756-130e-48fc-bcf9-008ab31f95cf\" (UID: \"47fa8756-130e-48fc-bcf9-008ab31f95cf\") " Oct 30 17:36:31.582235 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:31.582221 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6e99dad-79f2-4b63-87f9-1c761080cada-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6e99dad-79f2-4b63-87f9-1c761080cada-plugins-config-volume\") pod \"e6e99dad-79f2-4b63-87f9-1c761080cada\" (UID: \"e6e99dad-79f2-4b63-87f9-1c761080cada\") " Oct 30 17:36:31.582521 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:31.582399 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47fa8756-130e-48fc-bcf9-008ab31f95cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47fa8756-130e-48fc-bcf9-008ab31f95cf-plugins-config-volume\") pod \"47fa8756-130e-48fc-bcf9-008ab31f95cf\" (UID: \"47fa8756-130e-48fc-bcf9-008ab31f95cf\") " Oct 30 17:36:31.582521 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:31.582511 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6e99dad-79f2-4b63-87f9-1c761080cada-kube-api-access-mbtjr\" (UniqueName: \"kubernetes.io/projected/e6e99dad-79f2-4b63-87f9-1c761080cada-kube-api-access-mbtjr\") pod \"e6e99dad-79f2-4b63-87f9-1c761080cada\" (UID: \"e6e99dad-79f2-4b63-87f9-1c761080cada\") " Oct 30 17:36:31.583220 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:31.583163 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47fa8756-130e-48fc-bcf9-008ab31f95cf-plugins-config-volume" pod "47fa8756-130e-48fc-bcf9-008ab31f95cf" (UID: "47fa8756-130e-48fc-bcf9-008ab31f95cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:36:31.583552 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:31.583517 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6e99dad-79f2-4b63-87f9-1c761080cada-plugins-config-volume" pod "e6e99dad-79f2-4b63-87f9-1c761080cada" (UID: "e6e99dad-79f2-4b63-87f9-1c761080cada"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:36:31.586335 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:31.586299 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47fa8756-130e-48fc-bcf9-008ab31f95cf-kube-api-access-mt5qx" pod "47fa8756-130e-48fc-bcf9-008ab31f95cf" (UID: "47fa8756-130e-48fc-bcf9-008ab31f95cf"). InnerVolumeSpecName "kube-api-access-mt5qx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:36:31.586427 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:31.586368 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6e99dad-79f2-4b63-87f9-1c761080cada-kube-api-access-mbtjr" pod "e6e99dad-79f2-4b63-87f9-1c761080cada" (UID: "e6e99dad-79f2-4b63-87f9-1c761080cada"). InnerVolumeSpecName "kube-api-access-mbtjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:36:31.684101 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:31.684070 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbtjr\" (UniqueName: \"kubernetes.io/projected/e6e99dad-79f2-4b63-87f9-1c761080cada-kube-api-access-mbtjr\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:31.684101 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:31.684102 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mt5qx\" (UniqueName: \"kubernetes.io/projected/47fa8756-130e-48fc-bcf9-008ab31f95cf-kube-api-access-mt5qx\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:31.684340 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:31.684120 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6e99dad-79f2-4b63-87f9-1c761080cada-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:31.684340 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:31.684135 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47fa8756-130e-48fc-bcf9-008ab31f95cf-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:31.924002 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:31.923905 4522 generic.go:356] "Generic (PLEG): container finished" podID="47fa8756-130e-48fc-bcf9-008ab31f95cf" containerID="eaad7e2d0590e24706f9d1084db25ae7b57647af207968ccc4f4d99782071560" exitCode=0 Oct 30 17:36:31.924002 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:31.923942 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-db8417-bfdb94d4d-frgtn" event={"ID":"47fa8756-130e-48fc-bcf9-008ab31f95cf","Type":"ContainerDied","Data":"eaad7e2d0590e24706f9d1084db25ae7b57647af207968ccc4f4d99782071560"} Oct 30 17:36:31.924002 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:31.923999 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-db8417-bfdb94d4d-frgtn" Oct 30 17:36:31.924682 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:31.924015 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-db8417-bfdb94d4d-frgtn" event={"ID":"47fa8756-130e-48fc-bcf9-008ab31f95cf","Type":"ContainerDied","Data":"ef46a8683eb286b22516ae5662928b65954917ca4eca9c1f6c97795b2593ddef"} Oct 30 17:36:31.924682 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:31.924047 4522 scope.go:122] "RemoveContainer" containerID="eaad7e2d0590e24706f9d1084db25ae7b57647af207968ccc4f4d99782071560" Oct 30 17:36:31.931129 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:31.931099 4522 generic.go:356] "Generic (PLEG): container finished" podID="e6e99dad-79f2-4b63-87f9-1c761080cada" containerID="a0df871312b2d488be253a98a94a9998a29369b3f162066331873f625d033746" exitCode=0 Oct 30 17:36:31.931316 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:31.931160 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-db8417-74d9d84dc9-ddws5" Oct 30 17:36:31.931316 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:31.931186 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-db8417-74d9d84dc9-ddws5" event={"ID":"e6e99dad-79f2-4b63-87f9-1c761080cada","Type":"ContainerDied","Data":"a0df871312b2d488be253a98a94a9998a29369b3f162066331873f625d033746"} Oct 30 17:36:31.931407 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:31.931327 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-db8417-74d9d84dc9-ddws5" event={"ID":"e6e99dad-79f2-4b63-87f9-1c761080cada","Type":"ContainerDied","Data":"f108427ed3d5d904c10fe00da1547b90b263beb2c54e3dc45b8a969e86bc7ff7"} Oct 30 17:36:31.938243 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:31.938139 4522 scope.go:122] "RemoveContainer" containerID="eaad7e2d0590e24706f9d1084db25ae7b57647af207968ccc4f4d99782071560" Oct 30 17:36:31.938549 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:36:31.938523 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaad7e2d0590e24706f9d1084db25ae7b57647af207968ccc4f4d99782071560\": container with ID starting with eaad7e2d0590e24706f9d1084db25ae7b57647af207968ccc4f4d99782071560 not found: ID does not exist" containerID="eaad7e2d0590e24706f9d1084db25ae7b57647af207968ccc4f4d99782071560" Oct 30 17:36:31.938635 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:31.938593 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaad7e2d0590e24706f9d1084db25ae7b57647af207968ccc4f4d99782071560"} err="failed to get container status \"eaad7e2d0590e24706f9d1084db25ae7b57647af207968ccc4f4d99782071560\": rpc error: code = NotFound desc = could not find container \"eaad7e2d0590e24706f9d1084db25ae7b57647af207968ccc4f4d99782071560\": container with ID starting with eaad7e2d0590e24706f9d1084db25ae7b57647af207968ccc4f4d99782071560 not found: ID does not exist" Oct 30 17:36:31.938635 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:31.938620 4522 scope.go:122] "RemoveContainer" containerID="a0df871312b2d488be253a98a94a9998a29369b3f162066331873f625d033746" Oct 30 17:36:31.951067 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:31.951035 4522 scope.go:122] "RemoveContainer" containerID="a0df871312b2d488be253a98a94a9998a29369b3f162066331873f625d033746" Oct 30 17:36:31.951467 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:36:31.951446 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0df871312b2d488be253a98a94a9998a29369b3f162066331873f625d033746\": container with ID starting with a0df871312b2d488be253a98a94a9998a29369b3f162066331873f625d033746 not found: ID does not exist" containerID="a0df871312b2d488be253a98a94a9998a29369b3f162066331873f625d033746" Oct 30 17:36:31.951561 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:31.951479 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0df871312b2d488be253a98a94a9998a29369b3f162066331873f625d033746"} err="failed to get container status \"a0df871312b2d488be253a98a94a9998a29369b3f162066331873f625d033746\": rpc error: code = NotFound desc = could not find container \"a0df871312b2d488be253a98a94a9998a29369b3f162066331873f625d033746\": container with ID starting with a0df871312b2d488be253a98a94a9998a29369b3f162066331873f625d033746 not found: ID does not exist" Oct 30 17:36:31.999919 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:31.999888 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-db8417-bfdb94d4d-frgtn"] Oct 30 17:36:32.007270 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:32.007096 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-db8417-bfdb94d4d-frgtn"] Oct 30 17:36:32.047231 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:32.047197 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-db8417-74d9d84dc9-ddws5"] Oct 30 17:36:32.051106 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:32.051072 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-db8417-74d9d84dc9-ddws5"] Oct 30 17:36:32.099839 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:32.099810 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cb7367c-m8xfm" Oct 30 17:36:32.147208 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:32.147081 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47fa8756-130e-48fc-bcf9-008ab31f95cf" path="/var/lib/kubelet/pods/47fa8756-130e-48fc-bcf9-008ab31f95cf/volumes" Oct 30 17:36:32.147726 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:32.147702 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6e99dad-79f2-4b63-87f9-1c761080cada" path="/var/lib/kubelet/pods/e6e99dad-79f2-4b63-87f9-1c761080cada/volumes" Oct 30 17:36:32.190653 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:32.190579 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/444cd488-a651-413f-afc9-907bf73a1879-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/444cd488-a651-413f-afc9-907bf73a1879-maas-api-xks-ca\") pod \"444cd488-a651-413f-afc9-907bf73a1879\" (UID: \"444cd488-a651-413f-afc9-907bf73a1879\") " Oct 30 17:36:32.190653 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:32.190617 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/444cd488-a651-413f-afc9-907bf73a1879-kube-api-access-9j4gq\" (UniqueName: \"kubernetes.io/projected/444cd488-a651-413f-afc9-907bf73a1879-kube-api-access-9j4gq\") pod \"444cd488-a651-413f-afc9-907bf73a1879\" (UID: \"444cd488-a651-413f-afc9-907bf73a1879\") " Oct 30 17:36:32.190843 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:32.190764 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/444cd488-a651-413f-afc9-907bf73a1879-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/444cd488-a651-413f-afc9-907bf73a1879-maas-api-openshift-ca\") pod \"444cd488-a651-413f-afc9-907bf73a1879\" (UID: \"444cd488-a651-413f-afc9-907bf73a1879\") " Oct 30 17:36:32.191134 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:32.191110 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/444cd488-a651-413f-afc9-907bf73a1879-maas-api-openshift-ca" pod "444cd488-a651-413f-afc9-907bf73a1879" (UID: "444cd488-a651-413f-afc9-907bf73a1879"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:36:32.192743 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:32.192724 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/444cd488-a651-413f-afc9-907bf73a1879-maas-api-xks-ca" pod "444cd488-a651-413f-afc9-907bf73a1879" (UID: "444cd488-a651-413f-afc9-907bf73a1879"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:36:32.192823 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:32.192769 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/444cd488-a651-413f-afc9-907bf73a1879-kube-api-access-9j4gq" pod "444cd488-a651-413f-afc9-907bf73a1879" (UID: "444cd488-a651-413f-afc9-907bf73a1879"). InnerVolumeSpecName "kube-api-access-9j4gq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:36:32.291914 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:32.291886 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/444cd488-a651-413f-afc9-907bf73a1879-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:32.291914 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:32.291913 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/444cd488-a651-413f-afc9-907bf73a1879-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:32.292107 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:32.291929 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9j4gq\" (UniqueName: \"kubernetes.io/projected/444cd488-a651-413f-afc9-907bf73a1879-kube-api-access-9j4gq\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:32.948331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:32.948291 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cb7367c-m8xfm" Oct 30 17:36:32.948733 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:32.948344 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cb7367c-m8xfm" event={"ID":"444cd488-a651-413f-afc9-907bf73a1879","Type":"ContainerDied","Data":"65052c24983560a503db8cee440c23c61f31b6b0e051f7fc6309b4c3578d606f"} Oct 30 17:36:32.948733 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:32.948383 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65052c24983560a503db8cee440c23c61f31b6b0e051f7fc6309b4c3578d606f" Oct 30 17:36:34.621642 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:34.621587 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-82243b-7f4f9546b6-mg8hr"] Oct 30 17:36:34.693707 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:34.693671 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-82243b-6794bbb67-58ssq"] Oct 30 17:36:34.880122 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:34.880042 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq"] Oct 30 17:36:34.880366 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:34.880324 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq" podUID="06d654a9-d7c3-4b45-8bcc-abfad4c49970" containerName="istio-proxy" containerID="cri-o://be25d5e29c60fb7663216b9c320b6eb4d8d0e1b02e79d41efaf3a572b1552977" gracePeriod=30 Oct 30 17:36:35.138655 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:35.138573 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-04248e-6dfd587cc9-n8brs" podUID="80f8edc4-4e9f-435e-8518-e5f53d262030" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:36:35.483558 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:35.483513 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-b4f406-6d66f86b4b-bzcnr" podUID="519e4a59-cb60-4a1a-949b-56018ea6b89c" containerName="payload-processing" containerID="cri-o://d1e6ff4af5ae577ebeb135947213d7c776786a28925de3b5ce16349c3ea38b23" gracePeriod=25 Oct 30 17:36:35.522042 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:35.521985 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b4f406-786855ccd7-ns4j8" podUID="246bcd4d-bddf-466e-bf5b-9f18b81fa5fd" containerName="payload-pre-processing" containerID="cri-o://b64457195612210f19529d5ade53be8ee48a9839e49dc6d0e83feb62c511511d" gracePeriod=25 Oct 30 17:36:35.594694 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:35.594658 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b80d83-dhchx"] Oct 30 17:36:35.595270 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:35.595253 4522 cpu_manager.go:414] "containerMap: removing container" podUID="444cd488-a651-413f-afc9-907bf73a1879" containerName="revoke-keys" Oct 30 17:36:35.595341 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:35.595273 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="444cd488-a651-413f-afc9-907bf73a1879" containerName="revoke-keys" Oct 30 17:36:35.595341 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:35.595292 4522 cpu_manager.go:414] "containerMap: removing container" podUID="90cb278c-f9d6-4948-9843-0330b688521a" containerName="payload-pre-processing" Oct 30 17:36:35.595341 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:35.595298 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90cb278c-f9d6-4948-9843-0330b688521a" containerName="payload-pre-processing" Oct 30 17:36:35.595341 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:35.595320 4522 cpu_manager.go:414] "containerMap: removing container" podUID="e6e99dad-79f2-4b63-87f9-1c761080cada" containerName="payload-pre-processing" Oct 30 17:36:35.595341 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:35.595325 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6e99dad-79f2-4b63-87f9-1c761080cada" containerName="payload-pre-processing" Oct 30 17:36:35.595341 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:35.595342 4522 cpu_manager.go:414] "containerMap: removing container" podUID="47fa8756-130e-48fc-bcf9-008ab31f95cf" containerName="payload-processing" Oct 30 17:36:35.595540 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:35.595347 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47fa8756-130e-48fc-bcf9-008ab31f95cf" containerName="payload-processing" Oct 30 17:36:35.595540 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:35.595366 4522 cpu_manager.go:414] "containerMap: removing container" podUID="63ee8702-b07d-4e00-81d4-bffadca6608d" containerName="payload-processing" Oct 30 17:36:35.595540 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:35.595372 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63ee8702-b07d-4e00-81d4-bffadca6608d" containerName="payload-processing" Oct 30 17:36:35.595540 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:35.595382 4522 cpu_manager.go:414] "containerMap: removing container" podUID="ba5dbb57-7fdc-4509-a870-6c95778f650b" containerName="istio-proxy" Oct 30 17:36:35.595540 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:35.595386 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba5dbb57-7fdc-4509-a870-6c95778f650b" containerName="istio-proxy" Oct 30 17:36:35.595540 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:35.595469 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="63ee8702-b07d-4e00-81d4-bffadca6608d" containerName="payload-processing" Oct 30 17:36:35.595540 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:35.595480 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba5dbb57-7fdc-4509-a870-6c95778f650b" containerName="istio-proxy" Oct 30 17:36:35.595540 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:35.595488 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="444cd488-a651-413f-afc9-907bf73a1879" containerName="revoke-keys" Oct 30 17:36:35.595540 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:35.595497 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="90cb278c-f9d6-4948-9843-0330b688521a" containerName="payload-pre-processing" Oct 30 17:36:35.595540 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:35.595504 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6e99dad-79f2-4b63-87f9-1c761080cada" containerName="payload-pre-processing" Oct 30 17:36:35.595540 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:35.595511 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="47fa8756-130e-48fc-bcf9-008ab31f95cf" containerName="payload-processing" Oct 30 17:36:35.600236 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:35.600216 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b80d83-dhchx" Oct 30 17:36:35.620426 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:35.620397 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:36:35.620615 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:35.620598 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:36:35.622291 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:35.622267 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b80d83-dhchx"] Oct 30 17:36:35.733345 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:35.733294 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4b5d762-29b6-4913-99ff-4851f5c47bf6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-b80d83-dhchx\" (UID: \"a4b5d762-29b6-4913-99ff-4851f5c47bf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b80d83-dhchx" Oct 30 17:36:35.733539 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:35.733404 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4b5d762-29b6-4913-99ff-4851f5c47bf6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-b80d83-dhchx\" (UID: \"a4b5d762-29b6-4913-99ff-4851f5c47bf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b80d83-dhchx" Oct 30 17:36:35.733539 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:35.733476 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5ldc\" (UniqueName: \"kubernetes.io/projected/a4b5d762-29b6-4913-99ff-4851f5c47bf6-kube-api-access-p5ldc\") pod \"maas-api-revoke-keys-e2e-worker-w3-b80d83-dhchx\" (UID: \"a4b5d762-29b6-4913-99ff-4851f5c47bf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b80d83-dhchx" Oct 30 17:36:35.835272 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:35.835190 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4b5d762-29b6-4913-99ff-4851f5c47bf6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-b80d83-dhchx\" (UID: \"a4b5d762-29b6-4913-99ff-4851f5c47bf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b80d83-dhchx" Oct 30 17:36:35.835272 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:35.835247 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4b5d762-29b6-4913-99ff-4851f5c47bf6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-b80d83-dhchx\" (UID: \"a4b5d762-29b6-4913-99ff-4851f5c47bf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b80d83-dhchx" Oct 30 17:36:35.835485 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:35.835324 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5ldc\" (UniqueName: \"kubernetes.io/projected/a4b5d762-29b6-4913-99ff-4851f5c47bf6-kube-api-access-p5ldc\") pod \"maas-api-revoke-keys-e2e-worker-w3-b80d83-dhchx\" (UID: \"a4b5d762-29b6-4913-99ff-4851f5c47bf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b80d83-dhchx" Oct 30 17:36:35.836116 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:35.836091 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4b5d762-29b6-4913-99ff-4851f5c47bf6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-b80d83-dhchx\" (UID: \"a4b5d762-29b6-4913-99ff-4851f5c47bf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b80d83-dhchx" Oct 30 17:36:35.837721 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:35.837701 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4b5d762-29b6-4913-99ff-4851f5c47bf6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-b80d83-dhchx\" (UID: \"a4b5d762-29b6-4913-99ff-4851f5c47bf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b80d83-dhchx" Oct 30 17:36:35.865830 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:35.865792 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5ldc\" (UniqueName: \"kubernetes.io/projected/a4b5d762-29b6-4913-99ff-4851f5c47bf6-kube-api-access-p5ldc\") pod \"maas-api-revoke-keys-e2e-worker-w3-b80d83-dhchx\" (UID: \"a4b5d762-29b6-4913-99ff-4851f5c47bf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b80d83-dhchx" Oct 30 17:36:35.913567 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:35.913531 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b80d83-dhchx" Oct 30 17:36:36.248276 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:36:36.248248 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4b5d762_29b6_4913_99ff_4851f5c47bf6.slice/crio-65d6707b052549b22598f75aadf4d0e75a7138cf8ef64e47ed48d490273306a8 WatchSource:0}: Error finding container 65d6707b052549b22598f75aadf4d0e75a7138cf8ef64e47ed48d490273306a8: Status 404 returned error can't find the container with id 65d6707b052549b22598f75aadf4d0e75a7138cf8ef64e47ed48d490273306a8 Oct 30 17:36:36.248820 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:36.248798 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b80d83-dhchx"] Oct 30 17:36:37.039839 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:37.039803 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b80d83-dhchx" event={"ID":"a4b5d762-29b6-4913-99ff-4851f5c47bf6","Type":"ContainerStarted","Data":"ce6c5675593741b541f1e00d0109a5bd0fda93dcb0d3fbf9ddecf639ab057b35"} Oct 30 17:36:37.040269 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:37.039860 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b80d83-dhchx" event={"ID":"a4b5d762-29b6-4913-99ff-4851f5c47bf6","Type":"ContainerStarted","Data":"65d6707b052549b22598f75aadf4d0e75a7138cf8ef64e47ed48d490273306a8"} Oct 30 17:36:37.091552 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:37.091488 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b80d83-dhchx" podStartSLOduration=2.091471504 podStartE2EDuration="2.091471504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:36:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:36:37.088406555 +0000 UTC m=+2971.089161482" watchObservedRunningTime="2026-10-30 17:36:37.091471504 +0000 UTC m=+2971.092226405" Oct 30 17:36:38.048981 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:38.048941 4522 generic.go:356] "Generic (PLEG): container finished" podID="a4b5d762-29b6-4913-99ff-4851f5c47bf6" containerID="ce6c5675593741b541f1e00d0109a5bd0fda93dcb0d3fbf9ddecf639ab057b35" exitCode=0 Oct 30 17:36:38.049495 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:38.049029 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b80d83-dhchx" event={"ID":"a4b5d762-29b6-4913-99ff-4851f5c47bf6","Type":"ContainerDied","Data":"ce6c5675593741b541f1e00d0109a5bd0fda93dcb0d3fbf9ddecf639ab057b35"} Oct 30 17:36:38.092409 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:38.092374 4522 patch_prober.go:29] interesting pod/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:36:38.092572 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:38.092437 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq" podUID="06d654a9-d7c3-4b45-8bcc-abfad4c49970" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:36:39.211579 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:39.211551 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b80d83-dhchx" Oct 30 17:36:39.280086 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:39.280049 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4b5d762-29b6-4913-99ff-4851f5c47bf6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4b5d762-29b6-4913-99ff-4851f5c47bf6-maas-api-xks-ca\") pod \"a4b5d762-29b6-4913-99ff-4851f5c47bf6\" (UID: \"a4b5d762-29b6-4913-99ff-4851f5c47bf6\") " Oct 30 17:36:39.280677 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:39.280651 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4b5d762-29b6-4913-99ff-4851f5c47bf6-kube-api-access-p5ldc\" (UniqueName: \"kubernetes.io/projected/a4b5d762-29b6-4913-99ff-4851f5c47bf6-kube-api-access-p5ldc\") pod \"a4b5d762-29b6-4913-99ff-4851f5c47bf6\" (UID: \"a4b5d762-29b6-4913-99ff-4851f5c47bf6\") " Oct 30 17:36:39.281243 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:39.281221 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4b5d762-29b6-4913-99ff-4851f5c47bf6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4b5d762-29b6-4913-99ff-4851f5c47bf6-maas-api-openshift-ca\") pod \"a4b5d762-29b6-4913-99ff-4851f5c47bf6\" (UID: \"a4b5d762-29b6-4913-99ff-4851f5c47bf6\") " Oct 30 17:36:39.283555 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:39.283529 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4b5d762-29b6-4913-99ff-4851f5c47bf6-maas-api-openshift-ca" pod "a4b5d762-29b6-4913-99ff-4851f5c47bf6" (UID: "a4b5d762-29b6-4913-99ff-4851f5c47bf6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:36:39.289417 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:39.289387 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4b5d762-29b6-4913-99ff-4851f5c47bf6-kube-api-access-p5ldc" pod "a4b5d762-29b6-4913-99ff-4851f5c47bf6" (UID: "a4b5d762-29b6-4913-99ff-4851f5c47bf6"). InnerVolumeSpecName "kube-api-access-p5ldc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:36:39.289778 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:39.289754 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4b5d762-29b6-4913-99ff-4851f5c47bf6-maas-api-xks-ca" pod "a4b5d762-29b6-4913-99ff-4851f5c47bf6" (UID: "a4b5d762-29b6-4913-99ff-4851f5c47bf6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:36:39.383096 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:39.383003 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4b5d762-29b6-4913-99ff-4851f5c47bf6-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:39.383096 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:39.383044 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5ldc\" (UniqueName: \"kubernetes.io/projected/a4b5d762-29b6-4913-99ff-4851f5c47bf6-kube-api-access-p5ldc\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:39.383096 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:39.383056 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4b5d762-29b6-4913-99ff-4851f5c47bf6-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:39.622839 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:39.622795 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-82243b-7f4f9546b6-mg8hr" podUID="5333c134-0e1a-4096-a6d0-6d54f2373e28" containerName="payload-processing" containerID="cri-o://41112eefe8cb88efafdc400f68940bf161a6d97bb4d068ea8c498160480ed412" gracePeriod=25 Oct 30 17:36:39.694955 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:39.694910 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-82243b-6794bbb67-58ssq" podUID="5bdabfbc-c999-479c-a44e-d4949bd3c73d" containerName="payload-pre-processing" containerID="cri-o://1f71acee2445475d0860e021cf9070236c0d3fed188889c790e20f62d175672d" gracePeriod=25 Oct 30 17:36:39.955364 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:39.955338 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-04248e-6dfd587cc9-n8brs" Oct 30 17:36:40.065603 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.065569 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b80d83-dhchx" event={"ID":"a4b5d762-29b6-4913-99ff-4851f5c47bf6","Type":"ContainerDied","Data":"65d6707b052549b22598f75aadf4d0e75a7138cf8ef64e47ed48d490273306a8"} Oct 30 17:36:40.065699 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.065633 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65d6707b052549b22598f75aadf4d0e75a7138cf8ef64e47ed48d490273306a8" Oct 30 17:36:40.065753 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.065699 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b80d83-dhchx" Oct 30 17:36:40.070542 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.070515 4522 generic.go:356] "Generic (PLEG): container finished" podID="06d654a9-d7c3-4b45-8bcc-abfad4c49970" containerID="be25d5e29c60fb7663216b9c320b6eb4d8d0e1b02e79d41efaf3a572b1552977" exitCode=0 Oct 30 17:36:40.070733 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.070707 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq" event={"ID":"06d654a9-d7c3-4b45-8bcc-abfad4c49970","Type":"ContainerDied","Data":"be25d5e29c60fb7663216b9c320b6eb4d8d0e1b02e79d41efaf3a572b1552977"} Oct 30 17:36:40.073146 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.072558 4522 generic.go:356] "Generic (PLEG): container finished" podID="80f8edc4-4e9f-435e-8518-e5f53d262030" containerID="ee44950c663dabf7409634f43dd79fe985e5ec6c00f06f95918d4042c0d9a567" exitCode=0 Oct 30 17:36:40.073146 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.072606 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-04248e-6dfd587cc9-n8brs" event={"ID":"80f8edc4-4e9f-435e-8518-e5f53d262030","Type":"ContainerDied","Data":"ee44950c663dabf7409634f43dd79fe985e5ec6c00f06f95918d4042c0d9a567"} Oct 30 17:36:40.073146 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.072644 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-04248e-6dfd587cc9-n8brs" event={"ID":"80f8edc4-4e9f-435e-8518-e5f53d262030","Type":"ContainerDied","Data":"19683125e7d40ca44571c833a5144ab874ecf23ac4eeb525c1f994117c00288d"} Oct 30 17:36:40.073146 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.072673 4522 scope.go:122] "RemoveContainer" containerID="ee44950c663dabf7409634f43dd79fe985e5ec6c00f06f95918d4042c0d9a567" Oct 30 17:36:40.073146 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.072812 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-04248e-6dfd587cc9-n8brs" Oct 30 17:36:40.087709 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.087679 4522 scope.go:122] "RemoveContainer" containerID="ee44950c663dabf7409634f43dd79fe985e5ec6c00f06f95918d4042c0d9a567" Oct 30 17:36:40.088690 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:36:40.088659 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee44950c663dabf7409634f43dd79fe985e5ec6c00f06f95918d4042c0d9a567\": container with ID starting with ee44950c663dabf7409634f43dd79fe985e5ec6c00f06f95918d4042c0d9a567 not found: ID does not exist" containerID="ee44950c663dabf7409634f43dd79fe985e5ec6c00f06f95918d4042c0d9a567" Oct 30 17:36:40.088822 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.088698 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee44950c663dabf7409634f43dd79fe985e5ec6c00f06f95918d4042c0d9a567"} err="failed to get container status \"ee44950c663dabf7409634f43dd79fe985e5ec6c00f06f95918d4042c0d9a567\": rpc error: code = NotFound desc = could not find container \"ee44950c663dabf7409634f43dd79fe985e5ec6c00f06f95918d4042c0d9a567\": container with ID starting with ee44950c663dabf7409634f43dd79fe985e5ec6c00f06f95918d4042c0d9a567 not found: ID does not exist" Oct 30 17:36:40.092357 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.092333 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80f8edc4-4e9f-435e-8518-e5f53d262030-kube-api-access-hzldc\" (UniqueName: \"kubernetes.io/projected/80f8edc4-4e9f-435e-8518-e5f53d262030-kube-api-access-hzldc\") pod \"80f8edc4-4e9f-435e-8518-e5f53d262030\" (UID: \"80f8edc4-4e9f-435e-8518-e5f53d262030\") " Oct 30 17:36:40.092466 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.092385 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80f8edc4-4e9f-435e-8518-e5f53d262030-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80f8edc4-4e9f-435e-8518-e5f53d262030-plugins-config-volume\") pod \"80f8edc4-4e9f-435e-8518-e5f53d262030\" (UID: \"80f8edc4-4e9f-435e-8518-e5f53d262030\") " Oct 30 17:36:40.092765 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.092734 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80f8edc4-4e9f-435e-8518-e5f53d262030-plugins-config-volume" pod "80f8edc4-4e9f-435e-8518-e5f53d262030" (UID: "80f8edc4-4e9f-435e-8518-e5f53d262030"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:36:40.094719 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.094696 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80f8edc4-4e9f-435e-8518-e5f53d262030-kube-api-access-hzldc" pod "80f8edc4-4e9f-435e-8518-e5f53d262030" (UID: "80f8edc4-4e9f-435e-8518-e5f53d262030"). InnerVolumeSpecName "kube-api-access-hzldc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:36:40.153785 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.153754 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq" Oct 30 17:36:40.180856 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.180806 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-b4f406/distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb" podUID="d86fded1-2e04-4d73-bb22-ea0d9a61b961" containerName="main" containerID="cri-o://d0c833e26aeb6db1b92a905fa9433641e73eee8d02a39465c5ba761a3f45401c" gracePeriod=45 Oct 30 17:36:40.188119 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.188081 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-b4f406/distinct-w2-b4f406-kserve-5db4cc6969-7n846" podUID="b8810dbc-b769-46a4-90b9-72a68592bc1a" containerName="main" containerID="cri-o://c7b60475ed24d290180c64665644a3145ff12c2e5b4f2a6b0a432df5aa22bb6e" gracePeriod=45 Oct 30 17:36:40.194387 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.194363 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzldc\" (UniqueName: \"kubernetes.io/projected/80f8edc4-4e9f-435e-8518-e5f53d262030-kube-api-access-hzldc\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:40.194387 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.194388 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80f8edc4-4e9f-435e-8518-e5f53d262030-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:40.259814 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.259778 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-054acd-5fccd756d7-v75ht"] Oct 30 17:36:40.260719 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.260695 4522 cpu_manager.go:414] "containerMap: removing container" podUID="a4b5d762-29b6-4913-99ff-4851f5c47bf6" containerName="revoke-keys" Oct 30 17:36:40.260794 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.260722 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4b5d762-29b6-4913-99ff-4851f5c47bf6" containerName="revoke-keys" Oct 30 17:36:40.260794 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.260741 4522 cpu_manager.go:414] "containerMap: removing container" podUID="06d654a9-d7c3-4b45-8bcc-abfad4c49970" containerName="istio-proxy" Oct 30 17:36:40.260794 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.260750 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06d654a9-d7c3-4b45-8bcc-abfad4c49970" containerName="istio-proxy" Oct 30 17:36:40.260794 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.260788 4522 cpu_manager.go:414] "containerMap: removing container" podUID="80f8edc4-4e9f-435e-8518-e5f53d262030" containerName="payload-pre-processing" Oct 30 17:36:40.260923 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.260797 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80f8edc4-4e9f-435e-8518-e5f53d262030" containerName="payload-pre-processing" Oct 30 17:36:40.260972 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.260958 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="06d654a9-d7c3-4b45-8bcc-abfad4c49970" containerName="istio-proxy" Oct 30 17:36:40.261006 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.260985 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="80f8edc4-4e9f-435e-8518-e5f53d262030" containerName="payload-pre-processing" Oct 30 17:36:40.261043 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.261005 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4b5d762-29b6-4913-99ff-4851f5c47bf6" containerName="revoke-keys" Oct 30 17:36:40.265071 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.265047 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-054acd-5fccd756d7-v75ht" Oct 30 17:36:40.272018 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.271974 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/distinct-w2-b4f406-kserve-5db4cc6969-7n846" podUID="b8810dbc-b769-46a4-90b9-72a68592bc1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": dial tcp 10.134.0.114:8000: connect: connection refused" Oct 30 17:36:40.293849 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.293727 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-054acd-5fccd756d7-v75ht"] Oct 30 17:36:40.294213 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.294190 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-054acd\"" Oct 30 17:36:40.294414 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.294198 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-054acd-dockercfg-bkjcb\"" Oct 30 17:36:40.295390 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.295365 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06d654a9-d7c3-4b45-8bcc-abfad4c49970-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06d654a9-d7c3-4b45-8bcc-abfad4c49970-istio-data\") pod \"06d654a9-d7c3-4b45-8bcc-abfad4c49970\" (UID: \"06d654a9-d7c3-4b45-8bcc-abfad4c49970\") " Oct 30 17:36:40.295520 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.295502 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06d654a9-d7c3-4b45-8bcc-abfad4c49970-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06d654a9-d7c3-4b45-8bcc-abfad4c49970-istiod-ca-cert\") pod \"06d654a9-d7c3-4b45-8bcc-abfad4c49970\" (UID: \"06d654a9-d7c3-4b45-8bcc-abfad4c49970\") " Oct 30 17:36:40.295600 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.295545 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06d654a9-d7c3-4b45-8bcc-abfad4c49970-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06d654a9-d7c3-4b45-8bcc-abfad4c49970-credential-socket\") pod \"06d654a9-d7c3-4b45-8bcc-abfad4c49970\" (UID: \"06d654a9-d7c3-4b45-8bcc-abfad4c49970\") " Oct 30 17:36:40.295659 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.295625 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06d654a9-d7c3-4b45-8bcc-abfad4c49970-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06d654a9-d7c3-4b45-8bcc-abfad4c49970-workload-socket\") pod \"06d654a9-d7c3-4b45-8bcc-abfad4c49970\" (UID: \"06d654a9-d7c3-4b45-8bcc-abfad4c49970\") " Oct 30 17:36:40.295722 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.295657 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/06d654a9-d7c3-4b45-8bcc-abfad4c49970-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06d654a9-d7c3-4b45-8bcc-abfad4c49970-istio-podinfo\") pod \"06d654a9-d7c3-4b45-8bcc-abfad4c49970\" (UID: \"06d654a9-d7c3-4b45-8bcc-abfad4c49970\") " Oct 30 17:36:40.295722 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.295696 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06d654a9-d7c3-4b45-8bcc-abfad4c49970-kube-api-access-qwl69\" (UniqueName: \"kubernetes.io/projected/06d654a9-d7c3-4b45-8bcc-abfad4c49970-kube-api-access-qwl69\") pod \"06d654a9-d7c3-4b45-8bcc-abfad4c49970\" (UID: \"06d654a9-d7c3-4b45-8bcc-abfad4c49970\") " Oct 30 17:36:40.295836 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.295754 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06d654a9-d7c3-4b45-8bcc-abfad4c49970-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06d654a9-d7c3-4b45-8bcc-abfad4c49970-workload-certs\") pod \"06d654a9-d7c3-4b45-8bcc-abfad4c49970\" (UID: \"06d654a9-d7c3-4b45-8bcc-abfad4c49970\") " Oct 30 17:36:40.295836 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.295787 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06d654a9-d7c3-4b45-8bcc-abfad4c49970-istio-token\" (UniqueName: \"kubernetes.io/projected/06d654a9-d7c3-4b45-8bcc-abfad4c49970-istio-token\") pod \"06d654a9-d7c3-4b45-8bcc-abfad4c49970\" (UID: \"06d654a9-d7c3-4b45-8bcc-abfad4c49970\") " Oct 30 17:36:40.295836 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.295812 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06d654a9-d7c3-4b45-8bcc-abfad4c49970-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06d654a9-d7c3-4b45-8bcc-abfad4c49970-istio-envoy\") pod \"06d654a9-d7c3-4b45-8bcc-abfad4c49970\" (UID: \"06d654a9-d7c3-4b45-8bcc-abfad4c49970\") " Oct 30 17:36:40.298523 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.298410 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06d654a9-d7c3-4b45-8bcc-abfad4c49970-istio-data" pod "06d654a9-d7c3-4b45-8bcc-abfad4c49970" (UID: "06d654a9-d7c3-4b45-8bcc-abfad4c49970"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:40.299078 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.299055 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06d654a9-d7c3-4b45-8bcc-abfad4c49970-workload-certs" pod "06d654a9-d7c3-4b45-8bcc-abfad4c49970" (UID: "06d654a9-d7c3-4b45-8bcc-abfad4c49970"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:40.299834 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.299792 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06d654a9-d7c3-4b45-8bcc-abfad4c49970-credential-socket" pod "06d654a9-d7c3-4b45-8bcc-abfad4c49970" (UID: "06d654a9-d7c3-4b45-8bcc-abfad4c49970"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:40.300180 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.300156 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06d654a9-d7c3-4b45-8bcc-abfad4c49970-istiod-ca-cert" pod "06d654a9-d7c3-4b45-8bcc-abfad4c49970" (UID: "06d654a9-d7c3-4b45-8bcc-abfad4c49970"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:36:40.300844 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.300799 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06d654a9-d7c3-4b45-8bcc-abfad4c49970-workload-socket" pod "06d654a9-d7c3-4b45-8bcc-abfad4c49970" (UID: "06d654a9-d7c3-4b45-8bcc-abfad4c49970"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:40.301541 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.301509 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06d654a9-d7c3-4b45-8bcc-abfad4c49970-istio-envoy" pod "06d654a9-d7c3-4b45-8bcc-abfad4c49970" (UID: "06d654a9-d7c3-4b45-8bcc-abfad4c49970"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:40.301661 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.301639 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06d654a9-d7c3-4b45-8bcc-abfad4c49970-istio-token" pod "06d654a9-d7c3-4b45-8bcc-abfad4c49970" (UID: "06d654a9-d7c3-4b45-8bcc-abfad4c49970"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:36:40.302019 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.301998 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/06d654a9-d7c3-4b45-8bcc-abfad4c49970-istio-podinfo" pod "06d654a9-d7c3-4b45-8bcc-abfad4c49970" (UID: "06d654a9-d7c3-4b45-8bcc-abfad4c49970"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:36:40.302459 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.302435 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06d654a9-d7c3-4b45-8bcc-abfad4c49970-kube-api-access-qwl69" pod "06d654a9-d7c3-4b45-8bcc-abfad4c49970" (UID: "06d654a9-d7c3-4b45-8bcc-abfad4c49970"). InnerVolumeSpecName "kube-api-access-qwl69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:36:40.400363 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.384617 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s"] Oct 30 17:36:40.400363 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.385180 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-054acd-5694b584cb-wf7nq"] Oct 30 17:36:40.400363 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.391363 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-054acd-5694b584cb-wf7nq" Oct 30 17:36:40.400363 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.398275 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22a4271a-c94f-4902-9874-6712c9735cb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-054acd-5fccd756d7-v75ht\" (UID: \"22a4271a-c94f-4902-9874-6712c9735cb3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-054acd-5fccd756d7-v75ht" Oct 30 17:36:40.400363 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.398464 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xkb9\" (UniqueName: \"kubernetes.io/projected/22a4271a-c94f-4902-9874-6712c9735cb3-kube-api-access-8xkb9\") pod \"payload-pre-processing-e2e-ipp-b-054acd-5fccd756d7-v75ht\" (UID: \"22a4271a-c94f-4902-9874-6712c9735cb3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-054acd-5fccd756d7-v75ht" Oct 30 17:36:40.400363 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.398565 4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06d654a9-d7c3-4b45-8bcc-abfad4c49970-istiod-ca-cert\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:40.400363 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.398583 4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06d654a9-d7c3-4b45-8bcc-abfad4c49970-credential-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:40.400363 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.398598 4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06d654a9-d7c3-4b45-8bcc-abfad4c49970-workload-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:40.400363 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.398612 4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06d654a9-d7c3-4b45-8bcc-abfad4c49970-istio-podinfo\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:40.400363 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.398625 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwl69\" (UniqueName: \"kubernetes.io/projected/06d654a9-d7c3-4b45-8bcc-abfad4c49970-kube-api-access-qwl69\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:40.400363 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.398639 4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06d654a9-d7c3-4b45-8bcc-abfad4c49970-workload-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:40.400363 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.398652 4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06d654a9-d7c3-4b45-8bcc-abfad4c49970-istio-token\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:40.400363 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.398664 4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06d654a9-d7c3-4b45-8bcc-abfad4c49970-istio-envoy\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:40.400363 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.398675 4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06d654a9-d7c3-4b45-8bcc-abfad4c49970-istio-data\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:40.412381 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.405348 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz"] Oct 30 17:36:40.428077 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.428043 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-054acd-5694b584cb-wf7nq"] Oct 30 17:36:40.498032 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.497908 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-04248e-6dfd587cc9-n8brs"] Oct 30 17:36:40.499717 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.499685 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9734f86-d287-475b-982f-5c1b5443b16b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-054acd-5694b584cb-wf7nq\" (UID: \"c9734f86-d287-475b-982f-5c1b5443b16b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-054acd-5694b584cb-wf7nq" Oct 30 17:36:40.499885 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.499801 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22a4271a-c94f-4902-9874-6712c9735cb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-054acd-5fccd756d7-v75ht\" (UID: \"22a4271a-c94f-4902-9874-6712c9735cb3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-054acd-5fccd756d7-v75ht" Oct 30 17:36:40.499885 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.499833 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rq7q\" (UniqueName: \"kubernetes.io/projected/c9734f86-d287-475b-982f-5c1b5443b16b-kube-api-access-8rq7q\") pod \"payload-processing-e2e-ipp-b-054acd-5694b584cb-wf7nq\" (UID: \"c9734f86-d287-475b-982f-5c1b5443b16b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-054acd-5694b584cb-wf7nq" Oct 30 17:36:40.500064 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.499958 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xkb9\" (UniqueName: \"kubernetes.io/projected/22a4271a-c94f-4902-9874-6712c9735cb3-kube-api-access-8xkb9\") pod \"payload-pre-processing-e2e-ipp-b-054acd-5fccd756d7-v75ht\" (UID: \"22a4271a-c94f-4902-9874-6712c9735cb3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-054acd-5fccd756d7-v75ht" Oct 30 17:36:40.501224 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.501199 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22a4271a-c94f-4902-9874-6712c9735cb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-054acd-5fccd756d7-v75ht\" (UID: \"22a4271a-c94f-4902-9874-6712c9735cb3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-054acd-5fccd756d7-v75ht" Oct 30 17:36:40.503083 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.503058 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-04248e-6dfd587cc9-n8brs"] Oct 30 17:36:40.526529 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.526499 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xkb9\" (UniqueName: \"kubernetes.io/projected/22a4271a-c94f-4902-9874-6712c9735cb3-kube-api-access-8xkb9\") pod \"payload-pre-processing-e2e-ipp-b-054acd-5fccd756d7-v75ht\" (UID: \"22a4271a-c94f-4902-9874-6712c9735cb3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-054acd-5fccd756d7-v75ht" Oct 30 17:36:40.562757 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.562721 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b4f406/distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb" Oct 30 17:36:40.565424 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.565403 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b4f406/distinct-w2-b4f406-kserve-5db4cc6969-7n846" Oct 30 17:36:40.601056 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.601023 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9734f86-d287-475b-982f-5c1b5443b16b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-054acd-5694b584cb-wf7nq\" (UID: \"c9734f86-d287-475b-982f-5c1b5443b16b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-054acd-5694b584cb-wf7nq" Oct 30 17:36:40.601252 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.601137 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rq7q\" (UniqueName: \"kubernetes.io/projected/c9734f86-d287-475b-982f-5c1b5443b16b-kube-api-access-8rq7q\") pod \"payload-processing-e2e-ipp-b-054acd-5694b584cb-wf7nq\" (UID: \"c9734f86-d287-475b-982f-5c1b5443b16b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-054acd-5694b584cb-wf7nq" Oct 30 17:36:40.601863 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.601843 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9734f86-d287-475b-982f-5c1b5443b16b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-054acd-5694b584cb-wf7nq\" (UID: \"c9734f86-d287-475b-982f-5c1b5443b16b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-054acd-5694b584cb-wf7nq" Oct 30 17:36:40.635826 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.635797 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rq7q\" (UniqueName: \"kubernetes.io/projected/c9734f86-d287-475b-982f-5c1b5443b16b-kube-api-access-8rq7q\") pod \"payload-processing-e2e-ipp-b-054acd-5694b584cb-wf7nq\" (UID: \"c9734f86-d287-475b-982f-5c1b5443b16b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-054acd-5694b584cb-wf7nq" Oct 30 17:36:40.641814 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.641784 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-054acd-5fccd756d7-v75ht" Oct 30 17:36:40.704356 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.702073 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d86fded1-2e04-4d73-bb22-ea0d9a61b961-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d86fded1-2e04-4d73-bb22-ea0d9a61b961-model-cache\") pod \"d86fded1-2e04-4d73-bb22-ea0d9a61b961\" (UID: \"d86fded1-2e04-4d73-bb22-ea0d9a61b961\") " Oct 30 17:36:40.704356 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.702135 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8810dbc-b769-46a4-90b9-72a68592bc1a-tls-certs\" (UniqueName: \"kubernetes.io/secret/b8810dbc-b769-46a4-90b9-72a68592bc1a-tls-certs\") pod \"b8810dbc-b769-46a4-90b9-72a68592bc1a\" (UID: \"b8810dbc-b769-46a4-90b9-72a68592bc1a\") " Oct 30 17:36:40.704356 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.702400 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d86fded1-2e04-4d73-bb22-ea0d9a61b961-tls-certs\" (UniqueName: \"kubernetes.io/secret/d86fded1-2e04-4d73-bb22-ea0d9a61b961-tls-certs\") pod \"d86fded1-2e04-4d73-bb22-ea0d9a61b961\" (UID: \"d86fded1-2e04-4d73-bb22-ea0d9a61b961\") " Oct 30 17:36:40.704356 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.702437 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8810dbc-b769-46a4-90b9-72a68592bc1a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8810dbc-b769-46a4-90b9-72a68592bc1a-dshm\") pod \"b8810dbc-b769-46a4-90b9-72a68592bc1a\" (UID: \"b8810dbc-b769-46a4-90b9-72a68592bc1a\") " Oct 30 17:36:40.704356 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.702484 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8810dbc-b769-46a4-90b9-72a68592bc1a-home\" (UniqueName: \"kubernetes.io/empty-dir/b8810dbc-b769-46a4-90b9-72a68592bc1a-home\") pod \"b8810dbc-b769-46a4-90b9-72a68592bc1a\" (UID: \"b8810dbc-b769-46a4-90b9-72a68592bc1a\") " Oct 30 17:36:40.704356 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.702518 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d86fded1-2e04-4d73-bb22-ea0d9a61b961-kube-api-access-zk9kp\" (UniqueName: \"kubernetes.io/projected/d86fded1-2e04-4d73-bb22-ea0d9a61b961-kube-api-access-zk9kp\") pod \"d86fded1-2e04-4d73-bb22-ea0d9a61b961\" (UID: \"d86fded1-2e04-4d73-bb22-ea0d9a61b961\") " Oct 30 17:36:40.704356 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.702602 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d86fded1-2e04-4d73-bb22-ea0d9a61b961-home\" (UniqueName: \"kubernetes.io/empty-dir/d86fded1-2e04-4d73-bb22-ea0d9a61b961-home\") pod \"d86fded1-2e04-4d73-bb22-ea0d9a61b961\" (UID: \"d86fded1-2e04-4d73-bb22-ea0d9a61b961\") " Oct 30 17:36:40.704356 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.702644 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8810dbc-b769-46a4-90b9-72a68592bc1a-kube-api-access-t5gwv\" (UniqueName: \"kubernetes.io/projected/b8810dbc-b769-46a4-90b9-72a68592bc1a-kube-api-access-t5gwv\") pod \"b8810dbc-b769-46a4-90b9-72a68592bc1a\" (UID: \"b8810dbc-b769-46a4-90b9-72a68592bc1a\") " Oct 30 17:36:40.704356 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.702692 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d86fded1-2e04-4d73-bb22-ea0d9a61b961-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d86fded1-2e04-4d73-bb22-ea0d9a61b961-dshm\") pod \"d86fded1-2e04-4d73-bb22-ea0d9a61b961\" (UID: \"d86fded1-2e04-4d73-bb22-ea0d9a61b961\") " Oct 30 17:36:40.704356 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.702734 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8810dbc-b769-46a4-90b9-72a68592bc1a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8810dbc-b769-46a4-90b9-72a68592bc1a-tmp-dir\") pod \"b8810dbc-b769-46a4-90b9-72a68592bc1a\" (UID: \"b8810dbc-b769-46a4-90b9-72a68592bc1a\") " Oct 30 17:36:40.704356 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.702796 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8810dbc-b769-46a4-90b9-72a68592bc1a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8810dbc-b769-46a4-90b9-72a68592bc1a-model-cache\") pod \"b8810dbc-b769-46a4-90b9-72a68592bc1a\" (UID: \"b8810dbc-b769-46a4-90b9-72a68592bc1a\") " Oct 30 17:36:40.704356 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.702822 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d86fded1-2e04-4d73-bb22-ea0d9a61b961-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d86fded1-2e04-4d73-bb22-ea0d9a61b961-tmp-dir\") pod \"d86fded1-2e04-4d73-bb22-ea0d9a61b961\" (UID: \"d86fded1-2e04-4d73-bb22-ea0d9a61b961\") " Oct 30 17:36:40.704356 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.703623 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d86fded1-2e04-4d73-bb22-ea0d9a61b961-tmp-dir" pod "d86fded1-2e04-4d73-bb22-ea0d9a61b961" (UID: "d86fded1-2e04-4d73-bb22-ea0d9a61b961"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:40.704356 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.703810 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d86fded1-2e04-4d73-bb22-ea0d9a61b961-model-cache" pod "d86fded1-2e04-4d73-bb22-ea0d9a61b961" (UID: "d86fded1-2e04-4d73-bb22-ea0d9a61b961"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:40.705342 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.705161 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d86fded1-2e04-4d73-bb22-ea0d9a61b961-home" pod "d86fded1-2e04-4d73-bb22-ea0d9a61b961" (UID: "d86fded1-2e04-4d73-bb22-ea0d9a61b961"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:40.705573 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.705542 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8810dbc-b769-46a4-90b9-72a68592bc1a-home" pod "b8810dbc-b769-46a4-90b9-72a68592bc1a" (UID: "b8810dbc-b769-46a4-90b9-72a68592bc1a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:40.714429 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.705741 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8810dbc-b769-46a4-90b9-72a68592bc1a-tmp-dir" pod "b8810dbc-b769-46a4-90b9-72a68592bc1a" (UID: "b8810dbc-b769-46a4-90b9-72a68592bc1a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:40.714429 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.712596 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d86fded1-2e04-4d73-bb22-ea0d9a61b961-tls-certs" pod "d86fded1-2e04-4d73-bb22-ea0d9a61b961" (UID: "d86fded1-2e04-4d73-bb22-ea0d9a61b961"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:36:40.714429 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.712899 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d86fded1-2e04-4d73-bb22-ea0d9a61b961-kube-api-access-zk9kp" pod "d86fded1-2e04-4d73-bb22-ea0d9a61b961" (UID: "d86fded1-2e04-4d73-bb22-ea0d9a61b961"). InnerVolumeSpecName "kube-api-access-zk9kp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:36:40.714429 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.714394 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8810dbc-b769-46a4-90b9-72a68592bc1a-model-cache" pod "b8810dbc-b769-46a4-90b9-72a68592bc1a" (UID: "b8810dbc-b769-46a4-90b9-72a68592bc1a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:40.716417 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.716366 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8810dbc-b769-46a4-90b9-72a68592bc1a-tls-certs" pod "b8810dbc-b769-46a4-90b9-72a68592bc1a" (UID: "b8810dbc-b769-46a4-90b9-72a68592bc1a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:36:40.720470 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.720428 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d86fded1-2e04-4d73-bb22-ea0d9a61b961-dshm" pod "d86fded1-2e04-4d73-bb22-ea0d9a61b961" (UID: "d86fded1-2e04-4d73-bb22-ea0d9a61b961"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:40.720731 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.720700 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8810dbc-b769-46a4-90b9-72a68592bc1a-dshm" pod "b8810dbc-b769-46a4-90b9-72a68592bc1a" (UID: "b8810dbc-b769-46a4-90b9-72a68592bc1a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:40.737084 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.737051 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-054acd-5694b584cb-wf7nq" Oct 30 17:36:40.739662 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.739626 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8810dbc-b769-46a4-90b9-72a68592bc1a-kube-api-access-t5gwv" pod "b8810dbc-b769-46a4-90b9-72a68592bc1a" (UID: "b8810dbc-b769-46a4-90b9-72a68592bc1a"). InnerVolumeSpecName "kube-api-access-t5gwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:36:40.805759 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.805167 4522 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d86fded1-2e04-4d73-bb22-ea0d9a61b961-tls-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:40.805759 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.805207 4522 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8810dbc-b769-46a4-90b9-72a68592bc1a-dshm\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:40.805759 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.805220 4522 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8810dbc-b769-46a4-90b9-72a68592bc1a-home\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:40.805759 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.805234 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk9kp\" (UniqueName: \"kubernetes.io/projected/d86fded1-2e04-4d73-bb22-ea0d9a61b961-kube-api-access-zk9kp\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:40.805759 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.805250 4522 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d86fded1-2e04-4d73-bb22-ea0d9a61b961-home\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:40.805759 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.805263 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5gwv\" (UniqueName: \"kubernetes.io/projected/b8810dbc-b769-46a4-90b9-72a68592bc1a-kube-api-access-t5gwv\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:40.805759 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.805277 4522 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d86fded1-2e04-4d73-bb22-ea0d9a61b961-dshm\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:40.805759 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.805290 4522 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8810dbc-b769-46a4-90b9-72a68592bc1a-tmp-dir\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:40.805759 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.805319 4522 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8810dbc-b769-46a4-90b9-72a68592bc1a-model-cache\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:40.805759 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.805331 4522 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d86fded1-2e04-4d73-bb22-ea0d9a61b961-tmp-dir\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:40.805759 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.805343 4522 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d86fded1-2e04-4d73-bb22-ea0d9a61b961-model-cache\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:40.805759 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.805354 4522 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8810dbc-b769-46a4-90b9-72a68592bc1a-tls-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:40.964389 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:36:40.964357 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22a4271a_c94f_4902_9874_6712c9735cb3.slice/crio-2a89e2cf81d53898f36e5926d4571fa13e46a1495883bb12450bf26683085cd6 WatchSource:0}: Error finding container 2a89e2cf81d53898f36e5926d4571fa13e46a1495883bb12450bf26683085cd6: Status 404 returned error can't find the container with id 2a89e2cf81d53898f36e5926d4571fa13e46a1495883bb12450bf26683085cd6 Oct 30 17:36:40.965739 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:40.965717 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-054acd-5fccd756d7-v75ht"] Oct 30 17:36:41.057492 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:36:41.057458 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9734f86_d287_475b_982f_5c1b5443b16b.slice/crio-19ec91a5362e6bb37e63fac191de602a9252f0bf86eccd433f18e40db496429b WatchSource:0}: Error finding container 19ec91a5362e6bb37e63fac191de602a9252f0bf86eccd433f18e40db496429b: Status 404 returned error can't find the container with id 19ec91a5362e6bb37e63fac191de602a9252f0bf86eccd433f18e40db496429b Oct 30 17:36:41.058402 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:41.058379 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-054acd-5694b584cb-wf7nq"] Oct 30 17:36:41.082023 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:41.082000 4522 generic.go:356] "Generic (PLEG): container finished" podID="b8810dbc-b769-46a4-90b9-72a68592bc1a" containerID="c7b60475ed24d290180c64665644a3145ff12c2e5b4f2a6b0a432df5aa22bb6e" exitCode=0 Oct 30 17:36:41.082137 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:41.082043 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b4f406/distinct-w2-b4f406-kserve-5db4cc6969-7n846" Oct 30 17:36:41.082137 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:41.082081 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b4f406/distinct-w2-b4f406-kserve-5db4cc6969-7n846" event={"ID":"b8810dbc-b769-46a4-90b9-72a68592bc1a","Type":"ContainerDied","Data":"c7b60475ed24d290180c64665644a3145ff12c2e5b4f2a6b0a432df5aa22bb6e"} Oct 30 17:36:41.082137 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:41.082118 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b4f406/distinct-w2-b4f406-kserve-5db4cc6969-7n846" event={"ID":"b8810dbc-b769-46a4-90b9-72a68592bc1a","Type":"ContainerDied","Data":"e62870ba1362c4acd0e97b3c959da66020ce7cfbecf949c23062b22ad1be0dbc"} Oct 30 17:36:41.082274 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:41.082138 4522 scope.go:122] "RemoveContainer" containerID="c7b60475ed24d290180c64665644a3145ff12c2e5b4f2a6b0a432df5aa22bb6e" Oct 30 17:36:41.083918 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:41.083895 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-054acd-5fccd756d7-v75ht" event={"ID":"22a4271a-c94f-4902-9874-6712c9735cb3","Type":"ContainerStarted","Data":"2b05bab0b6431768dde8e2fab5c506eabac0957c6efcd23e2fe87f0bc8014dc9"} Oct 30 17:36:41.084017 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:41.083945 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-054acd-5fccd756d7-v75ht" event={"ID":"22a4271a-c94f-4902-9874-6712c9735cb3","Type":"ContainerStarted","Data":"2a89e2cf81d53898f36e5926d4571fa13e46a1495883bb12450bf26683085cd6"} Oct 30 17:36:41.084017 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:41.083993 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-054acd-5fccd756d7-v75ht" Oct 30 17:36:41.099997 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:41.099964 4522 generic.go:356] "Generic (PLEG): container finished" podID="d86fded1-2e04-4d73-bb22-ea0d9a61b961" containerID="d0c833e26aeb6db1b92a905fa9433641e73eee8d02a39465c5ba761a3f45401c" exitCode=0 Oct 30 17:36:41.100147 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:41.100128 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b4f406/distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb" event={"ID":"d86fded1-2e04-4d73-bb22-ea0d9a61b961","Type":"ContainerDied","Data":"d0c833e26aeb6db1b92a905fa9433641e73eee8d02a39465c5ba761a3f45401c"} Oct 30 17:36:41.101940 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:41.100173 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b4f406/distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb" event={"ID":"d86fded1-2e04-4d73-bb22-ea0d9a61b961","Type":"ContainerDied","Data":"cfe5071aafa8bdd008cf24ca8ec4ab0b8ea8212ce1675a5d16d2331af4e18222"} Oct 30 17:36:41.101940 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:41.100270 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b4f406/distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb" Oct 30 17:36:41.109584 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:41.106892 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq" event={"ID":"06d654a9-d7c3-4b45-8bcc-abfad4c49970","Type":"ContainerDied","Data":"17e1b0060e50944d33d43d9d4284a696e33758cd9817359e09eab9c069e91602"} Oct 30 17:36:41.109584 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:41.107078 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq" Oct 30 17:36:41.111245 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:41.111205 4522 scope.go:122] "RemoveContainer" containerID="c7b60475ed24d290180c64665644a3145ff12c2e5b4f2a6b0a432df5aa22bb6e" Oct 30 17:36:41.111743 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:36:41.111687 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7b60475ed24d290180c64665644a3145ff12c2e5b4f2a6b0a432df5aa22bb6e\": container with ID starting with c7b60475ed24d290180c64665644a3145ff12c2e5b4f2a6b0a432df5aa22bb6e not found: ID does not exist" containerID="c7b60475ed24d290180c64665644a3145ff12c2e5b4f2a6b0a432df5aa22bb6e" Oct 30 17:36:41.111743 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:41.111721 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7b60475ed24d290180c64665644a3145ff12c2e5b4f2a6b0a432df5aa22bb6e"} err="failed to get container status \"c7b60475ed24d290180c64665644a3145ff12c2e5b4f2a6b0a432df5aa22bb6e\": rpc error: code = NotFound desc = could not find container \"c7b60475ed24d290180c64665644a3145ff12c2e5b4f2a6b0a432df5aa22bb6e\": container with ID starting with c7b60475ed24d290180c64665644a3145ff12c2e5b4f2a6b0a432df5aa22bb6e not found: ID does not exist" Oct 30 17:36:41.111889 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:41.111745 4522 scope.go:122] "RemoveContainer" containerID="d0c833e26aeb6db1b92a905fa9433641e73eee8d02a39465c5ba761a3f45401c" Oct 30 17:36:41.111889 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:41.111773 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-054acd-5694b584cb-wf7nq" event={"ID":"c9734f86-d287-475b-982f-5c1b5443b16b","Type":"ContainerStarted","Data":"19ec91a5362e6bb37e63fac191de602a9252f0bf86eccd433f18e40db496429b"} Oct 30 17:36:41.124728 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:41.124704 4522 scope.go:122] "RemoveContainer" containerID="d0c833e26aeb6db1b92a905fa9433641e73eee8d02a39465c5ba761a3f45401c" Oct 30 17:36:41.125102 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:36:41.125079 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0c833e26aeb6db1b92a905fa9433641e73eee8d02a39465c5ba761a3f45401c\": container with ID starting with d0c833e26aeb6db1b92a905fa9433641e73eee8d02a39465c5ba761a3f45401c not found: ID does not exist" containerID="d0c833e26aeb6db1b92a905fa9433641e73eee8d02a39465c5ba761a3f45401c" Oct 30 17:36:41.125174 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:41.125110 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0c833e26aeb6db1b92a905fa9433641e73eee8d02a39465c5ba761a3f45401c"} err="failed to get container status \"d0c833e26aeb6db1b92a905fa9433641e73eee8d02a39465c5ba761a3f45401c\": rpc error: code = NotFound desc = could not find container \"d0c833e26aeb6db1b92a905fa9433641e73eee8d02a39465c5ba761a3f45401c\": container with ID starting with d0c833e26aeb6db1b92a905fa9433641e73eee8d02a39465c5ba761a3f45401c not found: ID does not exist" Oct 30 17:36:41.125174 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:41.125128 4522 scope.go:122] "RemoveContainer" containerID="be25d5e29c60fb7663216b9c320b6eb4d8d0e1b02e79d41efaf3a572b1552977" Oct 30 17:36:41.158503 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:41.158447 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-054acd-5fccd756d7-v75ht" podStartSLOduration=1.15843112 podStartE2EDuration="1.15843112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:36:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:36:41.158135432 +0000 UTC m=+2975.158890336" watchObservedRunningTime="2026-10-30 17:36:41.15843112 +0000 UTC m=+2975.159186021" Oct 30 17:36:41.199539 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:41.199507 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-b4f406/distinct-w2-b4f406-kserve-5db4cc6969-7n846"] Oct 30 17:36:41.212439 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:41.212412 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-b4f406/distinct-w2-b4f406-kserve-5db4cc6969-7n846"] Oct 30 17:36:41.251968 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:41.251936 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-b4f406/distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb"] Oct 30 17:36:41.255659 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:41.255625 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-b4f406/distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb"] Oct 30 17:36:41.298466 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:41.298385 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq"] Oct 30 17:36:41.301299 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:41.301277 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-jr6qq"] Oct 30 17:36:41.533993 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:41.533946 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/distinct-2-w2-b4f406-kserve-7544b89d86-fpjxb" podUID="d86fded1-2e04-4d73-bb22-ea0d9a61b961" containerName="main" probeResult="failure" output="Get \"https://10.134.0.127:8000/ready\": context deadline exceeded" Oct 30 17:36:42.125386 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:42.125346 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-054acd-5694b584cb-wf7nq" event={"ID":"c9734f86-d287-475b-982f-5c1b5443b16b","Type":"ContainerStarted","Data":"b3b2404a2dcc8a13a9a8605e9d1df4fd6b347a9993328e9d70ecd1db3e581ea1"} Oct 30 17:36:42.125574 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:42.125412 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-054acd-5694b584cb-wf7nq" Oct 30 17:36:42.145208 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:42.145177 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06d654a9-d7c3-4b45-8bcc-abfad4c49970" path="/var/lib/kubelet/pods/06d654a9-d7c3-4b45-8bcc-abfad4c49970/volumes" Oct 30 17:36:42.145633 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:42.145620 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80f8edc4-4e9f-435e-8518-e5f53d262030" path="/var/lib/kubelet/pods/80f8edc4-4e9f-435e-8518-e5f53d262030/volumes" Oct 30 17:36:42.146007 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:42.145995 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8810dbc-b769-46a4-90b9-72a68592bc1a" path="/var/lib/kubelet/pods/b8810dbc-b769-46a4-90b9-72a68592bc1a/volumes" Oct 30 17:36:42.146696 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:42.146681 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d86fded1-2e04-4d73-bb22-ea0d9a61b961" path="/var/lib/kubelet/pods/d86fded1-2e04-4d73-bb22-ea0d9a61b961/volumes" Oct 30 17:36:42.172119 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:42.172069 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-054acd-5694b584cb-wf7nq" podStartSLOduration=2.172057471 podStartE2EDuration="2.172057471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:36:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:36:42.169835912 +0000 UTC m=+2976.170590816" watchObservedRunningTime="2026-10-30 17:36:42.172057471 +0000 UTC m=+2976.172812373" Oct 30 17:36:42.415007 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:42.414915 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-82243b-6794bbb67-58ssq" podUID="5bdabfbc-c999-479c-a44e-d4949bd3c73d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:36:42.415454 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:42.414922 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-82243b-7f4f9546b6-mg8hr" podUID="5333c134-0e1a-4096-a6d0-6d54f2373e28" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:36:42.948161 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:36:42.948122 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:36:43.120014 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:43.119978 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-b4f406-6d66f86b4b-bzcnr" podUID="519e4a59-cb60-4a1a-949b-56018ea6b89c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:36:43.120014 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:43.120001 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b4f406-786855ccd7-ns4j8" podUID="246bcd4d-bddf-466e-bf5b-9f18b81fa5fd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:36:45.772146 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:45.772105 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-b80d83-58f8df6ff4-xt4dg"] Oct 30 17:36:48.247430 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:48.247389 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-82243b-nrh8z"] Oct 30 17:36:48.262069 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:48.262032 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-82243b-nrh8z"] Oct 30 17:36:50.034751 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:50.034708 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-031223-867764c779-6mp6s"] Oct 30 17:36:50.035490 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:50.035469 4522 cpu_manager.go:414] "containerMap: removing container" podUID="d86fded1-2e04-4d73-bb22-ea0d9a61b961" containerName="main" Oct 30 17:36:50.035610 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:50.035491 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d86fded1-2e04-4d73-bb22-ea0d9a61b961" containerName="main" Oct 30 17:36:50.035610 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:50.035507 4522 cpu_manager.go:414] "containerMap: removing container" podUID="b8810dbc-b769-46a4-90b9-72a68592bc1a" containerName="main" Oct 30 17:36:50.035610 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:50.035514 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8810dbc-b769-46a4-90b9-72a68592bc1a" containerName="main" Oct 30 17:36:50.035840 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:50.035706 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="d86fded1-2e04-4d73-bb22-ea0d9a61b961" containerName="main" Oct 30 17:36:50.035840 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:50.035720 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8810dbc-b769-46a4-90b9-72a68592bc1a" containerName="main" Oct 30 17:36:50.040511 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:50.040473 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-031223-867764c779-6mp6s" Oct 30 17:36:50.059998 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:50.059970 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-031223-867764c779-6mp6s"] Oct 30 17:36:50.068749 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:50.068722 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-031223-dockercfg-58ws7\"" Oct 30 17:36:50.068749 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:50.068744 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-031223\"" Oct 30 17:36:50.130433 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:50.130390 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-429bf\" (UniqueName: \"kubernetes.io/projected/6dab304c-05a7-4d7e-a8bc-46b808de8c8f-kube-api-access-429bf\") pod \"payload-pre-processing-e2e-ipp-b-031223-867764c779-6mp6s\" (UID: \"6dab304c-05a7-4d7e-a8bc-46b808de8c8f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-031223-867764c779-6mp6s" Oct 30 17:36:50.130609 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:50.130530 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dab304c-05a7-4d7e-a8bc-46b808de8c8f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-031223-867764c779-6mp6s\" (UID: \"6dab304c-05a7-4d7e-a8bc-46b808de8c8f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-031223-867764c779-6mp6s" Oct 30 17:36:50.144703 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:50.144678 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="043dfb0d-b8be-47b0-bd85-5842628a50f9" path="/var/lib/kubelet/pods/043dfb0d-b8be-47b0-bd85-5842628a50f9/volumes" Oct 30 17:36:50.203868 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:50.203835 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-031223-547d957db6-v6n6p"] Oct 30 17:36:50.208087 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:50.208069 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-031223-547d957db6-v6n6p" Oct 30 17:36:50.226573 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:50.226549 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-031223-547d957db6-v6n6p"] Oct 30 17:36:50.231787 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:50.231557 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-429bf\" (UniqueName: \"kubernetes.io/projected/6dab304c-05a7-4d7e-a8bc-46b808de8c8f-kube-api-access-429bf\") pod \"payload-pre-processing-e2e-ipp-b-031223-867764c779-6mp6s\" (UID: \"6dab304c-05a7-4d7e-a8bc-46b808de8c8f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-031223-867764c779-6mp6s" Oct 30 17:36:50.231787 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:50.231674 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dab304c-05a7-4d7e-a8bc-46b808de8c8f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-031223-867764c779-6mp6s\" (UID: \"6dab304c-05a7-4d7e-a8bc-46b808de8c8f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-031223-867764c779-6mp6s" Oct 30 17:36:50.231985 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:50.231823 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgdfp\" (UniqueName: \"kubernetes.io/projected/804e4c48-d1d9-4e76-9d19-bdabc3a805f3-kube-api-access-rgdfp\") pod \"payload-processing-e2e-ipp-b-031223-547d957db6-v6n6p\" (UID: \"804e4c48-d1d9-4e76-9d19-bdabc3a805f3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-031223-547d957db6-v6n6p" Oct 30 17:36:50.231985 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:50.231868 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/804e4c48-d1d9-4e76-9d19-bdabc3a805f3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-031223-547d957db6-v6n6p\" (UID: \"804e4c48-d1d9-4e76-9d19-bdabc3a805f3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-031223-547d957db6-v6n6p" Oct 30 17:36:50.233028 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:50.233003 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dab304c-05a7-4d7e-a8bc-46b808de8c8f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-031223-867764c779-6mp6s\" (UID: \"6dab304c-05a7-4d7e-a8bc-46b808de8c8f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-031223-867764c779-6mp6s" Oct 30 17:36:50.259089 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:50.259057 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-429bf\" (UniqueName: \"kubernetes.io/projected/6dab304c-05a7-4d7e-a8bc-46b808de8c8f-kube-api-access-429bf\") pod \"payload-pre-processing-e2e-ipp-b-031223-867764c779-6mp6s\" (UID: \"6dab304c-05a7-4d7e-a8bc-46b808de8c8f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-031223-867764c779-6mp6s" Oct 30 17:36:50.332689 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:50.332605 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgdfp\" (UniqueName: \"kubernetes.io/projected/804e4c48-d1d9-4e76-9d19-bdabc3a805f3-kube-api-access-rgdfp\") pod \"payload-processing-e2e-ipp-b-031223-547d957db6-v6n6p\" (UID: \"804e4c48-d1d9-4e76-9d19-bdabc3a805f3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-031223-547d957db6-v6n6p" Oct 30 17:36:50.332689 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:50.332649 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/804e4c48-d1d9-4e76-9d19-bdabc3a805f3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-031223-547d957db6-v6n6p\" (UID: \"804e4c48-d1d9-4e76-9d19-bdabc3a805f3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-031223-547d957db6-v6n6p" Oct 30 17:36:50.333349 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:50.333330 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/804e4c48-d1d9-4e76-9d19-bdabc3a805f3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-031223-547d957db6-v6n6p\" (UID: \"804e4c48-d1d9-4e76-9d19-bdabc3a805f3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-031223-547d957db6-v6n6p" Oct 30 17:36:50.355341 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:50.355292 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-031223-867764c779-6mp6s" Oct 30 17:36:50.357824 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:50.357793 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgdfp\" (UniqueName: \"kubernetes.io/projected/804e4c48-d1d9-4e76-9d19-bdabc3a805f3-kube-api-access-rgdfp\") pod \"payload-processing-e2e-ipp-b-031223-547d957db6-v6n6p\" (UID: \"804e4c48-d1d9-4e76-9d19-bdabc3a805f3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-031223-547d957db6-v6n6p" Oct 30 17:36:50.521795 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:50.521762 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-031223-547d957db6-v6n6p" Oct 30 17:36:50.663778 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:36:50.663751 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dab304c_05a7_4d7e_a8bc_46b808de8c8f.slice/crio-4cb2fe8450b916a20464c2a1d508a9cafa0f10e685290d6039a5a72cecf36099 WatchSource:0}: Error finding container 4cb2fe8450b916a20464c2a1d508a9cafa0f10e685290d6039a5a72cecf36099: Status 404 returned error can't find the container with id 4cb2fe8450b916a20464c2a1d508a9cafa0f10e685290d6039a5a72cecf36099 Oct 30 17:36:50.664732 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:50.664635 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-031223-867764c779-6mp6s"] Oct 30 17:36:50.773533 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:50.773493 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-b80d83-58f8df6ff4-xt4dg" podUID="bfdddc9b-c6c6-4cb1-bdd6-1634e76c4825" containerName="payload-processing" containerID="cri-o://dc92488ee977f9f3b44786c79d31b230099d2240d2a2a8f8c60abf0e04eb98e1" gracePeriod=25 Oct 30 17:36:50.831915 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:36:50.831887 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod804e4c48_d1d9_4e76_9d19_bdabc3a805f3.slice/crio-14620db30cd1d664837d5dfb0d9759789667ca028bb109ce06cf09ddea262240 WatchSource:0}: Error finding container 14620db30cd1d664837d5dfb0d9759789667ca028bb109ce06cf09ddea262240: Status 404 returned error can't find the container with id 14620db30cd1d664837d5dfb0d9759789667ca028bb109ce06cf09ddea262240 Oct 30 17:36:50.833463 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:50.833441 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-031223-547d957db6-v6n6p"] Oct 30 17:36:51.202351 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:51.202318 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-031223-547d957db6-v6n6p" event={"ID":"804e4c48-d1d9-4e76-9d19-bdabc3a805f3","Type":"ContainerStarted","Data":"58185126a3e3e35e9cd0db0bb94f05dcd9d439706ef977c782c097bf556bd3d2"} Oct 30 17:36:51.202840 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:51.202381 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-031223-547d957db6-v6n6p" event={"ID":"804e4c48-d1d9-4e76-9d19-bdabc3a805f3","Type":"ContainerStarted","Data":"14620db30cd1d664837d5dfb0d9759789667ca028bb109ce06cf09ddea262240"} Oct 30 17:36:51.202840 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:51.202575 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-031223-547d957db6-v6n6p" Oct 30 17:36:51.204255 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:51.204233 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-031223-867764c779-6mp6s" event={"ID":"6dab304c-05a7-4d7e-a8bc-46b808de8c8f","Type":"ContainerStarted","Data":"12bcdaeb76698779645db2ee39384850af1d01838d8028f0ef0e76743145043c"} Oct 30 17:36:51.204556 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:51.204280 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-031223-867764c779-6mp6s" Oct 30 17:36:51.204556 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:51.204317 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-031223-867764c779-6mp6s" event={"ID":"6dab304c-05a7-4d7e-a8bc-46b808de8c8f","Type":"ContainerStarted","Data":"4cb2fe8450b916a20464c2a1d508a9cafa0f10e685290d6039a5a72cecf36099"} Oct 30 17:36:51.250103 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:51.250052 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-031223-547d957db6-v6n6p" podStartSLOduration=1.250040128 podStartE2EDuration="1.250040128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:36:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:36:51.249125765 +0000 UTC m=+2985.249880667" watchObservedRunningTime="2026-10-30 17:36:51.250040128 +0000 UTC m=+2985.250795028" Oct 30 17:36:51.296942 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:51.296878 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-031223-867764c779-6mp6s" podStartSLOduration=1.296849627 podStartE2EDuration="1.296849627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:36:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:36:51.292203352 +0000 UTC m=+2985.292958255" watchObservedRunningTime="2026-10-30 17:36:51.296849627 +0000 UTC m=+2985.297604531" Oct 30 17:36:52.073781 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:52.073744 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daacd80b-7mkw4"] Oct 30 17:36:52.078080 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:52.078054 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daacd80b-7mkw4" Oct 30 17:36:52.105552 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:52.105512 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:36:52.113849 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:52.113821 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-b80d83-58f8df6ff4-xt4dg" podUID="bfdddc9b-c6c6-4cb1-bdd6-1634e76c4825" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:36:52.118736 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:52.118708 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:36:52.122299 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:52.122272 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daacd80b-7mkw4"] Oct 30 17:36:52.129722 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:52.129700 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-054acd-5fccd756d7-v75ht" Oct 30 17:36:52.151221 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:52.151196 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n82s\" (UniqueName: \"kubernetes.io/projected/d80fae1e-5c7b-4cb4-9b50-eae3ffc175ff-kube-api-access-2n82s\") pod \"maas-api-revoke-sub-daacd80b-7mkw4\" (UID: \"d80fae1e-5c7b-4cb4-9b50-eae3ffc175ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daacd80b-7mkw4" Oct 30 17:36:52.151412 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:52.151385 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d80fae1e-5c7b-4cb4-9b50-eae3ffc175ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-daacd80b-7mkw4\" (UID: \"d80fae1e-5c7b-4cb4-9b50-eae3ffc175ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daacd80b-7mkw4" Oct 30 17:36:52.151552 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:52.151527 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d80fae1e-5c7b-4cb4-9b50-eae3ffc175ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-daacd80b-7mkw4\" (UID: \"d80fae1e-5c7b-4cb4-9b50-eae3ffc175ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daacd80b-7mkw4" Oct 30 17:36:52.253120 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:52.253089 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d80fae1e-5c7b-4cb4-9b50-eae3ffc175ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-daacd80b-7mkw4\" (UID: \"d80fae1e-5c7b-4cb4-9b50-eae3ffc175ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daacd80b-7mkw4" Oct 30 17:36:52.253606 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:52.253143 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d80fae1e-5c7b-4cb4-9b50-eae3ffc175ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-daacd80b-7mkw4\" (UID: \"d80fae1e-5c7b-4cb4-9b50-eae3ffc175ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daacd80b-7mkw4" Oct 30 17:36:52.253606 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:52.253249 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n82s\" (UniqueName: \"kubernetes.io/projected/d80fae1e-5c7b-4cb4-9b50-eae3ffc175ff-kube-api-access-2n82s\") pod \"maas-api-revoke-sub-daacd80b-7mkw4\" (UID: \"d80fae1e-5c7b-4cb4-9b50-eae3ffc175ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daacd80b-7mkw4" Oct 30 17:36:52.253865 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:52.253836 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d80fae1e-5c7b-4cb4-9b50-eae3ffc175ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-daacd80b-7mkw4\" (UID: \"d80fae1e-5c7b-4cb4-9b50-eae3ffc175ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daacd80b-7mkw4" Oct 30 17:36:52.255633 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:52.255610 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d80fae1e-5c7b-4cb4-9b50-eae3ffc175ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-daacd80b-7mkw4\" (UID: \"d80fae1e-5c7b-4cb4-9b50-eae3ffc175ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daacd80b-7mkw4" Oct 30 17:36:52.279042 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:52.279015 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n82s\" (UniqueName: \"kubernetes.io/projected/d80fae1e-5c7b-4cb4-9b50-eae3ffc175ff-kube-api-access-2n82s\") pod \"maas-api-revoke-sub-daacd80b-7mkw4\" (UID: \"d80fae1e-5c7b-4cb4-9b50-eae3ffc175ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daacd80b-7mkw4" Oct 30 17:36:52.392712 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:52.392634 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daacd80b-7mkw4" Oct 30 17:36:52.415227 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:52.415193 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-82243b-6794bbb67-58ssq" podUID="5bdabfbc-c999-479c-a44e-d4949bd3c73d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:36:52.415383 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:52.415208 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-82243b-7f4f9546b6-mg8hr" podUID="5333c134-0e1a-4096-a6d0-6d54f2373e28" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:36:52.721738 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:36:52.721709 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd80fae1e_5c7b_4cb4_9b50_eae3ffc175ff.slice/crio-ca1f2513cc9cf716b2bb9352625a5ca2951944ffccb67bdf9702c5a4c7ec195a WatchSource:0}: Error finding container ca1f2513cc9cf716b2bb9352625a5ca2951944ffccb67bdf9702c5a4c7ec195a: Status 404 returned error can't find the container with id ca1f2513cc9cf716b2bb9352625a5ca2951944ffccb67bdf9702c5a4c7ec195a Oct 30 17:36:52.724834 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:52.724806 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daacd80b-7mkw4"] Oct 30 17:36:53.120401 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:53.120360 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-b4f406-6d66f86b4b-bzcnr" podUID="519e4a59-cb60-4a1a-949b-56018ea6b89c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:36:53.120584 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:53.120359 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b4f406-786855ccd7-ns4j8" podUID="246bcd4d-bddf-466e-bf5b-9f18b81fa5fd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:36:53.136977 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:53.136953 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-054acd-5694b584cb-wf7nq" Oct 30 17:36:53.222632 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:53.222587 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daacd80b-7mkw4" event={"ID":"d80fae1e-5c7b-4cb4-9b50-eae3ffc175ff","Type":"ContainerStarted","Data":"7da3be0e62b7aae34cc5414c014d5608325d722b4a6e469c5d750068348c7df9"} Oct 30 17:36:53.222824 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:53.222691 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daacd80b-7mkw4" event={"ID":"d80fae1e-5c7b-4cb4-9b50-eae3ffc175ff","Type":"ContainerStarted","Data":"ca1f2513cc9cf716b2bb9352625a5ca2951944ffccb67bdf9702c5a4c7ec195a"} Oct 30 17:36:53.266155 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:53.266057 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daacd80b-7mkw4" podStartSLOduration=1.266044819 podStartE2EDuration="1.266044819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:36:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:36:53.265346157 +0000 UTC m=+2987.266101082" watchObservedRunningTime="2026-10-30 17:36:53.266044819 +0000 UTC m=+2987.266799719" Oct 30 17:36:54.231202 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:54.231165 4522 generic.go:356] "Generic (PLEG): container finished" podID="d80fae1e-5c7b-4cb4-9b50-eae3ffc175ff" containerID="7da3be0e62b7aae34cc5414c014d5608325d722b4a6e469c5d750068348c7df9" exitCode=0 Oct 30 17:36:54.231415 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:54.231205 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daacd80b-7mkw4" event={"ID":"d80fae1e-5c7b-4cb4-9b50-eae3ffc175ff","Type":"ContainerDied","Data":"7da3be0e62b7aae34cc5414c014d5608325d722b4a6e469c5d750068348c7df9"} Oct 30 17:36:55.405112 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:55.405087 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daacd80b-7mkw4" Oct 30 17:36:55.466647 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:55.466599 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" podUID="9bfbc6d3-91fd-41a7-9585-40a8f252f743" containerName="main" containerID="cri-o://dd8ed889c56d104044f6b5565e22bb2e08a8caf2d9715bbf964f46275c2eac13" gracePeriod=45 Oct 30 17:36:55.476813 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:55.476777 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" podUID="75321c5d-8bf8-45c6-8811-81981fe3cad4" containerName="main" containerID="cri-o://ec5598f21e37163f7c4135de7a8a510a8fd2793f7c5e1b7e211068c818310f10" gracePeriod=45 Oct 30 17:36:55.491157 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:55.491130 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d80fae1e-5c7b-4cb4-9b50-eae3ffc175ff-kube-api-access-2n82s\" (UniqueName: \"kubernetes.io/projected/d80fae1e-5c7b-4cb4-9b50-eae3ffc175ff-kube-api-access-2n82s\") pod \"d80fae1e-5c7b-4cb4-9b50-eae3ffc175ff\" (UID: \"d80fae1e-5c7b-4cb4-9b50-eae3ffc175ff\") " Oct 30 17:36:55.492236 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:55.492160 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d80fae1e-5c7b-4cb4-9b50-eae3ffc175ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d80fae1e-5c7b-4cb4-9b50-eae3ffc175ff-maas-api-xks-ca\") pod \"d80fae1e-5c7b-4cb4-9b50-eae3ffc175ff\" (UID: \"d80fae1e-5c7b-4cb4-9b50-eae3ffc175ff\") " Oct 30 17:36:55.492381 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:55.492243 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d80fae1e-5c7b-4cb4-9b50-eae3ffc175ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d80fae1e-5c7b-4cb4-9b50-eae3ffc175ff-maas-api-openshift-ca\") pod \"d80fae1e-5c7b-4cb4-9b50-eae3ffc175ff\" (UID: \"d80fae1e-5c7b-4cb4-9b50-eae3ffc175ff\") " Oct 30 17:36:55.493248 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:55.493070 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d80fae1e-5c7b-4cb4-9b50-eae3ffc175ff-maas-api-openshift-ca" pod "d80fae1e-5c7b-4cb4-9b50-eae3ffc175ff" (UID: "d80fae1e-5c7b-4cb4-9b50-eae3ffc175ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:36:55.493892 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:55.493856 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d80fae1e-5c7b-4cb4-9b50-eae3ffc175ff-kube-api-access-2n82s" pod "d80fae1e-5c7b-4cb4-9b50-eae3ffc175ff" (UID: "d80fae1e-5c7b-4cb4-9b50-eae3ffc175ff"). InnerVolumeSpecName "kube-api-access-2n82s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:36:55.495277 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:55.495257 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d80fae1e-5c7b-4cb4-9b50-eae3ffc175ff-maas-api-xks-ca" pod "d80fae1e-5c7b-4cb4-9b50-eae3ffc175ff" (UID: "d80fae1e-5c7b-4cb4-9b50-eae3ffc175ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:36:55.593831 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:55.593801 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d80fae1e-5c7b-4cb4-9b50-eae3ffc175ff-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:55.593831 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:55.593831 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d80fae1e-5c7b-4cb4-9b50-eae3ffc175ff-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:55.593999 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:55.593846 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n82s\" (UniqueName: \"kubernetes.io/projected/d80fae1e-5c7b-4cb4-9b50-eae3ffc175ff-kube-api-access-2n82s\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:55.654577 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:55.654539 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a13a40c3-wjvk5"] Oct 30 17:36:55.655559 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:55.655531 4522 cpu_manager.go:414] "containerMap: removing container" podUID="d80fae1e-5c7b-4cb4-9b50-eae3ffc175ff" containerName="revoke-keys" Oct 30 17:36:55.655559 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:55.655560 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d80fae1e-5c7b-4cb4-9b50-eae3ffc175ff" containerName="revoke-keys" Oct 30 17:36:55.655755 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:55.655742 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="d80fae1e-5c7b-4cb4-9b50-eae3ffc175ff" containerName="revoke-keys" Oct 30 17:36:55.659979 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:55.659954 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a13a40c3-wjvk5" Oct 30 17:36:55.716434 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:55.696273 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srqcv\" (UniqueName: \"kubernetes.io/projected/78ab131d-5658-4d13-be06-4943fcdd609b-kube-api-access-srqcv\") pod \"maas-api-revoke-sub-a13a40c3-wjvk5\" (UID: \"78ab131d-5658-4d13-be06-4943fcdd609b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a13a40c3-wjvk5" Oct 30 17:36:55.716434 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:55.696584 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78ab131d-5658-4d13-be06-4943fcdd609b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a13a40c3-wjvk5\" (UID: \"78ab131d-5658-4d13-be06-4943fcdd609b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a13a40c3-wjvk5" Oct 30 17:36:55.716434 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:55.696652 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78ab131d-5658-4d13-be06-4943fcdd609b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a13a40c3-wjvk5\" (UID: \"78ab131d-5658-4d13-be06-4943fcdd609b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a13a40c3-wjvk5" Oct 30 17:36:55.716434 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:55.710570 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a13a40c3-wjvk5"] Oct 30 17:36:55.798112 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:55.798078 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78ab131d-5658-4d13-be06-4943fcdd609b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a13a40c3-wjvk5\" (UID: \"78ab131d-5658-4d13-be06-4943fcdd609b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a13a40c3-wjvk5" Oct 30 17:36:55.798297 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:55.798230 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srqcv\" (UniqueName: \"kubernetes.io/projected/78ab131d-5658-4d13-be06-4943fcdd609b-kube-api-access-srqcv\") pod \"maas-api-revoke-sub-a13a40c3-wjvk5\" (UID: \"78ab131d-5658-4d13-be06-4943fcdd609b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a13a40c3-wjvk5" Oct 30 17:36:55.798446 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:55.798426 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78ab131d-5658-4d13-be06-4943fcdd609b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a13a40c3-wjvk5\" (UID: \"78ab131d-5658-4d13-be06-4943fcdd609b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a13a40c3-wjvk5" Oct 30 17:36:55.799077 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:55.798894 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78ab131d-5658-4d13-be06-4943fcdd609b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a13a40c3-wjvk5\" (UID: \"78ab131d-5658-4d13-be06-4943fcdd609b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a13a40c3-wjvk5" Oct 30 17:36:55.801433 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:55.801412 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78ab131d-5658-4d13-be06-4943fcdd609b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a13a40c3-wjvk5\" (UID: \"78ab131d-5658-4d13-be06-4943fcdd609b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a13a40c3-wjvk5" Oct 30 17:36:55.825270 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:55.825189 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srqcv\" (UniqueName: \"kubernetes.io/projected/78ab131d-5658-4d13-be06-4943fcdd609b-kube-api-access-srqcv\") pod \"maas-api-revoke-sub-a13a40c3-wjvk5\" (UID: \"78ab131d-5658-4d13-be06-4943fcdd609b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a13a40c3-wjvk5" Oct 30 17:36:55.854616 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:55.854560 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" Oct 30 17:36:55.863666 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:55.863642 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" Oct 30 17:36:55.884627 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:55.884602 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b4f406-786855ccd7-ns4j8" Oct 30 17:36:55.889149 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:55.889127 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-b4f406-6d66f86b4b-bzcnr" Oct 30 17:36:55.899745 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:55.899720 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75321c5d-8bf8-45c6-8811-81981fe3cad4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/75321c5d-8bf8-45c6-8811-81981fe3cad4-dshm\") pod \"75321c5d-8bf8-45c6-8811-81981fe3cad4\" (UID: \"75321c5d-8bf8-45c6-8811-81981fe3cad4\") " Oct 30 17:36:55.899883 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:55.899766 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/246bcd4d-bddf-466e-bf5b-9f18b81fa5fd-kube-api-access-qzwnq\" (UniqueName: \"kubernetes.io/projected/246bcd4d-bddf-466e-bf5b-9f18b81fa5fd-kube-api-access-qzwnq\") pod \"246bcd4d-bddf-466e-bf5b-9f18b81fa5fd\" (UID: \"246bcd4d-bddf-466e-bf5b-9f18b81fa5fd\") " Oct 30 17:36:55.899883 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:55.899809 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bfbc6d3-91fd-41a7-9585-40a8f252f743-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bfbc6d3-91fd-41a7-9585-40a8f252f743-model-cache\") pod \"9bfbc6d3-91fd-41a7-9585-40a8f252f743\" (UID: \"9bfbc6d3-91fd-41a7-9585-40a8f252f743\") " Oct 30 17:36:55.899883 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:55.899839 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bfbc6d3-91fd-41a7-9585-40a8f252f743-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bfbc6d3-91fd-41a7-9585-40a8f252f743-tmp-dir\") pod \"9bfbc6d3-91fd-41a7-9585-40a8f252f743\" (UID: \"9bfbc6d3-91fd-41a7-9585-40a8f252f743\") " Oct 30 17:36:55.899883 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:55.899874 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75321c5d-8bf8-45c6-8811-81981fe3cad4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75321c5d-8bf8-45c6-8811-81981fe3cad4-model-cache\") pod \"75321c5d-8bf8-45c6-8811-81981fe3cad4\" (UID: \"75321c5d-8bf8-45c6-8811-81981fe3cad4\") " Oct 30 17:36:55.900123 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:55.899960 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75321c5d-8bf8-45c6-8811-81981fe3cad4-kube-api-access-bcwt6\" (UniqueName: \"kubernetes.io/projected/75321c5d-8bf8-45c6-8811-81981fe3cad4-kube-api-access-bcwt6\") pod \"75321c5d-8bf8-45c6-8811-81981fe3cad4\" (UID: \"75321c5d-8bf8-45c6-8811-81981fe3cad4\") " Oct 30 17:36:55.900123 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:55.900024 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bfbc6d3-91fd-41a7-9585-40a8f252f743-home\" (UniqueName: \"kubernetes.io/empty-dir/9bfbc6d3-91fd-41a7-9585-40a8f252f743-home\") pod \"9bfbc6d3-91fd-41a7-9585-40a8f252f743\" (UID: \"9bfbc6d3-91fd-41a7-9585-40a8f252f743\") " Oct 30 17:36:55.900123 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:55.900073 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bfbc6d3-91fd-41a7-9585-40a8f252f743-kube-api-access-g2nxn\" (UniqueName: \"kubernetes.io/projected/9bfbc6d3-91fd-41a7-9585-40a8f252f743-kube-api-access-g2nxn\") pod \"9bfbc6d3-91fd-41a7-9585-40a8f252f743\" (UID: \"9bfbc6d3-91fd-41a7-9585-40a8f252f743\") " Oct 30 17:36:55.900290 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:55.900168 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75321c5d-8bf8-45c6-8811-81981fe3cad4-tls-certs\" (UniqueName: \"kubernetes.io/secret/75321c5d-8bf8-45c6-8811-81981fe3cad4-tls-certs\") pod \"75321c5d-8bf8-45c6-8811-81981fe3cad4\" (UID: \"75321c5d-8bf8-45c6-8811-81981fe3cad4\") " Oct 30 17:36:55.900290 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:55.900220 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bfbc6d3-91fd-41a7-9585-40a8f252f743-tls-certs\" (UniqueName: \"kubernetes.io/secret/9bfbc6d3-91fd-41a7-9585-40a8f252f743-tls-certs\") pod \"9bfbc6d3-91fd-41a7-9585-40a8f252f743\" (UID: \"9bfbc6d3-91fd-41a7-9585-40a8f252f743\") " Oct 30 17:36:55.900290 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:55.900261 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bfbc6d3-91fd-41a7-9585-40a8f252f743-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bfbc6d3-91fd-41a7-9585-40a8f252f743-dshm\") pod \"9bfbc6d3-91fd-41a7-9585-40a8f252f743\" (UID: \"9bfbc6d3-91fd-41a7-9585-40a8f252f743\") " Oct 30 17:36:55.900546 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:55.900368 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75321c5d-8bf8-45c6-8811-81981fe3cad4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75321c5d-8bf8-45c6-8811-81981fe3cad4-tmp-dir\") pod \"75321c5d-8bf8-45c6-8811-81981fe3cad4\" (UID: \"75321c5d-8bf8-45c6-8811-81981fe3cad4\") " Oct 30 17:36:55.900546 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:55.900416 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/246bcd4d-bddf-466e-bf5b-9f18b81fa5fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/246bcd4d-bddf-466e-bf5b-9f18b81fa5fd-plugins-config-volume\") pod \"246bcd4d-bddf-466e-bf5b-9f18b81fa5fd\" (UID: \"246bcd4d-bddf-466e-bf5b-9f18b81fa5fd\") " Oct 30 17:36:55.900546 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:55.900449 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75321c5d-8bf8-45c6-8811-81981fe3cad4-home\" (UniqueName: \"kubernetes.io/empty-dir/75321c5d-8bf8-45c6-8811-81981fe3cad4-home\") pod \"75321c5d-8bf8-45c6-8811-81981fe3cad4\" (UID: \"75321c5d-8bf8-45c6-8811-81981fe3cad4\") " Oct 30 17:36:55.901117 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:55.901088 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75321c5d-8bf8-45c6-8811-81981fe3cad4-home" pod "75321c5d-8bf8-45c6-8811-81981fe3cad4" (UID: "75321c5d-8bf8-45c6-8811-81981fe3cad4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:55.901285 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:55.901268 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bfbc6d3-91fd-41a7-9585-40a8f252f743-home" pod "9bfbc6d3-91fd-41a7-9585-40a8f252f743" (UID: "9bfbc6d3-91fd-41a7-9585-40a8f252f743"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:55.901951 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:55.901921 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bfbc6d3-91fd-41a7-9585-40a8f252f743-model-cache" pod "9bfbc6d3-91fd-41a7-9585-40a8f252f743" (UID: "9bfbc6d3-91fd-41a7-9585-40a8f252f743"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:55.902425 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:55.902074 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75321c5d-8bf8-45c6-8811-81981fe3cad4-tmp-dir" pod "75321c5d-8bf8-45c6-8811-81981fe3cad4" (UID: "75321c5d-8bf8-45c6-8811-81981fe3cad4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:55.902425 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:55.902284 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/246bcd4d-bddf-466e-bf5b-9f18b81fa5fd-plugins-config-volume" pod "246bcd4d-bddf-466e-bf5b-9f18b81fa5fd" (UID: "246bcd4d-bddf-466e-bf5b-9f18b81fa5fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:36:55.902773 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:55.902616 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bfbc6d3-91fd-41a7-9585-40a8f252f743-tmp-dir" pod "9bfbc6d3-91fd-41a7-9585-40a8f252f743" (UID: "9bfbc6d3-91fd-41a7-9585-40a8f252f743"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:55.902843 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:55.902807 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75321c5d-8bf8-45c6-8811-81981fe3cad4-model-cache" pod "75321c5d-8bf8-45c6-8811-81981fe3cad4" (UID: "75321c5d-8bf8-45c6-8811-81981fe3cad4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:55.904103 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:55.903936 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75321c5d-8bf8-45c6-8811-81981fe3cad4-dshm" pod "75321c5d-8bf8-45c6-8811-81981fe3cad4" (UID: "75321c5d-8bf8-45c6-8811-81981fe3cad4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:55.904103 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:55.904054 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bfbc6d3-91fd-41a7-9585-40a8f252f743-kube-api-access-g2nxn" pod "9bfbc6d3-91fd-41a7-9585-40a8f252f743" (UID: "9bfbc6d3-91fd-41a7-9585-40a8f252f743"). InnerVolumeSpecName "kube-api-access-g2nxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:36:55.904103 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:55.904079 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75321c5d-8bf8-45c6-8811-81981fe3cad4-kube-api-access-bcwt6" pod "75321c5d-8bf8-45c6-8811-81981fe3cad4" (UID: "75321c5d-8bf8-45c6-8811-81981fe3cad4"). InnerVolumeSpecName "kube-api-access-bcwt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:36:55.905651 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:55.905626 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bfbc6d3-91fd-41a7-9585-40a8f252f743-tls-certs" pod "9bfbc6d3-91fd-41a7-9585-40a8f252f743" (UID: "9bfbc6d3-91fd-41a7-9585-40a8f252f743"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:36:55.906076 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:55.906055 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bfbc6d3-91fd-41a7-9585-40a8f252f743-dshm" pod "9bfbc6d3-91fd-41a7-9585-40a8f252f743" (UID: "9bfbc6d3-91fd-41a7-9585-40a8f252f743"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:55.908956 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:55.908922 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75321c5d-8bf8-45c6-8811-81981fe3cad4-tls-certs" pod "75321c5d-8bf8-45c6-8811-81981fe3cad4" (UID: "75321c5d-8bf8-45c6-8811-81981fe3cad4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:36:55.909072 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:55.909014 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/246bcd4d-bddf-466e-bf5b-9f18b81fa5fd-kube-api-access-qzwnq" pod "246bcd4d-bddf-466e-bf5b-9f18b81fa5fd" (UID: "246bcd4d-bddf-466e-bf5b-9f18b81fa5fd"). InnerVolumeSpecName "kube-api-access-qzwnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:36:56.001383 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.001352 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/519e4a59-cb60-4a1a-949b-56018ea6b89c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/519e4a59-cb60-4a1a-949b-56018ea6b89c-plugins-config-volume\") pod \"519e4a59-cb60-4a1a-949b-56018ea6b89c\" (UID: \"519e4a59-cb60-4a1a-949b-56018ea6b89c\") " Oct 30 17:36:56.001576 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.001455 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/519e4a59-cb60-4a1a-949b-56018ea6b89c-kube-api-access-h4w72\" (UniqueName: \"kubernetes.io/projected/519e4a59-cb60-4a1a-949b-56018ea6b89c-kube-api-access-h4w72\") pod \"519e4a59-cb60-4a1a-949b-56018ea6b89c\" (UID: \"519e4a59-cb60-4a1a-949b-56018ea6b89c\") " Oct 30 17:36:56.001811 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.001782 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/519e4a59-cb60-4a1a-949b-56018ea6b89c-plugins-config-volume" pod "519e4a59-cb60-4a1a-949b-56018ea6b89c" (UID: "519e4a59-cb60-4a1a-949b-56018ea6b89c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:36:56.002198 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.001992 4522 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bfbc6d3-91fd-41a7-9585-40a8f252f743-model-cache\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:56.002198 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.002007 4522 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bfbc6d3-91fd-41a7-9585-40a8f252f743-tmp-dir\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:56.002198 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.002018 4522 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75321c5d-8bf8-45c6-8811-81981fe3cad4-model-cache\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:56.002198 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.002030 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcwt6\" (UniqueName: \"kubernetes.io/projected/75321c5d-8bf8-45c6-8811-81981fe3cad4-kube-api-access-bcwt6\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:56.002198 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.002051 4522 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bfbc6d3-91fd-41a7-9585-40a8f252f743-home\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:56.002198 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.002070 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2nxn\" (UniqueName: \"kubernetes.io/projected/9bfbc6d3-91fd-41a7-9585-40a8f252f743-kube-api-access-g2nxn\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:56.002198 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.002085 4522 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75321c5d-8bf8-45c6-8811-81981fe3cad4-tls-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:56.002198 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.002096 4522 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bfbc6d3-91fd-41a7-9585-40a8f252f743-tls-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:56.002198 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.002107 4522 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bfbc6d3-91fd-41a7-9585-40a8f252f743-dshm\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:56.002198 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.002118 4522 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75321c5d-8bf8-45c6-8811-81981fe3cad4-tmp-dir\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:56.002198 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.002130 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/246bcd4d-bddf-466e-bf5b-9f18b81fa5fd-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:56.002198 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.002140 4522 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75321c5d-8bf8-45c6-8811-81981fe3cad4-home\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:56.002198 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.002152 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/519e4a59-cb60-4a1a-949b-56018ea6b89c-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:56.002198 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.002164 4522 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75321c5d-8bf8-45c6-8811-81981fe3cad4-dshm\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:56.002198 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.002175 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzwnq\" (UniqueName: \"kubernetes.io/projected/246bcd4d-bddf-466e-bf5b-9f18b81fa5fd-kube-api-access-qzwnq\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:56.003868 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.003838 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/519e4a59-cb60-4a1a-949b-56018ea6b89c-kube-api-access-h4w72" pod "519e4a59-cb60-4a1a-949b-56018ea6b89c" (UID: "519e4a59-cb60-4a1a-949b-56018ea6b89c"). InnerVolumeSpecName "kube-api-access-h4w72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:36:56.034632 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.034555 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a13a40c3-wjvk5" Oct 30 17:36:56.103415 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.103385 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4w72\" (UniqueName: \"kubernetes.io/projected/519e4a59-cb60-4a1a-949b-56018ea6b89c-kube-api-access-h4w72\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:56.265190 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.265157 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daacd80b-7mkw4" Oct 30 17:36:56.265190 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.265175 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daacd80b-7mkw4" event={"ID":"d80fae1e-5c7b-4cb4-9b50-eae3ffc175ff","Type":"ContainerDied","Data":"ca1f2513cc9cf716b2bb9352625a5ca2951944ffccb67bdf9702c5a4c7ec195a"} Oct 30 17:36:56.265525 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.265255 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca1f2513cc9cf716b2bb9352625a5ca2951944ffccb67bdf9702c5a4c7ec195a" Oct 30 17:36:56.267557 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.267531 4522 generic.go:356] "Generic (PLEG): container finished" podID="9bfbc6d3-91fd-41a7-9585-40a8f252f743" containerID="dd8ed889c56d104044f6b5565e22bb2e08a8caf2d9715bbf964f46275c2eac13" exitCode=0 Oct 30 17:36:56.267807 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.267564 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" event={"ID":"9bfbc6d3-91fd-41a7-9585-40a8f252f743","Type":"ContainerDied","Data":"dd8ed889c56d104044f6b5565e22bb2e08a8caf2d9715bbf964f46275c2eac13"} Oct 30 17:36:56.267940 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.267583 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" Oct 30 17:36:56.267940 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.267831 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" event={"ID":"9bfbc6d3-91fd-41a7-9585-40a8f252f743","Type":"ContainerDied","Data":"d4d16356dd88e628a12267bdb7d9ee580a99c6041ec3b662361299ab9c044575"} Oct 30 17:36:56.267940 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.267861 4522 scope.go:122] "RemoveContainer" containerID="dd8ed889c56d104044f6b5565e22bb2e08a8caf2d9715bbf964f46275c2eac13" Oct 30 17:36:56.270259 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.270178 4522 generic.go:356] "Generic (PLEG): container finished" podID="75321c5d-8bf8-45c6-8811-81981fe3cad4" containerID="ec5598f21e37163f7c4135de7a8a510a8fd2793f7c5e1b7e211068c818310f10" exitCode=0 Oct 30 17:36:56.270422 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.270403 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" event={"ID":"75321c5d-8bf8-45c6-8811-81981fe3cad4","Type":"ContainerDied","Data":"ec5598f21e37163f7c4135de7a8a510a8fd2793f7c5e1b7e211068c818310f10"} Oct 30 17:36:56.270513 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.270444 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" event={"ID":"75321c5d-8bf8-45c6-8811-81981fe3cad4","Type":"ContainerDied","Data":"acca2f0754083aafdf2a8ae1c88996ca9db7f96d6d5f14269eeb8277642dbc25"} Oct 30 17:36:56.270893 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.270866 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz" Oct 30 17:36:56.271835 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.271811 4522 generic.go:356] "Generic (PLEG): container finished" podID="519e4a59-cb60-4a1a-949b-56018ea6b89c" containerID="d1e6ff4af5ae577ebeb135947213d7c776786a28925de3b5ce16349c3ea38b23" exitCode=0 Oct 30 17:36:56.271989 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.271970 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-b4f406-6d66f86b4b-bzcnr" Oct 30 17:36:56.272082 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.272014 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-b4f406-6d66f86b4b-bzcnr" event={"ID":"519e4a59-cb60-4a1a-949b-56018ea6b89c","Type":"ContainerDied","Data":"d1e6ff4af5ae577ebeb135947213d7c776786a28925de3b5ce16349c3ea38b23"} Oct 30 17:36:56.272082 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.272066 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-b4f406-6d66f86b4b-bzcnr" event={"ID":"519e4a59-cb60-4a1a-949b-56018ea6b89c","Type":"ContainerDied","Data":"b58f5723477f9f02e998a1c35ff1758e1ecc49f5db5ea2aee90a88acf6a8cfd7"} Oct 30 17:36:56.274277 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.274246 4522 generic.go:356] "Generic (PLEG): container finished" podID="246bcd4d-bddf-466e-bf5b-9f18b81fa5fd" containerID="b64457195612210f19529d5ade53be8ee48a9839e49dc6d0e83feb62c511511d" exitCode=0 Oct 30 17:36:56.277429 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.274338 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b4f406-786855ccd7-ns4j8" event={"ID":"246bcd4d-bddf-466e-bf5b-9f18b81fa5fd","Type":"ContainerDied","Data":"b64457195612210f19529d5ade53be8ee48a9839e49dc6d0e83feb62c511511d"} Oct 30 17:36:56.277429 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.274375 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b4f406-786855ccd7-ns4j8" event={"ID":"246bcd4d-bddf-466e-bf5b-9f18b81fa5fd","Type":"ContainerDied","Data":"10f7f7cbf2ea647a9b80b9166bcde0585f253affbab8e9e741c9810b89a7c21e"} Oct 30 17:36:56.277429 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.274444 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b4f406-786855ccd7-ns4j8" Oct 30 17:36:56.282806 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.282785 4522 scope.go:122] "RemoveContainer" containerID="dd8ed889c56d104044f6b5565e22bb2e08a8caf2d9715bbf964f46275c2eac13" Oct 30 17:36:56.283378 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:36:56.283352 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd8ed889c56d104044f6b5565e22bb2e08a8caf2d9715bbf964f46275c2eac13\": container with ID starting with dd8ed889c56d104044f6b5565e22bb2e08a8caf2d9715bbf964f46275c2eac13 not found: ID does not exist" containerID="dd8ed889c56d104044f6b5565e22bb2e08a8caf2d9715bbf964f46275c2eac13" Oct 30 17:36:56.283455 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.283386 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd8ed889c56d104044f6b5565e22bb2e08a8caf2d9715bbf964f46275c2eac13"} err="failed to get container status \"dd8ed889c56d104044f6b5565e22bb2e08a8caf2d9715bbf964f46275c2eac13\": rpc error: code = NotFound desc = could not find container \"dd8ed889c56d104044f6b5565e22bb2e08a8caf2d9715bbf964f46275c2eac13\": container with ID starting with dd8ed889c56d104044f6b5565e22bb2e08a8caf2d9715bbf964f46275c2eac13 not found: ID does not exist" Oct 30 17:36:56.283455 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.283404 4522 scope.go:122] "RemoveContainer" containerID="ec5598f21e37163f7c4135de7a8a510a8fd2793f7c5e1b7e211068c818310f10" Oct 30 17:36:56.293533 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.293513 4522 scope.go:122] "RemoveContainer" containerID="ec5598f21e37163f7c4135de7a8a510a8fd2793f7c5e1b7e211068c818310f10" Oct 30 17:36:56.293818 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:36:56.293796 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec5598f21e37163f7c4135de7a8a510a8fd2793f7c5e1b7e211068c818310f10\": container with ID starting with ec5598f21e37163f7c4135de7a8a510a8fd2793f7c5e1b7e211068c818310f10 not found: ID does not exist" containerID="ec5598f21e37163f7c4135de7a8a510a8fd2793f7c5e1b7e211068c818310f10" Oct 30 17:36:56.293897 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.293832 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec5598f21e37163f7c4135de7a8a510a8fd2793f7c5e1b7e211068c818310f10"} err="failed to get container status \"ec5598f21e37163f7c4135de7a8a510a8fd2793f7c5e1b7e211068c818310f10\": rpc error: code = NotFound desc = could not find container \"ec5598f21e37163f7c4135de7a8a510a8fd2793f7c5e1b7e211068c818310f10\": container with ID starting with ec5598f21e37163f7c4135de7a8a510a8fd2793f7c5e1b7e211068c818310f10 not found: ID does not exist" Oct 30 17:36:56.293897 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.293859 4522 scope.go:122] "RemoveContainer" containerID="d1e6ff4af5ae577ebeb135947213d7c776786a28925de3b5ce16349c3ea38b23" Oct 30 17:36:56.304379 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.304362 4522 scope.go:122] "RemoveContainer" containerID="d1e6ff4af5ae577ebeb135947213d7c776786a28925de3b5ce16349c3ea38b23" Oct 30 17:36:56.304675 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:36:56.304650 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1e6ff4af5ae577ebeb135947213d7c776786a28925de3b5ce16349c3ea38b23\": container with ID starting with d1e6ff4af5ae577ebeb135947213d7c776786a28925de3b5ce16349c3ea38b23 not found: ID does not exist" containerID="d1e6ff4af5ae577ebeb135947213d7c776786a28925de3b5ce16349c3ea38b23" Oct 30 17:36:56.304727 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.304681 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1e6ff4af5ae577ebeb135947213d7c776786a28925de3b5ce16349c3ea38b23"} err="failed to get container status \"d1e6ff4af5ae577ebeb135947213d7c776786a28925de3b5ce16349c3ea38b23\": rpc error: code = NotFound desc = could not find container \"d1e6ff4af5ae577ebeb135947213d7c776786a28925de3b5ce16349c3ea38b23\": container with ID starting with d1e6ff4af5ae577ebeb135947213d7c776786a28925de3b5ce16349c3ea38b23 not found: ID does not exist" Oct 30 17:36:56.304727 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.304699 4522 scope.go:122] "RemoveContainer" containerID="b64457195612210f19529d5ade53be8ee48a9839e49dc6d0e83feb62c511511d" Oct 30 17:36:56.327457 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.327357 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz"] Oct 30 17:36:56.327531 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.327504 4522 scope.go:122] "RemoveContainer" containerID="b64457195612210f19529d5ade53be8ee48a9839e49dc6d0e83feb62c511511d" Oct 30 17:36:56.327915 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:36:56.327888 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b64457195612210f19529d5ade53be8ee48a9839e49dc6d0e83feb62c511511d\": container with ID starting with b64457195612210f19529d5ade53be8ee48a9839e49dc6d0e83feb62c511511d not found: ID does not exist" containerID="b64457195612210f19529d5ade53be8ee48a9839e49dc6d0e83feb62c511511d" Oct 30 17:36:56.328041 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.327923 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b64457195612210f19529d5ade53be8ee48a9839e49dc6d0e83feb62c511511d"} err="failed to get container status \"b64457195612210f19529d5ade53be8ee48a9839e49dc6d0e83feb62c511511d\": rpc error: code = NotFound desc = could not find container \"b64457195612210f19529d5ade53be8ee48a9839e49dc6d0e83feb62c511511d\": container with ID starting with b64457195612210f19529d5ade53be8ee48a9839e49dc6d0e83feb62c511511d not found: ID does not exist" Oct 30 17:36:56.331571 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.331551 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-b80d83/premium-w3-b80d83-kserve-5d5b58dc78-xw7kz"] Oct 30 17:36:56.349761 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:36:56.349738 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78ab131d_5658_4d13_be06_4943fcdd609b.slice/crio-96d1cc45572a39c9785a45fcda47762502e96b9d4db86f4bb9808dd2c4d1d25d WatchSource:0}: Error finding container 96d1cc45572a39c9785a45fcda47762502e96b9d4db86f4bb9808dd2c4d1d25d: Status 404 returned error can't find the container with id 96d1cc45572a39c9785a45fcda47762502e96b9d4db86f4bb9808dd2c4d1d25d Oct 30 17:36:56.350083 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.350061 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a13a40c3-wjvk5"] Oct 30 17:36:56.387454 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.387431 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-b4f406-6d66f86b4b-bzcnr"] Oct 30 17:36:56.392838 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.392813 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-b4f406-6d66f86b4b-bzcnr"] Oct 30 17:36:56.427806 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.427776 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-b4f406-786855ccd7-ns4j8"] Oct 30 17:36:56.432727 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.432701 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-b4f406-786855ccd7-ns4j8"] Oct 30 17:36:56.470457 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.470419 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s"] Oct 30 17:36:56.473510 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.473485 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s"] Oct 30 17:36:56.524358 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.524324 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm"] Oct 30 17:36:56.524610 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.524585 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm" podUID="40baffd6-35b6-412d-a7bd-f3155ed306db" containerName="istio-proxy" containerID="cri-o://3f880d54c1afb9e761182f68fcd858d1e2da519556f253e8e462e711925cd876" gracePeriod=30 Oct 30 17:36:56.820774 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:56.820728 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-w3-b80d83-kserve-8697d79dd5-rjk8s" podUID="9bfbc6d3-91fd-41a7-9585-40a8f252f743" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:36:57.289448 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:57.289420 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-86f50a-ntln2"] Oct 30 17:36:57.290026 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:57.290011 4522 cpu_manager.go:414] "containerMap: removing container" podUID="246bcd4d-bddf-466e-bf5b-9f18b81fa5fd" containerName="payload-pre-processing" Oct 30 17:36:57.290076 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:57.290030 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="246bcd4d-bddf-466e-bf5b-9f18b81fa5fd" containerName="payload-pre-processing" Oct 30 17:36:57.290076 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:57.290045 4522 cpu_manager.go:414] "containerMap: removing container" podUID="9bfbc6d3-91fd-41a7-9585-40a8f252f743" containerName="main" Oct 30 17:36:57.290076 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:57.290050 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bfbc6d3-91fd-41a7-9585-40a8f252f743" containerName="main" Oct 30 17:36:57.290076 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:57.290073 4522 cpu_manager.go:414] "containerMap: removing container" podUID="519e4a59-cb60-4a1a-949b-56018ea6b89c" containerName="payload-processing" Oct 30 17:36:57.290212 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:57.290078 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="519e4a59-cb60-4a1a-949b-56018ea6b89c" containerName="payload-processing" Oct 30 17:36:57.290212 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:57.290086 4522 cpu_manager.go:414] "containerMap: removing container" podUID="75321c5d-8bf8-45c6-8811-81981fe3cad4" containerName="main" Oct 30 17:36:57.290212 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:57.290091 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75321c5d-8bf8-45c6-8811-81981fe3cad4" containerName="main" Oct 30 17:36:57.290212 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:57.290179 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="246bcd4d-bddf-466e-bf5b-9f18b81fa5fd" containerName="payload-pre-processing" Oct 30 17:36:57.290212 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:57.290189 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bfbc6d3-91fd-41a7-9585-40a8f252f743" containerName="main" Oct 30 17:36:57.290212 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:57.290199 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="519e4a59-cb60-4a1a-949b-56018ea6b89c" containerName="payload-processing" Oct 30 17:36:57.290212 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:57.290205 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="75321c5d-8bf8-45c6-8811-81981fe3cad4" containerName="main" Oct 30 17:36:57.293651 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:57.293630 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-86f50a-ntln2" Oct 30 17:36:57.296257 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:57.296229 4522 generic.go:356] "Generic (PLEG): container finished" podID="78ab131d-5658-4d13-be06-4943fcdd609b" containerID="65cae631033de8d75467565ad77e58739fc7eaeb3908f1a7ae0d4ebc1be0d5b4" exitCode=0 Oct 30 17:36:57.296420 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:57.296273 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a13a40c3-wjvk5" event={"ID":"78ab131d-5658-4d13-be06-4943fcdd609b","Type":"ContainerDied","Data":"65cae631033de8d75467565ad77e58739fc7eaeb3908f1a7ae0d4ebc1be0d5b4"} Oct 30 17:36:57.296420 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:57.296328 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a13a40c3-wjvk5" event={"ID":"78ab131d-5658-4d13-be06-4943fcdd609b","Type":"ContainerStarted","Data":"96d1cc45572a39c9785a45fcda47762502e96b9d4db86f4bb9808dd2c4d1d25d"} Oct 30 17:36:57.314276 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:57.314252 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-86f50a-ntln2"] Oct 30 17:36:57.318784 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:57.318750 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/039ed493-8a78-4672-bc24-4e2b37cee2e4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-86f50a-ntln2\" (UID: \"039ed493-8a78-4672-bc24-4e2b37cee2e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-86f50a-ntln2" Oct 30 17:36:57.318926 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:57.318853 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/039ed493-8a78-4672-bc24-4e2b37cee2e4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-86f50a-ntln2\" (UID: \"039ed493-8a78-4672-bc24-4e2b37cee2e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-86f50a-ntln2" Oct 30 17:36:57.319019 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:57.318997 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdpfz\" (UniqueName: \"kubernetes.io/projected/039ed493-8a78-4672-bc24-4e2b37cee2e4-kube-api-access-zdpfz\") pod \"maas-api-revoke-keys-e2e-worker-w5-86f50a-ntln2\" (UID: \"039ed493-8a78-4672-bc24-4e2b37cee2e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-86f50a-ntln2" Oct 30 17:36:57.420815 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:57.420782 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdpfz\" (UniqueName: \"kubernetes.io/projected/039ed493-8a78-4672-bc24-4e2b37cee2e4-kube-api-access-zdpfz\") pod \"maas-api-revoke-keys-e2e-worker-w5-86f50a-ntln2\" (UID: \"039ed493-8a78-4672-bc24-4e2b37cee2e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-86f50a-ntln2" Oct 30 17:36:57.421010 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:57.420885 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/039ed493-8a78-4672-bc24-4e2b37cee2e4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-86f50a-ntln2\" (UID: \"039ed493-8a78-4672-bc24-4e2b37cee2e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-86f50a-ntln2" Oct 30 17:36:57.421010 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:57.420914 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/039ed493-8a78-4672-bc24-4e2b37cee2e4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-86f50a-ntln2\" (UID: \"039ed493-8a78-4672-bc24-4e2b37cee2e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-86f50a-ntln2" Oct 30 17:36:57.421674 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:57.421649 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/039ed493-8a78-4672-bc24-4e2b37cee2e4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-86f50a-ntln2\" (UID: \"039ed493-8a78-4672-bc24-4e2b37cee2e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-86f50a-ntln2" Oct 30 17:36:57.423479 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:57.423453 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/039ed493-8a78-4672-bc24-4e2b37cee2e4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-86f50a-ntln2\" (UID: \"039ed493-8a78-4672-bc24-4e2b37cee2e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-86f50a-ntln2" Oct 30 17:36:57.446004 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:57.445968 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdpfz\" (UniqueName: \"kubernetes.io/projected/039ed493-8a78-4672-bc24-4e2b37cee2e4-kube-api-access-zdpfz\") pod \"maas-api-revoke-keys-e2e-worker-w5-86f50a-ntln2\" (UID: \"039ed493-8a78-4672-bc24-4e2b37cee2e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-86f50a-ntln2" Oct 30 17:36:57.609818 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:57.609733 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-86f50a-ntln2" Oct 30 17:36:57.918561 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:36:57.918505 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod039ed493_8a78_4672_bc24_4e2b37cee2e4.slice/crio-a88ede9fb5a872ca67e37c50191d97ee4e6281b330161d55a821aa1e9e326ff2 WatchSource:0}: Error finding container a88ede9fb5a872ca67e37c50191d97ee4e6281b330161d55a821aa1e9e326ff2: Status 404 returned error can't find the container with id a88ede9fb5a872ca67e37c50191d97ee4e6281b330161d55a821aa1e9e326ff2 Oct 30 17:36:57.921745 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:57.920454 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-86f50a-ntln2"] Oct 30 17:36:58.146035 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:58.146002 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="246bcd4d-bddf-466e-bf5b-9f18b81fa5fd" path="/var/lib/kubelet/pods/246bcd4d-bddf-466e-bf5b-9f18b81fa5fd/volumes" Oct 30 17:36:58.146505 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:58.146490 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="519e4a59-cb60-4a1a-949b-56018ea6b89c" path="/var/lib/kubelet/pods/519e4a59-cb60-4a1a-949b-56018ea6b89c/volumes" Oct 30 17:36:58.146927 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:58.146909 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75321c5d-8bf8-45c6-8811-81981fe3cad4" path="/var/lib/kubelet/pods/75321c5d-8bf8-45c6-8811-81981fe3cad4/volumes" Oct 30 17:36:58.147686 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:58.147665 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bfbc6d3-91fd-41a7-9585-40a8f252f743" path="/var/lib/kubelet/pods/9bfbc6d3-91fd-41a7-9585-40a8f252f743/volumes" Oct 30 17:36:58.306376 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:58.306349 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-86f50a-ntln2" event={"ID":"039ed493-8a78-4672-bc24-4e2b37cee2e4","Type":"ContainerStarted","Data":"a88ede9fb5a872ca67e37c50191d97ee4e6281b330161d55a821aa1e9e326ff2"} Oct 30 17:36:58.426925 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:58.426897 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a13a40c3-wjvk5" Oct 30 17:36:58.535168 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:58.535130 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78ab131d-5658-4d13-be06-4943fcdd609b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78ab131d-5658-4d13-be06-4943fcdd609b-maas-api-xks-ca\") pod \"78ab131d-5658-4d13-be06-4943fcdd609b\" (UID: \"78ab131d-5658-4d13-be06-4943fcdd609b\") " Oct 30 17:36:58.535734 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:58.535214 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78ab131d-5658-4d13-be06-4943fcdd609b-kube-api-access-srqcv\" (UniqueName: \"kubernetes.io/projected/78ab131d-5658-4d13-be06-4943fcdd609b-kube-api-access-srqcv\") pod \"78ab131d-5658-4d13-be06-4943fcdd609b\" (UID: \"78ab131d-5658-4d13-be06-4943fcdd609b\") " Oct 30 17:36:58.535734 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:58.535336 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78ab131d-5658-4d13-be06-4943fcdd609b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78ab131d-5658-4d13-be06-4943fcdd609b-maas-api-openshift-ca\") pod \"78ab131d-5658-4d13-be06-4943fcdd609b\" (UID: \"78ab131d-5658-4d13-be06-4943fcdd609b\") " Oct 30 17:36:58.536019 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:58.535986 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78ab131d-5658-4d13-be06-4943fcdd609b-maas-api-openshift-ca" pod "78ab131d-5658-4d13-be06-4943fcdd609b" (UID: "78ab131d-5658-4d13-be06-4943fcdd609b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:36:58.537573 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:58.537546 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78ab131d-5658-4d13-be06-4943fcdd609b-maas-api-xks-ca" pod "78ab131d-5658-4d13-be06-4943fcdd609b" (UID: "78ab131d-5658-4d13-be06-4943fcdd609b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:36:58.537670 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:58.537601 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78ab131d-5658-4d13-be06-4943fcdd609b-kube-api-access-srqcv" pod "78ab131d-5658-4d13-be06-4943fcdd609b" (UID: "78ab131d-5658-4d13-be06-4943fcdd609b"). InnerVolumeSpecName "kube-api-access-srqcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:36:58.637257 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:58.637167 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78ab131d-5658-4d13-be06-4943fcdd609b-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:58.637257 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:58.637202 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srqcv\" (UniqueName: \"kubernetes.io/projected/78ab131d-5658-4d13-be06-4943fcdd609b-kube-api-access-srqcv\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:58.637257 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:58.637246 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78ab131d-5658-4d13-be06-4943fcdd609b-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:36:58.653927 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:58.653888 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-04248e-6xkgh"] Oct 30 17:36:58.671112 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:58.671078 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-04248e-6xkgh"] Oct 30 17:36:59.315376 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:59.315333 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a13a40c3-wjvk5" event={"ID":"78ab131d-5658-4d13-be06-4943fcdd609b","Type":"ContainerDied","Data":"96d1cc45572a39c9785a45fcda47762502e96b9d4db86f4bb9808dd2c4d1d25d"} Oct 30 17:36:59.315562 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:59.315397 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96d1cc45572a39c9785a45fcda47762502e96b9d4db86f4bb9808dd2c4d1d25d" Oct 30 17:36:59.315562 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:59.315340 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a13a40c3-wjvk5" Oct 30 17:36:59.317170 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:59.317147 4522 generic.go:356] "Generic (PLEG): container finished" podID="039ed493-8a78-4672-bc24-4e2b37cee2e4" containerID="a73ec6e06360f6b4f2b9899b33b1b598dbb3ab6e751daf14c3c055b5a1aee4f5" exitCode=0 Oct 30 17:36:59.317291 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:59.317181 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-86f50a-ntln2" event={"ID":"039ed493-8a78-4672-bc24-4e2b37cee2e4","Type":"ContainerDied","Data":"a73ec6e06360f6b4f2b9899b33b1b598dbb3ab6e751daf14c3c055b5a1aee4f5"} Oct 30 17:36:59.984423 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:36:59.984395 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-82243b-7f4f9546b6-mg8hr" Oct 30 17:37:00.004035 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:00.004008 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-82243b-6794bbb67-58ssq" Oct 30 17:37:00.054822 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:00.054792 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5333c134-0e1a-4096-a6d0-6d54f2373e28-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5333c134-0e1a-4096-a6d0-6d54f2373e28-plugins-config-volume\") pod \"5333c134-0e1a-4096-a6d0-6d54f2373e28\" (UID: \"5333c134-0e1a-4096-a6d0-6d54f2373e28\") " Oct 30 17:37:00.055028 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:00.054879 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5333c134-0e1a-4096-a6d0-6d54f2373e28-kube-api-access-8l2tw\" (UniqueName: \"kubernetes.io/projected/5333c134-0e1a-4096-a6d0-6d54f2373e28-kube-api-access-8l2tw\") pod \"5333c134-0e1a-4096-a6d0-6d54f2373e28\" (UID: \"5333c134-0e1a-4096-a6d0-6d54f2373e28\") " Oct 30 17:37:00.055028 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:00.054974 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bdabfbc-c999-479c-a44e-d4949bd3c73d-kube-api-access-m6jtl\" (UniqueName: \"kubernetes.io/projected/5bdabfbc-c999-479c-a44e-d4949bd3c73d-kube-api-access-m6jtl\") pod \"5bdabfbc-c999-479c-a44e-d4949bd3c73d\" (UID: \"5bdabfbc-c999-479c-a44e-d4949bd3c73d\") " Oct 30 17:37:00.055174 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:00.055038 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bdabfbc-c999-479c-a44e-d4949bd3c73d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bdabfbc-c999-479c-a44e-d4949bd3c73d-plugins-config-volume\") pod \"5bdabfbc-c999-479c-a44e-d4949bd3c73d\" (UID: \"5bdabfbc-c999-479c-a44e-d4949bd3c73d\") " Oct 30 17:37:00.055427 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:00.055244 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5333c134-0e1a-4096-a6d0-6d54f2373e28-plugins-config-volume" pod "5333c134-0e1a-4096-a6d0-6d54f2373e28" (UID: "5333c134-0e1a-4096-a6d0-6d54f2373e28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:37:00.055588 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:00.055446 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bdabfbc-c999-479c-a44e-d4949bd3c73d-plugins-config-volume" pod "5bdabfbc-c999-479c-a44e-d4949bd3c73d" (UID: "5bdabfbc-c999-479c-a44e-d4949bd3c73d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:37:00.055720 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:00.055696 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bdabfbc-c999-479c-a44e-d4949bd3c73d-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:37:00.055840 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:00.055723 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5333c134-0e1a-4096-a6d0-6d54f2373e28-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:37:00.057396 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:00.057373 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5333c134-0e1a-4096-a6d0-6d54f2373e28-kube-api-access-8l2tw" pod "5333c134-0e1a-4096-a6d0-6d54f2373e28" (UID: "5333c134-0e1a-4096-a6d0-6d54f2373e28"). InnerVolumeSpecName "kube-api-access-8l2tw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:37:00.057500 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:00.057485 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bdabfbc-c999-479c-a44e-d4949bd3c73d-kube-api-access-m6jtl" pod "5bdabfbc-c999-479c-a44e-d4949bd3c73d" (UID: "5bdabfbc-c999-479c-a44e-d4949bd3c73d"). InnerVolumeSpecName "kube-api-access-m6jtl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:37:00.154161 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:00.154125 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b9ebedf-4812-4ebf-82cd-c80d06d1886a" path="/var/lib/kubelet/pods/6b9ebedf-4812-4ebf-82cd-c80d06d1886a/volumes" Oct 30 17:37:00.156297 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:00.156268 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l2tw\" (UniqueName: \"kubernetes.io/projected/5333c134-0e1a-4096-a6d0-6d54f2373e28-kube-api-access-8l2tw\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:37:00.156297 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:00.156296 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6jtl\" (UniqueName: \"kubernetes.io/projected/5bdabfbc-c999-479c-a44e-d4949bd3c73d-kube-api-access-m6jtl\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:37:00.326134 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:00.326097 4522 generic.go:356] "Generic (PLEG): container finished" podID="5333c134-0e1a-4096-a6d0-6d54f2373e28" containerID="41112eefe8cb88efafdc400f68940bf161a6d97bb4d068ea8c498160480ed412" exitCode=0 Oct 30 17:37:00.326355 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:00.326156 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-82243b-7f4f9546b6-mg8hr" Oct 30 17:37:00.326355 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:00.326182 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-82243b-7f4f9546b6-mg8hr" event={"ID":"5333c134-0e1a-4096-a6d0-6d54f2373e28","Type":"ContainerDied","Data":"41112eefe8cb88efafdc400f68940bf161a6d97bb4d068ea8c498160480ed412"} Oct 30 17:37:00.326355 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:00.326245 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-82243b-7f4f9546b6-mg8hr" event={"ID":"5333c134-0e1a-4096-a6d0-6d54f2373e28","Type":"ContainerDied","Data":"6df712a7483e3401ff802366868309605064ccbc8dfc88fbea0c5fa518b34b2c"} Oct 30 17:37:00.326355 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:00.326275 4522 scope.go:122] "RemoveContainer" containerID="41112eefe8cb88efafdc400f68940bf161a6d97bb4d068ea8c498160480ed412" Oct 30 17:37:00.327669 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:00.327646 4522 generic.go:356] "Generic (PLEG): container finished" podID="5bdabfbc-c999-479c-a44e-d4949bd3c73d" containerID="1f71acee2445475d0860e021cf9070236c0d3fed188889c790e20f62d175672d" exitCode=0 Oct 30 17:37:00.327776 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:00.327703 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-82243b-6794bbb67-58ssq" Oct 30 17:37:00.327776 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:00.327738 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-82243b-6794bbb67-58ssq" event={"ID":"5bdabfbc-c999-479c-a44e-d4949bd3c73d","Type":"ContainerDied","Data":"1f71acee2445475d0860e021cf9070236c0d3fed188889c790e20f62d175672d"} Oct 30 17:37:00.327886 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:00.327788 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-82243b-6794bbb67-58ssq" event={"ID":"5bdabfbc-c999-479c-a44e-d4949bd3c73d","Type":"ContainerDied","Data":"f2c04590fc9684b79f217f9788e7ecac19d14ac60f2c7478bfb1d04488f31ac1"} Oct 30 17:37:00.338450 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:00.338429 4522 scope.go:122] "RemoveContainer" containerID="41112eefe8cb88efafdc400f68940bf161a6d97bb4d068ea8c498160480ed412" Oct 30 17:37:00.338733 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:37:00.338712 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41112eefe8cb88efafdc400f68940bf161a6d97bb4d068ea8c498160480ed412\": container with ID starting with 41112eefe8cb88efafdc400f68940bf161a6d97bb4d068ea8c498160480ed412 not found: ID does not exist" containerID="41112eefe8cb88efafdc400f68940bf161a6d97bb4d068ea8c498160480ed412" Oct 30 17:37:00.338791 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:00.338741 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41112eefe8cb88efafdc400f68940bf161a6d97bb4d068ea8c498160480ed412"} err="failed to get container status \"41112eefe8cb88efafdc400f68940bf161a6d97bb4d068ea8c498160480ed412\": rpc error: code = NotFound desc = could not find container \"41112eefe8cb88efafdc400f68940bf161a6d97bb4d068ea8c498160480ed412\": container with ID starting with 41112eefe8cb88efafdc400f68940bf161a6d97bb4d068ea8c498160480ed412 not found: ID does not exist" Oct 30 17:37:00.338791 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:00.338763 4522 scope.go:122] "RemoveContainer" containerID="1f71acee2445475d0860e021cf9070236c0d3fed188889c790e20f62d175672d" Oct 30 17:37:00.352665 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:00.352643 4522 scope.go:122] "RemoveContainer" containerID="1f71acee2445475d0860e021cf9070236c0d3fed188889c790e20f62d175672d" Oct 30 17:37:00.352970 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:37:00.352946 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f71acee2445475d0860e021cf9070236c0d3fed188889c790e20f62d175672d\": container with ID starting with 1f71acee2445475d0860e021cf9070236c0d3fed188889c790e20f62d175672d not found: ID does not exist" containerID="1f71acee2445475d0860e021cf9070236c0d3fed188889c790e20f62d175672d" Oct 30 17:37:00.353055 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:00.352982 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f71acee2445475d0860e021cf9070236c0d3fed188889c790e20f62d175672d"} err="failed to get container status \"1f71acee2445475d0860e021cf9070236c0d3fed188889c790e20f62d175672d\": rpc error: code = NotFound desc = could not find container \"1f71acee2445475d0860e021cf9070236c0d3fed188889c790e20f62d175672d\": container with ID starting with 1f71acee2445475d0860e021cf9070236c0d3fed188889c790e20f62d175672d not found: ID does not exist" Oct 30 17:37:00.384191 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:00.384159 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-82243b-6794bbb67-58ssq"] Oct 30 17:37:00.398477 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:00.398452 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-82243b-6794bbb67-58ssq"] Oct 30 17:37:00.439174 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:00.439079 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-82243b-7f4f9546b6-mg8hr"] Oct 30 17:37:00.444446 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:00.444414 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-82243b-7f4f9546b6-mg8hr"] Oct 30 17:37:00.456482 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:00.456455 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-86f50a-ntln2" Oct 30 17:37:00.559906 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:00.559876 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/039ed493-8a78-4672-bc24-4e2b37cee2e4-kube-api-access-zdpfz\" (UniqueName: \"kubernetes.io/projected/039ed493-8a78-4672-bc24-4e2b37cee2e4-kube-api-access-zdpfz\") pod \"039ed493-8a78-4672-bc24-4e2b37cee2e4\" (UID: \"039ed493-8a78-4672-bc24-4e2b37cee2e4\") " Oct 30 17:37:00.560089 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:00.559965 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/039ed493-8a78-4672-bc24-4e2b37cee2e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/039ed493-8a78-4672-bc24-4e2b37cee2e4-maas-api-xks-ca\") pod \"039ed493-8a78-4672-bc24-4e2b37cee2e4\" (UID: \"039ed493-8a78-4672-bc24-4e2b37cee2e4\") " Oct 30 17:37:00.560089 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:00.560053 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/039ed493-8a78-4672-bc24-4e2b37cee2e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/039ed493-8a78-4672-bc24-4e2b37cee2e4-maas-api-openshift-ca\") pod \"039ed493-8a78-4672-bc24-4e2b37cee2e4\" (UID: \"039ed493-8a78-4672-bc24-4e2b37cee2e4\") " Oct 30 17:37:00.560435 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:00.560413 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/039ed493-8a78-4672-bc24-4e2b37cee2e4-maas-api-openshift-ca" pod "039ed493-8a78-4672-bc24-4e2b37cee2e4" (UID: "039ed493-8a78-4672-bc24-4e2b37cee2e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:37:00.560613 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:00.560596 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/039ed493-8a78-4672-bc24-4e2b37cee2e4-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:37:00.562043 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:00.562022 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/039ed493-8a78-4672-bc24-4e2b37cee2e4-maas-api-xks-ca" pod "039ed493-8a78-4672-bc24-4e2b37cee2e4" (UID: "039ed493-8a78-4672-bc24-4e2b37cee2e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:37:00.562111 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:00.562092 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/039ed493-8a78-4672-bc24-4e2b37cee2e4-kube-api-access-zdpfz" pod "039ed493-8a78-4672-bc24-4e2b37cee2e4" (UID: "039ed493-8a78-4672-bc24-4e2b37cee2e4"). InnerVolumeSpecName "kube-api-access-zdpfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:37:00.662295 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:00.662257 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdpfz\" (UniqueName: \"kubernetes.io/projected/039ed493-8a78-4672-bc24-4e2b37cee2e4-kube-api-access-zdpfz\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:37:00.662295 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:00.662289 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/039ed493-8a78-4672-bc24-4e2b37cee2e4-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:37:01.338726 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:01.338688 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-86f50a-ntln2" event={"ID":"039ed493-8a78-4672-bc24-4e2b37cee2e4","Type":"ContainerDied","Data":"a88ede9fb5a872ca67e37c50191d97ee4e6281b330161d55a821aa1e9e326ff2"} Oct 30 17:37:01.339133 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:01.338741 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a88ede9fb5a872ca67e37c50191d97ee4e6281b330161d55a821aa1e9e326ff2" Oct 30 17:37:01.339133 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:01.338708 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-86f50a-ntln2" Oct 30 17:37:01.798766 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:01.798740 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm" Oct 30 17:37:01.877202 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:01.877172 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40baffd6-35b6-412d-a7bd-f3155ed306db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40baffd6-35b6-412d-a7bd-f3155ed306db-credential-socket\") pod \"40baffd6-35b6-412d-a7bd-f3155ed306db\" (UID: \"40baffd6-35b6-412d-a7bd-f3155ed306db\") " Oct 30 17:37:01.877454 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:01.877227 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40baffd6-35b6-412d-a7bd-f3155ed306db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40baffd6-35b6-412d-a7bd-f3155ed306db-istio-envoy\") pod \"40baffd6-35b6-412d-a7bd-f3155ed306db\" (UID: \"40baffd6-35b6-412d-a7bd-f3155ed306db\") " Oct 30 17:37:01.877454 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:01.877249 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40baffd6-35b6-412d-a7bd-f3155ed306db-kube-api-access-pwwvq\" (UniqueName: \"kubernetes.io/projected/40baffd6-35b6-412d-a7bd-f3155ed306db-kube-api-access-pwwvq\") pod \"40baffd6-35b6-412d-a7bd-f3155ed306db\" (UID: \"40baffd6-35b6-412d-a7bd-f3155ed306db\") " Oct 30 17:37:01.877454 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:01.877382 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40baffd6-35b6-412d-a7bd-f3155ed306db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40baffd6-35b6-412d-a7bd-f3155ed306db-workload-certs\") pod \"40baffd6-35b6-412d-a7bd-f3155ed306db\" (UID: \"40baffd6-35b6-412d-a7bd-f3155ed306db\") " Oct 30 17:37:01.877633 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:01.877495 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40baffd6-35b6-412d-a7bd-f3155ed306db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40baffd6-35b6-412d-a7bd-f3155ed306db-istio-data\") pod \"40baffd6-35b6-412d-a7bd-f3155ed306db\" (UID: \"40baffd6-35b6-412d-a7bd-f3155ed306db\") " Oct 30 17:37:01.877633 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:01.877544 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40baffd6-35b6-412d-a7bd-f3155ed306db-credential-socket" pod "40baffd6-35b6-412d-a7bd-f3155ed306db" (UID: "40baffd6-35b6-412d-a7bd-f3155ed306db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:37:01.877633 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:01.877588 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40baffd6-35b6-412d-a7bd-f3155ed306db-istio-token\" (UniqueName: \"kubernetes.io/projected/40baffd6-35b6-412d-a7bd-f3155ed306db-istio-token\") pod \"40baffd6-35b6-412d-a7bd-f3155ed306db\" (UID: \"40baffd6-35b6-412d-a7bd-f3155ed306db\") " Oct 30 17:37:01.877633 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:01.877619 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40baffd6-35b6-412d-a7bd-f3155ed306db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40baffd6-35b6-412d-a7bd-f3155ed306db-istiod-ca-cert\") pod \"40baffd6-35b6-412d-a7bd-f3155ed306db\" (UID: \"40baffd6-35b6-412d-a7bd-f3155ed306db\") " Oct 30 17:37:01.877913 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:01.877655 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40baffd6-35b6-412d-a7bd-f3155ed306db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40baffd6-35b6-412d-a7bd-f3155ed306db-workload-socket\") pod \"40baffd6-35b6-412d-a7bd-f3155ed306db\" (UID: \"40baffd6-35b6-412d-a7bd-f3155ed306db\") " Oct 30 17:37:01.877913 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:01.877671 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40baffd6-35b6-412d-a7bd-f3155ed306db-workload-certs" pod "40baffd6-35b6-412d-a7bd-f3155ed306db" (UID: "40baffd6-35b6-412d-a7bd-f3155ed306db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:37:01.877913 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:01.877684 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/40baffd6-35b6-412d-a7bd-f3155ed306db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40baffd6-35b6-412d-a7bd-f3155ed306db-istio-podinfo\") pod \"40baffd6-35b6-412d-a7bd-f3155ed306db\" (UID: \"40baffd6-35b6-412d-a7bd-f3155ed306db\") " Oct 30 17:37:01.877913 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:01.877778 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40baffd6-35b6-412d-a7bd-f3155ed306db-istio-data" pod "40baffd6-35b6-412d-a7bd-f3155ed306db" (UID: "40baffd6-35b6-412d-a7bd-f3155ed306db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:37:01.878250 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:01.878220 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40baffd6-35b6-412d-a7bd-f3155ed306db-istiod-ca-cert" pod "40baffd6-35b6-412d-a7bd-f3155ed306db" (UID: "40baffd6-35b6-412d-a7bd-f3155ed306db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:37:01.878522 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:01.878440 4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40baffd6-35b6-412d-a7bd-f3155ed306db-credential-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:37:01.878522 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:01.878476 4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40baffd6-35b6-412d-a7bd-f3155ed306db-workload-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:37:01.878522 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:01.878490 4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40baffd6-35b6-412d-a7bd-f3155ed306db-istio-data\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:37:01.878522 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:01.878443 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40baffd6-35b6-412d-a7bd-f3155ed306db-workload-socket" pod "40baffd6-35b6-412d-a7bd-f3155ed306db" (UID: "40baffd6-35b6-412d-a7bd-f3155ed306db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:37:01.880608 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:01.880575 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40baffd6-35b6-412d-a7bd-f3155ed306db-kube-api-access-pwwvq" pod "40baffd6-35b6-412d-a7bd-f3155ed306db" (UID: "40baffd6-35b6-412d-a7bd-f3155ed306db"). InnerVolumeSpecName "kube-api-access-pwwvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:37:01.880708 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:01.880649 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40baffd6-35b6-412d-a7bd-f3155ed306db-istio-podinfo" pod "40baffd6-35b6-412d-a7bd-f3155ed306db" (UID: "40baffd6-35b6-412d-a7bd-f3155ed306db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:37:01.880944 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:01.880926 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40baffd6-35b6-412d-a7bd-f3155ed306db-istio-envoy" pod "40baffd6-35b6-412d-a7bd-f3155ed306db" (UID: "40baffd6-35b6-412d-a7bd-f3155ed306db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:37:01.884460 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:01.884427 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40baffd6-35b6-412d-a7bd-f3155ed306db-istio-token" pod "40baffd6-35b6-412d-a7bd-f3155ed306db" (UID: "40baffd6-35b6-412d-a7bd-f3155ed306db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:37:01.980064 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:01.980029 4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40baffd6-35b6-412d-a7bd-f3155ed306db-istio-token\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:37:01.980064 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:01.980059 4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40baffd6-35b6-412d-a7bd-f3155ed306db-istiod-ca-cert\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:37:01.980064 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:01.980069 4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40baffd6-35b6-412d-a7bd-f3155ed306db-workload-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:37:01.980301 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:01.980078 4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40baffd6-35b6-412d-a7bd-f3155ed306db-istio-podinfo\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:37:01.980301 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:01.980088 4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40baffd6-35b6-412d-a7bd-f3155ed306db-istio-envoy\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:37:01.980301 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:01.980096 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwwvq\" (UniqueName: \"kubernetes.io/projected/40baffd6-35b6-412d-a7bd-f3155ed306db-kube-api-access-pwwvq\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:37:02.036596 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:02.036563 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9"] Oct 30 17:37:02.114761 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:02.114725 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-b80d83-58f8df6ff4-xt4dg" podUID="bfdddc9b-c6c6-4cb1-bdd6-1634e76c4825" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:37:02.145566 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:02.145484 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5333c134-0e1a-4096-a6d0-6d54f2373e28" path="/var/lib/kubelet/pods/5333c134-0e1a-4096-a6d0-6d54f2373e28/volumes" Oct 30 17:37:02.145912 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:02.145899 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bdabfbc-c999-479c-a44e-d4949bd3c73d" path="/var/lib/kubelet/pods/5bdabfbc-c999-479c-a44e-d4949bd3c73d/volumes" Oct 30 17:37:02.215625 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:02.215595 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-031223-547d957db6-v6n6p" Oct 30 17:37:02.215822 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:02.215780 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-031223-867764c779-6mp6s" Oct 30 17:37:02.351175 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:02.351147 4522 generic.go:356] "Generic (PLEG): container finished" podID="40baffd6-35b6-412d-a7bd-f3155ed306db" containerID="3f880d54c1afb9e761182f68fcd858d1e2da519556f253e8e462e711925cd876" exitCode=0 Oct 30 17:37:02.351647 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:02.351199 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm" Oct 30 17:37:02.351647 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:02.351235 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm" event={"ID":"40baffd6-35b6-412d-a7bd-f3155ed306db","Type":"ContainerDied","Data":"3f880d54c1afb9e761182f68fcd858d1e2da519556f253e8e462e711925cd876"} Oct 30 17:37:02.351647 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:02.351322 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm" event={"ID":"40baffd6-35b6-412d-a7bd-f3155ed306db","Type":"ContainerDied","Data":"3e26e774e771a4a9cbdae1d769844d202187b3583f81ad3661ed9b499af96c72"} Oct 30 17:37:02.351647 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:02.351343 4522 scope.go:122] "RemoveContainer" containerID="3f880d54c1afb9e761182f68fcd858d1e2da519556f253e8e462e711925cd876" Oct 30 17:37:02.363260 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:02.363240 4522 scope.go:122] "RemoveContainer" containerID="674dcc578e0637f40a21dfbc09d9216f80795539a9e7fd2fd1c29bf09d61fc05" Oct 30 17:37:02.373572 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:02.373553 4522 scope.go:122] "RemoveContainer" containerID="3f880d54c1afb9e761182f68fcd858d1e2da519556f253e8e462e711925cd876" Oct 30 17:37:02.373867 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:37:02.373847 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f880d54c1afb9e761182f68fcd858d1e2da519556f253e8e462e711925cd876\": container with ID starting with 3f880d54c1afb9e761182f68fcd858d1e2da519556f253e8e462e711925cd876 not found: ID does not exist" containerID="3f880d54c1afb9e761182f68fcd858d1e2da519556f253e8e462e711925cd876" Oct 30 17:37:02.373950 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:02.373881 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f880d54c1afb9e761182f68fcd858d1e2da519556f253e8e462e711925cd876"} err="failed to get container status \"3f880d54c1afb9e761182f68fcd858d1e2da519556f253e8e462e711925cd876\": rpc error: code = NotFound desc = could not find container \"3f880d54c1afb9e761182f68fcd858d1e2da519556f253e8e462e711925cd876\": container with ID starting with 3f880d54c1afb9e761182f68fcd858d1e2da519556f253e8e462e711925cd876 not found: ID does not exist" Oct 30 17:37:02.373950 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:02.373908 4522 scope.go:122] "RemoveContainer" containerID="674dcc578e0637f40a21dfbc09d9216f80795539a9e7fd2fd1c29bf09d61fc05" Oct 30 17:37:02.374202 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:37:02.374183 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"674dcc578e0637f40a21dfbc09d9216f80795539a9e7fd2fd1c29bf09d61fc05\": container with ID starting with 674dcc578e0637f40a21dfbc09d9216f80795539a9e7fd2fd1c29bf09d61fc05 not found: ID does not exist" containerID="674dcc578e0637f40a21dfbc09d9216f80795539a9e7fd2fd1c29bf09d61fc05" Oct 30 17:37:02.374252 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:02.374210 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"674dcc578e0637f40a21dfbc09d9216f80795539a9e7fd2fd1c29bf09d61fc05"} err="failed to get container status \"674dcc578e0637f40a21dfbc09d9216f80795539a9e7fd2fd1c29bf09d61fc05\": rpc error: code = NotFound desc = could not find container \"674dcc578e0637f40a21dfbc09d9216f80795539a9e7fd2fd1c29bf09d61fc05\": container with ID starting with 674dcc578e0637f40a21dfbc09d9216f80795539a9e7fd2fd1c29bf09d61fc05 not found: ID does not exist" Oct 30 17:37:02.409199 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:02.409110 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm"] Oct 30 17:37:02.413196 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:02.413168 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-9qwcm"] Oct 30 17:37:04.148091 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:04.148044 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40baffd6-35b6-412d-a7bd-f3155ed306db" path="/var/lib/kubelet/pods/40baffd6-35b6-412d-a7bd-f3155ed306db/volumes" Oct 30 17:37:04.273567 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:04.273530 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a949e7-rm5bd"] Oct 30 17:37:04.274123 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:04.274104 4522 cpu_manager.go:414] "containerMap: removing container" podUID="40baffd6-35b6-412d-a7bd-f3155ed306db" containerName="istio-proxy" Oct 30 17:37:04.274222 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:04.274125 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40baffd6-35b6-412d-a7bd-f3155ed306db" containerName="istio-proxy" Oct 30 17:37:04.274222 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:04.274142 4522 cpu_manager.go:414] "containerMap: removing container" podUID="40baffd6-35b6-412d-a7bd-f3155ed306db" containerName="istio-proxy" Oct 30 17:37:04.274222 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:04.274150 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40baffd6-35b6-412d-a7bd-f3155ed306db" containerName="istio-proxy" Oct 30 17:37:04.274222 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:04.274197 4522 cpu_manager.go:414] "containerMap: removing container" podUID="039ed493-8a78-4672-bc24-4e2b37cee2e4" containerName="revoke-keys" Oct 30 17:37:04.274222 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:04.274205 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="039ed493-8a78-4672-bc24-4e2b37cee2e4" containerName="revoke-keys" Oct 30 17:37:04.274222 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:04.274219 4522 cpu_manager.go:414] "containerMap: removing container" podUID="5333c134-0e1a-4096-a6d0-6d54f2373e28" containerName="payload-processing" Oct 30 17:37:04.274566 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:04.274227 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5333c134-0e1a-4096-a6d0-6d54f2373e28" containerName="payload-processing" Oct 30 17:37:04.274566 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:04.274253 4522 cpu_manager.go:414] "containerMap: removing container" podUID="78ab131d-5658-4d13-be06-4943fcdd609b" containerName="revoke-keys" Oct 30 17:37:04.274566 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:04.274261 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78ab131d-5658-4d13-be06-4943fcdd609b" containerName="revoke-keys" Oct 30 17:37:04.274566 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:04.274280 4522 cpu_manager.go:414] "containerMap: removing container" podUID="5bdabfbc-c999-479c-a44e-d4949bd3c73d" containerName="payload-pre-processing" Oct 30 17:37:04.274566 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:04.274288 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bdabfbc-c999-479c-a44e-d4949bd3c73d" containerName="payload-pre-processing" Oct 30 17:37:04.274566 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:04.274430 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="40baffd6-35b6-412d-a7bd-f3155ed306db" containerName="istio-proxy" Oct 30 17:37:04.274566 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:04.274452 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="039ed493-8a78-4672-bc24-4e2b37cee2e4" containerName="revoke-keys" Oct 30 17:37:04.274566 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:04.274462 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="5333c134-0e1a-4096-a6d0-6d54f2373e28" containerName="payload-processing" Oct 30 17:37:04.274566 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:04.274472 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="78ab131d-5658-4d13-be06-4943fcdd609b" containerName="revoke-keys" Oct 30 17:37:04.274566 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:04.274482 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bdabfbc-c999-479c-a44e-d4949bd3c73d" containerName="payload-pre-processing" Oct 30 17:37:04.281792 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:04.281768 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a949e7-rm5bd" Oct 30 17:37:04.298777 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:04.298742 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a949e7-rm5bd"] Oct 30 17:37:04.307672 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:04.307641 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:37:04.307905 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:04.307739 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:37:04.414129 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:04.414045 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d13dde97-6b5f-4f2e-bf24-6ba2e2a29109-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a949e7-rm5bd\" (UID: \"d13dde97-6b5f-4f2e-bf24-6ba2e2a29109\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a949e7-rm5bd" Oct 30 17:37:04.414129 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:04.414115 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxq47\" (UniqueName: \"kubernetes.io/projected/d13dde97-6b5f-4f2e-bf24-6ba2e2a29109-kube-api-access-xxq47\") pod \"maas-api-revoke-keys-e2e-ipp-a-a949e7-rm5bd\" (UID: \"d13dde97-6b5f-4f2e-bf24-6ba2e2a29109\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a949e7-rm5bd" Oct 30 17:37:04.414365 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:04.414209 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d13dde97-6b5f-4f2e-bf24-6ba2e2a29109-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a949e7-rm5bd\" (UID: \"d13dde97-6b5f-4f2e-bf24-6ba2e2a29109\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a949e7-rm5bd" Oct 30 17:37:04.514989 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:04.514950 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d13dde97-6b5f-4f2e-bf24-6ba2e2a29109-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a949e7-rm5bd\" (UID: \"d13dde97-6b5f-4f2e-bf24-6ba2e2a29109\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a949e7-rm5bd" Oct 30 17:37:04.515186 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:04.515022 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxq47\" (UniqueName: \"kubernetes.io/projected/d13dde97-6b5f-4f2e-bf24-6ba2e2a29109-kube-api-access-xxq47\") pod \"maas-api-revoke-keys-e2e-ipp-a-a949e7-rm5bd\" (UID: \"d13dde97-6b5f-4f2e-bf24-6ba2e2a29109\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a949e7-rm5bd" Oct 30 17:37:04.515186 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:04.515058 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d13dde97-6b5f-4f2e-bf24-6ba2e2a29109-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a949e7-rm5bd\" (UID: \"d13dde97-6b5f-4f2e-bf24-6ba2e2a29109\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a949e7-rm5bd" Oct 30 17:37:04.515869 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:04.515846 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d13dde97-6b5f-4f2e-bf24-6ba2e2a29109-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a949e7-rm5bd\" (UID: \"d13dde97-6b5f-4f2e-bf24-6ba2e2a29109\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a949e7-rm5bd" Oct 30 17:37:04.517356 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:04.517331 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d13dde97-6b5f-4f2e-bf24-6ba2e2a29109-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a949e7-rm5bd\" (UID: \"d13dde97-6b5f-4f2e-bf24-6ba2e2a29109\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a949e7-rm5bd" Oct 30 17:37:04.542021 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:04.541992 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxq47\" (UniqueName: \"kubernetes.io/projected/d13dde97-6b5f-4f2e-bf24-6ba2e2a29109-kube-api-access-xxq47\") pod \"maas-api-revoke-keys-e2e-ipp-a-a949e7-rm5bd\" (UID: \"d13dde97-6b5f-4f2e-bf24-6ba2e2a29109\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a949e7-rm5bd" Oct 30 17:37:04.598649 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:04.598615 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a949e7-rm5bd" Oct 30 17:37:04.906489 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:37:04.906447 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd13dde97_6b5f_4f2e_bf24_6ba2e2a29109.slice/crio-b1de954b4c49a156df71a8595a40b5400e484d7d727702e1746447d883f5a873 WatchSource:0}: Error finding container b1de954b4c49a156df71a8595a40b5400e484d7d727702e1746447d883f5a873: Status 404 returned error can't find the container with id b1de954b4c49a156df71a8595a40b5400e484d7d727702e1746447d883f5a873 Oct 30 17:37:04.906945 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:04.906920 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a949e7-rm5bd"] Oct 30 17:37:05.387159 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:05.387122 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a949e7-rm5bd" event={"ID":"d13dde97-6b5f-4f2e-bf24-6ba2e2a29109","Type":"ContainerStarted","Data":"411d2d43af71dcc876bde3ca58f4e4358edc601f34b00fef5ff0f932fb3ebffd"} Oct 30 17:37:05.387665 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:05.387173 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a949e7-rm5bd" event={"ID":"d13dde97-6b5f-4f2e-bf24-6ba2e2a29109","Type":"ContainerStarted","Data":"b1de954b4c49a156df71a8595a40b5400e484d7d727702e1746447d883f5a873"} Oct 30 17:37:05.443265 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:05.443209 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a949e7-rm5bd" podStartSLOduration=1.443194177 podStartE2EDuration="1.443194177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:37:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:37:05.440752774 +0000 UTC m=+2999.441507695" watchObservedRunningTime="2026-10-30 17:37:05.443194177 +0000 UTC m=+2999.443949077" Oct 30 17:37:06.396321 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:06.396285 4522 generic.go:356] "Generic (PLEG): container finished" podID="d13dde97-6b5f-4f2e-bf24-6ba2e2a29109" containerID="411d2d43af71dcc876bde3ca58f4e4358edc601f34b00fef5ff0f932fb3ebffd" exitCode=0 Oct 30 17:37:06.396733 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:06.396376 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a949e7-rm5bd" event={"ID":"d13dde97-6b5f-4f2e-bf24-6ba2e2a29109","Type":"ContainerDied","Data":"411d2d43af71dcc876bde3ca58f4e4358edc601f34b00fef5ff0f932fb3ebffd"} Oct 30 17:37:07.608787 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:07.608757 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a949e7-rm5bd" Oct 30 17:37:07.756651 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:07.756620 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d13dde97-6b5f-4f2e-bf24-6ba2e2a29109-kube-api-access-xxq47\" (UniqueName: \"kubernetes.io/projected/d13dde97-6b5f-4f2e-bf24-6ba2e2a29109-kube-api-access-xxq47\") pod \"d13dde97-6b5f-4f2e-bf24-6ba2e2a29109\" (UID: \"d13dde97-6b5f-4f2e-bf24-6ba2e2a29109\") " Oct 30 17:37:07.756860 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:07.756729 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d13dde97-6b5f-4f2e-bf24-6ba2e2a29109-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d13dde97-6b5f-4f2e-bf24-6ba2e2a29109-maas-api-openshift-ca\") pod \"d13dde97-6b5f-4f2e-bf24-6ba2e2a29109\" (UID: \"d13dde97-6b5f-4f2e-bf24-6ba2e2a29109\") " Oct 30 17:37:07.756860 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:07.756794 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d13dde97-6b5f-4f2e-bf24-6ba2e2a29109-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d13dde97-6b5f-4f2e-bf24-6ba2e2a29109-maas-api-xks-ca\") pod \"d13dde97-6b5f-4f2e-bf24-6ba2e2a29109\" (UID: \"d13dde97-6b5f-4f2e-bf24-6ba2e2a29109\") " Oct 30 17:37:07.757139 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:07.757113 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d13dde97-6b5f-4f2e-bf24-6ba2e2a29109-maas-api-openshift-ca" pod "d13dde97-6b5f-4f2e-bf24-6ba2e2a29109" (UID: "d13dde97-6b5f-4f2e-bf24-6ba2e2a29109"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:37:07.758797 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:07.758775 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d13dde97-6b5f-4f2e-bf24-6ba2e2a29109-kube-api-access-xxq47" pod "d13dde97-6b5f-4f2e-bf24-6ba2e2a29109" (UID: "d13dde97-6b5f-4f2e-bf24-6ba2e2a29109"). InnerVolumeSpecName "kube-api-access-xxq47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:37:07.758943 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:07.758922 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d13dde97-6b5f-4f2e-bf24-6ba2e2a29109-maas-api-xks-ca" pod "d13dde97-6b5f-4f2e-bf24-6ba2e2a29109" (UID: "d13dde97-6b5f-4f2e-bf24-6ba2e2a29109"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:37:07.858827 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:07.858790 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d13dde97-6b5f-4f2e-bf24-6ba2e2a29109-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:37:07.858827 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:07.858822 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d13dde97-6b5f-4f2e-bf24-6ba2e2a29109-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:37:07.858827 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:07.858831 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxq47\" (UniqueName: \"kubernetes.io/projected/d13dde97-6b5f-4f2e-bf24-6ba2e2a29109-kube-api-access-xxq47\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:37:08.412255 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:08.412166 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a949e7-rm5bd" Oct 30 17:37:08.412255 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:08.412198 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a949e7-rm5bd" event={"ID":"d13dde97-6b5f-4f2e-bf24-6ba2e2a29109","Type":"ContainerDied","Data":"b1de954b4c49a156df71a8595a40b5400e484d7d727702e1746447d883f5a873"} Oct 30 17:37:08.412255 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:08.412245 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1de954b4c49a156df71a8595a40b5400e484d7d727702e1746447d883f5a873" Oct 30 17:37:10.869774 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:10.869738 4522 scope.go:122] "RemoveContainer" containerID="572781c294d4c4998833ab61e73180cc1906d96a853ffdfd4759075e00f6aea5" Oct 30 17:37:10.886921 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:10.886901 4522 scope.go:122] "RemoveContainer" containerID="4c14bf70328cefea257af5321836cfa1200ef5131d702d5aacdb427560bce077" Oct 30 17:37:11.056057 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:11.056034 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-b80d83-58f8df6ff4-xt4dg" Oct 30 17:37:11.196876 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:11.196841 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfdddc9b-c6c6-4cb1-bdd6-1634e76c4825-kube-api-access-n94df\" (UniqueName: \"kubernetes.io/projected/bfdddc9b-c6c6-4cb1-bdd6-1634e76c4825-kube-api-access-n94df\") pod \"bfdddc9b-c6c6-4cb1-bdd6-1634e76c4825\" (UID: \"bfdddc9b-c6c6-4cb1-bdd6-1634e76c4825\") " Oct 30 17:37:11.197057 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:11.197032 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfdddc9b-c6c6-4cb1-bdd6-1634e76c4825-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfdddc9b-c6c6-4cb1-bdd6-1634e76c4825-plugins-config-volume\") pod \"bfdddc9b-c6c6-4cb1-bdd6-1634e76c4825\" (UID: \"bfdddc9b-c6c6-4cb1-bdd6-1634e76c4825\") " Oct 30 17:37:11.197430 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:11.197404 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfdddc9b-c6c6-4cb1-bdd6-1634e76c4825-plugins-config-volume" pod "bfdddc9b-c6c6-4cb1-bdd6-1634e76c4825" (UID: "bfdddc9b-c6c6-4cb1-bdd6-1634e76c4825"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:37:11.199089 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:11.199067 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfdddc9b-c6c6-4cb1-bdd6-1634e76c4825-kube-api-access-n94df" pod "bfdddc9b-c6c6-4cb1-bdd6-1634e76c4825" (UID: "bfdddc9b-c6c6-4cb1-bdd6-1634e76c4825"). InnerVolumeSpecName "kube-api-access-n94df". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:37:11.298609 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:11.298526 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfdddc9b-c6c6-4cb1-bdd6-1634e76c4825-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:37:11.298609 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:11.298553 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n94df\" (UniqueName: \"kubernetes.io/projected/bfdddc9b-c6c6-4cb1-bdd6-1634e76c4825-kube-api-access-n94df\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:37:11.435384 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:11.435349 4522 generic.go:356] "Generic (PLEG): container finished" podID="bfdddc9b-c6c6-4cb1-bdd6-1634e76c4825" containerID="dc92488ee977f9f3b44786c79d31b230099d2240d2a2a8f8c60abf0e04eb98e1" exitCode=0 Oct 30 17:37:11.435569 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:11.435399 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-b80d83-58f8df6ff4-xt4dg" Oct 30 17:37:11.435569 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:11.435429 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-b80d83-58f8df6ff4-xt4dg" event={"ID":"bfdddc9b-c6c6-4cb1-bdd6-1634e76c4825","Type":"ContainerDied","Data":"dc92488ee977f9f3b44786c79d31b230099d2240d2a2a8f8c60abf0e04eb98e1"} Oct 30 17:37:11.435569 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:11.435483 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-b80d83-58f8df6ff4-xt4dg" event={"ID":"bfdddc9b-c6c6-4cb1-bdd6-1634e76c4825","Type":"ContainerDied","Data":"6085afd8e31317b1fe94941995f246a5493eb2a7966a00d9f2302ade8bef3dab"} Oct 30 17:37:11.435569 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:11.435504 4522 scope.go:122] "RemoveContainer" containerID="dc92488ee977f9f3b44786c79d31b230099d2240d2a2a8f8c60abf0e04eb98e1" Oct 30 17:37:11.446272 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:11.446250 4522 scope.go:122] "RemoveContainer" containerID="dc92488ee977f9f3b44786c79d31b230099d2240d2a2a8f8c60abf0e04eb98e1" Oct 30 17:37:11.446621 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:37:11.446598 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc92488ee977f9f3b44786c79d31b230099d2240d2a2a8f8c60abf0e04eb98e1\": container with ID starting with dc92488ee977f9f3b44786c79d31b230099d2240d2a2a8f8c60abf0e04eb98e1 not found: ID does not exist" containerID="dc92488ee977f9f3b44786c79d31b230099d2240d2a2a8f8c60abf0e04eb98e1" Oct 30 17:37:11.446701 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:11.446633 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc92488ee977f9f3b44786c79d31b230099d2240d2a2a8f8c60abf0e04eb98e1"} err="failed to get container status \"dc92488ee977f9f3b44786c79d31b230099d2240d2a2a8f8c60abf0e04eb98e1\": rpc error: code = NotFound desc = could not find container \"dc92488ee977f9f3b44786c79d31b230099d2240d2a2a8f8c60abf0e04eb98e1\": container with ID starting with dc92488ee977f9f3b44786c79d31b230099d2240d2a2a8f8c60abf0e04eb98e1 not found: ID does not exist" Oct 30 17:37:11.492793 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:11.492723 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-b80d83-58f8df6ff4-xt4dg"] Oct 30 17:37:11.505961 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:11.505932 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-b80d83-58f8df6ff4-xt4dg"] Oct 30 17:37:12.146105 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:12.146073 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfdddc9b-c6c6-4cb1-bdd6-1634e76c4825" path="/var/lib/kubelet/pods/bfdddc9b-c6c6-4cb1-bdd6-1634e76c4825/volumes" Oct 30 17:37:12.948196 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:37:12.948154 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:37:17.058015 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:17.057953 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" podUID="4b4ad752-e053-4cea-a3bc-fe2be6f92021" containerName="main" containerID="cri-o://4f08eb9d781721239b79b3f6e885d7b55d3347c64721b985ed55c8e50914f1c2" gracePeriod=45 Oct 30 17:37:17.324967 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:17.324906 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" Oct 30 17:37:17.364962 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:17.364923 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b4ad752-e053-4cea-a3bc-fe2be6f92021-kube-api-access-9sxx8\" (UniqueName: \"kubernetes.io/projected/4b4ad752-e053-4cea-a3bc-fe2be6f92021-kube-api-access-9sxx8\") pod \"4b4ad752-e053-4cea-a3bc-fe2be6f92021\" (UID: \"4b4ad752-e053-4cea-a3bc-fe2be6f92021\") " Oct 30 17:37:17.365193 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:17.365016 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b4ad752-e053-4cea-a3bc-fe2be6f92021-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b4ad752-e053-4cea-a3bc-fe2be6f92021-model-cache\") pod \"4b4ad752-e053-4cea-a3bc-fe2be6f92021\" (UID: \"4b4ad752-e053-4cea-a3bc-fe2be6f92021\") " Oct 30 17:37:17.365193 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:17.365092 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b4ad752-e053-4cea-a3bc-fe2be6f92021-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b4ad752-e053-4cea-a3bc-fe2be6f92021-tmp-dir\") pod \"4b4ad752-e053-4cea-a3bc-fe2be6f92021\" (UID: \"4b4ad752-e053-4cea-a3bc-fe2be6f92021\") " Oct 30 17:37:17.365542 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:17.365517 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b4ad752-e053-4cea-a3bc-fe2be6f92021-model-cache" pod "4b4ad752-e053-4cea-a3bc-fe2be6f92021" (UID: "4b4ad752-e053-4cea-a3bc-fe2be6f92021"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:37:17.366725 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:17.366679 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b4ad752-e053-4cea-a3bc-fe2be6f92021-tmp-dir" pod "4b4ad752-e053-4cea-a3bc-fe2be6f92021" (UID: "4b4ad752-e053-4cea-a3bc-fe2be6f92021"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:37:17.367172 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:17.367141 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b4ad752-e053-4cea-a3bc-fe2be6f92021-tls-certs\" (UniqueName: \"kubernetes.io/secret/4b4ad752-e053-4cea-a3bc-fe2be6f92021-tls-certs\") pod \"4b4ad752-e053-4cea-a3bc-fe2be6f92021\" (UID: \"4b4ad752-e053-4cea-a3bc-fe2be6f92021\") " Oct 30 17:37:17.367270 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:17.367253 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b4ad752-e053-4cea-a3bc-fe2be6f92021-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b4ad752-e053-4cea-a3bc-fe2be6f92021-dshm\") pod \"4b4ad752-e053-4cea-a3bc-fe2be6f92021\" (UID: \"4b4ad752-e053-4cea-a3bc-fe2be6f92021\") " Oct 30 17:37:17.367384 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:17.367289 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b4ad752-e053-4cea-a3bc-fe2be6f92021-home\" (UniqueName: \"kubernetes.io/empty-dir/4b4ad752-e053-4cea-a3bc-fe2be6f92021-home\") pod \"4b4ad752-e053-4cea-a3bc-fe2be6f92021\" (UID: \"4b4ad752-e053-4cea-a3bc-fe2be6f92021\") " Oct 30 17:37:17.367928 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:17.367905 4522 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b4ad752-e053-4cea-a3bc-fe2be6f92021-model-cache\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:37:17.368071 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:17.367934 4522 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b4ad752-e053-4cea-a3bc-fe2be6f92021-tmp-dir\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:37:17.368179 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:17.368114 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b4ad752-e053-4cea-a3bc-fe2be6f92021-home" pod "4b4ad752-e053-4cea-a3bc-fe2be6f92021" (UID: "4b4ad752-e053-4cea-a3bc-fe2be6f92021"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:37:17.383799 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:17.383508 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b4ad752-e053-4cea-a3bc-fe2be6f92021-dshm" pod "4b4ad752-e053-4cea-a3bc-fe2be6f92021" (UID: "4b4ad752-e053-4cea-a3bc-fe2be6f92021"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:37:17.383799 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:17.383514 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b4ad752-e053-4cea-a3bc-fe2be6f92021-tls-certs" pod "4b4ad752-e053-4cea-a3bc-fe2be6f92021" (UID: "4b4ad752-e053-4cea-a3bc-fe2be6f92021"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:37:17.383799 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:17.383560 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b4ad752-e053-4cea-a3bc-fe2be6f92021-kube-api-access-9sxx8" pod "4b4ad752-e053-4cea-a3bc-fe2be6f92021" (UID: "4b4ad752-e053-4cea-a3bc-fe2be6f92021"). InnerVolumeSpecName "kube-api-access-9sxx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:37:17.469116 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:17.469085 4522 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b4ad752-e053-4cea-a3bc-fe2be6f92021-tls-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:37:17.469116 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:17.469112 4522 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b4ad752-e053-4cea-a3bc-fe2be6f92021-dshm\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:37:17.469116 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:17.469123 4522 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b4ad752-e053-4cea-a3bc-fe2be6f92021-home\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:37:17.469388 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:17.469132 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sxx8\" (UniqueName: \"kubernetes.io/projected/4b4ad752-e053-4cea-a3bc-fe2be6f92021-kube-api-access-9sxx8\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:37:17.479590 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:17.479561 4522 generic.go:356] "Generic (PLEG): container finished" podID="4b4ad752-e053-4cea-a3bc-fe2be6f92021" containerID="4f08eb9d781721239b79b3f6e885d7b55d3347c64721b985ed55c8e50914f1c2" exitCode=0 Oct 30 17:37:17.479820 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:17.479621 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" Oct 30 17:37:17.479820 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:17.479643 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" event={"ID":"4b4ad752-e053-4cea-a3bc-fe2be6f92021","Type":"ContainerDied","Data":"4f08eb9d781721239b79b3f6e885d7b55d3347c64721b985ed55c8e50914f1c2"} Oct 30 17:37:17.479820 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:17.479706 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" event={"ID":"4b4ad752-e053-4cea-a3bc-fe2be6f92021","Type":"ContainerDied","Data":"87f8ffa1118f3b10a9102de0bd5d730d3e2b5ca9ba004b9cf081e9fd7a64b3b8"} Oct 30 17:37:17.479820 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:17.479734 4522 scope.go:122] "RemoveContainer" containerID="4f08eb9d781721239b79b3f6e885d7b55d3347c64721b985ed55c8e50914f1c2" Oct 30 17:37:17.491560 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:17.491542 4522 scope.go:122] "RemoveContainer" containerID="4f08eb9d781721239b79b3f6e885d7b55d3347c64721b985ed55c8e50914f1c2" Oct 30 17:37:17.491848 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:37:17.491826 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f08eb9d781721239b79b3f6e885d7b55d3347c64721b985ed55c8e50914f1c2\": container with ID starting with 4f08eb9d781721239b79b3f6e885d7b55d3347c64721b985ed55c8e50914f1c2 not found: ID does not exist" containerID="4f08eb9d781721239b79b3f6e885d7b55d3347c64721b985ed55c8e50914f1c2" Oct 30 17:37:17.491944 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:17.491855 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f08eb9d781721239b79b3f6e885d7b55d3347c64721b985ed55c8e50914f1c2"} err="failed to get container status \"4f08eb9d781721239b79b3f6e885d7b55d3347c64721b985ed55c8e50914f1c2\": rpc error: code = NotFound desc = could not find container \"4f08eb9d781721239b79b3f6e885d7b55d3347c64721b985ed55c8e50914f1c2\": container with ID starting with 4f08eb9d781721239b79b3f6e885d7b55d3347c64721b985ed55c8e50914f1c2 not found: ID does not exist" Oct 30 17:37:17.546942 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:17.546907 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9"] Oct 30 17:37:17.554778 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:17.554750 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9"] Oct 30 17:37:18.144502 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:18.144460 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b4ad752-e053-4cea-a3bc-fe2be6f92021" path="/var/lib/kubelet/pods/4b4ad752-e053-4cea-a3bc-fe2be6f92021/volumes" Oct 30 17:37:18.273804 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:18.273766 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-w5-86f50a-kserve-994c9b44-vssh9" podUID="4b4ad752-e053-4cea-a3bc-fe2be6f92021" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Oct 30 17:37:21.136922 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:21.136837 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b80d83-dhchx"] Oct 30 17:37:21.151777 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:21.151749 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b80d83-dhchx"] Oct 30 17:37:22.144902 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:22.144868 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4b5d762-29b6-4913-99ff-4851f5c47bf6" path="/var/lib/kubelet/pods/a4b5d762-29b6-4913-99ff-4851f5c47bf6/volumes" Oct 30 17:37:23.841377 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:23.841342 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b4f406-7cw59"] Oct 30 17:37:23.855174 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:23.855135 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b4f406-7cw59"] Oct 30 17:37:24.144514 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:24.144426 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="450fab43-5d99-4c34-97e6-59a6d64de032" path="/var/lib/kubelet/pods/450fab43-5d99-4c34-97e6-59a6d64de032/volumes" Oct 30 17:37:24.588477 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:24.588440 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a949e7-rm5bd"] Oct 30 17:37:24.602488 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:24.602447 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a949e7-rm5bd"] Oct 30 17:37:26.146691 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:26.146658 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d13dde97-6b5f-4f2e-bf24-6ba2e2a29109" path="/var/lib/kubelet/pods/d13dde97-6b5f-4f2e-bf24-6ba2e2a29109/volumes" Oct 30 17:37:28.531620 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:28.531586 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da10aa3f-zkrjd"] Oct 30 17:37:28.532165 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:28.532151 4522 cpu_manager.go:414] "containerMap: removing container" podUID="4b4ad752-e053-4cea-a3bc-fe2be6f92021" containerName="main" Oct 30 17:37:28.532216 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:28.532166 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b4ad752-e053-4cea-a3bc-fe2be6f92021" containerName="main" Oct 30 17:37:28.532216 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:28.532183 4522 cpu_manager.go:414] "containerMap: removing container" podUID="bfdddc9b-c6c6-4cb1-bdd6-1634e76c4825" containerName="payload-processing" Oct 30 17:37:28.532216 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:28.532189 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfdddc9b-c6c6-4cb1-bdd6-1634e76c4825" containerName="payload-processing" Oct 30 17:37:28.532216 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:28.532208 4522 cpu_manager.go:414] "containerMap: removing container" podUID="d13dde97-6b5f-4f2e-bf24-6ba2e2a29109" containerName="revoke-keys" Oct 30 17:37:28.532216 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:28.532213 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d13dde97-6b5f-4f2e-bf24-6ba2e2a29109" containerName="revoke-keys" Oct 30 17:37:28.532404 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:28.532288 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b4ad752-e053-4cea-a3bc-fe2be6f92021" containerName="main" Oct 30 17:37:28.532404 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:28.532297 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfdddc9b-c6c6-4cb1-bdd6-1634e76c4825" containerName="payload-processing" Oct 30 17:37:28.532404 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:28.532318 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="40baffd6-35b6-412d-a7bd-f3155ed306db" containerName="istio-proxy" Oct 30 17:37:28.532404 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:28.532326 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="d13dde97-6b5f-4f2e-bf24-6ba2e2a29109" containerName="revoke-keys" Oct 30 17:37:28.537389 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:28.537372 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da10aa3f-zkrjd" Oct 30 17:37:28.558896 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:28.558869 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da10aa3f-zkrjd"] Oct 30 17:37:28.601820 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:28.601786 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4831a6b7-f606-47ee-82bc-d31a0b0288df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da10aa3f-zkrjd\" (UID: \"4831a6b7-f606-47ee-82bc-d31a0b0288df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da10aa3f-zkrjd" Oct 30 17:37:28.601988 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:28.601897 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsrjn\" (UniqueName: \"kubernetes.io/projected/4831a6b7-f606-47ee-82bc-d31a0b0288df-kube-api-access-jsrjn\") pod \"maas-api-revoke-sub-da10aa3f-zkrjd\" (UID: \"4831a6b7-f606-47ee-82bc-d31a0b0288df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da10aa3f-zkrjd" Oct 30 17:37:28.601988 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:28.601950 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4831a6b7-f606-47ee-82bc-d31a0b0288df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da10aa3f-zkrjd\" (UID: \"4831a6b7-f606-47ee-82bc-d31a0b0288df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da10aa3f-zkrjd" Oct 30 17:37:28.652160 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:28.652121 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:37:28.652358 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:28.652175 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:37:28.703013 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:28.702978 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsrjn\" (UniqueName: \"kubernetes.io/projected/4831a6b7-f606-47ee-82bc-d31a0b0288df-kube-api-access-jsrjn\") pod \"maas-api-revoke-sub-da10aa3f-zkrjd\" (UID: \"4831a6b7-f606-47ee-82bc-d31a0b0288df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da10aa3f-zkrjd" Oct 30 17:37:28.703216 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:28.703046 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4831a6b7-f606-47ee-82bc-d31a0b0288df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da10aa3f-zkrjd\" (UID: \"4831a6b7-f606-47ee-82bc-d31a0b0288df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da10aa3f-zkrjd" Oct 30 17:37:28.703216 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:28.703192 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4831a6b7-f606-47ee-82bc-d31a0b0288df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da10aa3f-zkrjd\" (UID: \"4831a6b7-f606-47ee-82bc-d31a0b0288df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da10aa3f-zkrjd" Oct 30 17:37:28.703976 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:28.703875 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4831a6b7-f606-47ee-82bc-d31a0b0288df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da10aa3f-zkrjd\" (UID: \"4831a6b7-f606-47ee-82bc-d31a0b0288df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da10aa3f-zkrjd" Oct 30 17:37:28.706052 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:28.706025 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4831a6b7-f606-47ee-82bc-d31a0b0288df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da10aa3f-zkrjd\" (UID: \"4831a6b7-f606-47ee-82bc-d31a0b0288df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da10aa3f-zkrjd" Oct 30 17:37:28.729548 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:28.729517 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsrjn\" (UniqueName: \"kubernetes.io/projected/4831a6b7-f606-47ee-82bc-d31a0b0288df-kube-api-access-jsrjn\") pod \"maas-api-revoke-sub-da10aa3f-zkrjd\" (UID: \"4831a6b7-f606-47ee-82bc-d31a0b0288df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da10aa3f-zkrjd" Oct 30 17:37:28.851263 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:28.851179 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da10aa3f-zkrjd" Oct 30 17:37:29.162460 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:37:29.162428 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4831a6b7_f606_47ee_82bc_d31a0b0288df.slice/crio-195e0c9a4964a900e0327897fae3328a24269a279565a98dcabd945d82652173 WatchSource:0}: Error finding container 195e0c9a4964a900e0327897fae3328a24269a279565a98dcabd945d82652173: Status 404 returned error can't find the container with id 195e0c9a4964a900e0327897fae3328a24269a279565a98dcabd945d82652173 Oct 30 17:37:29.163989 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:29.163963 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da10aa3f-zkrjd"] Oct 30 17:37:29.423025 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:29.422989 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-054acd-5694b584cb-wf7nq"] Oct 30 17:37:29.467024 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:29.466989 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-054acd-5fccd756d7-v75ht"] Oct 30 17:37:29.572934 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:29.572891 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da10aa3f-zkrjd" event={"ID":"4831a6b7-f606-47ee-82bc-d31a0b0288df","Type":"ContainerStarted","Data":"90c473fffbbc1486f63cb4c5d77dc271409b5da0395994c7b051963269ba05fb"} Oct 30 17:37:29.573469 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:29.572954 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da10aa3f-zkrjd" event={"ID":"4831a6b7-f606-47ee-82bc-d31a0b0288df","Type":"ContainerStarted","Data":"195e0c9a4964a900e0327897fae3328a24269a279565a98dcabd945d82652173"} Oct 30 17:37:29.622265 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:29.622213 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da10aa3f-zkrjd" podStartSLOduration=1.622196362 podStartE2EDuration="1.622196362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:37:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:37:29.61962237 +0000 UTC m=+3023.620377271" watchObservedRunningTime="2026-10-30 17:37:29.622196362 +0000 UTC m=+3023.622951263" Oct 30 17:37:30.580768 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:30.580733 4522 generic.go:356] "Generic (PLEG): container finished" podID="4831a6b7-f606-47ee-82bc-d31a0b0288df" containerID="90c473fffbbc1486f63cb4c5d77dc271409b5da0395994c7b051963269ba05fb" exitCode=0 Oct 30 17:37:30.581189 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:30.580810 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da10aa3f-zkrjd" event={"ID":"4831a6b7-f606-47ee-82bc-d31a0b0288df","Type":"ContainerDied","Data":"90c473fffbbc1486f63cb4c5d77dc271409b5da0395994c7b051963269ba05fb"} Oct 30 17:37:31.733641 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:31.733610 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da10aa3f-zkrjd" Oct 30 17:37:31.842982 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:31.842886 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4831a6b7-f606-47ee-82bc-d31a0b0288df-kube-api-access-jsrjn\" (UniqueName: \"kubernetes.io/projected/4831a6b7-f606-47ee-82bc-d31a0b0288df-kube-api-access-jsrjn\") pod \"4831a6b7-f606-47ee-82bc-d31a0b0288df\" (UID: \"4831a6b7-f606-47ee-82bc-d31a0b0288df\") " Oct 30 17:37:31.842982 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:31.842940 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4831a6b7-f606-47ee-82bc-d31a0b0288df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4831a6b7-f606-47ee-82bc-d31a0b0288df-maas-api-openshift-ca\") pod \"4831a6b7-f606-47ee-82bc-d31a0b0288df\" (UID: \"4831a6b7-f606-47ee-82bc-d31a0b0288df\") " Oct 30 17:37:31.843244 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:31.843046 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4831a6b7-f606-47ee-82bc-d31a0b0288df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4831a6b7-f606-47ee-82bc-d31a0b0288df-maas-api-xks-ca\") pod \"4831a6b7-f606-47ee-82bc-d31a0b0288df\" (UID: \"4831a6b7-f606-47ee-82bc-d31a0b0288df\") " Oct 30 17:37:31.843439 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:31.843401 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4831a6b7-f606-47ee-82bc-d31a0b0288df-maas-api-openshift-ca" pod "4831a6b7-f606-47ee-82bc-d31a0b0288df" (UID: "4831a6b7-f606-47ee-82bc-d31a0b0288df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:37:31.843532 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:31.843520 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4831a6b7-f606-47ee-82bc-d31a0b0288df-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:37:31.845136 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:31.845110 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4831a6b7-f606-47ee-82bc-d31a0b0288df-kube-api-access-jsrjn" pod "4831a6b7-f606-47ee-82bc-d31a0b0288df" (UID: "4831a6b7-f606-47ee-82bc-d31a0b0288df"). InnerVolumeSpecName "kube-api-access-jsrjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:37:31.845421 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:31.845396 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4831a6b7-f606-47ee-82bc-d31a0b0288df-maas-api-xks-ca" pod "4831a6b7-f606-47ee-82bc-d31a0b0288df" (UID: "4831a6b7-f606-47ee-82bc-d31a0b0288df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:37:31.959392 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:31.954229 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsrjn\" (UniqueName: \"kubernetes.io/projected/4831a6b7-f606-47ee-82bc-d31a0b0288df-kube-api-access-jsrjn\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:37:31.959392 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:31.954273 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4831a6b7-f606-47ee-82bc-d31a0b0288df-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:37:32.536431 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:32.536378 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77"] Oct 30 17:37:32.536756 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:32.536719 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77" podUID="bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46" containerName="istio-proxy" containerID="cri-o://d695d95539345ee85b4ebf5ed1bb5cc069868b79a2cba30654927701839d9f5a" gracePeriod=30 Oct 30 17:37:32.595421 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:32.595392 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da10aa3f-zkrjd" Oct 30 17:37:32.595610 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:32.595434 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da10aa3f-zkrjd" event={"ID":"4831a6b7-f606-47ee-82bc-d31a0b0288df","Type":"ContainerDied","Data":"195e0c9a4964a900e0327897fae3328a24269a279565a98dcabd945d82652173"} Oct 30 17:37:32.595610 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:32.595483 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="195e0c9a4964a900e0327897fae3328a24269a279565a98dcabd945d82652173" Oct 30 17:37:32.973955 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:32.973916 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-031223-gtwqd"] Oct 30 17:37:32.974564 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:32.974548 4522 cpu_manager.go:414] "containerMap: removing container" podUID="4831a6b7-f606-47ee-82bc-d31a0b0288df" containerName="revoke-keys" Oct 30 17:37:32.974616 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:32.974565 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4831a6b7-f606-47ee-82bc-d31a0b0288df" containerName="revoke-keys" Oct 30 17:37:32.974687 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:32.974678 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="4831a6b7-f606-47ee-82bc-d31a0b0288df" containerName="revoke-keys" Oct 30 17:37:32.977989 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:32.977972 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-031223-gtwqd" Oct 30 17:37:32.998709 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:32.998682 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:37:32.998853 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:32.998837 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:37:33.013861 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:33.013825 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-031223-gtwqd"] Oct 30 17:37:33.069869 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:33.069834 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2966625c-2ca6-4a73-b0e6-3eb648a2745c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-031223-gtwqd\" (UID: \"2966625c-2ca6-4a73-b0e6-3eb648a2745c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-031223-gtwqd" Oct 30 17:37:33.070017 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:33.069908 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2966625c-2ca6-4a73-b0e6-3eb648a2745c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-031223-gtwqd\" (UID: \"2966625c-2ca6-4a73-b0e6-3eb648a2745c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-031223-gtwqd" Oct 30 17:37:33.070017 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:33.069973 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2m2b\" (UniqueName: \"kubernetes.io/projected/2966625c-2ca6-4a73-b0e6-3eb648a2745c-kube-api-access-x2m2b\") pod \"maas-api-revoke-keys-e2e-ipp-b-031223-gtwqd\" (UID: \"2966625c-2ca6-4a73-b0e6-3eb648a2745c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-031223-gtwqd" Oct 30 17:37:33.170993 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:33.170957 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2966625c-2ca6-4a73-b0e6-3eb648a2745c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-031223-gtwqd\" (UID: \"2966625c-2ca6-4a73-b0e6-3eb648a2745c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-031223-gtwqd" Oct 30 17:37:33.171214 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:33.171128 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2966625c-2ca6-4a73-b0e6-3eb648a2745c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-031223-gtwqd\" (UID: \"2966625c-2ca6-4a73-b0e6-3eb648a2745c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-031223-gtwqd" Oct 30 17:37:33.171214 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:33.171179 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2m2b\" (UniqueName: \"kubernetes.io/projected/2966625c-2ca6-4a73-b0e6-3eb648a2745c-kube-api-access-x2m2b\") pod \"maas-api-revoke-keys-e2e-ipp-b-031223-gtwqd\" (UID: \"2966625c-2ca6-4a73-b0e6-3eb648a2745c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-031223-gtwqd" Oct 30 17:37:33.171797 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:33.171777 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2966625c-2ca6-4a73-b0e6-3eb648a2745c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-031223-gtwqd\" (UID: \"2966625c-2ca6-4a73-b0e6-3eb648a2745c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-031223-gtwqd" Oct 30 17:37:33.173407 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:33.173383 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2966625c-2ca6-4a73-b0e6-3eb648a2745c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-031223-gtwqd\" (UID: \"2966625c-2ca6-4a73-b0e6-3eb648a2745c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-031223-gtwqd" Oct 30 17:37:33.197803 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:33.197777 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2m2b\" (UniqueName: \"kubernetes.io/projected/2966625c-2ca6-4a73-b0e6-3eb648a2745c-kube-api-access-x2m2b\") pod \"maas-api-revoke-keys-e2e-ipp-b-031223-gtwqd\" (UID: \"2966625c-2ca6-4a73-b0e6-3eb648a2745c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-031223-gtwqd" Oct 30 17:37:33.291819 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:33.291742 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-031223-gtwqd" Oct 30 17:37:33.608215 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:33.608188 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-031223-gtwqd"] Oct 30 17:37:33.608546 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:37:33.608523 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2966625c_2ca6_4a73_b0e6_3eb648a2745c.slice/crio-75bf6e136f0e969abb49bc9e93783a5774c07d524ad71920df122cc8b5f6b2c5 WatchSource:0}: Error finding container 75bf6e136f0e969abb49bc9e93783a5774c07d524ad71920df122cc8b5f6b2c5: Status 404 returned error can't find the container with id 75bf6e136f0e969abb49bc9e93783a5774c07d524ad71920df122cc8b5f6b2c5 Oct 30 17:37:34.423734 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:34.423690 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-054acd-5694b584cb-wf7nq" podUID="c9734f86-d287-475b-982f-5c1b5443b16b" containerName="payload-processing" containerID="cri-o://b3b2404a2dcc8a13a9a8605e9d1df4fd6b347a9993328e9d70ecd1db3e581ea1" gracePeriod=25 Oct 30 17:37:34.467612 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:34.467544 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-054acd-5fccd756d7-v75ht" podUID="22a4271a-c94f-4902-9874-6712c9735cb3" containerName="payload-pre-processing" containerID="cri-o://2b05bab0b6431768dde8e2fab5c506eabac0957c6efcd23e2fe87f0bc8014dc9" gracePeriod=25 Oct 30 17:37:34.612575 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:34.612482 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-031223-gtwqd" event={"ID":"2966625c-2ca6-4a73-b0e6-3eb648a2745c","Type":"ContainerStarted","Data":"555856a636bd042252bd7fb3eab6ea56ef05ffde618561785f046cb2b7a51a32"} Oct 30 17:37:34.612575 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:34.612545 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-031223-gtwqd" event={"ID":"2966625c-2ca6-4a73-b0e6-3eb648a2745c","Type":"ContainerStarted","Data":"75bf6e136f0e969abb49bc9e93783a5774c07d524ad71920df122cc8b5f6b2c5"} Oct 30 17:37:34.614491 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:34.614466 4522 generic.go:356] "Generic (PLEG): container finished" podID="c9734f86-d287-475b-982f-5c1b5443b16b" containerID="b3b2404a2dcc8a13a9a8605e9d1df4fd6b347a9993328e9d70ecd1db3e581ea1" exitCode=0 Oct 30 17:37:34.614625 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:34.614541 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-054acd-5694b584cb-wf7nq" event={"ID":"c9734f86-d287-475b-982f-5c1b5443b16b","Type":"ContainerDied","Data":"b3b2404a2dcc8a13a9a8605e9d1df4fd6b347a9993328e9d70ecd1db3e581ea1"} Oct 30 17:37:34.616268 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:34.616246 4522 generic.go:356] "Generic (PLEG): container finished" podID="22a4271a-c94f-4902-9874-6712c9735cb3" containerID="2b05bab0b6431768dde8e2fab5c506eabac0957c6efcd23e2fe87f0bc8014dc9" exitCode=0 Oct 30 17:37:34.616436 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:34.616348 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-054acd-5fccd756d7-v75ht" event={"ID":"22a4271a-c94f-4902-9874-6712c9735cb3","Type":"ContainerDied","Data":"2b05bab0b6431768dde8e2fab5c506eabac0957c6efcd23e2fe87f0bc8014dc9"} Oct 30 17:37:34.660366 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:34.659229 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-031223-gtwqd" podStartSLOduration=2.6592094790000003 podStartE2EDuration="2.659209479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:37:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:37:34.658841198 +0000 UTC m=+3028.659596102" watchObservedRunningTime="2026-10-30 17:37:34.659209479 +0000 UTC m=+3028.659964384" Oct 30 17:37:34.735660 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:34.735634 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-054acd-5694b584cb-wf7nq" Oct 30 17:37:34.760024 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:34.760000 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-054acd-5fccd756d7-v75ht" Oct 30 17:37:34.889904 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:34.889873 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22a4271a-c94f-4902-9874-6712c9735cb3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22a4271a-c94f-4902-9874-6712c9735cb3-plugins-config-volume\") pod \"22a4271a-c94f-4902-9874-6712c9735cb3\" (UID: \"22a4271a-c94f-4902-9874-6712c9735cb3\") " Oct 30 17:37:34.890107 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:34.889973 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9734f86-d287-475b-982f-5c1b5443b16b-kube-api-access-8rq7q\" (UniqueName: \"kubernetes.io/projected/c9734f86-d287-475b-982f-5c1b5443b16b-kube-api-access-8rq7q\") pod \"c9734f86-d287-475b-982f-5c1b5443b16b\" (UID: \"c9734f86-d287-475b-982f-5c1b5443b16b\") " Oct 30 17:37:34.890107 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:34.890095 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22a4271a-c94f-4902-9874-6712c9735cb3-kube-api-access-8xkb9\" (UniqueName: \"kubernetes.io/projected/22a4271a-c94f-4902-9874-6712c9735cb3-kube-api-access-8xkb9\") pod \"22a4271a-c94f-4902-9874-6712c9735cb3\" (UID: \"22a4271a-c94f-4902-9874-6712c9735cb3\") " Oct 30 17:37:34.890228 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:34.890125 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9734f86-d287-475b-982f-5c1b5443b16b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9734f86-d287-475b-982f-5c1b5443b16b-plugins-config-volume\") pod \"c9734f86-d287-475b-982f-5c1b5443b16b\" (UID: \"c9734f86-d287-475b-982f-5c1b5443b16b\") " Oct 30 17:37:34.890378 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:34.890351 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22a4271a-c94f-4902-9874-6712c9735cb3-plugins-config-volume" pod "22a4271a-c94f-4902-9874-6712c9735cb3" (UID: "22a4271a-c94f-4902-9874-6712c9735cb3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:37:34.890634 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:34.890604 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9734f86-d287-475b-982f-5c1b5443b16b-plugins-config-volume" pod "c9734f86-d287-475b-982f-5c1b5443b16b" (UID: "c9734f86-d287-475b-982f-5c1b5443b16b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:37:34.890795 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:34.890772 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9734f86-d287-475b-982f-5c1b5443b16b-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:37:34.890912 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:34.890798 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22a4271a-c94f-4902-9874-6712c9735cb3-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:37:34.892202 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:34.892171 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9734f86-d287-475b-982f-5c1b5443b16b-kube-api-access-8rq7q" pod "c9734f86-d287-475b-982f-5c1b5443b16b" (UID: "c9734f86-d287-475b-982f-5c1b5443b16b"). InnerVolumeSpecName "kube-api-access-8rq7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:37:34.892638 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:34.892612 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22a4271a-c94f-4902-9874-6712c9735cb3-kube-api-access-8xkb9" pod "22a4271a-c94f-4902-9874-6712c9735cb3" (UID: "22a4271a-c94f-4902-9874-6712c9735cb3"). InnerVolumeSpecName "kube-api-access-8xkb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:37:34.991830 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:34.991794 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xkb9\" (UniqueName: \"kubernetes.io/projected/22a4271a-c94f-4902-9874-6712c9735cb3-kube-api-access-8xkb9\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:37:34.991830 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:34.991826 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rq7q\" (UniqueName: \"kubernetes.io/projected/c9734f86-d287-475b-982f-5c1b5443b16b-kube-api-access-8rq7q\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:37:35.625039 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:35.625008 4522 generic.go:356] "Generic (PLEG): container finished" podID="2966625c-2ca6-4a73-b0e6-3eb648a2745c" containerID="555856a636bd042252bd7fb3eab6ea56ef05ffde618561785f046cb2b7a51a32" exitCode=0 Oct 30 17:37:35.625549 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:35.625093 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-031223-gtwqd" event={"ID":"2966625c-2ca6-4a73-b0e6-3eb648a2745c","Type":"ContainerDied","Data":"555856a636bd042252bd7fb3eab6ea56ef05ffde618561785f046cb2b7a51a32"} Oct 30 17:37:35.626602 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:35.626585 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-054acd-5694b584cb-wf7nq" Oct 30 17:37:35.626685 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:35.626621 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-054acd-5694b584cb-wf7nq" event={"ID":"c9734f86-d287-475b-982f-5c1b5443b16b","Type":"ContainerDied","Data":"19ec91a5362e6bb37e63fac191de602a9252f0bf86eccd433f18e40db496429b"} Oct 30 17:37:35.626685 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:35.626675 4522 scope.go:122] "RemoveContainer" containerID="b3b2404a2dcc8a13a9a8605e9d1df4fd6b347a9993328e9d70ecd1db3e581ea1" Oct 30 17:37:35.628177 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:35.628159 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-054acd-5fccd756d7-v75ht" event={"ID":"22a4271a-c94f-4902-9874-6712c9735cb3","Type":"ContainerDied","Data":"2a89e2cf81d53898f36e5926d4571fa13e46a1495883bb12450bf26683085cd6"} Oct 30 17:37:35.628259 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:35.628174 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-054acd-5fccd756d7-v75ht" Oct 30 17:37:35.638791 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:35.638772 4522 scope.go:122] "RemoveContainer" containerID="2b05bab0b6431768dde8e2fab5c506eabac0957c6efcd23e2fe87f0bc8014dc9" Oct 30 17:37:35.717179 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:35.717145 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-054acd-5fccd756d7-v75ht"] Oct 30 17:37:35.721509 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:35.721479 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-054acd-5fccd756d7-v75ht"] Oct 30 17:37:35.765563 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:35.765529 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-054acd-5694b584cb-wf7nq"] Oct 30 17:37:35.771847 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:35.771818 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-054acd-5694b584cb-wf7nq"] Oct 30 17:37:36.145192 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:36.145160 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22a4271a-c94f-4902-9874-6712c9735cb3" path="/var/lib/kubelet/pods/22a4271a-c94f-4902-9874-6712c9735cb3/volumes" Oct 30 17:37:36.145657 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:36.145642 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9734f86-d287-475b-982f-5c1b5443b16b" path="/var/lib/kubelet/pods/c9734f86-d287-475b-982f-5c1b5443b16b/volumes" Oct 30 17:37:36.801909 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:36.801881 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-031223-gtwqd" Oct 30 17:37:36.916415 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:36.916331 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2966625c-2ca6-4a73-b0e6-3eb648a2745c-kube-api-access-x2m2b\" (UniqueName: \"kubernetes.io/projected/2966625c-2ca6-4a73-b0e6-3eb648a2745c-kube-api-access-x2m2b\") pod \"2966625c-2ca6-4a73-b0e6-3eb648a2745c\" (UID: \"2966625c-2ca6-4a73-b0e6-3eb648a2745c\") " Oct 30 17:37:36.916638 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:36.916430 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2966625c-2ca6-4a73-b0e6-3eb648a2745c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2966625c-2ca6-4a73-b0e6-3eb648a2745c-maas-api-openshift-ca\") pod \"2966625c-2ca6-4a73-b0e6-3eb648a2745c\" (UID: \"2966625c-2ca6-4a73-b0e6-3eb648a2745c\") " Oct 30 17:37:36.916638 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:36.916512 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2966625c-2ca6-4a73-b0e6-3eb648a2745c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2966625c-2ca6-4a73-b0e6-3eb648a2745c-maas-api-xks-ca\") pod \"2966625c-2ca6-4a73-b0e6-3eb648a2745c\" (UID: \"2966625c-2ca6-4a73-b0e6-3eb648a2745c\") " Oct 30 17:37:36.916819 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:36.916785 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2966625c-2ca6-4a73-b0e6-3eb648a2745c-maas-api-openshift-ca" pod "2966625c-2ca6-4a73-b0e6-3eb648a2745c" (UID: "2966625c-2ca6-4a73-b0e6-3eb648a2745c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:37:36.916981 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:36.916967 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2966625c-2ca6-4a73-b0e6-3eb648a2745c-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:37:36.918537 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:36.918515 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2966625c-2ca6-4a73-b0e6-3eb648a2745c-maas-api-xks-ca" pod "2966625c-2ca6-4a73-b0e6-3eb648a2745c" (UID: "2966625c-2ca6-4a73-b0e6-3eb648a2745c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:37:36.918645 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:36.918614 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2966625c-2ca6-4a73-b0e6-3eb648a2745c-kube-api-access-x2m2b" pod "2966625c-2ca6-4a73-b0e6-3eb648a2745c" (UID: "2966625c-2ca6-4a73-b0e6-3eb648a2745c"). InnerVolumeSpecName "kube-api-access-x2m2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:37:37.018624 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:37.018590 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2m2b\" (UniqueName: \"kubernetes.io/projected/2966625c-2ca6-4a73-b0e6-3eb648a2745c-kube-api-access-x2m2b\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:37:37.018624 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:37.018622 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2966625c-2ca6-4a73-b0e6-3eb648a2745c-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:37:37.648565 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:37.648527 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-031223-gtwqd" Oct 30 17:37:37.648565 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:37.648535 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-031223-gtwqd" event={"ID":"2966625c-2ca6-4a73-b0e6-3eb648a2745c","Type":"ContainerDied","Data":"75bf6e136f0e969abb49bc9e93783a5774c07d524ad71920df122cc8b5f6b2c5"} Oct 30 17:37:37.648792 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:37.648585 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75bf6e136f0e969abb49bc9e93783a5774c07d524ad71920df122cc8b5f6b2c5" Oct 30 17:37:37.815823 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:37.815762 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77" Oct 30 17:37:37.932797 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:37.932756 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46-workload-certs\") pod \"bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46\" (UID: \"bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46\") " Oct 30 17:37:37.932996 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:37.932857 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46-istio-data\") pod \"bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46\" (UID: \"bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46\") " Oct 30 17:37:37.932996 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:37.932915 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46-istio-podinfo\") pod \"bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46\" (UID: \"bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46\") " Oct 30 17:37:37.932996 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:37.932972 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46-istio-token\" (UniqueName: \"kubernetes.io/projected/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46-istio-token\") pod \"bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46\" (UID: \"bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46\") " Oct 30 17:37:37.933193 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:37.933001 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46-kube-api-access-qfb55\" (UniqueName: \"kubernetes.io/projected/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46-kube-api-access-qfb55\") pod \"bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46\" (UID: \"bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46\") " Oct 30 17:37:37.933193 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:37.933061 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46-workload-certs" pod "bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46" (UID: "bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:37:37.933193 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:37.933131 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46-workload-socket\") pod \"bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46\" (UID: \"bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46\") " Oct 30 17:37:37.933193 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:37.933168 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46-istio-data" pod "bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46" (UID: "bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:37:37.933193 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:37.933179 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46-istio-envoy\") pod \"bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46\" (UID: \"bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46\") " Oct 30 17:37:37.933523 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:37.933392 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46-credential-socket\") pod \"bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46\" (UID: \"bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46\") " Oct 30 17:37:37.933523 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:37.933438 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46-istiod-ca-cert\") pod \"bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46\" (UID: \"bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46\") " Oct 30 17:37:37.933523 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:37.933431 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46-workload-socket" pod "bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46" (UID: "bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:37:37.933709 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:37.933643 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46-credential-socket" pod "bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46" (UID: "bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:37:37.933910 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:37.933873 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46-istiod-ca-cert" pod "bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46" (UID: "bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:37:37.933996 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:37.933957 4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46-credential-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:37:37.933996 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:37.933980 4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46-workload-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:37:37.934073 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:37.933995 4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46-istio-data\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:37:37.934073 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:37.934010 4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46-workload-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:37:37.935409 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:37.935357 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46-istio-podinfo" pod "bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46" (UID: "bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:37:37.935522 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:37.935498 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46-istio-token" pod "bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46" (UID: "bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:37:37.935754 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:37.935737 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46-kube-api-access-qfb55" pod "bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46" (UID: "bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46"). InnerVolumeSpecName "kube-api-access-qfb55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:37:37.935823 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:37.935808 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46-istio-envoy" pod "bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46" (UID: "bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:37:38.034832 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:38.034801 4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46-istio-podinfo\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:37:38.034832 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:38.034829 4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46-istio-token\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:37:38.034832 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:38.034840 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfb55\" (UniqueName: \"kubernetes.io/projected/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46-kube-api-access-qfb55\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:37:38.035086 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:38.034849 4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46-istio-envoy\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:37:38.035086 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:38.034860 4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46-istiod-ca-cert\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:37:38.657285 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:38.657253 4522 generic.go:356] "Generic (PLEG): container finished" podID="bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46" containerID="d695d95539345ee85b4ebf5ed1bb5cc069868b79a2cba30654927701839d9f5a" exitCode=0 Oct 30 17:37:38.657465 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:38.657298 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77" Oct 30 17:37:38.657465 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:38.657339 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77" event={"ID":"bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46","Type":"ContainerDied","Data":"d695d95539345ee85b4ebf5ed1bb5cc069868b79a2cba30654927701839d9f5a"} Oct 30 17:37:38.657465 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:38.657395 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77" event={"ID":"bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46","Type":"ContainerDied","Data":"2333e5e5184282dfe451f443d30fae98c23ea2967cbf44bb7ffadeb5202f39d4"} Oct 30 17:37:38.657465 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:38.657425 4522 scope.go:122] "RemoveContainer" containerID="d695d95539345ee85b4ebf5ed1bb5cc069868b79a2cba30654927701839d9f5a" Oct 30 17:37:38.668946 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:38.668688 4522 scope.go:122] "RemoveContainer" containerID="d695d95539345ee85b4ebf5ed1bb5cc069868b79a2cba30654927701839d9f5a" Oct 30 17:37:38.669022 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:37:38.668967 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d695d95539345ee85b4ebf5ed1bb5cc069868b79a2cba30654927701839d9f5a\": container with ID starting with d695d95539345ee85b4ebf5ed1bb5cc069868b79a2cba30654927701839d9f5a not found: ID does not exist" containerID="d695d95539345ee85b4ebf5ed1bb5cc069868b79a2cba30654927701839d9f5a" Oct 30 17:37:38.669022 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:38.668987 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d695d95539345ee85b4ebf5ed1bb5cc069868b79a2cba30654927701839d9f5a"} err="failed to get container status \"d695d95539345ee85b4ebf5ed1bb5cc069868b79a2cba30654927701839d9f5a\": rpc error: code = NotFound desc = could not find container \"d695d95539345ee85b4ebf5ed1bb5cc069868b79a2cba30654927701839d9f5a\": container with ID starting with d695d95539345ee85b4ebf5ed1bb5cc069868b79a2cba30654927701839d9f5a not found: ID does not exist" Oct 30 17:37:38.734569 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:38.734536 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77"] Oct 30 17:37:38.748817 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:38.748782 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-tcx77"] Oct 30 17:37:40.144056 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:40.144018 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46" path="/var/lib/kubelet/pods/bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46/volumes" Oct 30 17:37:42.179685 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:42.179650 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2psr5"] Oct 30 17:37:42.180115 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:42.179850 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2psr5" podUID="7e69801d-f91f-47b2-96e6-fc4d9bf46678" containerName="discovery" containerID="cri-o://f3303ff42eef707940516e2c1f1f9ecbe1d81d04fc19dc2db0bc0ceff8316e3b" gracePeriod=30 Oct 30 17:37:42.450493 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:42.450464 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2psr5" Oct 30 17:37:42.478732 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:42.478700 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e69801d-f91f-47b2-96e6-fc4d9bf46678-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7e69801d-f91f-47b2-96e6-fc4d9bf46678-istio-csr-ca-configmap\") pod \"7e69801d-f91f-47b2-96e6-fc4d9bf46678\" (UID: \"7e69801d-f91f-47b2-96e6-fc4d9bf46678\") " Oct 30 17:37:42.478889 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:42.478754 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e69801d-f91f-47b2-96e6-fc4d9bf46678-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7e69801d-f91f-47b2-96e6-fc4d9bf46678-istio-csr-dns-cert\") pod \"7e69801d-f91f-47b2-96e6-fc4d9bf46678\" (UID: \"7e69801d-f91f-47b2-96e6-fc4d9bf46678\") " Oct 30 17:37:42.478889 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:42.478810 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e69801d-f91f-47b2-96e6-fc4d9bf46678-cacerts\" (UniqueName: \"kubernetes.io/secret/7e69801d-f91f-47b2-96e6-fc4d9bf46678-cacerts\") pod \"7e69801d-f91f-47b2-96e6-fc4d9bf46678\" (UID: \"7e69801d-f91f-47b2-96e6-fc4d9bf46678\") " Oct 30 17:37:42.478889 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:42.478839 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e69801d-f91f-47b2-96e6-fc4d9bf46678-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7e69801d-f91f-47b2-96e6-fc4d9bf46678-istio-kubeconfig\") pod \"7e69801d-f91f-47b2-96e6-fc4d9bf46678\" (UID: \"7e69801d-f91f-47b2-96e6-fc4d9bf46678\") " Oct 30 17:37:42.478889 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:42.478882 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e69801d-f91f-47b2-96e6-fc4d9bf46678-kube-api-access-t9j7t\" (UniqueName: \"kubernetes.io/projected/7e69801d-f91f-47b2-96e6-fc4d9bf46678-kube-api-access-t9j7t\") pod \"7e69801d-f91f-47b2-96e6-fc4d9bf46678\" (UID: \"7e69801d-f91f-47b2-96e6-fc4d9bf46678\") " Oct 30 17:37:42.479081 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:42.478924 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e69801d-f91f-47b2-96e6-fc4d9bf46678-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e69801d-f91f-47b2-96e6-fc4d9bf46678-local-certs\") pod \"7e69801d-f91f-47b2-96e6-fc4d9bf46678\" (UID: \"7e69801d-f91f-47b2-96e6-fc4d9bf46678\") " Oct 30 17:37:42.479081 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:42.478952 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e69801d-f91f-47b2-96e6-fc4d9bf46678-istio-token\" (UniqueName: \"kubernetes.io/projected/7e69801d-f91f-47b2-96e6-fc4d9bf46678-istio-token\") pod \"7e69801d-f91f-47b2-96e6-fc4d9bf46678\" (UID: \"7e69801d-f91f-47b2-96e6-fc4d9bf46678\") " Oct 30 17:37:42.479195 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:42.479114 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e69801d-f91f-47b2-96e6-fc4d9bf46678-istio-csr-ca-configmap" pod "7e69801d-f91f-47b2-96e6-fc4d9bf46678" (UID: "7e69801d-f91f-47b2-96e6-fc4d9bf46678"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:37:42.479484 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:42.479458 4522 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7e69801d-f91f-47b2-96e6-fc4d9bf46678-istio-csr-ca-configmap\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:37:42.481467 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:42.481425 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e69801d-f91f-47b2-96e6-fc4d9bf46678-cacerts" pod "7e69801d-f91f-47b2-96e6-fc4d9bf46678" (UID: "7e69801d-f91f-47b2-96e6-fc4d9bf46678"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:37:42.481467 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:42.481448 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e69801d-f91f-47b2-96e6-fc4d9bf46678-istio-kubeconfig" pod "7e69801d-f91f-47b2-96e6-fc4d9bf46678" (UID: "7e69801d-f91f-47b2-96e6-fc4d9bf46678"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:37:42.481602 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:42.481550 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e69801d-f91f-47b2-96e6-fc4d9bf46678-istio-csr-dns-cert" pod "7e69801d-f91f-47b2-96e6-fc4d9bf46678" (UID: "7e69801d-f91f-47b2-96e6-fc4d9bf46678"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:37:42.481652 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:42.481605 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e69801d-f91f-47b2-96e6-fc4d9bf46678-local-certs" pod "7e69801d-f91f-47b2-96e6-fc4d9bf46678" (UID: "7e69801d-f91f-47b2-96e6-fc4d9bf46678"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:37:42.481697 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:42.481657 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e69801d-f91f-47b2-96e6-fc4d9bf46678-kube-api-access-t9j7t" pod "7e69801d-f91f-47b2-96e6-fc4d9bf46678" (UID: "7e69801d-f91f-47b2-96e6-fc4d9bf46678"). InnerVolumeSpecName "kube-api-access-t9j7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:37:42.481782 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:42.481767 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e69801d-f91f-47b2-96e6-fc4d9bf46678-istio-token" pod "7e69801d-f91f-47b2-96e6-fc4d9bf46678" (UID: "7e69801d-f91f-47b2-96e6-fc4d9bf46678"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:37:42.580575 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:42.580540 4522 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7e69801d-f91f-47b2-96e6-fc4d9bf46678-istio-csr-dns-cert\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:37:42.580575 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:42.580569 4522 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7e69801d-f91f-47b2-96e6-fc4d9bf46678-cacerts\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:37:42.580575 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:42.580581 4522 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7e69801d-f91f-47b2-96e6-fc4d9bf46678-istio-kubeconfig\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:37:42.580822 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:42.580590 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9j7t\" (UniqueName: \"kubernetes.io/projected/7e69801d-f91f-47b2-96e6-fc4d9bf46678-kube-api-access-t9j7t\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:37:42.580822 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:42.580600 4522 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e69801d-f91f-47b2-96e6-fc4d9bf46678-local-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:37:42.580822 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:42.580609 4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e69801d-f91f-47b2-96e6-fc4d9bf46678-istio-token\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:37:42.686926 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:42.686846 4522 generic.go:356] "Generic (PLEG): container finished" podID="7e69801d-f91f-47b2-96e6-fc4d9bf46678" containerID="f3303ff42eef707940516e2c1f1f9ecbe1d81d04fc19dc2db0bc0ceff8316e3b" exitCode=0 Oct 30 17:37:42.686926 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:42.686907 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2psr5" Oct 30 17:37:42.687159 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:42.686936 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2psr5" event={"ID":"7e69801d-f91f-47b2-96e6-fc4d9bf46678","Type":"ContainerDied","Data":"f3303ff42eef707940516e2c1f1f9ecbe1d81d04fc19dc2db0bc0ceff8316e3b"} Oct 30 17:37:42.687159 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:42.686998 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2psr5" event={"ID":"7e69801d-f91f-47b2-96e6-fc4d9bf46678","Type":"ContainerDied","Data":"c2ef87155fa68e60d55b35d020f0570f0168200e118e54dd8c6382ca9dbd9b28"} Oct 30 17:37:42.687159 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:42.687026 4522 scope.go:122] "RemoveContainer" containerID="f3303ff42eef707940516e2c1f1f9ecbe1d81d04fc19dc2db0bc0ceff8316e3b" Oct 30 17:37:42.698113 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:42.698093 4522 scope.go:122] "RemoveContainer" containerID="f3303ff42eef707940516e2c1f1f9ecbe1d81d04fc19dc2db0bc0ceff8316e3b" Oct 30 17:37:42.698423 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:37:42.698404 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3303ff42eef707940516e2c1f1f9ecbe1d81d04fc19dc2db0bc0ceff8316e3b\": container with ID starting with f3303ff42eef707940516e2c1f1f9ecbe1d81d04fc19dc2db0bc0ceff8316e3b not found: ID does not exist" containerID="f3303ff42eef707940516e2c1f1f9ecbe1d81d04fc19dc2db0bc0ceff8316e3b" Oct 30 17:37:42.698510 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:42.698430 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3303ff42eef707940516e2c1f1f9ecbe1d81d04fc19dc2db0bc0ceff8316e3b"} err="failed to get container status \"f3303ff42eef707940516e2c1f1f9ecbe1d81d04fc19dc2db0bc0ceff8316e3b\": rpc error: code = NotFound desc = could not find container \"f3303ff42eef707940516e2c1f1f9ecbe1d81d04fc19dc2db0bc0ceff8316e3b\": container with ID starting with f3303ff42eef707940516e2c1f1f9ecbe1d81d04fc19dc2db0bc0ceff8316e3b not found: ID does not exist" Oct 30 17:37:42.747685 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:42.747654 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2psr5"] Oct 30 17:37:42.752169 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:42.752137 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2psr5"] Oct 30 17:37:42.947643 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:37:42.947561 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:37:44.144432 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:44.144401 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e69801d-f91f-47b2-96e6-fc4d9bf46678" path="/var/lib/kubelet/pods/7e69801d-f91f-47b2-96e6-fc4d9bf46678/volumes" Oct 30 17:37:46.483247 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:46.483214 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-054acd-8fb78649f-2tvjd"] Oct 30 17:37:46.483767 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:46.483750 4522 cpu_manager.go:414] "containerMap: removing container" podUID="22a4271a-c94f-4902-9874-6712c9735cb3" containerName="payload-pre-processing" Oct 30 17:37:46.483816 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:46.483769 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22a4271a-c94f-4902-9874-6712c9735cb3" containerName="payload-pre-processing" Oct 30 17:37:46.483816 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:46.483780 4522 cpu_manager.go:414] "containerMap: removing container" podUID="2966625c-2ca6-4a73-b0e6-3eb648a2745c" containerName="revoke-keys" Oct 30 17:37:46.483816 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:46.483787 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2966625c-2ca6-4a73-b0e6-3eb648a2745c" containerName="revoke-keys" Oct 30 17:37:46.483928 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:46.483821 4522 cpu_manager.go:414] "containerMap: removing container" podUID="7e69801d-f91f-47b2-96e6-fc4d9bf46678" containerName="discovery" Oct 30 17:37:46.483928 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:46.483827 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e69801d-f91f-47b2-96e6-fc4d9bf46678" containerName="discovery" Oct 30 17:37:46.483928 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:46.483835 4522 cpu_manager.go:414] "containerMap: removing container" podUID="bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46" containerName="istio-proxy" Oct 30 17:37:46.483928 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:46.483839 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46" containerName="istio-proxy" Oct 30 17:37:46.483928 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:46.483852 4522 cpu_manager.go:414] "containerMap: removing container" podUID="c9734f86-d287-475b-982f-5c1b5443b16b" containerName="payload-processing" Oct 30 17:37:46.483928 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:46.483857 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9734f86-d287-475b-982f-5c1b5443b16b" containerName="payload-processing" Oct 30 17:37:46.484118 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:46.483951 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="22a4271a-c94f-4902-9874-6712c9735cb3" containerName="payload-pre-processing" Oct 30 17:37:46.484118 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:46.483962 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="2966625c-2ca6-4a73-b0e6-3eb648a2745c" containerName="revoke-keys" Oct 30 17:37:46.484118 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:46.483973 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e69801d-f91f-47b2-96e6-fc4d9bf46678" containerName="discovery" Oct 30 17:37:46.484118 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:46.483979 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc3cd556-0c9a-4b61-8f9b-7cbb39e3fd46" containerName="istio-proxy" Oct 30 17:37:46.484118 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:46.483987 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9734f86-d287-475b-982f-5c1b5443b16b" containerName="payload-processing" Oct 30 17:37:46.488726 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:46.488706 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-054acd-8fb78649f-2tvjd" Oct 30 17:37:46.523682 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:46.523296 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f17482c-7c2f-441a-9a58-fcfae062d99d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-054acd-8fb78649f-2tvjd\" (UID: \"0f17482c-7c2f-441a-9a58-fcfae062d99d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-054acd-8fb78649f-2tvjd" Oct 30 17:37:46.523682 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:46.523418 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x8vt\" (UniqueName: \"kubernetes.io/projected/0f17482c-7c2f-441a-9a58-fcfae062d99d-kube-api-access-5x8vt\") pod \"payload-pre-processing-e2e-ipp-b-054acd-8fb78649f-2tvjd\" (UID: \"0f17482c-7c2f-441a-9a58-fcfae062d99d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-054acd-8fb78649f-2tvjd" Oct 30 17:37:46.531616 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:46.531587 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-054acd-dockercfg-xwwz4\"" Oct 30 17:37:46.531937 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:46.531917 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-054acd\"" Oct 30 17:37:46.542042 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:46.542009 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-054acd-8fb78649f-2tvjd"] Oct 30 17:37:46.615616 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:46.615570 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-054acd-5bd75fbb7d-b8qks"] Oct 30 17:37:46.620283 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:46.620263 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-054acd-5bd75fbb7d-b8qks" Oct 30 17:37:46.624182 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:46.624149 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f17482c-7c2f-441a-9a58-fcfae062d99d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-054acd-8fb78649f-2tvjd\" (UID: \"0f17482c-7c2f-441a-9a58-fcfae062d99d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-054acd-8fb78649f-2tvjd" Oct 30 17:37:46.624334 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:46.624224 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x8vt\" (UniqueName: \"kubernetes.io/projected/0f17482c-7c2f-441a-9a58-fcfae062d99d-kube-api-access-5x8vt\") pod \"payload-pre-processing-e2e-ipp-b-054acd-8fb78649f-2tvjd\" (UID: \"0f17482c-7c2f-441a-9a58-fcfae062d99d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-054acd-8fb78649f-2tvjd" Oct 30 17:37:46.624892 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:46.624871 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f17482c-7c2f-441a-9a58-fcfae062d99d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-054acd-8fb78649f-2tvjd\" (UID: \"0f17482c-7c2f-441a-9a58-fcfae062d99d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-054acd-8fb78649f-2tvjd" Oct 30 17:37:46.645073 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:46.645042 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-054acd-5bd75fbb7d-b8qks"] Oct 30 17:37:46.655044 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:46.655014 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x8vt\" (UniqueName: \"kubernetes.io/projected/0f17482c-7c2f-441a-9a58-fcfae062d99d-kube-api-access-5x8vt\") pod \"payload-pre-processing-e2e-ipp-b-054acd-8fb78649f-2tvjd\" (UID: \"0f17482c-7c2f-441a-9a58-fcfae062d99d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-054acd-8fb78649f-2tvjd" Oct 30 17:37:46.725215 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:46.725175 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2d64419-530d-4740-affc-9730206d31a1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-054acd-5bd75fbb7d-b8qks\" (UID: \"d2d64419-530d-4740-affc-9730206d31a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-054acd-5bd75fbb7d-b8qks" Oct 30 17:37:46.725433 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:46.725378 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78f9z\" (UniqueName: \"kubernetes.io/projected/d2d64419-530d-4740-affc-9730206d31a1-kube-api-access-78f9z\") pod \"payload-processing-e2e-ipp-b-054acd-5bd75fbb7d-b8qks\" (UID: \"d2d64419-530d-4740-affc-9730206d31a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-054acd-5bd75fbb7d-b8qks" Oct 30 17:37:46.803294 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:46.803201 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-054acd-8fb78649f-2tvjd" Oct 30 17:37:46.827109 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:46.827065 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2d64419-530d-4740-affc-9730206d31a1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-054acd-5bd75fbb7d-b8qks\" (UID: \"d2d64419-530d-4740-affc-9730206d31a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-054acd-5bd75fbb7d-b8qks" Oct 30 17:37:46.827262 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:46.827224 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78f9z\" (UniqueName: \"kubernetes.io/projected/d2d64419-530d-4740-affc-9730206d31a1-kube-api-access-78f9z\") pod \"payload-processing-e2e-ipp-b-054acd-5bd75fbb7d-b8qks\" (UID: \"d2d64419-530d-4740-affc-9730206d31a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-054acd-5bd75fbb7d-b8qks" Oct 30 17:37:46.827841 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:46.827817 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2d64419-530d-4740-affc-9730206d31a1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-054acd-5bd75fbb7d-b8qks\" (UID: \"d2d64419-530d-4740-affc-9730206d31a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-054acd-5bd75fbb7d-b8qks" Oct 30 17:37:46.854396 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:46.854355 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78f9z\" (UniqueName: \"kubernetes.io/projected/d2d64419-530d-4740-affc-9730206d31a1-kube-api-access-78f9z\") pod \"payload-processing-e2e-ipp-b-054acd-5bd75fbb7d-b8qks\" (UID: \"d2d64419-530d-4740-affc-9730206d31a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-054acd-5bd75fbb7d-b8qks" Oct 30 17:37:46.936139 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:46.936101 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-054acd-5bd75fbb7d-b8qks" Oct 30 17:37:47.114651 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:37:47.114617 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f17482c_7c2f_441a_9a58_fcfae062d99d.slice/crio-31ada452cb838e99cebaa0fcfded306e5fc7f2b3fceaea4a0db5b9462e4eb57c WatchSource:0}: Error finding container 31ada452cb838e99cebaa0fcfded306e5fc7f2b3fceaea4a0db5b9462e4eb57c: Status 404 returned error can't find the container with id 31ada452cb838e99cebaa0fcfded306e5fc7f2b3fceaea4a0db5b9462e4eb57c Oct 30 17:37:47.115137 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:47.115117 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-054acd-8fb78649f-2tvjd"] Oct 30 17:37:47.247534 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:37:47.247504 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2d64419_530d_4740_affc_9730206d31a1.slice/crio-57bfa4df1c9ff42f61e4f1128f52cc07128841427b7b3d4d339c4a14cb8adda1 WatchSource:0}: Error finding container 57bfa4df1c9ff42f61e4f1128f52cc07128841427b7b3d4d339c4a14cb8adda1: Status 404 returned error can't find the container with id 57bfa4df1c9ff42f61e4f1128f52cc07128841427b7b3d4d339c4a14cb8adda1 Oct 30 17:37:47.247996 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:47.247971 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-054acd-5bd75fbb7d-b8qks"] Oct 30 17:37:47.727078 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:47.727044 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-054acd-8fb78649f-2tvjd" event={"ID":"0f17482c-7c2f-441a-9a58-fcfae062d99d","Type":"ContainerStarted","Data":"19801787dda5be12957a876aaee44aeef1e0075976408a357c4e58c0f07274c4"} Oct 30 17:37:47.727573 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:47.727095 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-054acd-8fb78649f-2tvjd" event={"ID":"0f17482c-7c2f-441a-9a58-fcfae062d99d","Type":"ContainerStarted","Data":"31ada452cb838e99cebaa0fcfded306e5fc7f2b3fceaea4a0db5b9462e4eb57c"} Oct 30 17:37:47.727573 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:47.727142 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-054acd-8fb78649f-2tvjd" Oct 30 17:37:47.728962 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:47.728941 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-054acd-5bd75fbb7d-b8qks" event={"ID":"d2d64419-530d-4740-affc-9730206d31a1","Type":"ContainerStarted","Data":"3c8e45b4a07105dc7a9e1b7620db5f2b861d339b2777c646f469f12db79e3834"} Oct 30 17:37:47.729097 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:47.728975 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-054acd-5bd75fbb7d-b8qks" event={"ID":"d2d64419-530d-4740-affc-9730206d31a1","Type":"ContainerStarted","Data":"57bfa4df1c9ff42f61e4f1128f52cc07128841427b7b3d4d339c4a14cb8adda1"} Oct 30 17:37:47.729097 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:47.729071 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-054acd-5bd75fbb7d-b8qks" Oct 30 17:37:47.776374 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:47.776323 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-054acd-8fb78649f-2tvjd" podStartSLOduration=1.776294338 podStartE2EDuration="1.776294338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:37:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:37:47.774487255 +0000 UTC m=+3041.775242156" watchObservedRunningTime="2026-10-30 17:37:47.776294338 +0000 UTC m=+3041.777049233" Oct 30 17:37:47.830493 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:47.830444 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-054acd-5bd75fbb7d-b8qks" podStartSLOduration=1.830430749 podStartE2EDuration="1.830430749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:37:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:37:47.827953518 +0000 UTC m=+3041.828708440" watchObservedRunningTime="2026-10-30 17:37:47.830430749 +0000 UTC m=+3041.831185649" Oct 30 17:37:54.362111 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:54.362057 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-031223-79c4b96cc9-4twf6"] Oct 30 17:37:54.362795 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:54.362325 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-031223-79c4b96cc9-4twf6" podUID="9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4" containerName="maas-api" containerID="cri-o://0bef0c6853ef220fb2e888c67d5e6a4a2f65f1643b2a8e3755a4570864d77973" gracePeriod=30 Oct 30 17:37:54.533126 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:54.533054 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-031223-547d957db6-v6n6p"] Oct 30 17:37:54.533565 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:54.533296 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-031223-547d957db6-v6n6p" podUID="804e4c48-d1d9-4e76-9d19-bdabc3a805f3" containerName="payload-processing" containerID="cri-o://58185126a3e3e35e9cd0db0bb94f05dcd9d439706ef977c782c097bf556bd3d2" gracePeriod=30 Oct 30 17:37:54.558800 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:54.558052 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-031223-867764c779-6mp6s"] Oct 30 17:37:54.558800 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:54.558288 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-031223-867764c779-6mp6s" podUID="6dab304c-05a7-4d7e-a8bc-46b808de8c8f" containerName="payload-pre-processing" containerID="cri-o://12bcdaeb76698779645db2ee39384850af1d01838d8028f0ef0e76743145043c" gracePeriod=30 Oct 30 17:37:54.703908 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:54.703872 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-031223-79c4b96cc9-4twf6" Oct 30 17:37:54.783848 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:54.783818 4522 generic.go:356] "Generic (PLEG): container finished" podID="6dab304c-05a7-4d7e-a8bc-46b808de8c8f" containerID="12bcdaeb76698779645db2ee39384850af1d01838d8028f0ef0e76743145043c" exitCode=0 Oct 30 17:37:54.784036 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:54.783916 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-031223-867764c779-6mp6s" event={"ID":"6dab304c-05a7-4d7e-a8bc-46b808de8c8f","Type":"ContainerDied","Data":"12bcdaeb76698779645db2ee39384850af1d01838d8028f0ef0e76743145043c"} Oct 30 17:37:54.785944 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:54.785903 4522 generic.go:356] "Generic (PLEG): container finished" podID="9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4" containerID="0bef0c6853ef220fb2e888c67d5e6a4a2f65f1643b2a8e3755a4570864d77973" exitCode=0 Oct 30 17:37:54.786403 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:54.786338 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-031223-79c4b96cc9-4twf6" Oct 30 17:37:54.786719 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:54.786680 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-031223-79c4b96cc9-4twf6" event={"ID":"9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4","Type":"ContainerDied","Data":"0bef0c6853ef220fb2e888c67d5e6a4a2f65f1643b2a8e3755a4570864d77973"} Oct 30 17:37:54.787573 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:54.786737 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-031223-79c4b96cc9-4twf6" event={"ID":"9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4","Type":"ContainerDied","Data":"2dc69ed45b8734caf315f58b6ebb60c2d9b2c24d700d99be0b72b7206f5e302d"} Oct 30 17:37:54.787573 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:54.786768 4522 scope.go:122] "RemoveContainer" containerID="0bef0c6853ef220fb2e888c67d5e6a4a2f65f1643b2a8e3755a4570864d77973" Oct 30 17:37:54.792198 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:54.792173 4522 generic.go:356] "Generic (PLEG): container finished" podID="804e4c48-d1d9-4e76-9d19-bdabc3a805f3" containerID="58185126a3e3e35e9cd0db0bb94f05dcd9d439706ef977c782c097bf556bd3d2" exitCode=0 Oct 30 17:37:54.792381 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:54.792347 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-031223-547d957db6-v6n6p" event={"ID":"804e4c48-d1d9-4e76-9d19-bdabc3a805f3","Type":"ContainerDied","Data":"58185126a3e3e35e9cd0db0bb94f05dcd9d439706ef977c782c097bf556bd3d2"} Oct 30 17:37:54.806155 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:54.806094 4522 scope.go:122] "RemoveContainer" containerID="0bef0c6853ef220fb2e888c67d5e6a4a2f65f1643b2a8e3755a4570864d77973" Oct 30 17:37:54.806534 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:37:54.806508 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bef0c6853ef220fb2e888c67d5e6a4a2f65f1643b2a8e3755a4570864d77973\": container with ID starting with 0bef0c6853ef220fb2e888c67d5e6a4a2f65f1643b2a8e3755a4570864d77973 not found: ID does not exist" containerID="0bef0c6853ef220fb2e888c67d5e6a4a2f65f1643b2a8e3755a4570864d77973" Oct 30 17:37:54.806636 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:54.806546 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bef0c6853ef220fb2e888c67d5e6a4a2f65f1643b2a8e3755a4570864d77973"} err="failed to get container status \"0bef0c6853ef220fb2e888c67d5e6a4a2f65f1643b2a8e3755a4570864d77973\": rpc error: code = NotFound desc = could not find container \"0bef0c6853ef220fb2e888c67d5e6a4a2f65f1643b2a8e3755a4570864d77973\": container with ID starting with 0bef0c6853ef220fb2e888c67d5e6a4a2f65f1643b2a8e3755a4570864d77973 not found: ID does not exist" Oct 30 17:37:54.814442 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:54.814408 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-86f50a-ntln2"] Oct 30 17:37:54.818903 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:54.818780 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4-kube-api-access-4xkc5\" (UniqueName: \"kubernetes.io/projected/9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4-kube-api-access-4xkc5\") pod \"9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4\" (UID: \"9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4\") " Oct 30 17:37:54.818903 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:54.818797 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-86f50a-ntln2"] Oct 30 17:37:54.818903 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:54.818886 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4-metrics-certs\") pod \"9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4\" (UID: \"9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4\") " Oct 30 17:37:54.819216 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:54.818929 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4-maas-api-tls\") pod \"9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4\" (UID: \"9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4\") " Oct 30 17:37:54.819216 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:54.819002 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4-openshift-service-ca\") pod \"9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4\" (UID: \"9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4\") " Oct 30 17:37:54.819645 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:54.819619 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4-openshift-service-ca" pod "9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4" (UID: "9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:37:54.822845 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:54.822813 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4-kube-api-access-4xkc5" pod "9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4" (UID: "9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4"). InnerVolumeSpecName "kube-api-access-4xkc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:37:54.822845 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:54.822826 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4-metrics-certs" pod "9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4" (UID: "9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:37:54.823029 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:54.822863 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4-maas-api-tls" pod "9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4" (UID: "9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:37:54.835190 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:54.835167 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-031223-547d957db6-v6n6p" Oct 30 17:37:54.850586 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:54.850565 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-031223-867764c779-6mp6s" Oct 30 17:37:54.919769 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:54.919691 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/804e4c48-d1d9-4e76-9d19-bdabc3a805f3-kube-api-access-rgdfp\" (UniqueName: \"kubernetes.io/projected/804e4c48-d1d9-4e76-9d19-bdabc3a805f3-kube-api-access-rgdfp\") pod \"804e4c48-d1d9-4e76-9d19-bdabc3a805f3\" (UID: \"804e4c48-d1d9-4e76-9d19-bdabc3a805f3\") " Oct 30 17:37:54.919928 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:54.919800 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dab304c-05a7-4d7e-a8bc-46b808de8c8f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dab304c-05a7-4d7e-a8bc-46b808de8c8f-plugins-config-volume\") pod \"6dab304c-05a7-4d7e-a8bc-46b808de8c8f\" (UID: \"6dab304c-05a7-4d7e-a8bc-46b808de8c8f\") " Oct 30 17:37:54.920006 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:54.919987 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/804e4c48-d1d9-4e76-9d19-bdabc3a805f3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/804e4c48-d1d9-4e76-9d19-bdabc3a805f3-plugins-config-volume\") pod \"804e4c48-d1d9-4e76-9d19-bdabc3a805f3\" (UID: \"804e4c48-d1d9-4e76-9d19-bdabc3a805f3\") " Oct 30 17:37:54.920122 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:54.920104 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dab304c-05a7-4d7e-a8bc-46b808de8c8f-kube-api-access-429bf\" (UniqueName: \"kubernetes.io/projected/6dab304c-05a7-4d7e-a8bc-46b808de8c8f-kube-api-access-429bf\") pod \"6dab304c-05a7-4d7e-a8bc-46b808de8c8f\" (UID: \"6dab304c-05a7-4d7e-a8bc-46b808de8c8f\") " Oct 30 17:37:54.920200 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:54.920158 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dab304c-05a7-4d7e-a8bc-46b808de8c8f-plugins-config-volume" pod "6dab304c-05a7-4d7e-a8bc-46b808de8c8f" (UID: "6dab304c-05a7-4d7e-a8bc-46b808de8c8f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:37:54.920447 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:54.920418 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/804e4c48-d1d9-4e76-9d19-bdabc3a805f3-plugins-config-volume" pod "804e4c48-d1d9-4e76-9d19-bdabc3a805f3" (UID: "804e4c48-d1d9-4e76-9d19-bdabc3a805f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:37:54.920623 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:54.920483 4522 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4-metrics-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:37:54.920623 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:54.920515 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4-maas-api-tls\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:37:54.920623 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:54.920529 4522 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4-openshift-service-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:37:54.920623 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:54.920543 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xkc5\" (UniqueName: \"kubernetes.io/projected/9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4-kube-api-access-4xkc5\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:37:54.920623 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:54.920559 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dab304c-05a7-4d7e-a8bc-46b808de8c8f-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:37:54.922386 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:54.922358 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dab304c-05a7-4d7e-a8bc-46b808de8c8f-kube-api-access-429bf" pod "6dab304c-05a7-4d7e-a8bc-46b808de8c8f" (UID: "6dab304c-05a7-4d7e-a8bc-46b808de8c8f"). InnerVolumeSpecName "kube-api-access-429bf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:37:54.922536 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:54.922516 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/804e4c48-d1d9-4e76-9d19-bdabc3a805f3-kube-api-access-rgdfp" pod "804e4c48-d1d9-4e76-9d19-bdabc3a805f3" (UID: "804e4c48-d1d9-4e76-9d19-bdabc3a805f3"). InnerVolumeSpecName "kube-api-access-rgdfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:37:55.022024 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:55.021985 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/804e4c48-d1d9-4e76-9d19-bdabc3a805f3-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:37:55.022024 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:55.022020 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-429bf\" (UniqueName: \"kubernetes.io/projected/6dab304c-05a7-4d7e-a8bc-46b808de8c8f-kube-api-access-429bf\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:37:55.022024 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:55.022031 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgdfp\" (UniqueName: \"kubernetes.io/projected/804e4c48-d1d9-4e76-9d19-bdabc3a805f3-kube-api-access-rgdfp\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:37:55.139724 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:55.139689 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-031223-79c4b96cc9-4twf6"] Oct 30 17:37:55.142195 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:55.142167 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-031223-79c4b96cc9-4twf6"] Oct 30 17:37:55.287250 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:55.287220 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-031223-gtwqd"] Oct 30 17:37:55.289905 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:55.289885 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-031223-gtwqd"] Oct 30 17:37:55.806083 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:55.806053 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-031223-547d957db6-v6n6p" Oct 30 17:37:55.806083 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:55.806072 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-031223-547d957db6-v6n6p" event={"ID":"804e4c48-d1d9-4e76-9d19-bdabc3a805f3","Type":"ContainerDied","Data":"14620db30cd1d664837d5dfb0d9759789667ca028bb109ce06cf09ddea262240"} Oct 30 17:37:55.806623 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:55.806159 4522 scope.go:122] "RemoveContainer" containerID="58185126a3e3e35e9cd0db0bb94f05dcd9d439706ef977c782c097bf556bd3d2" Oct 30 17:37:55.807791 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:55.807755 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-031223-867764c779-6mp6s" event={"ID":"6dab304c-05a7-4d7e-a8bc-46b808de8c8f","Type":"ContainerDied","Data":"4cb2fe8450b916a20464c2a1d508a9cafa0f10e685290d6039a5a72cecf36099"} Oct 30 17:37:55.807912 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:55.807764 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-031223-867764c779-6mp6s" Oct 30 17:37:55.817492 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:55.817470 4522 scope.go:122] "RemoveContainer" containerID="12bcdaeb76698779645db2ee39384850af1d01838d8028f0ef0e76743145043c" Oct 30 17:37:55.876271 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:55.876236 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-031223-867764c779-6mp6s"] Oct 30 17:37:55.878567 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:55.878544 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-031223-867764c779-6mp6s"] Oct 30 17:37:55.915448 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:55.915413 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-031223-547d957db6-v6n6p"] Oct 30 17:37:55.919818 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:55.919789 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-031223-547d957db6-v6n6p"] Oct 30 17:37:56.144827 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:56.144748 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="039ed493-8a78-4672-bc24-4e2b37cee2e4" path="/var/lib/kubelet/pods/039ed493-8a78-4672-bc24-4e2b37cee2e4/volumes" Oct 30 17:37:56.145200 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:56.145186 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2966625c-2ca6-4a73-b0e6-3eb648a2745c" path="/var/lib/kubelet/pods/2966625c-2ca6-4a73-b0e6-3eb648a2745c/volumes" Oct 30 17:37:56.145586 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:56.145573 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dab304c-05a7-4d7e-a8bc-46b808de8c8f" path="/var/lib/kubelet/pods/6dab304c-05a7-4d7e-a8bc-46b808de8c8f/volumes" Oct 30 17:37:56.146196 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:56.146182 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="804e4c48-d1d9-4e76-9d19-bdabc3a805f3" path="/var/lib/kubelet/pods/804e4c48-d1d9-4e76-9d19-bdabc3a805f3/volumes" Oct 30 17:37:56.146595 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:56.146582 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4" path="/var/lib/kubelet/pods/9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4/volumes" Oct 30 17:37:58.738216 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:58.738189 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-054acd-5bd75fbb7d-b8qks" Oct 30 17:37:58.738714 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:37:58.738257 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-054acd-8fb78649f-2tvjd" Oct 30 17:38:02.476868 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:02.476829 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7"] Oct 30 17:38:02.477336 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:02.477082 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7" podUID="c6ffa9d7-4de9-4b98-8e7e-016db7370f08" containerName="istio-proxy" containerID="cri-o://a75994c0c66a7d2db476768b52b2221a36b163fdf6b06263a5613f123bf21b1d" gracePeriod=30 Oct 30 17:38:03.048392 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:03.048354 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-8f92ef/ipp-route-8f92ef-kserve-7948cdcdf-r2dkv"] Oct 30 17:38:03.049119 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:03.049097 4522 cpu_manager.go:414] "containerMap: removing container" podUID="9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4" containerName="maas-api" Oct 30 17:38:03.049181 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:03.049124 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4" containerName="maas-api" Oct 30 17:38:03.049181 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:03.049173 4522 cpu_manager.go:414] "containerMap: removing container" podUID="804e4c48-d1d9-4e76-9d19-bdabc3a805f3" containerName="payload-processing" Oct 30 17:38:03.049256 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:03.049182 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="804e4c48-d1d9-4e76-9d19-bdabc3a805f3" containerName="payload-processing" Oct 30 17:38:03.049256 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:03.049214 4522 cpu_manager.go:414] "containerMap: removing container" podUID="6dab304c-05a7-4d7e-a8bc-46b808de8c8f" containerName="payload-pre-processing" Oct 30 17:38:03.049256 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:03.049223 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dab304c-05a7-4d7e-a8bc-46b808de8c8f" containerName="payload-pre-processing" Oct 30 17:38:03.049429 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:03.049412 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f24bd0b-f62f-4634-91f3-5f4ae16c9fb4" containerName="maas-api" Oct 30 17:38:03.049483 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:03.049443 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="804e4c48-d1d9-4e76-9d19-bdabc3a805f3" containerName="payload-processing" Oct 30 17:38:03.049483 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:03.049454 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dab304c-05a7-4d7e-a8bc-46b808de8c8f" containerName="payload-pre-processing" Oct 30 17:38:03.054941 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:03.054916 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-8f92ef/ipp-route-8f92ef-kserve-7948cdcdf-r2dkv" Oct 30 17:38:03.072779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:03.072740 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-8f92ef\"/\"default-dockercfg-tpdpq\"" Oct 30 17:38:03.072960 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:03.072832 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-8f92ef\"/\"ipp-route-8f92ef-kserve-self-signed-certs\"" Oct 30 17:38:03.073720 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:03.073700 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-8f92ef/ipp-route-8f92ef-kserve-7948cdcdf-r2dkv"] Oct 30 17:38:03.084831 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:03.084804 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-8f92ef\"/\"kube-root-ca.crt\"" Oct 30 17:38:03.084992 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:03.084832 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-8f92ef\"/\"openshift-service-ca.crt\"" Oct 30 17:38:03.214000 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:03.213959 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a834b00-2f75-49d9-9624-ed1545b1a989-home\") pod \"ipp-route-8f92ef-kserve-7948cdcdf-r2dkv\" (UID: \"8a834b00-2f75-49d9-9624-ed1545b1a989\") " pod="ai-tenant-e2e-ipp-a-8f92ef/ipp-route-8f92ef-kserve-7948cdcdf-r2dkv" Oct 30 17:38:03.214201 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:03.214045 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a834b00-2f75-49d9-9624-ed1545b1a989-model-cache\") pod \"ipp-route-8f92ef-kserve-7948cdcdf-r2dkv\" (UID: \"8a834b00-2f75-49d9-9624-ed1545b1a989\") " pod="ai-tenant-e2e-ipp-a-8f92ef/ipp-route-8f92ef-kserve-7948cdcdf-r2dkv" Oct 30 17:38:03.214201 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:03.214094 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a834b00-2f75-49d9-9624-ed1545b1a989-tmp-dir\") pod \"ipp-route-8f92ef-kserve-7948cdcdf-r2dkv\" (UID: \"8a834b00-2f75-49d9-9624-ed1545b1a989\") " pod="ai-tenant-e2e-ipp-a-8f92ef/ipp-route-8f92ef-kserve-7948cdcdf-r2dkv" Oct 30 17:38:03.214201 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:03.214144 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a834b00-2f75-49d9-9624-ed1545b1a989-tls-certs\") pod \"ipp-route-8f92ef-kserve-7948cdcdf-r2dkv\" (UID: \"8a834b00-2f75-49d9-9624-ed1545b1a989\") " pod="ai-tenant-e2e-ipp-a-8f92ef/ipp-route-8f92ef-kserve-7948cdcdf-r2dkv" Oct 30 17:38:03.214201 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:03.214197 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a834b00-2f75-49d9-9624-ed1545b1a989-dshm\") pod \"ipp-route-8f92ef-kserve-7948cdcdf-r2dkv\" (UID: \"8a834b00-2f75-49d9-9624-ed1545b1a989\") " pod="ai-tenant-e2e-ipp-a-8f92ef/ipp-route-8f92ef-kserve-7948cdcdf-r2dkv" Oct 30 17:38:03.214390 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:03.214245 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6shdm\" (UniqueName: \"kubernetes.io/projected/8a834b00-2f75-49d9-9624-ed1545b1a989-kube-api-access-6shdm\") pod \"ipp-route-8f92ef-kserve-7948cdcdf-r2dkv\" (UID: \"8a834b00-2f75-49d9-9624-ed1545b1a989\") " pod="ai-tenant-e2e-ipp-a-8f92ef/ipp-route-8f92ef-kserve-7948cdcdf-r2dkv" Oct 30 17:38:03.315554 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:03.315455 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a834b00-2f75-49d9-9624-ed1545b1a989-tmp-dir\") pod \"ipp-route-8f92ef-kserve-7948cdcdf-r2dkv\" (UID: \"8a834b00-2f75-49d9-9624-ed1545b1a989\") " pod="ai-tenant-e2e-ipp-a-8f92ef/ipp-route-8f92ef-kserve-7948cdcdf-r2dkv" Oct 30 17:38:03.315554 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:03.315525 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a834b00-2f75-49d9-9624-ed1545b1a989-tls-certs\") pod \"ipp-route-8f92ef-kserve-7948cdcdf-r2dkv\" (UID: \"8a834b00-2f75-49d9-9624-ed1545b1a989\") " pod="ai-tenant-e2e-ipp-a-8f92ef/ipp-route-8f92ef-kserve-7948cdcdf-r2dkv" Oct 30 17:38:03.315554 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:03.315549 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a834b00-2f75-49d9-9624-ed1545b1a989-dshm\") pod \"ipp-route-8f92ef-kserve-7948cdcdf-r2dkv\" (UID: \"8a834b00-2f75-49d9-9624-ed1545b1a989\") " pod="ai-tenant-e2e-ipp-a-8f92ef/ipp-route-8f92ef-kserve-7948cdcdf-r2dkv" Oct 30 17:38:03.315882 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:03.315585 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6shdm\" (UniqueName: \"kubernetes.io/projected/8a834b00-2f75-49d9-9624-ed1545b1a989-kube-api-access-6shdm\") pod \"ipp-route-8f92ef-kserve-7948cdcdf-r2dkv\" (UID: \"8a834b00-2f75-49d9-9624-ed1545b1a989\") " pod="ai-tenant-e2e-ipp-a-8f92ef/ipp-route-8f92ef-kserve-7948cdcdf-r2dkv" Oct 30 17:38:03.315882 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:03.315753 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a834b00-2f75-49d9-9624-ed1545b1a989-home\") pod \"ipp-route-8f92ef-kserve-7948cdcdf-r2dkv\" (UID: \"8a834b00-2f75-49d9-9624-ed1545b1a989\") " pod="ai-tenant-e2e-ipp-a-8f92ef/ipp-route-8f92ef-kserve-7948cdcdf-r2dkv" Oct 30 17:38:03.315882 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:03.315778 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a834b00-2f75-49d9-9624-ed1545b1a989-model-cache\") pod \"ipp-route-8f92ef-kserve-7948cdcdf-r2dkv\" (UID: \"8a834b00-2f75-49d9-9624-ed1545b1a989\") " pod="ai-tenant-e2e-ipp-a-8f92ef/ipp-route-8f92ef-kserve-7948cdcdf-r2dkv" Oct 30 17:38:03.316070 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:03.315911 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a834b00-2f75-49d9-9624-ed1545b1a989-tmp-dir\") pod \"ipp-route-8f92ef-kserve-7948cdcdf-r2dkv\" (UID: \"8a834b00-2f75-49d9-9624-ed1545b1a989\") " pod="ai-tenant-e2e-ipp-a-8f92ef/ipp-route-8f92ef-kserve-7948cdcdf-r2dkv" Oct 30 17:38:03.316070 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:03.316031 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a834b00-2f75-49d9-9624-ed1545b1a989-home\") pod \"ipp-route-8f92ef-kserve-7948cdcdf-r2dkv\" (UID: \"8a834b00-2f75-49d9-9624-ed1545b1a989\") " pod="ai-tenant-e2e-ipp-a-8f92ef/ipp-route-8f92ef-kserve-7948cdcdf-r2dkv" Oct 30 17:38:03.316177 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:03.316119 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a834b00-2f75-49d9-9624-ed1545b1a989-model-cache\") pod \"ipp-route-8f92ef-kserve-7948cdcdf-r2dkv\" (UID: \"8a834b00-2f75-49d9-9624-ed1545b1a989\") " pod="ai-tenant-e2e-ipp-a-8f92ef/ipp-route-8f92ef-kserve-7948cdcdf-r2dkv" Oct 30 17:38:03.317855 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:03.317833 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a834b00-2f75-49d9-9624-ed1545b1a989-dshm\") pod \"ipp-route-8f92ef-kserve-7948cdcdf-r2dkv\" (UID: \"8a834b00-2f75-49d9-9624-ed1545b1a989\") " pod="ai-tenant-e2e-ipp-a-8f92ef/ipp-route-8f92ef-kserve-7948cdcdf-r2dkv" Oct 30 17:38:03.318138 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:03.318118 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a834b00-2f75-49d9-9624-ed1545b1a989-tls-certs\") pod \"ipp-route-8f92ef-kserve-7948cdcdf-r2dkv\" (UID: \"8a834b00-2f75-49d9-9624-ed1545b1a989\") " pod="ai-tenant-e2e-ipp-a-8f92ef/ipp-route-8f92ef-kserve-7948cdcdf-r2dkv" Oct 30 17:38:03.354682 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:03.354647 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6shdm\" (UniqueName: \"kubernetes.io/projected/8a834b00-2f75-49d9-9624-ed1545b1a989-kube-api-access-6shdm\") pod \"ipp-route-8f92ef-kserve-7948cdcdf-r2dkv\" (UID: \"8a834b00-2f75-49d9-9624-ed1545b1a989\") " pod="ai-tenant-e2e-ipp-a-8f92ef/ipp-route-8f92ef-kserve-7948cdcdf-r2dkv" Oct 30 17:38:03.369501 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:03.369475 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-8f92ef/ipp-route-8f92ef-kserve-7948cdcdf-r2dkv" Oct 30 17:38:03.681581 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:03.681548 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-8f92ef/ipp-route-8f92ef-kserve-7948cdcdf-r2dkv"] Oct 30 17:38:03.684059 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:38:03.684021 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a834b00_2f75_49d9_9624_ed1545b1a989.slice/crio-6ea4b6b42cdc2cba1343fd9f088c1871dd2b931d5442723748ba8f4a5d465d0d WatchSource:0}: Error finding container 6ea4b6b42cdc2cba1343fd9f088c1871dd2b931d5442723748ba8f4a5d465d0d: Status 404 returned error can't find the container with id 6ea4b6b42cdc2cba1343fd9f088c1871dd2b931d5442723748ba8f4a5d465d0d Oct 30 17:38:03.868184 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:03.868146 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-8f92ef/ipp-route-8f92ef-kserve-7948cdcdf-r2dkv" event={"ID":"8a834b00-2f75-49d9-9624-ed1545b1a989","Type":"ContainerStarted","Data":"cffac3184332cc22b788902b791e0d1500d4655e5c2e596e044353ac363ca097"} Oct 30 17:38:03.868396 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:03.868194 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-8f92ef/ipp-route-8f92ef-kserve-7948cdcdf-r2dkv" event={"ID":"8a834b00-2f75-49d9-9624-ed1545b1a989","Type":"ContainerStarted","Data":"6ea4b6b42cdc2cba1343fd9f088c1871dd2b931d5442723748ba8f4a5d465d0d"} Oct 30 17:38:03.915995 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:03.915934 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-8f92ef/ipp-route-8f92ef-kserve-7948cdcdf-r2dkv" podStartSLOduration=0.915920089 podStartE2EDuration="915.920089ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:38:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:38:03.91440305 +0000 UTC m=+3057.915157951" watchObservedRunningTime="2026-10-30 17:38:03.915920089 +0000 UTC m=+3057.916674989" Oct 30 17:38:04.259392 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:04.259358 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-469ad7-cblh2"] Oct 30 17:38:04.263178 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:04.263159 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-469ad7-cblh2" Oct 30 17:38:04.282339 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:04.281610 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:38:04.282339 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:04.281692 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:38:04.285565 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:04.285540 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-469ad7-cblh2"] Oct 30 17:38:04.370283 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:04.370253 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-8f92ef/ipp-route-8f92ef-kserve-7948cdcdf-r2dkv" Oct 30 17:38:04.427858 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:04.427820 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mtnv\" (UniqueName: \"kubernetes.io/projected/0321e5b7-aeed-43af-b484-0829a9a06f5f-kube-api-access-7mtnv\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-469ad7-cblh2\" (UID: \"0321e5b7-aeed-43af-b484-0829a9a06f5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-469ad7-cblh2" Oct 30 17:38:04.428057 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:04.427946 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0321e5b7-aeed-43af-b484-0829a9a06f5f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-469ad7-cblh2\" (UID: \"0321e5b7-aeed-43af-b484-0829a9a06f5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-469ad7-cblh2" Oct 30 17:38:04.428057 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:04.428033 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0321e5b7-aeed-43af-b484-0829a9a06f5f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-469ad7-cblh2\" (UID: \"0321e5b7-aeed-43af-b484-0829a9a06f5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-469ad7-cblh2" Oct 30 17:38:04.529745 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:04.529658 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0321e5b7-aeed-43af-b484-0829a9a06f5f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-469ad7-cblh2\" (UID: \"0321e5b7-aeed-43af-b484-0829a9a06f5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-469ad7-cblh2" Oct 30 17:38:04.529900 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:04.529804 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mtnv\" (UniqueName: \"kubernetes.io/projected/0321e5b7-aeed-43af-b484-0829a9a06f5f-kube-api-access-7mtnv\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-469ad7-cblh2\" (UID: \"0321e5b7-aeed-43af-b484-0829a9a06f5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-469ad7-cblh2" Oct 30 17:38:04.529900 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:04.529861 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0321e5b7-aeed-43af-b484-0829a9a06f5f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-469ad7-cblh2\" (UID: \"0321e5b7-aeed-43af-b484-0829a9a06f5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-469ad7-cblh2" Oct 30 17:38:04.530423 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:04.530398 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0321e5b7-aeed-43af-b484-0829a9a06f5f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-469ad7-cblh2\" (UID: \"0321e5b7-aeed-43af-b484-0829a9a06f5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-469ad7-cblh2" Oct 30 17:38:04.532217 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:04.532192 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0321e5b7-aeed-43af-b484-0829a9a06f5f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-469ad7-cblh2\" (UID: \"0321e5b7-aeed-43af-b484-0829a9a06f5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-469ad7-cblh2" Oct 30 17:38:04.556501 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:04.556460 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mtnv\" (UniqueName: \"kubernetes.io/projected/0321e5b7-aeed-43af-b484-0829a9a06f5f-kube-api-access-7mtnv\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-469ad7-cblh2\" (UID: \"0321e5b7-aeed-43af-b484-0829a9a06f5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-469ad7-cblh2" Oct 30 17:38:04.576690 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:04.576663 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-469ad7-cblh2" Oct 30 17:38:04.881238 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:04.881207 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-469ad7-cblh2"] Oct 30 17:38:04.881681 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:38:04.881233 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0321e5b7_aeed_43af_b484_0829a9a06f5f.slice/crio-15e6ba19a3cb8451ba27ea4ce637c697d44077c7346e673e46590041542f80a4 WatchSource:0}: Error finding container 15e6ba19a3cb8451ba27ea4ce637c697d44077c7346e673e46590041542f80a4: Status 404 returned error can't find the container with id 15e6ba19a3cb8451ba27ea4ce637c697d44077c7346e673e46590041542f80a4 Oct 30 17:38:05.883574 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:05.883542 4522 generic.go:356] "Generic (PLEG): container finished" podID="0321e5b7-aeed-43af-b484-0829a9a06f5f" containerID="161d1894517498a2c81f823624697cccca7454eae5c550d7a3adb492a08a7f6e" exitCode=0 Oct 30 17:38:05.883965 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:05.883626 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-469ad7-cblh2" event={"ID":"0321e5b7-aeed-43af-b484-0829a9a06f5f","Type":"ContainerDied","Data":"161d1894517498a2c81f823624697cccca7454eae5c550d7a3adb492a08a7f6e"} Oct 30 17:38:05.883965 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:05.883678 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-469ad7-cblh2" event={"ID":"0321e5b7-aeed-43af-b484-0829a9a06f5f","Type":"ContainerStarted","Data":"15e6ba19a3cb8451ba27ea4ce637c697d44077c7346e673e46590041542f80a4"} Oct 30 17:38:07.033082 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:07.033054 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-469ad7-cblh2" Oct 30 17:38:07.158522 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:07.158433 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0321e5b7-aeed-43af-b484-0829a9a06f5f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0321e5b7-aeed-43af-b484-0829a9a06f5f-maas-api-xks-ca\") pod \"0321e5b7-aeed-43af-b484-0829a9a06f5f\" (UID: \"0321e5b7-aeed-43af-b484-0829a9a06f5f\") " Oct 30 17:38:07.158522 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:07.158478 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0321e5b7-aeed-43af-b484-0829a9a06f5f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0321e5b7-aeed-43af-b484-0829a9a06f5f-maas-api-openshift-ca\") pod \"0321e5b7-aeed-43af-b484-0829a9a06f5f\" (UID: \"0321e5b7-aeed-43af-b484-0829a9a06f5f\") " Oct 30 17:38:07.158724 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:07.158592 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0321e5b7-aeed-43af-b484-0829a9a06f5f-kube-api-access-7mtnv\" (UniqueName: \"kubernetes.io/projected/0321e5b7-aeed-43af-b484-0829a9a06f5f-kube-api-access-7mtnv\") pod \"0321e5b7-aeed-43af-b484-0829a9a06f5f\" (UID: \"0321e5b7-aeed-43af-b484-0829a9a06f5f\") " Oct 30 17:38:07.158918 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:07.158865 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0321e5b7-aeed-43af-b484-0829a9a06f5f-maas-api-openshift-ca" pod "0321e5b7-aeed-43af-b484-0829a9a06f5f" (UID: "0321e5b7-aeed-43af-b484-0829a9a06f5f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:38:07.159088 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:07.159073 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0321e5b7-aeed-43af-b484-0829a9a06f5f-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:38:07.160745 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:07.160721 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0321e5b7-aeed-43af-b484-0829a9a06f5f-maas-api-xks-ca" pod "0321e5b7-aeed-43af-b484-0829a9a06f5f" (UID: "0321e5b7-aeed-43af-b484-0829a9a06f5f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:38:07.160839 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:07.160726 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0321e5b7-aeed-43af-b484-0829a9a06f5f-kube-api-access-7mtnv" pod "0321e5b7-aeed-43af-b484-0829a9a06f5f" (UID: "0321e5b7-aeed-43af-b484-0829a9a06f5f"). InnerVolumeSpecName "kube-api-access-7mtnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:38:07.259698 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:07.259659 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0321e5b7-aeed-43af-b484-0829a9a06f5f-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:38:07.259698 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:07.259693 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mtnv\" (UniqueName: \"kubernetes.io/projected/0321e5b7-aeed-43af-b484-0829a9a06f5f-kube-api-access-7mtnv\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:38:07.728291 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:07.728264 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7" Oct 30 17:38:07.866480 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:07.866386 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c6ffa9d7-4de9-4b98-8e7e-016db7370f08-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6ffa9d7-4de9-4b98-8e7e-016db7370f08-istio-podinfo\") pod \"c6ffa9d7-4de9-4b98-8e7e-016db7370f08\" (UID: \"c6ffa9d7-4de9-4b98-8e7e-016db7370f08\") " Oct 30 17:38:07.866480 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:07.866460 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6ffa9d7-4de9-4b98-8e7e-016db7370f08-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ffa9d7-4de9-4b98-8e7e-016db7370f08-workload-socket\") pod \"c6ffa9d7-4de9-4b98-8e7e-016db7370f08\" (UID: \"c6ffa9d7-4de9-4b98-8e7e-016db7370f08\") " Oct 30 17:38:07.866735 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:07.866493 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6ffa9d7-4de9-4b98-8e7e-016db7370f08-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6ffa9d7-4de9-4b98-8e7e-016db7370f08-istio-envoy\") pod \"c6ffa9d7-4de9-4b98-8e7e-016db7370f08\" (UID: \"c6ffa9d7-4de9-4b98-8e7e-016db7370f08\") " Oct 30 17:38:07.866735 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:07.866536 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6ffa9d7-4de9-4b98-8e7e-016db7370f08-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6ffa9d7-4de9-4b98-8e7e-016db7370f08-istio-data\") pod \"c6ffa9d7-4de9-4b98-8e7e-016db7370f08\" (UID: \"c6ffa9d7-4de9-4b98-8e7e-016db7370f08\") " Oct 30 17:38:07.866735 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:07.866592 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6ffa9d7-4de9-4b98-8e7e-016db7370f08-kube-api-access-rc6v5\" (UniqueName: \"kubernetes.io/projected/c6ffa9d7-4de9-4b98-8e7e-016db7370f08-kube-api-access-rc6v5\") pod \"c6ffa9d7-4de9-4b98-8e7e-016db7370f08\" (UID: \"c6ffa9d7-4de9-4b98-8e7e-016db7370f08\") " Oct 30 17:38:07.866735 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:07.866630 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6ffa9d7-4de9-4b98-8e7e-016db7370f08-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6ffa9d7-4de9-4b98-8e7e-016db7370f08-istiod-ca-cert\") pod \"c6ffa9d7-4de9-4b98-8e7e-016db7370f08\" (UID: \"c6ffa9d7-4de9-4b98-8e7e-016db7370f08\") " Oct 30 17:38:07.866735 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:07.866658 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6ffa9d7-4de9-4b98-8e7e-016db7370f08-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ffa9d7-4de9-4b98-8e7e-016db7370f08-credential-socket\") pod \"c6ffa9d7-4de9-4b98-8e7e-016db7370f08\" (UID: \"c6ffa9d7-4de9-4b98-8e7e-016db7370f08\") " Oct 30 17:38:07.866735 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:07.866716 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6ffa9d7-4de9-4b98-8e7e-016db7370f08-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6ffa9d7-4de9-4b98-8e7e-016db7370f08-workload-certs\") pod \"c6ffa9d7-4de9-4b98-8e7e-016db7370f08\" (UID: \"c6ffa9d7-4de9-4b98-8e7e-016db7370f08\") " Oct 30 17:38:07.867140 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:07.866772 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6ffa9d7-4de9-4b98-8e7e-016db7370f08-istio-token\" (UniqueName: \"kubernetes.io/projected/c6ffa9d7-4de9-4b98-8e7e-016db7370f08-istio-token\") pod \"c6ffa9d7-4de9-4b98-8e7e-016db7370f08\" (UID: \"c6ffa9d7-4de9-4b98-8e7e-016db7370f08\") " Oct 30 17:38:07.867140 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:07.866841 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6ffa9d7-4de9-4b98-8e7e-016db7370f08-workload-socket" pod "c6ffa9d7-4de9-4b98-8e7e-016db7370f08" (UID: "c6ffa9d7-4de9-4b98-8e7e-016db7370f08"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:07.867140 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:07.866996 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6ffa9d7-4de9-4b98-8e7e-016db7370f08-workload-certs" pod "c6ffa9d7-4de9-4b98-8e7e-016db7370f08" (UID: "c6ffa9d7-4de9-4b98-8e7e-016db7370f08"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:07.867140 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:07.867014 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6ffa9d7-4de9-4b98-8e7e-016db7370f08-credential-socket" pod "c6ffa9d7-4de9-4b98-8e7e-016db7370f08" (UID: "c6ffa9d7-4de9-4b98-8e7e-016db7370f08"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:07.867405 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:07.867160 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6ffa9d7-4de9-4b98-8e7e-016db7370f08-istiod-ca-cert" pod "c6ffa9d7-4de9-4b98-8e7e-016db7370f08" (UID: "c6ffa9d7-4de9-4b98-8e7e-016db7370f08"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:38:07.867405 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:07.867297 4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ffa9d7-4de9-4b98-8e7e-016db7370f08-workload-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:38:07.867405 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:07.867368 4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6ffa9d7-4de9-4b98-8e7e-016db7370f08-istiod-ca-cert\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:38:07.867405 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:07.867383 4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ffa9d7-4de9-4b98-8e7e-016db7370f08-credential-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:38:07.867405 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:07.867398 4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6ffa9d7-4de9-4b98-8e7e-016db7370f08-workload-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:38:07.867698 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:07.867521 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6ffa9d7-4de9-4b98-8e7e-016db7370f08-istio-data" pod "c6ffa9d7-4de9-4b98-8e7e-016db7370f08" (UID: "c6ffa9d7-4de9-4b98-8e7e-016db7370f08"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:07.868896 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:07.868868 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6ffa9d7-4de9-4b98-8e7e-016db7370f08-istio-envoy" pod "c6ffa9d7-4de9-4b98-8e7e-016db7370f08" (UID: "c6ffa9d7-4de9-4b98-8e7e-016db7370f08"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:07.869217 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:07.869199 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6ffa9d7-4de9-4b98-8e7e-016db7370f08-istio-token" pod "c6ffa9d7-4de9-4b98-8e7e-016db7370f08" (UID: "c6ffa9d7-4de9-4b98-8e7e-016db7370f08"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:38:07.869505 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:07.869481 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c6ffa9d7-4de9-4b98-8e7e-016db7370f08-istio-podinfo" pod "c6ffa9d7-4de9-4b98-8e7e-016db7370f08" (UID: "c6ffa9d7-4de9-4b98-8e7e-016db7370f08"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:38:07.869505 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:07.869500 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6ffa9d7-4de9-4b98-8e7e-016db7370f08-kube-api-access-rc6v5" pod "c6ffa9d7-4de9-4b98-8e7e-016db7370f08" (UID: "c6ffa9d7-4de9-4b98-8e7e-016db7370f08"). InnerVolumeSpecName "kube-api-access-rc6v5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:38:07.898044 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:07.898009 4522 generic.go:356] "Generic (PLEG): container finished" podID="c6ffa9d7-4de9-4b98-8e7e-016db7370f08" containerID="a75994c0c66a7d2db476768b52b2221a36b163fdf6b06263a5613f123bf21b1d" exitCode=0 Oct 30 17:38:07.898259 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:07.898068 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7" Oct 30 17:38:07.898259 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:07.898089 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7" event={"ID":"c6ffa9d7-4de9-4b98-8e7e-016db7370f08","Type":"ContainerDied","Data":"a75994c0c66a7d2db476768b52b2221a36b163fdf6b06263a5613f123bf21b1d"} Oct 30 17:38:07.898259 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:07.898142 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7" event={"ID":"c6ffa9d7-4de9-4b98-8e7e-016db7370f08","Type":"ContainerDied","Data":"80a40a456651254920b6a1016e1ac5993359853d0a2258e25a886abceef83a52"} Oct 30 17:38:07.898259 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:07.898165 4522 scope.go:122] "RemoveContainer" containerID="a75994c0c66a7d2db476768b52b2221a36b163fdf6b06263a5613f123bf21b1d" Oct 30 17:38:07.900002 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:07.899980 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-469ad7-cblh2" event={"ID":"0321e5b7-aeed-43af-b484-0829a9a06f5f","Type":"ContainerDied","Data":"15e6ba19a3cb8451ba27ea4ce637c697d44077c7346e673e46590041542f80a4"} Oct 30 17:38:07.900078 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:07.900007 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-469ad7-cblh2" Oct 30 17:38:07.900078 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:07.900022 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15e6ba19a3cb8451ba27ea4ce637c697d44077c7346e673e46590041542f80a4" Oct 30 17:38:07.910724 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:07.910704 4522 scope.go:122] "RemoveContainer" containerID="325d2d8270cefdb0a5d94cf6e729d4271779e426379459d5f6628641cf644b40" Oct 30 17:38:07.920319 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:07.920281 4522 scope.go:122] "RemoveContainer" containerID="a75994c0c66a7d2db476768b52b2221a36b163fdf6b06263a5613f123bf21b1d" Oct 30 17:38:07.920690 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:38:07.920666 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a75994c0c66a7d2db476768b52b2221a36b163fdf6b06263a5613f123bf21b1d\": container with ID starting with a75994c0c66a7d2db476768b52b2221a36b163fdf6b06263a5613f123bf21b1d not found: ID does not exist" containerID="a75994c0c66a7d2db476768b52b2221a36b163fdf6b06263a5613f123bf21b1d" Oct 30 17:38:07.920773 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:07.920702 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a75994c0c66a7d2db476768b52b2221a36b163fdf6b06263a5613f123bf21b1d"} err="failed to get container status \"a75994c0c66a7d2db476768b52b2221a36b163fdf6b06263a5613f123bf21b1d\": rpc error: code = NotFound desc = could not find container \"a75994c0c66a7d2db476768b52b2221a36b163fdf6b06263a5613f123bf21b1d\": container with ID starting with a75994c0c66a7d2db476768b52b2221a36b163fdf6b06263a5613f123bf21b1d not found: ID does not exist" Oct 30 17:38:07.920773 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:07.920726 4522 scope.go:122] "RemoveContainer" containerID="325d2d8270cefdb0a5d94cf6e729d4271779e426379459d5f6628641cf644b40" Oct 30 17:38:07.921003 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:38:07.920984 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"325d2d8270cefdb0a5d94cf6e729d4271779e426379459d5f6628641cf644b40\": container with ID starting with 325d2d8270cefdb0a5d94cf6e729d4271779e426379459d5f6628641cf644b40 not found: ID does not exist" containerID="325d2d8270cefdb0a5d94cf6e729d4271779e426379459d5f6628641cf644b40" Oct 30 17:38:07.921044 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:07.921012 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"325d2d8270cefdb0a5d94cf6e729d4271779e426379459d5f6628641cf644b40"} err="failed to get container status \"325d2d8270cefdb0a5d94cf6e729d4271779e426379459d5f6628641cf644b40\": rpc error: code = NotFound desc = could not find container \"325d2d8270cefdb0a5d94cf6e729d4271779e426379459d5f6628641cf644b40\": container with ID starting with 325d2d8270cefdb0a5d94cf6e729d4271779e426379459d5f6628641cf644b40 not found: ID does not exist" Oct 30 17:38:07.959148 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:07.959115 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7"] Oct 30 17:38:07.962855 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:07.962831 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jhlc7"] Oct 30 17:38:07.968577 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:07.968557 4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6ffa9d7-4de9-4b98-8e7e-016db7370f08-istio-token\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:38:07.968650 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:07.968579 4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6ffa9d7-4de9-4b98-8e7e-016db7370f08-istio-podinfo\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:38:07.968650 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:07.968588 4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6ffa9d7-4de9-4b98-8e7e-016db7370f08-istio-envoy\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:38:07.968650 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:07.968597 4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6ffa9d7-4de9-4b98-8e7e-016db7370f08-istio-data\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:38:07.968650 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:07.968605 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc6v5\" (UniqueName: \"kubernetes.io/projected/c6ffa9d7-4de9-4b98-8e7e-016db7370f08-kube-api-access-rc6v5\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:38:08.144523 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:08.144443 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6ffa9d7-4de9-4b98-8e7e-016db7370f08" path="/var/lib/kubelet/pods/c6ffa9d7-4de9-4b98-8e7e-016db7370f08/volumes" Oct 30 17:38:11.040198 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:11.040165 4522 scope.go:122] "RemoveContainer" containerID="d2025a855d019536d113422d396f034e729edc9861a5aed3a3a41374ba88005b" Oct 30 17:38:11.050474 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:11.050449 4522 scope.go:122] "RemoveContainer" containerID="2d3854802a66fc1c4cccb6ec05e682b10c74f6dd082ffb802eb9d98413762c49" Oct 30 17:38:12.948196 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:38:12.948160 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:38:13.370754 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:13.370722 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-8f92ef/ipp-route-8f92ef-kserve-7948cdcdf-r2dkv" Oct 30 17:38:13.383921 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:13.383895 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-8f92ef/ipp-route-8f92ef-kserve-7948cdcdf-r2dkv" Oct 30 17:38:13.964732 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:13.964704 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-8f92ef/ipp-route-8f92ef-kserve-7948cdcdf-r2dkv" Oct 30 17:38:17.265044 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:17.265008 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-469ad7-cblh2"] Oct 30 17:38:17.280383 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:17.280350 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-469ad7-cblh2"] Oct 30 17:38:18.144639 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:18.144609 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0321e5b7-aeed-43af-b484-0829a9a06f5f" path="/var/lib/kubelet/pods/0321e5b7-aeed-43af-b484-0829a9a06f5f/volumes" Oct 30 17:38:25.300341 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:25.300289 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75586d-cf8zq"] Oct 30 17:38:25.300848 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:25.300815 4522 cpu_manager.go:414] "containerMap: removing container" podUID="c6ffa9d7-4de9-4b98-8e7e-016db7370f08" containerName="istio-proxy" Oct 30 17:38:25.300848 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:25.300829 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6ffa9d7-4de9-4b98-8e7e-016db7370f08" containerName="istio-proxy" Oct 30 17:38:25.300929 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:25.300852 4522 cpu_manager.go:414] "containerMap: removing container" podUID="c6ffa9d7-4de9-4b98-8e7e-016db7370f08" containerName="istio-proxy" Oct 30 17:38:25.300929 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:25.300861 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6ffa9d7-4de9-4b98-8e7e-016db7370f08" containerName="istio-proxy" Oct 30 17:38:25.300929 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:25.300873 4522 cpu_manager.go:414] "containerMap: removing container" podUID="0321e5b7-aeed-43af-b484-0829a9a06f5f" containerName="revoke-keys" Oct 30 17:38:25.300929 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:25.300879 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0321e5b7-aeed-43af-b484-0829a9a06f5f" containerName="revoke-keys" Oct 30 17:38:25.301326 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:25.300968 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6ffa9d7-4de9-4b98-8e7e-016db7370f08" containerName="istio-proxy" Oct 30 17:38:25.301326 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:25.300977 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="0321e5b7-aeed-43af-b484-0829a9a06f5f" containerName="revoke-keys" Oct 30 17:38:25.301326 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:25.300988 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6ffa9d7-4de9-4b98-8e7e-016db7370f08" containerName="istio-proxy" Oct 30 17:38:25.305986 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:25.305957 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75586d-cf8zq" Oct 30 17:38:25.323753 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:25.323721 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75586d-cf8zq"] Oct 30 17:38:25.352698 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:25.352659 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znxz4\" (UniqueName: \"kubernetes.io/projected/e6f0370f-c963-409d-9228-39a520135eaa-kube-api-access-znxz4\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-75586d-cf8zq\" (UID: \"e6f0370f-c963-409d-9228-39a520135eaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75586d-cf8zq" Oct 30 17:38:25.352888 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:25.352794 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6f0370f-c963-409d-9228-39a520135eaa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-75586d-cf8zq\" (UID: \"e6f0370f-c963-409d-9228-39a520135eaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75586d-cf8zq" Oct 30 17:38:25.352988 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:25.352966 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6f0370f-c963-409d-9228-39a520135eaa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-75586d-cf8zq\" (UID: \"e6f0370f-c963-409d-9228-39a520135eaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75586d-cf8zq" Oct 30 17:38:25.411102 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:25.411072 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:38:25.411260 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:25.411144 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:38:25.454136 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:25.454104 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6f0370f-c963-409d-9228-39a520135eaa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-75586d-cf8zq\" (UID: \"e6f0370f-c963-409d-9228-39a520135eaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75586d-cf8zq" Oct 30 17:38:25.454350 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:25.454150 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znxz4\" (UniqueName: \"kubernetes.io/projected/e6f0370f-c963-409d-9228-39a520135eaa-kube-api-access-znxz4\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-75586d-cf8zq\" (UID: \"e6f0370f-c963-409d-9228-39a520135eaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75586d-cf8zq" Oct 30 17:38:25.454350 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:25.454201 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6f0370f-c963-409d-9228-39a520135eaa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-75586d-cf8zq\" (UID: \"e6f0370f-c963-409d-9228-39a520135eaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75586d-cf8zq" Oct 30 17:38:25.454886 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:25.454852 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6f0370f-c963-409d-9228-39a520135eaa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-75586d-cf8zq\" (UID: \"e6f0370f-c963-409d-9228-39a520135eaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75586d-cf8zq" Oct 30 17:38:25.456616 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:25.456592 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6f0370f-c963-409d-9228-39a520135eaa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-75586d-cf8zq\" (UID: \"e6f0370f-c963-409d-9228-39a520135eaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75586d-cf8zq" Oct 30 17:38:25.481167 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:25.481142 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znxz4\" (UniqueName: \"kubernetes.io/projected/e6f0370f-c963-409d-9228-39a520135eaa-kube-api-access-znxz4\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-75586d-cf8zq\" (UID: \"e6f0370f-c963-409d-9228-39a520135eaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75586d-cf8zq" Oct 30 17:38:25.619621 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:25.619536 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75586d-cf8zq" Oct 30 17:38:25.924272 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:38:25.924228 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6f0370f_c963_409d_9228_39a520135eaa.slice/crio-a18bdd49062463caab17bb3c2c9244064cc4fee0be3946ea24375e19a2df65f0 WatchSource:0}: Error finding container a18bdd49062463caab17bb3c2c9244064cc4fee0be3946ea24375e19a2df65f0: Status 404 returned error can't find the container with id a18bdd49062463caab17bb3c2c9244064cc4fee0be3946ea24375e19a2df65f0 Oct 30 17:38:25.924752 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:25.924726 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75586d-cf8zq"] Oct 30 17:38:26.040897 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:26.040853 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75586d-cf8zq" event={"ID":"e6f0370f-c963-409d-9228-39a520135eaa","Type":"ContainerStarted","Data":"a18bdd49062463caab17bb3c2c9244064cc4fee0be3946ea24375e19a2df65f0"} Oct 30 17:38:27.049135 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:27.049046 4522 generic.go:356] "Generic (PLEG): container finished" podID="e6f0370f-c963-409d-9228-39a520135eaa" containerID="02debccec32f9af2cc35d5f20849b899e16491db40cdb4980d66cc4bb41a0ca2" exitCode=0 Oct 30 17:38:27.049528 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:27.049133 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75586d-cf8zq" event={"ID":"e6f0370f-c963-409d-9228-39a520135eaa","Type":"ContainerDied","Data":"02debccec32f9af2cc35d5f20849b899e16491db40cdb4980d66cc4bb41a0ca2"} Oct 30 17:38:28.204411 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:28.204384 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75586d-cf8zq" Oct 30 17:38:28.289988 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:28.289959 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6f0370f-c963-409d-9228-39a520135eaa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6f0370f-c963-409d-9228-39a520135eaa-maas-api-openshift-ca\") pod \"e6f0370f-c963-409d-9228-39a520135eaa\" (UID: \"e6f0370f-c963-409d-9228-39a520135eaa\") " Oct 30 17:38:28.290136 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:28.290094 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6f0370f-c963-409d-9228-39a520135eaa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6f0370f-c963-409d-9228-39a520135eaa-maas-api-xks-ca\") pod \"e6f0370f-c963-409d-9228-39a520135eaa\" (UID: \"e6f0370f-c963-409d-9228-39a520135eaa\") " Oct 30 17:38:28.290199 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:28.290139 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6f0370f-c963-409d-9228-39a520135eaa-kube-api-access-znxz4\" (UniqueName: \"kubernetes.io/projected/e6f0370f-c963-409d-9228-39a520135eaa-kube-api-access-znxz4\") pod \"e6f0370f-c963-409d-9228-39a520135eaa\" (UID: \"e6f0370f-c963-409d-9228-39a520135eaa\") " Oct 30 17:38:28.290280 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:28.290259 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6f0370f-c963-409d-9228-39a520135eaa-maas-api-openshift-ca" pod "e6f0370f-c963-409d-9228-39a520135eaa" (UID: "e6f0370f-c963-409d-9228-39a520135eaa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:38:28.290609 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:28.290591 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6f0370f-c963-409d-9228-39a520135eaa-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:38:28.292249 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:28.292228 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6f0370f-c963-409d-9228-39a520135eaa-maas-api-xks-ca" pod "e6f0370f-c963-409d-9228-39a520135eaa" (UID: "e6f0370f-c963-409d-9228-39a520135eaa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:38:28.292357 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:28.292335 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6f0370f-c963-409d-9228-39a520135eaa-kube-api-access-znxz4" pod "e6f0370f-c963-409d-9228-39a520135eaa" (UID: "e6f0370f-c963-409d-9228-39a520135eaa"). InnerVolumeSpecName "kube-api-access-znxz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:38:28.392229 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:28.392138 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6f0370f-c963-409d-9228-39a520135eaa-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:38:28.392229 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:28.392175 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znxz4\" (UniqueName: \"kubernetes.io/projected/e6f0370f-c963-409d-9228-39a520135eaa-kube-api-access-znxz4\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:38:29.063408 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:29.063373 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75586d-cf8zq" Oct 30 17:38:29.063596 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:29.063376 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75586d-cf8zq" event={"ID":"e6f0370f-c963-409d-9228-39a520135eaa","Type":"ContainerDied","Data":"a18bdd49062463caab17bb3c2c9244064cc4fee0be3946ea24375e19a2df65f0"} Oct 30 17:38:29.063596 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:29.063503 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a18bdd49062463caab17bb3c2c9244064cc4fee0be3946ea24375e19a2df65f0" Oct 30 17:38:35.431477 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:35.431439 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75586d-79556b99-k6d8b"] Oct 30 17:38:37.941192 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:37.941155 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75586d-cf8zq"] Oct 30 17:38:37.956239 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:37.956206 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75586d-cf8zq"] Oct 30 17:38:38.144175 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:38.144143 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6f0370f-c963-409d-9228-39a520135eaa" path="/var/lib/kubelet/pods/e6f0370f-c963-409d-9228-39a520135eaa/volumes" Oct 30 17:38:39.459467 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:39.459427 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-8f92ef-85f6b78466-9mbs9_bca92d15-cd50-43f0-bf26-e5603842657c/payload-processing/0.log" Oct 30 17:38:40.083175 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:40.083133 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-054acd-5bd75fbb7d-b8qks_d2d64419-530d-4740-affc-9730206d31a1/payload-processing/0.log" Oct 30 17:38:40.432455 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:40.432363 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75586d-79556b99-k6d8b" podUID="e25622db-17ca-433c-bd0a-94589349d821" containerName="payload-pre-processing" containerID="cri-o://96874ed144797dc90b7ec6c43d856e9a91e7c3ccbef6c0a44f93f92dddf13dec" gracePeriod=25 Oct 30 17:38:40.641265 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:40.641234 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8f92ef-hbcrl"] Oct 30 17:38:40.641744 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:40.641730 4522 cpu_manager.go:414] "containerMap: removing container" podUID="e6f0370f-c963-409d-9228-39a520135eaa" containerName="revoke-keys" Oct 30 17:38:40.641786 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:40.641744 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6f0370f-c963-409d-9228-39a520135eaa" containerName="revoke-keys" Oct 30 17:38:40.641842 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:40.641832 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6f0370f-c963-409d-9228-39a520135eaa" containerName="revoke-keys" Oct 30 17:38:40.645121 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:40.645103 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8f92ef-hbcrl" Oct 30 17:38:40.668134 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:40.668103 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8f92ef-hbcrl"] Oct 30 17:38:40.678750 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:40.678721 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:38:40.678874 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:40.678782 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:38:40.828762 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:40.828721 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/024635fe-4b1e-4366-82c6-8cb91b132a66-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8f92ef-hbcrl\" (UID: \"024635fe-4b1e-4366-82c6-8cb91b132a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8f92ef-hbcrl" Oct 30 17:38:40.828980 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:40.828792 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wxnk\" (UniqueName: \"kubernetes.io/projected/024635fe-4b1e-4366-82c6-8cb91b132a66-kube-api-access-4wxnk\") pod \"maas-api-revoke-keys-e2e-ipp-a-8f92ef-hbcrl\" (UID: \"024635fe-4b1e-4366-82c6-8cb91b132a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8f92ef-hbcrl" Oct 30 17:38:40.828980 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:40.828947 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/024635fe-4b1e-4366-82c6-8cb91b132a66-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8f92ef-hbcrl\" (UID: \"024635fe-4b1e-4366-82c6-8cb91b132a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8f92ef-hbcrl" Oct 30 17:38:40.930527 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:40.930494 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/024635fe-4b1e-4366-82c6-8cb91b132a66-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8f92ef-hbcrl\" (UID: \"024635fe-4b1e-4366-82c6-8cb91b132a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8f92ef-hbcrl" Oct 30 17:38:40.930693 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:40.930603 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/024635fe-4b1e-4366-82c6-8cb91b132a66-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8f92ef-hbcrl\" (UID: \"024635fe-4b1e-4366-82c6-8cb91b132a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8f92ef-hbcrl" Oct 30 17:38:40.930693 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:40.930639 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wxnk\" (UniqueName: \"kubernetes.io/projected/024635fe-4b1e-4366-82c6-8cb91b132a66-kube-api-access-4wxnk\") pod \"maas-api-revoke-keys-e2e-ipp-a-8f92ef-hbcrl\" (UID: \"024635fe-4b1e-4366-82c6-8cb91b132a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8f92ef-hbcrl" Oct 30 17:38:40.931180 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:40.931158 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/024635fe-4b1e-4366-82c6-8cb91b132a66-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8f92ef-hbcrl\" (UID: \"024635fe-4b1e-4366-82c6-8cb91b132a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8f92ef-hbcrl" Oct 30 17:38:40.933200 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:40.933162 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/024635fe-4b1e-4366-82c6-8cb91b132a66-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8f92ef-hbcrl\" (UID: \"024635fe-4b1e-4366-82c6-8cb91b132a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8f92ef-hbcrl" Oct 30 17:38:40.959512 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:40.959480 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wxnk\" (UniqueName: \"kubernetes.io/projected/024635fe-4b1e-4366-82c6-8cb91b132a66-kube-api-access-4wxnk\") pod \"maas-api-revoke-keys-e2e-ipp-a-8f92ef-hbcrl\" (UID: \"024635fe-4b1e-4366-82c6-8cb91b132a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8f92ef-hbcrl" Oct 30 17:38:41.257998 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:41.257946 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8f92ef-hbcrl" Oct 30 17:38:41.564352 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:41.564325 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8f92ef-hbcrl"] Oct 30 17:38:41.564901 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:38:41.564875 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod024635fe_4b1e_4366_82c6_8cb91b132a66.slice/crio-d0e2652f5fd99db88be284a8f26555e6bc230ad8bbd4d4351aad0b5da09f49d5 WatchSource:0}: Error finding container d0e2652f5fd99db88be284a8f26555e6bc230ad8bbd4d4351aad0b5da09f49d5: Status 404 returned error can't find the container with id d0e2652f5fd99db88be284a8f26555e6bc230ad8bbd4d4351aad0b5da09f49d5 Oct 30 17:38:42.151698 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:42.151661 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8f92ef-hbcrl" event={"ID":"024635fe-4b1e-4366-82c6-8cb91b132a66","Type":"ContainerStarted","Data":"fb25fd622a003cb825a415fa30d9ccc0fb9a0052fce1f3eafebf76e55ee27112"} Oct 30 17:38:42.152143 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:42.151709 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8f92ef-hbcrl" event={"ID":"024635fe-4b1e-4366-82c6-8cb91b132a66","Type":"ContainerStarted","Data":"d0e2652f5fd99db88be284a8f26555e6bc230ad8bbd4d4351aad0b5da09f49d5"} Oct 30 17:38:42.199085 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:42.198986 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8f92ef-hbcrl" podStartSLOduration=2.198973582 podStartE2EDuration="2.198973582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:38:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:38:42.198666929 +0000 UTC m=+3096.199421857" watchObservedRunningTime="2026-10-30 17:38:42.198973582 +0000 UTC m=+3096.199728483" Oct 30 17:38:42.948507 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:38:42.948411 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:38:43.160128 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:43.160087 4522 generic.go:356] "Generic (PLEG): container finished" podID="024635fe-4b1e-4366-82c6-8cb91b132a66" containerID="fb25fd622a003cb825a415fa30d9ccc0fb9a0052fce1f3eafebf76e55ee27112" exitCode=0 Oct 30 17:38:43.160616 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:43.160168 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8f92ef-hbcrl" event={"ID":"024635fe-4b1e-4366-82c6-8cb91b132a66","Type":"ContainerDied","Data":"fb25fd622a003cb825a415fa30d9ccc0fb9a0052fce1f3eafebf76e55ee27112"} Oct 30 17:38:43.602624 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:43.602589 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75586d-79556b99-k6d8b" podUID="e25622db-17ca-433c-bd0a-94589349d821" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:38:44.301844 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:44.301819 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8f92ef-hbcrl" Oct 30 17:38:44.466183 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:44.466151 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/024635fe-4b1e-4366-82c6-8cb91b132a66-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/024635fe-4b1e-4366-82c6-8cb91b132a66-maas-api-openshift-ca\") pod \"024635fe-4b1e-4366-82c6-8cb91b132a66\" (UID: \"024635fe-4b1e-4366-82c6-8cb91b132a66\") " Oct 30 17:38:44.466396 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:44.466281 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/024635fe-4b1e-4366-82c6-8cb91b132a66-kube-api-access-4wxnk\" (UniqueName: \"kubernetes.io/projected/024635fe-4b1e-4366-82c6-8cb91b132a66-kube-api-access-4wxnk\") pod \"024635fe-4b1e-4366-82c6-8cb91b132a66\" (UID: \"024635fe-4b1e-4366-82c6-8cb91b132a66\") " Oct 30 17:38:44.466396 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:44.466339 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/024635fe-4b1e-4366-82c6-8cb91b132a66-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/024635fe-4b1e-4366-82c6-8cb91b132a66-maas-api-xks-ca\") pod \"024635fe-4b1e-4366-82c6-8cb91b132a66\" (UID: \"024635fe-4b1e-4366-82c6-8cb91b132a66\") " Oct 30 17:38:44.466525 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:44.466491 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/024635fe-4b1e-4366-82c6-8cb91b132a66-maas-api-openshift-ca" pod "024635fe-4b1e-4366-82c6-8cb91b132a66" (UID: "024635fe-4b1e-4366-82c6-8cb91b132a66"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:38:44.466633 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:44.466619 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/024635fe-4b1e-4366-82c6-8cb91b132a66-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:38:44.468473 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:44.468444 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/024635fe-4b1e-4366-82c6-8cb91b132a66-maas-api-xks-ca" pod "024635fe-4b1e-4366-82c6-8cb91b132a66" (UID: "024635fe-4b1e-4366-82c6-8cb91b132a66"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:38:44.468597 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:44.468534 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/024635fe-4b1e-4366-82c6-8cb91b132a66-kube-api-access-4wxnk" pod "024635fe-4b1e-4366-82c6-8cb91b132a66" (UID: "024635fe-4b1e-4366-82c6-8cb91b132a66"). InnerVolumeSpecName "kube-api-access-4wxnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:38:44.567535 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:44.567501 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wxnk\" (UniqueName: \"kubernetes.io/projected/024635fe-4b1e-4366-82c6-8cb91b132a66-kube-api-access-4wxnk\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:38:44.567535 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:44.567532 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/024635fe-4b1e-4366-82c6-8cb91b132a66-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:38:45.174290 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:45.174254 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8f92ef-hbcrl" event={"ID":"024635fe-4b1e-4366-82c6-8cb91b132a66","Type":"ContainerDied","Data":"d0e2652f5fd99db88be284a8f26555e6bc230ad8bbd4d4351aad0b5da09f49d5"} Oct 30 17:38:45.174290 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:45.174283 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8f92ef-hbcrl" Oct 30 17:38:45.174521 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:45.174339 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0e2652f5fd99db88be284a8f26555e6bc230ad8bbd4d4351aad0b5da09f49d5" Oct 30 17:38:46.299047 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:46.299015 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-096d75-hgnzk"] Oct 30 17:38:46.300521 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:46.300482 4522 cpu_manager.go:414] "containerMap: removing container" podUID="024635fe-4b1e-4366-82c6-8cb91b132a66" containerName="revoke-keys" Oct 30 17:38:46.300682 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:46.300525 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="024635fe-4b1e-4366-82c6-8cb91b132a66" containerName="revoke-keys" Oct 30 17:38:46.300783 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:46.300769 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="024635fe-4b1e-4366-82c6-8cb91b132a66" containerName="revoke-keys" Oct 30 17:38:46.305617 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:46.305590 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-096d75-hgnzk" Oct 30 17:38:46.323904 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:46.323875 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-096d75-hgnzk"] Oct 30 17:38:46.335122 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:46.335093 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:38:46.335122 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:46.335094 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:38:46.383893 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:46.383860 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61b4a7c2-6306-4c7d-8968-a1fe3f753547-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-096d75-hgnzk\" (UID: \"61b4a7c2-6306-4c7d-8968-a1fe3f753547\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-096d75-hgnzk" Oct 30 17:38:46.383893 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:46.383894 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6l86\" (UniqueName: \"kubernetes.io/projected/61b4a7c2-6306-4c7d-8968-a1fe3f753547-kube-api-access-w6l86\") pod \"maas-api-revoke-keys-e2e-worker-w0-096d75-hgnzk\" (UID: \"61b4a7c2-6306-4c7d-8968-a1fe3f753547\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-096d75-hgnzk" Oct 30 17:38:46.384103 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:46.384017 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61b4a7c2-6306-4c7d-8968-a1fe3f753547-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-096d75-hgnzk\" (UID: \"61b4a7c2-6306-4c7d-8968-a1fe3f753547\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-096d75-hgnzk" Oct 30 17:38:46.484597 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:46.484560 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61b4a7c2-6306-4c7d-8968-a1fe3f753547-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-096d75-hgnzk\" (UID: \"61b4a7c2-6306-4c7d-8968-a1fe3f753547\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-096d75-hgnzk" Oct 30 17:38:46.484597 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:46.484603 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6l86\" (UniqueName: \"kubernetes.io/projected/61b4a7c2-6306-4c7d-8968-a1fe3f753547-kube-api-access-w6l86\") pod \"maas-api-revoke-keys-e2e-worker-w0-096d75-hgnzk\" (UID: \"61b4a7c2-6306-4c7d-8968-a1fe3f753547\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-096d75-hgnzk" Oct 30 17:38:46.484856 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:46.484699 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61b4a7c2-6306-4c7d-8968-a1fe3f753547-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-096d75-hgnzk\" (UID: \"61b4a7c2-6306-4c7d-8968-a1fe3f753547\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-096d75-hgnzk" Oct 30 17:38:46.485274 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:46.485252 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61b4a7c2-6306-4c7d-8968-a1fe3f753547-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-096d75-hgnzk\" (UID: \"61b4a7c2-6306-4c7d-8968-a1fe3f753547\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-096d75-hgnzk" Oct 30 17:38:46.487113 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:46.487089 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61b4a7c2-6306-4c7d-8968-a1fe3f753547-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-096d75-hgnzk\" (UID: \"61b4a7c2-6306-4c7d-8968-a1fe3f753547\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-096d75-hgnzk" Oct 30 17:38:46.510486 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:46.510457 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6l86\" (UniqueName: \"kubernetes.io/projected/61b4a7c2-6306-4c7d-8968-a1fe3f753547-kube-api-access-w6l86\") pod \"maas-api-revoke-keys-e2e-worker-w0-096d75-hgnzk\" (UID: \"61b4a7c2-6306-4c7d-8968-a1fe3f753547\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-096d75-hgnzk" Oct 30 17:38:46.620690 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:46.620608 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-096d75-hgnzk" Oct 30 17:38:46.924035 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:38:46.924003 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61b4a7c2_6306_4c7d_8968_a1fe3f753547.slice/crio-1e038de213650da098f0214ea2e99bc80aa4b91972b68c9c8531eb0cf80c6249 WatchSource:0}: Error finding container 1e038de213650da098f0214ea2e99bc80aa4b91972b68c9c8531eb0cf80c6249: Status 404 returned error can't find the container with id 1e038de213650da098f0214ea2e99bc80aa4b91972b68c9c8531eb0cf80c6249 Oct 30 17:38:46.924198 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:46.924170 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-096d75-hgnzk"] Oct 30 17:38:47.188816 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:47.188732 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-096d75-hgnzk" event={"ID":"61b4a7c2-6306-4c7d-8968-a1fe3f753547","Type":"ContainerStarted","Data":"1e038de213650da098f0214ea2e99bc80aa4b91972b68c9c8531eb0cf80c6249"} Oct 30 17:38:48.032575 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:48.032533 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e220f21c-5bcb2"] Oct 30 17:38:48.046867 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:48.046803 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e220f21c-5bcb2"] Oct 30 17:38:48.144691 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:48.144660 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f18989d-1278-4f5f-a6b7-89179eff0676" path="/var/lib/kubelet/pods/7f18989d-1278-4f5f-a6b7-89179eff0676/volumes" Oct 30 17:38:48.196794 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:48.196765 4522 generic.go:356] "Generic (PLEG): container finished" podID="61b4a7c2-6306-4c7d-8968-a1fe3f753547" containerID="c88856ddce22cc0a772dc518b80070d64c925d9fad1371e345563ddd74961fab" exitCode=0 Oct 30 17:38:48.196967 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:48.196856 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-096d75-hgnzk" event={"ID":"61b4a7c2-6306-4c7d-8968-a1fe3f753547","Type":"ContainerDied","Data":"c88856ddce22cc0a772dc518b80070d64c925d9fad1371e345563ddd74961fab"} Oct 30 17:38:49.335905 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:49.335879 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-096d75-hgnzk" Oct 30 17:38:49.416826 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:49.416791 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61b4a7c2-6306-4c7d-8968-a1fe3f753547-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61b4a7c2-6306-4c7d-8968-a1fe3f753547-maas-api-xks-ca\") pod \"61b4a7c2-6306-4c7d-8968-a1fe3f753547\" (UID: \"61b4a7c2-6306-4c7d-8968-a1fe3f753547\") " Oct 30 17:38:49.416998 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:49.416859 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61b4a7c2-6306-4c7d-8968-a1fe3f753547-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61b4a7c2-6306-4c7d-8968-a1fe3f753547-maas-api-openshift-ca\") pod \"61b4a7c2-6306-4c7d-8968-a1fe3f753547\" (UID: \"61b4a7c2-6306-4c7d-8968-a1fe3f753547\") " Oct 30 17:38:49.416998 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:49.416958 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61b4a7c2-6306-4c7d-8968-a1fe3f753547-kube-api-access-w6l86\" (UniqueName: \"kubernetes.io/projected/61b4a7c2-6306-4c7d-8968-a1fe3f753547-kube-api-access-w6l86\") pod \"61b4a7c2-6306-4c7d-8968-a1fe3f753547\" (UID: \"61b4a7c2-6306-4c7d-8968-a1fe3f753547\") " Oct 30 17:38:49.417294 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:49.417265 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61b4a7c2-6306-4c7d-8968-a1fe3f753547-maas-api-openshift-ca" pod "61b4a7c2-6306-4c7d-8968-a1fe3f753547" (UID: "61b4a7c2-6306-4c7d-8968-a1fe3f753547"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:38:49.417442 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:49.417414 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61b4a7c2-6306-4c7d-8968-a1fe3f753547-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:38:49.418890 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:49.418861 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61b4a7c2-6306-4c7d-8968-a1fe3f753547-maas-api-xks-ca" pod "61b4a7c2-6306-4c7d-8968-a1fe3f753547" (UID: "61b4a7c2-6306-4c7d-8968-a1fe3f753547"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:38:49.418983 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:49.418961 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61b4a7c2-6306-4c7d-8968-a1fe3f753547-kube-api-access-w6l86" pod "61b4a7c2-6306-4c7d-8968-a1fe3f753547" (UID: "61b4a7c2-6306-4c7d-8968-a1fe3f753547"). InnerVolumeSpecName "kube-api-access-w6l86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:38:49.518624 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:49.518542 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6l86\" (UniqueName: \"kubernetes.io/projected/61b4a7c2-6306-4c7d-8968-a1fe3f753547-kube-api-access-w6l86\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:38:49.518624 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:49.518572 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61b4a7c2-6306-4c7d-8968-a1fe3f753547-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:38:50.211849 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:50.211816 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-096d75-hgnzk" Oct 30 17:38:50.211849 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:50.211843 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-096d75-hgnzk" event={"ID":"61b4a7c2-6306-4c7d-8968-a1fe3f753547","Type":"ContainerDied","Data":"1e038de213650da098f0214ea2e99bc80aa4b91972b68c9c8531eb0cf80c6249"} Oct 30 17:38:50.212080 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:50.211889 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e038de213650da098f0214ea2e99bc80aa4b91972b68c9c8531eb0cf80c6249" Oct 30 17:38:50.965998 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:50.965968 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc1e4760-bm4jl"] Oct 30 17:38:50.966518 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:50.966499 4522 cpu_manager.go:414] "containerMap: removing container" podUID="61b4a7c2-6306-4c7d-8968-a1fe3f753547" containerName="revoke-keys" Oct 30 17:38:50.966603 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:50.966520 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61b4a7c2-6306-4c7d-8968-a1fe3f753547" containerName="revoke-keys" Oct 30 17:38:50.966683 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:50.966671 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="61b4a7c2-6306-4c7d-8968-a1fe3f753547" containerName="revoke-keys" Oct 30 17:38:50.970367 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:50.970345 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1e4760-bm4jl" Oct 30 17:38:50.990935 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:50.990910 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc1e4760-bm4jl"] Oct 30 17:38:51.001407 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:51.001387 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:38:51.001509 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:51.001458 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:38:51.030847 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:51.030812 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz4qf\" (UniqueName: \"kubernetes.io/projected/82b766ef-96a7-4c73-886f-56e20fc1422c-kube-api-access-nz4qf\") pod \"maas-api-revoke-sub-bc1e4760-bm4jl\" (UID: \"82b766ef-96a7-4c73-886f-56e20fc1422c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1e4760-bm4jl" Oct 30 17:38:51.031004 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:51.030934 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82b766ef-96a7-4c73-886f-56e20fc1422c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc1e4760-bm4jl\" (UID: \"82b766ef-96a7-4c73-886f-56e20fc1422c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1e4760-bm4jl" Oct 30 17:38:51.031004 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:51.030983 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82b766ef-96a7-4c73-886f-56e20fc1422c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc1e4760-bm4jl\" (UID: \"82b766ef-96a7-4c73-886f-56e20fc1422c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1e4760-bm4jl" Oct 30 17:38:51.093250 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:51.093216 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl"] Oct 30 17:38:51.103539 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:51.103512 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p"] Oct 30 17:38:51.132096 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:51.132067 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz4qf\" (UniqueName: \"kubernetes.io/projected/82b766ef-96a7-4c73-886f-56e20fc1422c-kube-api-access-nz4qf\") pod \"maas-api-revoke-sub-bc1e4760-bm4jl\" (UID: \"82b766ef-96a7-4c73-886f-56e20fc1422c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1e4760-bm4jl" Oct 30 17:38:51.132261 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:51.132138 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82b766ef-96a7-4c73-886f-56e20fc1422c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc1e4760-bm4jl\" (UID: \"82b766ef-96a7-4c73-886f-56e20fc1422c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1e4760-bm4jl" Oct 30 17:38:51.132261 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:51.132174 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82b766ef-96a7-4c73-886f-56e20fc1422c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc1e4760-bm4jl\" (UID: \"82b766ef-96a7-4c73-886f-56e20fc1422c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1e4760-bm4jl" Oct 30 17:38:51.132785 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:51.132760 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82b766ef-96a7-4c73-886f-56e20fc1422c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc1e4760-bm4jl\" (UID: \"82b766ef-96a7-4c73-886f-56e20fc1422c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1e4760-bm4jl" Oct 30 17:38:51.134469 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:51.134449 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82b766ef-96a7-4c73-886f-56e20fc1422c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc1e4760-bm4jl\" (UID: \"82b766ef-96a7-4c73-886f-56e20fc1422c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1e4760-bm4jl" Oct 30 17:38:51.158114 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:51.158084 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz4qf\" (UniqueName: \"kubernetes.io/projected/82b766ef-96a7-4c73-886f-56e20fc1422c-kube-api-access-nz4qf\") pod \"maas-api-revoke-sub-bc1e4760-bm4jl\" (UID: \"82b766ef-96a7-4c73-886f-56e20fc1422c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1e4760-bm4jl" Oct 30 17:38:51.287773 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:51.287686 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1e4760-bm4jl" Oct 30 17:38:51.599247 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:38:51.599217 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82b766ef_96a7_4c73_886f_56e20fc1422c.slice/crio-bed7567bb0126588494450574a267dae8080a0f1ad206293aecdd3fa8aec26a1 WatchSource:0}: Error finding container bed7567bb0126588494450574a267dae8080a0f1ad206293aecdd3fa8aec26a1: Status 404 returned error can't find the container with id bed7567bb0126588494450574a267dae8080a0f1ad206293aecdd3fa8aec26a1 Oct 30 17:38:51.599601 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:51.599575 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc1e4760-bm4jl"] Oct 30 17:38:52.226630 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:52.226597 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1e4760-bm4jl" event={"ID":"82b766ef-96a7-4c73-886f-56e20fc1422c","Type":"ContainerStarted","Data":"3b62497adc43b67740b7180160088bb8e64027da57fe2c3936a3acb514504366"} Oct 30 17:38:52.227015 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:52.226649 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1e4760-bm4jl" event={"ID":"82b766ef-96a7-4c73-886f-56e20fc1422c","Type":"ContainerStarted","Data":"bed7567bb0126588494450574a267dae8080a0f1ad206293aecdd3fa8aec26a1"} Oct 30 17:38:52.298904 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:52.298856 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1e4760-bm4jl" podStartSLOduration=2.298841801 podStartE2EDuration="2.298841801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:38:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:38:52.297421166 +0000 UTC m=+3106.298176070" watchObservedRunningTime="2026-10-30 17:38:52.298841801 +0000 UTC m=+3106.299596701" Oct 30 17:38:53.234740 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:53.234706 4522 generic.go:356] "Generic (PLEG): container finished" podID="82b766ef-96a7-4c73-886f-56e20fc1422c" containerID="3b62497adc43b67740b7180160088bb8e64027da57fe2c3936a3acb514504366" exitCode=0 Oct 30 17:38:53.235147 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:53.234797 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1e4760-bm4jl" event={"ID":"82b766ef-96a7-4c73-886f-56e20fc1422c","Type":"ContainerDied","Data":"3b62497adc43b67740b7180160088bb8e64027da57fe2c3936a3acb514504366"} Oct 30 17:38:53.603116 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:53.603032 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75586d-79556b99-k6d8b" podUID="e25622db-17ca-433c-bd0a-94589349d821" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:38:54.376729 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:54.376705 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1e4760-bm4jl" Oct 30 17:38:54.464060 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:54.464031 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82b766ef-96a7-4c73-886f-56e20fc1422c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82b766ef-96a7-4c73-886f-56e20fc1422c-maas-api-openshift-ca\") pod \"82b766ef-96a7-4c73-886f-56e20fc1422c\" (UID: \"82b766ef-96a7-4c73-886f-56e20fc1422c\") " Oct 30 17:38:54.464235 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:54.464116 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82b766ef-96a7-4c73-886f-56e20fc1422c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82b766ef-96a7-4c73-886f-56e20fc1422c-maas-api-xks-ca\") pod \"82b766ef-96a7-4c73-886f-56e20fc1422c\" (UID: \"82b766ef-96a7-4c73-886f-56e20fc1422c\") " Oct 30 17:38:54.464235 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:54.464162 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82b766ef-96a7-4c73-886f-56e20fc1422c-kube-api-access-nz4qf\" (UniqueName: \"kubernetes.io/projected/82b766ef-96a7-4c73-886f-56e20fc1422c-kube-api-access-nz4qf\") pod \"82b766ef-96a7-4c73-886f-56e20fc1422c\" (UID: \"82b766ef-96a7-4c73-886f-56e20fc1422c\") " Oct 30 17:38:54.464407 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:54.464383 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82b766ef-96a7-4c73-886f-56e20fc1422c-maas-api-openshift-ca" pod "82b766ef-96a7-4c73-886f-56e20fc1422c" (UID: "82b766ef-96a7-4c73-886f-56e20fc1422c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:38:54.464555 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:54.464540 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82b766ef-96a7-4c73-886f-56e20fc1422c-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:38:54.466247 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:54.466218 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82b766ef-96a7-4c73-886f-56e20fc1422c-maas-api-xks-ca" pod "82b766ef-96a7-4c73-886f-56e20fc1422c" (UID: "82b766ef-96a7-4c73-886f-56e20fc1422c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:38:54.466368 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:54.466320 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82b766ef-96a7-4c73-886f-56e20fc1422c-kube-api-access-nz4qf" pod "82b766ef-96a7-4c73-886f-56e20fc1422c" (UID: "82b766ef-96a7-4c73-886f-56e20fc1422c"). InnerVolumeSpecName "kube-api-access-nz4qf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:38:54.566108 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:54.566015 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82b766ef-96a7-4c73-886f-56e20fc1422c-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:38:54.566108 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:54.566044 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz4qf\" (UniqueName: \"kubernetes.io/projected/82b766ef-96a7-4c73-886f-56e20fc1422c-kube-api-access-nz4qf\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:38:55.249065 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:55.249030 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1e4760-bm4jl" Oct 30 17:38:55.249248 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:55.249061 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1e4760-bm4jl" event={"ID":"82b766ef-96a7-4c73-886f-56e20fc1422c","Type":"ContainerDied","Data":"bed7567bb0126588494450574a267dae8080a0f1ad206293aecdd3fa8aec26a1"} Oct 30 17:38:55.249248 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:55.249112 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bed7567bb0126588494450574a267dae8080a0f1ad206293aecdd3fa8aec26a1" Oct 30 17:38:56.411771 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:38:56.411731 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-096d75-67ff8648-hmszh"] Oct 30 17:39:00.227892 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:00.227855 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a3d7477-hv4h8"] Oct 30 17:39:00.228409 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:00.228393 4522 cpu_manager.go:414] "containerMap: removing container" podUID="82b766ef-96a7-4c73-886f-56e20fc1422c" containerName="revoke-keys" Oct 30 17:39:00.228464 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:00.228410 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82b766ef-96a7-4c73-886f-56e20fc1422c" containerName="revoke-keys" Oct 30 17:39:00.228516 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:00.228499 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="82b766ef-96a7-4c73-886f-56e20fc1422c" containerName="revoke-keys" Oct 30 17:39:00.231961 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:00.231945 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3d7477-hv4h8" Oct 30 17:39:00.254915 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:00.254879 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a3d7477-hv4h8"] Oct 30 17:39:00.266247 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:00.266218 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:39:00.266952 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:00.266935 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:39:00.270833 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:00.270805 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bc5d398-p6pgs"] Oct 30 17:39:00.275687 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:00.275666 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bc5d398-p6pgs" Oct 30 17:39:00.289508 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:00.289486 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bc5d398-p6pgs"] Oct 30 17:39:00.329087 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:00.329055 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg4tk\" (UniqueName: \"kubernetes.io/projected/41a11823-1100-41a5-bd01-b67ec1ed5191-kube-api-access-vg4tk\") pod \"maas-api-revoke-sub-2bc5d398-p6pgs\" (UID: \"41a11823-1100-41a5-bd01-b67ec1ed5191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bc5d398-p6pgs" Oct 30 17:39:00.329335 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:00.329188 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtf52\" (UniqueName: \"kubernetes.io/projected/123ecbf4-9360-4086-8c6f-6fbad0ae75d2-kube-api-access-jtf52\") pod \"maas-api-revoke-sub-9a3d7477-hv4h8\" (UID: \"123ecbf4-9360-4086-8c6f-6fbad0ae75d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3d7477-hv4h8" Oct 30 17:39:00.329459 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:00.329358 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41a11823-1100-41a5-bd01-b67ec1ed5191-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2bc5d398-p6pgs\" (UID: \"41a11823-1100-41a5-bd01-b67ec1ed5191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bc5d398-p6pgs" Oct 30 17:39:00.329530 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:00.329467 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/123ecbf4-9360-4086-8c6f-6fbad0ae75d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a3d7477-hv4h8\" (UID: \"123ecbf4-9360-4086-8c6f-6fbad0ae75d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3d7477-hv4h8" Oct 30 17:39:00.329530 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:00.329521 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41a11823-1100-41a5-bd01-b67ec1ed5191-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2bc5d398-p6pgs\" (UID: \"41a11823-1100-41a5-bd01-b67ec1ed5191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bc5d398-p6pgs" Oct 30 17:39:00.329704 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:00.329680 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/123ecbf4-9360-4086-8c6f-6fbad0ae75d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a3d7477-hv4h8\" (UID: \"123ecbf4-9360-4086-8c6f-6fbad0ae75d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3d7477-hv4h8" Oct 30 17:39:00.430543 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:00.430507 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtf52\" (UniqueName: \"kubernetes.io/projected/123ecbf4-9360-4086-8c6f-6fbad0ae75d2-kube-api-access-jtf52\") pod \"maas-api-revoke-sub-9a3d7477-hv4h8\" (UID: \"123ecbf4-9360-4086-8c6f-6fbad0ae75d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3d7477-hv4h8" Oct 30 17:39:00.430766 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:00.430555 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41a11823-1100-41a5-bd01-b67ec1ed5191-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2bc5d398-p6pgs\" (UID: \"41a11823-1100-41a5-bd01-b67ec1ed5191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bc5d398-p6pgs" Oct 30 17:39:00.430766 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:00.430606 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/123ecbf4-9360-4086-8c6f-6fbad0ae75d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a3d7477-hv4h8\" (UID: \"123ecbf4-9360-4086-8c6f-6fbad0ae75d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3d7477-hv4h8" Oct 30 17:39:00.430766 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:00.430646 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41a11823-1100-41a5-bd01-b67ec1ed5191-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2bc5d398-p6pgs\" (UID: \"41a11823-1100-41a5-bd01-b67ec1ed5191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bc5d398-p6pgs" Oct 30 17:39:00.430766 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:00.430683 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/123ecbf4-9360-4086-8c6f-6fbad0ae75d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a3d7477-hv4h8\" (UID: \"123ecbf4-9360-4086-8c6f-6fbad0ae75d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3d7477-hv4h8" Oct 30 17:39:00.430766 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:00.430719 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg4tk\" (UniqueName: \"kubernetes.io/projected/41a11823-1100-41a5-bd01-b67ec1ed5191-kube-api-access-vg4tk\") pod \"maas-api-revoke-sub-2bc5d398-p6pgs\" (UID: \"41a11823-1100-41a5-bd01-b67ec1ed5191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bc5d398-p6pgs" Oct 30 17:39:00.431390 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:00.431358 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41a11823-1100-41a5-bd01-b67ec1ed5191-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2bc5d398-p6pgs\" (UID: \"41a11823-1100-41a5-bd01-b67ec1ed5191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bc5d398-p6pgs" Oct 30 17:39:00.431523 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:00.431357 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/123ecbf4-9360-4086-8c6f-6fbad0ae75d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a3d7477-hv4h8\" (UID: \"123ecbf4-9360-4086-8c6f-6fbad0ae75d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3d7477-hv4h8" Oct 30 17:39:00.433196 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:00.433168 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41a11823-1100-41a5-bd01-b67ec1ed5191-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2bc5d398-p6pgs\" (UID: \"41a11823-1100-41a5-bd01-b67ec1ed5191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bc5d398-p6pgs" Oct 30 17:39:00.433196 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:00.433189 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/123ecbf4-9360-4086-8c6f-6fbad0ae75d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a3d7477-hv4h8\" (UID: \"123ecbf4-9360-4086-8c6f-6fbad0ae75d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3d7477-hv4h8" Oct 30 17:39:00.459917 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:00.459887 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg4tk\" (UniqueName: \"kubernetes.io/projected/41a11823-1100-41a5-bd01-b67ec1ed5191-kube-api-access-vg4tk\") pod \"maas-api-revoke-sub-2bc5d398-p6pgs\" (UID: \"41a11823-1100-41a5-bd01-b67ec1ed5191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bc5d398-p6pgs" Oct 30 17:39:00.460015 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:00.459987 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtf52\" (UniqueName: \"kubernetes.io/projected/123ecbf4-9360-4086-8c6f-6fbad0ae75d2-kube-api-access-jtf52\") pod \"maas-api-revoke-sub-9a3d7477-hv4h8\" (UID: \"123ecbf4-9360-4086-8c6f-6fbad0ae75d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3d7477-hv4h8" Oct 30 17:39:00.544537 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:00.544457 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3d7477-hv4h8" Oct 30 17:39:00.606974 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:00.606927 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bc5d398-p6pgs" Oct 30 17:39:00.732980 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:00.732954 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75586d-79556b99-k6d8b" Oct 30 17:39:00.835680 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:00.835606 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e25622db-17ca-433c-bd0a-94589349d821-kube-api-access-vvplq\" (UniqueName: \"kubernetes.io/projected/e25622db-17ca-433c-bd0a-94589349d821-kube-api-access-vvplq\") pod \"e25622db-17ca-433c-bd0a-94589349d821\" (UID: \"e25622db-17ca-433c-bd0a-94589349d821\") " Oct 30 17:39:00.835680 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:00.835662 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e25622db-17ca-433c-bd0a-94589349d821-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e25622db-17ca-433c-bd0a-94589349d821-plugins-config-volume\") pod \"e25622db-17ca-433c-bd0a-94589349d821\" (UID: \"e25622db-17ca-433c-bd0a-94589349d821\") " Oct 30 17:39:00.836085 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:00.836040 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e25622db-17ca-433c-bd0a-94589349d821-plugins-config-volume" pod "e25622db-17ca-433c-bd0a-94589349d821" (UID: "e25622db-17ca-433c-bd0a-94589349d821"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:39:00.837869 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:00.837845 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e25622db-17ca-433c-bd0a-94589349d821-kube-api-access-vvplq" pod "e25622db-17ca-433c-bd0a-94589349d821" (UID: "e25622db-17ca-433c-bd0a-94589349d821"). InnerVolumeSpecName "kube-api-access-vvplq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:39:00.870479 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:00.870448 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a3d7477-hv4h8"] Oct 30 17:39:00.870599 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:39:00.870497 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod123ecbf4_9360_4086_8c6f_6fbad0ae75d2.slice/crio-2675250566ac4cc171cc5d568e98ae893fbbffa63f95dc7a567343dee2d448fc WatchSource:0}: Error finding container 2675250566ac4cc171cc5d568e98ae893fbbffa63f95dc7a567343dee2d448fc: Status 404 returned error can't find the container with id 2675250566ac4cc171cc5d568e98ae893fbbffa63f95dc7a567343dee2d448fc Oct 30 17:39:00.936957 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:00.936925 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvplq\" (UniqueName: \"kubernetes.io/projected/e25622db-17ca-433c-bd0a-94589349d821-kube-api-access-vvplq\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:39:00.936957 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:00.936953 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e25622db-17ca-433c-bd0a-94589349d821-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:39:00.941477 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:00.941453 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bc5d398-p6pgs"] Oct 30 17:39:00.942127 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:39:00.942102 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41a11823_1100_41a5_bd01_b67ec1ed5191.slice/crio-f4196f1f6fe57c3ee9c196d483be100945613fb0d861cd5263ef866fd014f6e1 WatchSource:0}: Error finding container f4196f1f6fe57c3ee9c196d483be100945613fb0d861cd5263ef866fd014f6e1: Status 404 returned error can't find the container with id f4196f1f6fe57c3ee9c196d483be100945613fb0d861cd5263ef866fd014f6e1 Oct 30 17:39:01.302725 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:01.302693 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3d7477-hv4h8" event={"ID":"123ecbf4-9360-4086-8c6f-6fbad0ae75d2","Type":"ContainerStarted","Data":"2675250566ac4cc171cc5d568e98ae893fbbffa63f95dc7a567343dee2d448fc"} Oct 30 17:39:01.304331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:01.304274 4522 generic.go:356] "Generic (PLEG): container finished" podID="e25622db-17ca-433c-bd0a-94589349d821" containerID="96874ed144797dc90b7ec6c43d856e9a91e7c3ccbef6c0a44f93f92dddf13dec" exitCode=0 Oct 30 17:39:01.304460 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:01.304402 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75586d-79556b99-k6d8b" Oct 30 17:39:01.305514 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:01.305485 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75586d-79556b99-k6d8b" event={"ID":"e25622db-17ca-433c-bd0a-94589349d821","Type":"ContainerDied","Data":"96874ed144797dc90b7ec6c43d856e9a91e7c3ccbef6c0a44f93f92dddf13dec"} Oct 30 17:39:01.305608 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:01.305546 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75586d-79556b99-k6d8b" event={"ID":"e25622db-17ca-433c-bd0a-94589349d821","Type":"ContainerDied","Data":"17975eede459bff93b5d17a5f63ed78719e302a208602962186b3ddfc7fb122a"} Oct 30 17:39:01.305608 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:01.305573 4522 scope.go:122] "RemoveContainer" containerID="96874ed144797dc90b7ec6c43d856e9a91e7c3ccbef6c0a44f93f92dddf13dec" Oct 30 17:39:01.308027 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:01.308002 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bc5d398-p6pgs" event={"ID":"41a11823-1100-41a5-bd01-b67ec1ed5191","Type":"ContainerStarted","Data":"95902780cf4b2b8bb8a8a8f94487cfa8f3b0c4e5da332faae4fb502e7efbe5c1"} Oct 30 17:39:01.308133 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:01.308052 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bc5d398-p6pgs" event={"ID":"41a11823-1100-41a5-bd01-b67ec1ed5191","Type":"ContainerStarted","Data":"f4196f1f6fe57c3ee9c196d483be100945613fb0d861cd5263ef866fd014f6e1"} Oct 30 17:39:01.316450 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:01.316431 4522 scope.go:122] "RemoveContainer" containerID="96874ed144797dc90b7ec6c43d856e9a91e7c3ccbef6c0a44f93f92dddf13dec" Oct 30 17:39:01.316709 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:39:01.316689 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96874ed144797dc90b7ec6c43d856e9a91e7c3ccbef6c0a44f93f92dddf13dec\": container with ID starting with 96874ed144797dc90b7ec6c43d856e9a91e7c3ccbef6c0a44f93f92dddf13dec not found: ID does not exist" containerID="96874ed144797dc90b7ec6c43d856e9a91e7c3ccbef6c0a44f93f92dddf13dec" Oct 30 17:39:01.316773 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:01.316719 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96874ed144797dc90b7ec6c43d856e9a91e7c3ccbef6c0a44f93f92dddf13dec"} err="failed to get container status \"96874ed144797dc90b7ec6c43d856e9a91e7c3ccbef6c0a44f93f92dddf13dec\": rpc error: code = NotFound desc = could not find container \"96874ed144797dc90b7ec6c43d856e9a91e7c3ccbef6c0a44f93f92dddf13dec\": container with ID starting with 96874ed144797dc90b7ec6c43d856e9a91e7c3ccbef6c0a44f93f92dddf13dec not found: ID does not exist" Oct 30 17:39:01.380670 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:01.380579 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bc5d398-p6pgs" podStartSLOduration=1.380566589 podStartE2EDuration="1.380566589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:39:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:39:01.378842924 +0000 UTC m=+3115.379597819" watchObservedRunningTime="2026-10-30 17:39:01.380566589 +0000 UTC m=+3115.381321490" Oct 30 17:39:01.412367 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:01.412330 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-096d75-67ff8648-hmszh" podUID="0e716095-6ba5-49f8-a79b-5ec7a226336b" containerName="payload-processing" containerID="cri-o://1bc673e20285e377f207342390515b137239392b2cfb111957e6a7214a22fbd8" gracePeriod=25 Oct 30 17:39:01.424126 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:01.424088 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75586d-79556b99-k6d8b"] Oct 30 17:39:01.435928 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:01.435901 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-75586d-79556b99-k6d8b"] Oct 30 17:39:02.039516 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:02.039432 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d63772ed-6sfht"] Oct 30 17:39:02.053805 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:02.053777 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d63772ed-6sfht"] Oct 30 17:39:02.144946 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:02.144915 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f5abdf5-256f-425f-b4bf-09cd1cefe7d9" path="/var/lib/kubelet/pods/2f5abdf5-256f-425f-b4bf-09cd1cefe7d9/volumes" Oct 30 17:39:02.145339 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:02.145325 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e25622db-17ca-433c-bd0a-94589349d821" path="/var/lib/kubelet/pods/e25622db-17ca-433c-bd0a-94589349d821/volumes" Oct 30 17:39:02.317073 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:02.316982 4522 generic.go:356] "Generic (PLEG): container finished" podID="41a11823-1100-41a5-bd01-b67ec1ed5191" containerID="95902780cf4b2b8bb8a8a8f94487cfa8f3b0c4e5da332faae4fb502e7efbe5c1" exitCode=0 Oct 30 17:39:02.317549 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:02.317064 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bc5d398-p6pgs" event={"ID":"41a11823-1100-41a5-bd01-b67ec1ed5191","Type":"ContainerDied","Data":"95902780cf4b2b8bb8a8a8f94487cfa8f3b0c4e5da332faae4fb502e7efbe5c1"} Oct 30 17:39:02.318610 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:02.318587 4522 generic.go:356] "Generic (PLEG): container finished" podID="123ecbf4-9360-4086-8c6f-6fbad0ae75d2" containerID="82c71da55d9bb43a180922abadad2ccfb26c105bc2f347cd5c857a00c673612d" exitCode=0 Oct 30 17:39:02.318734 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:02.318652 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3d7477-hv4h8" event={"ID":"123ecbf4-9360-4086-8c6f-6fbad0ae75d2","Type":"ContainerDied","Data":"82c71da55d9bb43a180922abadad2ccfb26c105bc2f347cd5c857a00c673612d"} Oct 30 17:39:03.120564 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:03.120524 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-096d75-67ff8648-hmszh" podUID="0e716095-6ba5-49f8-a79b-5ec7a226336b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:39:03.476361 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:03.476331 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bc5d398-p6pgs" Oct 30 17:39:03.506128 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:03.506101 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3d7477-hv4h8" Oct 30 17:39:03.565256 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:03.565222 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/123ecbf4-9360-4086-8c6f-6fbad0ae75d2-kube-api-access-jtf52\" (UniqueName: \"kubernetes.io/projected/123ecbf4-9360-4086-8c6f-6fbad0ae75d2-kube-api-access-jtf52\") pod \"123ecbf4-9360-4086-8c6f-6fbad0ae75d2\" (UID: \"123ecbf4-9360-4086-8c6f-6fbad0ae75d2\") " Oct 30 17:39:03.565564 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:03.565265 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41a11823-1100-41a5-bd01-b67ec1ed5191-kube-api-access-vg4tk\" (UniqueName: \"kubernetes.io/projected/41a11823-1100-41a5-bd01-b67ec1ed5191-kube-api-access-vg4tk\") pod \"41a11823-1100-41a5-bd01-b67ec1ed5191\" (UID: \"41a11823-1100-41a5-bd01-b67ec1ed5191\") " Oct 30 17:39:03.565564 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:03.565332 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/123ecbf4-9360-4086-8c6f-6fbad0ae75d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/123ecbf4-9360-4086-8c6f-6fbad0ae75d2-maas-api-openshift-ca\") pod \"123ecbf4-9360-4086-8c6f-6fbad0ae75d2\" (UID: \"123ecbf4-9360-4086-8c6f-6fbad0ae75d2\") " Oct 30 17:39:03.565564 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:03.565380 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41a11823-1100-41a5-bd01-b67ec1ed5191-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41a11823-1100-41a5-bd01-b67ec1ed5191-maas-api-openshift-ca\") pod \"41a11823-1100-41a5-bd01-b67ec1ed5191\" (UID: \"41a11823-1100-41a5-bd01-b67ec1ed5191\") " Oct 30 17:39:03.565564 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:03.565450 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/123ecbf4-9360-4086-8c6f-6fbad0ae75d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/123ecbf4-9360-4086-8c6f-6fbad0ae75d2-maas-api-xks-ca\") pod \"123ecbf4-9360-4086-8c6f-6fbad0ae75d2\" (UID: \"123ecbf4-9360-4086-8c6f-6fbad0ae75d2\") " Oct 30 17:39:03.565564 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:03.565544 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41a11823-1100-41a5-bd01-b67ec1ed5191-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41a11823-1100-41a5-bd01-b67ec1ed5191-maas-api-xks-ca\") pod \"41a11823-1100-41a5-bd01-b67ec1ed5191\" (UID: \"41a11823-1100-41a5-bd01-b67ec1ed5191\") " Oct 30 17:39:03.565874 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:03.565752 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/123ecbf4-9360-4086-8c6f-6fbad0ae75d2-maas-api-openshift-ca" pod "123ecbf4-9360-4086-8c6f-6fbad0ae75d2" (UID: "123ecbf4-9360-4086-8c6f-6fbad0ae75d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:39:03.565995 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:03.565968 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/123ecbf4-9360-4086-8c6f-6fbad0ae75d2-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:39:03.566210 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:03.566155 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41a11823-1100-41a5-bd01-b67ec1ed5191-maas-api-openshift-ca" pod "41a11823-1100-41a5-bd01-b67ec1ed5191" (UID: "41a11823-1100-41a5-bd01-b67ec1ed5191"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:39:03.568361 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:03.568329 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41a11823-1100-41a5-bd01-b67ec1ed5191-maas-api-xks-ca" pod "41a11823-1100-41a5-bd01-b67ec1ed5191" (UID: "41a11823-1100-41a5-bd01-b67ec1ed5191"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:39:03.568457 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:03.568416 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/123ecbf4-9360-4086-8c6f-6fbad0ae75d2-maas-api-xks-ca" pod "123ecbf4-9360-4086-8c6f-6fbad0ae75d2" (UID: "123ecbf4-9360-4086-8c6f-6fbad0ae75d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:39:03.568457 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:03.568419 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/123ecbf4-9360-4086-8c6f-6fbad0ae75d2-kube-api-access-jtf52" pod "123ecbf4-9360-4086-8c6f-6fbad0ae75d2" (UID: "123ecbf4-9360-4086-8c6f-6fbad0ae75d2"). InnerVolumeSpecName "kube-api-access-jtf52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:39:03.568457 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:03.568438 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41a11823-1100-41a5-bd01-b67ec1ed5191-kube-api-access-vg4tk" pod "41a11823-1100-41a5-bd01-b67ec1ed5191" (UID: "41a11823-1100-41a5-bd01-b67ec1ed5191"). InnerVolumeSpecName "kube-api-access-vg4tk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:39:03.667954 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:03.667869 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41a11823-1100-41a5-bd01-b67ec1ed5191-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:39:03.667954 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:03.667899 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtf52\" (UniqueName: \"kubernetes.io/projected/123ecbf4-9360-4086-8c6f-6fbad0ae75d2-kube-api-access-jtf52\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:39:03.667954 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:03.667909 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg4tk\" (UniqueName: \"kubernetes.io/projected/41a11823-1100-41a5-bd01-b67ec1ed5191-kube-api-access-vg4tk\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:39:03.667954 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:03.667918 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41a11823-1100-41a5-bd01-b67ec1ed5191-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:39:03.667954 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:03.667927 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/123ecbf4-9360-4086-8c6f-6fbad0ae75d2-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:39:04.333299 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:04.333263 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bc5d398-p6pgs" event={"ID":"41a11823-1100-41a5-bd01-b67ec1ed5191","Type":"ContainerDied","Data":"f4196f1f6fe57c3ee9c196d483be100945613fb0d861cd5263ef866fd014f6e1"} Oct 30 17:39:04.333528 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:04.333331 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4196f1f6fe57c3ee9c196d483be100945613fb0d861cd5263ef866fd014f6e1" Oct 30 17:39:04.333528 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:04.333271 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bc5d398-p6pgs" Oct 30 17:39:04.334885 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:04.334865 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3d7477-hv4h8" Oct 30 17:39:04.335010 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:04.334905 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3d7477-hv4h8" event={"ID":"123ecbf4-9360-4086-8c6f-6fbad0ae75d2","Type":"ContainerDied","Data":"2675250566ac4cc171cc5d568e98ae893fbbffa63f95dc7a567343dee2d448fc"} Oct 30 17:39:04.335010 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:04.334953 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2675250566ac4cc171cc5d568e98ae893fbbffa63f95dc7a567343dee2d448fc" Oct 30 17:39:05.592846 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:05.592811 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8f92ef-85f6b78466-9mbs9"] Oct 30 17:39:05.593254 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:05.592983 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-8f92ef-85f6b78466-9mbs9" podUID="bca92d15-cd50-43f0-bf26-e5603842657c" containerName="payload-processing" containerID="cri-o://74144b8c162b483de434cbdb732535e0ae24d78d39bd3b8175d1c263f81147df" gracePeriod=30 Oct 30 17:39:05.618923 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:05.618888 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8f92ef-5bd8786b7-52rzg"] Oct 30 17:39:05.619170 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:05.619140 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f92ef-5bd8786b7-52rzg" podUID="0e442fef-3f96-4523-8cae-814507c01bab" containerName="payload-pre-processing" containerID="cri-o://914bd156d6dc284fe2405f8f1d4d7e14691d8401a2d385659c3678d7dce2ec97" gracePeriod=30 Oct 30 17:39:05.905026 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:05.904995 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8f92ef-85f6b78466-9mbs9" Oct 30 17:39:05.921791 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:05.921766 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f92ef-5bd8786b7-52rzg" Oct 30 17:39:05.991651 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:05.991616 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e442fef-3f96-4523-8cae-814507c01bab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e442fef-3f96-4523-8cae-814507c01bab-plugins-config-volume\") pod \"0e442fef-3f96-4523-8cae-814507c01bab\" (UID: \"0e442fef-3f96-4523-8cae-814507c01bab\") " Oct 30 17:39:05.991813 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:05.991719 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bca92d15-cd50-43f0-bf26-e5603842657c-kube-api-access-fmcmc\" (UniqueName: \"kubernetes.io/projected/bca92d15-cd50-43f0-bf26-e5603842657c-kube-api-access-fmcmc\") pod \"bca92d15-cd50-43f0-bf26-e5603842657c\" (UID: \"bca92d15-cd50-43f0-bf26-e5603842657c\") " Oct 30 17:39:05.991813 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:05.991782 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e442fef-3f96-4523-8cae-814507c01bab-kube-api-access-4klbj\" (UniqueName: \"kubernetes.io/projected/0e442fef-3f96-4523-8cae-814507c01bab-kube-api-access-4klbj\") pod \"0e442fef-3f96-4523-8cae-814507c01bab\" (UID: \"0e442fef-3f96-4523-8cae-814507c01bab\") " Oct 30 17:39:05.991968 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:05.991848 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bca92d15-cd50-43f0-bf26-e5603842657c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bca92d15-cd50-43f0-bf26-e5603842657c-plugins-config-volume\") pod \"bca92d15-cd50-43f0-bf26-e5603842657c\" (UID: \"bca92d15-cd50-43f0-bf26-e5603842657c\") " Oct 30 17:39:05.992060 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:05.992028 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e442fef-3f96-4523-8cae-814507c01bab-plugins-config-volume" pod "0e442fef-3f96-4523-8cae-814507c01bab" (UID: "0e442fef-3f96-4523-8cae-814507c01bab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:39:05.992262 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:05.992240 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bca92d15-cd50-43f0-bf26-e5603842657c-plugins-config-volume" pod "bca92d15-cd50-43f0-bf26-e5603842657c" (UID: "bca92d15-cd50-43f0-bf26-e5603842657c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:39:05.992364 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:05.992341 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e442fef-3f96-4523-8cae-814507c01bab-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:39:05.992447 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:05.992369 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bca92d15-cd50-43f0-bf26-e5603842657c-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:39:05.994273 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:05.994255 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bca92d15-cd50-43f0-bf26-e5603842657c-kube-api-access-fmcmc" pod "bca92d15-cd50-43f0-bf26-e5603842657c" (UID: "bca92d15-cd50-43f0-bf26-e5603842657c"). InnerVolumeSpecName "kube-api-access-fmcmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:39:05.996066 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:05.996044 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e442fef-3f96-4523-8cae-814507c01bab-kube-api-access-4klbj" pod "0e442fef-3f96-4523-8cae-814507c01bab" (UID: "0e442fef-3f96-4523-8cae-814507c01bab"). InnerVolumeSpecName "kube-api-access-4klbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:39:06.093428 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.093393 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4klbj\" (UniqueName: \"kubernetes.io/projected/0e442fef-3f96-4523-8cae-814507c01bab-kube-api-access-4klbj\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:39:06.093428 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.093422 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmcmc\" (UniqueName: \"kubernetes.io/projected/bca92d15-cd50-43f0-bf26-e5603842657c-kube-api-access-fmcmc\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:39:06.114297 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.114261 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" podUID="2f1e3bbb-ab1c-455d-a05d-d62f72c5d009" containerName="main" containerID="cri-o://56fda00ed5e16c409cc69d65fb9063109ae58c0bd6c429b41977240d42a559c8" gracePeriod=45 Oct 30 17:39:06.123714 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.123686 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" podUID="c9ad71b7-ad90-4c8c-bf27-0f10b23f1593" containerName="main" containerID="cri-o://bf05f7463cc12ae7785f536115fbfa2a4ff775a268d2b6698e6b34725cdded74" gracePeriod=45 Oct 30 17:39:06.273699 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.273658 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" podUID="2f1e3bbb-ab1c-455d-a05d-d62f72c5d009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": dial tcp 10.134.0.76:8000: connect: connection refused" Oct 30 17:39:06.287249 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.287214 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8f92ef-hbcrl"] Oct 30 17:39:06.290134 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.290093 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8f92ef-hbcrl"] Oct 30 17:39:06.352369 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.351841 4522 generic.go:356] "Generic (PLEG): container finished" podID="bca92d15-cd50-43f0-bf26-e5603842657c" containerID="74144b8c162b483de434cbdb732535e0ae24d78d39bd3b8175d1c263f81147df" exitCode=0 Oct 30 17:39:06.352369 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.351891 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8f92ef-85f6b78466-9mbs9" Oct 30 17:39:06.352369 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.351924 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8f92ef-85f6b78466-9mbs9" event={"ID":"bca92d15-cd50-43f0-bf26-e5603842657c","Type":"ContainerDied","Data":"74144b8c162b483de434cbdb732535e0ae24d78d39bd3b8175d1c263f81147df"} Oct 30 17:39:06.352369 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.351991 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8f92ef-85f6b78466-9mbs9" event={"ID":"bca92d15-cd50-43f0-bf26-e5603842657c","Type":"ContainerDied","Data":"784944fff68fe43bff6ac285bf918203d2125f8b59237fa0edcc430aaa25d778"} Oct 30 17:39:06.352369 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.352024 4522 scope.go:122] "RemoveContainer" containerID="74144b8c162b483de434cbdb732535e0ae24d78d39bd3b8175d1c263f81147df" Oct 30 17:39:06.355441 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.355413 4522 generic.go:356] "Generic (PLEG): container finished" podID="2f1e3bbb-ab1c-455d-a05d-d62f72c5d009" containerID="56fda00ed5e16c409cc69d65fb9063109ae58c0bd6c429b41977240d42a559c8" exitCode=0 Oct 30 17:39:06.355554 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.355540 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" event={"ID":"2f1e3bbb-ab1c-455d-a05d-d62f72c5d009","Type":"ContainerDied","Data":"56fda00ed5e16c409cc69d65fb9063109ae58c0bd6c429b41977240d42a559c8"} Oct 30 17:39:06.357480 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.357455 4522 generic.go:356] "Generic (PLEG): container finished" podID="0e442fef-3f96-4523-8cae-814507c01bab" containerID="914bd156d6dc284fe2405f8f1d4d7e14691d8401a2d385659c3678d7dce2ec97" exitCode=0 Oct 30 17:39:06.357646 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.357555 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f92ef-5bd8786b7-52rzg" Oct 30 17:39:06.357646 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.357599 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f92ef-5bd8786b7-52rzg" event={"ID":"0e442fef-3f96-4523-8cae-814507c01bab","Type":"ContainerDied","Data":"914bd156d6dc284fe2405f8f1d4d7e14691d8401a2d385659c3678d7dce2ec97"} Oct 30 17:39:06.357646 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.357642 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f92ef-5bd8786b7-52rzg" event={"ID":"0e442fef-3f96-4523-8cae-814507c01bab","Type":"ContainerDied","Data":"6091bf82d790a0026aa8f8d7262c0a49d0d314b1b2c261408ef35ef1c298f217"} Oct 30 17:39:06.361351 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.360340 4522 generic.go:356] "Generic (PLEG): container finished" podID="c9ad71b7-ad90-4c8c-bf27-0f10b23f1593" containerID="bf05f7463cc12ae7785f536115fbfa2a4ff775a268d2b6698e6b34725cdded74" exitCode=0 Oct 30 17:39:06.361351 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.360392 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" event={"ID":"c9ad71b7-ad90-4c8c-bf27-0f10b23f1593","Type":"ContainerDied","Data":"bf05f7463cc12ae7785f536115fbfa2a4ff775a268d2b6698e6b34725cdded74"} Oct 30 17:39:06.366567 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.366549 4522 scope.go:122] "RemoveContainer" containerID="74144b8c162b483de434cbdb732535e0ae24d78d39bd3b8175d1c263f81147df" Oct 30 17:39:06.366934 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:39:06.366906 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74144b8c162b483de434cbdb732535e0ae24d78d39bd3b8175d1c263f81147df\": container with ID starting with 74144b8c162b483de434cbdb732535e0ae24d78d39bd3b8175d1c263f81147df not found: ID does not exist" containerID="74144b8c162b483de434cbdb732535e0ae24d78d39bd3b8175d1c263f81147df" Oct 30 17:39:06.368371 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.366943 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74144b8c162b483de434cbdb732535e0ae24d78d39bd3b8175d1c263f81147df"} err="failed to get container status \"74144b8c162b483de434cbdb732535e0ae24d78d39bd3b8175d1c263f81147df\": rpc error: code = NotFound desc = could not find container \"74144b8c162b483de434cbdb732535e0ae24d78d39bd3b8175d1c263f81147df\": container with ID starting with 74144b8c162b483de434cbdb732535e0ae24d78d39bd3b8175d1c263f81147df not found: ID does not exist" Oct 30 17:39:06.368469 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.368376 4522 scope.go:122] "RemoveContainer" containerID="914bd156d6dc284fe2405f8f1d4d7e14691d8401a2d385659c3678d7dce2ec97" Oct 30 17:39:06.380593 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.380569 4522 scope.go:122] "RemoveContainer" containerID="914bd156d6dc284fe2405f8f1d4d7e14691d8401a2d385659c3678d7dce2ec97" Oct 30 17:39:06.380978 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:39:06.380957 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"914bd156d6dc284fe2405f8f1d4d7e14691d8401a2d385659c3678d7dce2ec97\": container with ID starting with 914bd156d6dc284fe2405f8f1d4d7e14691d8401a2d385659c3678d7dce2ec97 not found: ID does not exist" containerID="914bd156d6dc284fe2405f8f1d4d7e14691d8401a2d385659c3678d7dce2ec97" Oct 30 17:39:06.381091 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.380989 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"914bd156d6dc284fe2405f8f1d4d7e14691d8401a2d385659c3678d7dce2ec97"} err="failed to get container status \"914bd156d6dc284fe2405f8f1d4d7e14691d8401a2d385659c3678d7dce2ec97\": rpc error: code = NotFound desc = could not find container \"914bd156d6dc284fe2405f8f1d4d7e14691d8401a2d385659c3678d7dce2ec97\": container with ID starting with 914bd156d6dc284fe2405f8f1d4d7e14691d8401a2d385659c3678d7dce2ec97 not found: ID does not exist" Oct 30 17:39:06.381091 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.381012 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" Oct 30 17:39:06.389534 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.389516 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" Oct 30 17:39:06.414162 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.414133 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8f92ef-85f6b78466-9mbs9"] Oct 30 17:39:06.423057 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.423029 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8f92ef-85f6b78466-9mbs9"] Oct 30 17:39:06.463722 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.463691 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8f92ef-5bd8786b7-52rzg"] Oct 30 17:39:06.468006 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.467978 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8f92ef-5bd8786b7-52rzg"] Oct 30 17:39:06.497439 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.497400 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9ad71b7-ad90-4c8c-bf27-0f10b23f1593-kube-api-access-5lrmc\" (UniqueName: \"kubernetes.io/projected/c9ad71b7-ad90-4c8c-bf27-0f10b23f1593-kube-api-access-5lrmc\") pod \"c9ad71b7-ad90-4c8c-bf27-0f10b23f1593\" (UID: \"c9ad71b7-ad90-4c8c-bf27-0f10b23f1593\") " Oct 30 17:39:06.497614 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.497450 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9ad71b7-ad90-4c8c-bf27-0f10b23f1593-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9ad71b7-ad90-4c8c-bf27-0f10b23f1593-model-cache\") pod \"c9ad71b7-ad90-4c8c-bf27-0f10b23f1593\" (UID: \"c9ad71b7-ad90-4c8c-bf27-0f10b23f1593\") " Oct 30 17:39:06.497614 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.497487 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9ad71b7-ad90-4c8c-bf27-0f10b23f1593-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9ad71b7-ad90-4c8c-bf27-0f10b23f1593-tmp-dir\") pod \"c9ad71b7-ad90-4c8c-bf27-0f10b23f1593\" (UID: \"c9ad71b7-ad90-4c8c-bf27-0f10b23f1593\") " Oct 30 17:39:06.497614 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.497526 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f1e3bbb-ab1c-455d-a05d-d62f72c5d009-kube-api-access-p47sj\" (UniqueName: \"kubernetes.io/projected/2f1e3bbb-ab1c-455d-a05d-d62f72c5d009-kube-api-access-p47sj\") pod \"2f1e3bbb-ab1c-455d-a05d-d62f72c5d009\" (UID: \"2f1e3bbb-ab1c-455d-a05d-d62f72c5d009\") " Oct 30 17:39:06.497614 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.497585 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f1e3bbb-ab1c-455d-a05d-d62f72c5d009-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f1e3bbb-ab1c-455d-a05d-d62f72c5d009-dshm\") pod \"2f1e3bbb-ab1c-455d-a05d-d62f72c5d009\" (UID: \"2f1e3bbb-ab1c-455d-a05d-d62f72c5d009\") " Oct 30 17:39:06.497866 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.497633 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9ad71b7-ad90-4c8c-bf27-0f10b23f1593-tls-certs\" (UniqueName: \"kubernetes.io/secret/c9ad71b7-ad90-4c8c-bf27-0f10b23f1593-tls-certs\") pod \"c9ad71b7-ad90-4c8c-bf27-0f10b23f1593\" (UID: \"c9ad71b7-ad90-4c8c-bf27-0f10b23f1593\") " Oct 30 17:39:06.497866 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.497681 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f1e3bbb-ab1c-455d-a05d-d62f72c5d009-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f1e3bbb-ab1c-455d-a05d-d62f72c5d009-tmp-dir\") pod \"2f1e3bbb-ab1c-455d-a05d-d62f72c5d009\" (UID: \"2f1e3bbb-ab1c-455d-a05d-d62f72c5d009\") " Oct 30 17:39:06.497866 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.497720 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f1e3bbb-ab1c-455d-a05d-d62f72c5d009-home\" (UniqueName: \"kubernetes.io/empty-dir/2f1e3bbb-ab1c-455d-a05d-d62f72c5d009-home\") pod \"2f1e3bbb-ab1c-455d-a05d-d62f72c5d009\" (UID: \"2f1e3bbb-ab1c-455d-a05d-d62f72c5d009\") " Oct 30 17:39:06.497866 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.497774 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f1e3bbb-ab1c-455d-a05d-d62f72c5d009-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f1e3bbb-ab1c-455d-a05d-d62f72c5d009-model-cache\") pod \"2f1e3bbb-ab1c-455d-a05d-d62f72c5d009\" (UID: \"2f1e3bbb-ab1c-455d-a05d-d62f72c5d009\") " Oct 30 17:39:06.497866 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.497859 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9ad71b7-ad90-4c8c-bf27-0f10b23f1593-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9ad71b7-ad90-4c8c-bf27-0f10b23f1593-dshm\") pod \"c9ad71b7-ad90-4c8c-bf27-0f10b23f1593\" (UID: \"c9ad71b7-ad90-4c8c-bf27-0f10b23f1593\") " Oct 30 17:39:06.498145 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.497898 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9ad71b7-ad90-4c8c-bf27-0f10b23f1593-home\" (UniqueName: \"kubernetes.io/empty-dir/c9ad71b7-ad90-4c8c-bf27-0f10b23f1593-home\") pod \"c9ad71b7-ad90-4c8c-bf27-0f10b23f1593\" (UID: \"c9ad71b7-ad90-4c8c-bf27-0f10b23f1593\") " Oct 30 17:39:06.498145 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.497967 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f1e3bbb-ab1c-455d-a05d-d62f72c5d009-tls-certs\" (UniqueName: \"kubernetes.io/secret/2f1e3bbb-ab1c-455d-a05d-d62f72c5d009-tls-certs\") pod \"2f1e3bbb-ab1c-455d-a05d-d62f72c5d009\" (UID: \"2f1e3bbb-ab1c-455d-a05d-d62f72c5d009\") " Oct 30 17:39:06.498261 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.498217 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f1e3bbb-ab1c-455d-a05d-d62f72c5d009-home" pod "2f1e3bbb-ab1c-455d-a05d-d62f72c5d009" (UID: "2f1e3bbb-ab1c-455d-a05d-d62f72c5d009"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:06.499737 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.498429 4522 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f1e3bbb-ab1c-455d-a05d-d62f72c5d009-home\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:39:06.499737 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.498642 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f1e3bbb-ab1c-455d-a05d-d62f72c5d009-tmp-dir" pod "2f1e3bbb-ab1c-455d-a05d-d62f72c5d009" (UID: "2f1e3bbb-ab1c-455d-a05d-d62f72c5d009"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:06.499737 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.498774 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f1e3bbb-ab1c-455d-a05d-d62f72c5d009-model-cache" pod "2f1e3bbb-ab1c-455d-a05d-d62f72c5d009" (UID: "2f1e3bbb-ab1c-455d-a05d-d62f72c5d009"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:06.499737 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.498850 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9ad71b7-ad90-4c8c-bf27-0f10b23f1593-tmp-dir" pod "c9ad71b7-ad90-4c8c-bf27-0f10b23f1593" (UID: "c9ad71b7-ad90-4c8c-bf27-0f10b23f1593"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:06.499737 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.499441 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9ad71b7-ad90-4c8c-bf27-0f10b23f1593-home" pod "c9ad71b7-ad90-4c8c-bf27-0f10b23f1593" (UID: "c9ad71b7-ad90-4c8c-bf27-0f10b23f1593"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:06.499737 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.499522 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9ad71b7-ad90-4c8c-bf27-0f10b23f1593-model-cache" pod "c9ad71b7-ad90-4c8c-bf27-0f10b23f1593" (UID: "c9ad71b7-ad90-4c8c-bf27-0f10b23f1593"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:06.500543 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.500485 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9ad71b7-ad90-4c8c-bf27-0f10b23f1593-kube-api-access-5lrmc" pod "c9ad71b7-ad90-4c8c-bf27-0f10b23f1593" (UID: "c9ad71b7-ad90-4c8c-bf27-0f10b23f1593"). InnerVolumeSpecName "kube-api-access-5lrmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:39:06.500748 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.500720 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f1e3bbb-ab1c-455d-a05d-d62f72c5d009-tls-certs" pod "2f1e3bbb-ab1c-455d-a05d-d62f72c5d009" (UID: "2f1e3bbb-ab1c-455d-a05d-d62f72c5d009"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:39:06.501334 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.501162 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f1e3bbb-ab1c-455d-a05d-d62f72c5d009-dshm" pod "2f1e3bbb-ab1c-455d-a05d-d62f72c5d009" (UID: "2f1e3bbb-ab1c-455d-a05d-d62f72c5d009"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:06.501334 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.501220 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f1e3bbb-ab1c-455d-a05d-d62f72c5d009-kube-api-access-p47sj" pod "2f1e3bbb-ab1c-455d-a05d-d62f72c5d009" (UID: "2f1e3bbb-ab1c-455d-a05d-d62f72c5d009"). InnerVolumeSpecName "kube-api-access-p47sj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:39:06.502046 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.502021 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9ad71b7-ad90-4c8c-bf27-0f10b23f1593-tls-certs" pod "c9ad71b7-ad90-4c8c-bf27-0f10b23f1593" (UID: "c9ad71b7-ad90-4c8c-bf27-0f10b23f1593"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:39:06.502239 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.502221 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9ad71b7-ad90-4c8c-bf27-0f10b23f1593-dshm" pod "c9ad71b7-ad90-4c8c-bf27-0f10b23f1593" (UID: "c9ad71b7-ad90-4c8c-bf27-0f10b23f1593"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:06.599721 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.599688 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lrmc\" (UniqueName: \"kubernetes.io/projected/c9ad71b7-ad90-4c8c-bf27-0f10b23f1593-kube-api-access-5lrmc\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:39:06.599721 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.599720 4522 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9ad71b7-ad90-4c8c-bf27-0f10b23f1593-model-cache\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:39:06.599721 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.599729 4522 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9ad71b7-ad90-4c8c-bf27-0f10b23f1593-tmp-dir\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:39:06.600184 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.599738 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p47sj\" (UniqueName: \"kubernetes.io/projected/2f1e3bbb-ab1c-455d-a05d-d62f72c5d009-kube-api-access-p47sj\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:39:06.600184 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.599746 4522 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f1e3bbb-ab1c-455d-a05d-d62f72c5d009-dshm\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:39:06.600184 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.599755 4522 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9ad71b7-ad90-4c8c-bf27-0f10b23f1593-tls-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:39:06.600184 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.599764 4522 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f1e3bbb-ab1c-455d-a05d-d62f72c5d009-tmp-dir\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:39:06.600184 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.599772 4522 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f1e3bbb-ab1c-455d-a05d-d62f72c5d009-model-cache\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:39:06.600184 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.599779 4522 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9ad71b7-ad90-4c8c-bf27-0f10b23f1593-dshm\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:39:06.600184 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.599786 4522 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9ad71b7-ad90-4c8c-bf27-0f10b23f1593-home\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:39:06.600184 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:06.599793 4522 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f1e3bbb-ab1c-455d-a05d-d62f72c5d009-tls-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:39:07.367954 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:07.367914 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" Oct 30 17:39:07.368190 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:07.367958 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl" event={"ID":"2f1e3bbb-ab1c-455d-a05d-d62f72c5d009","Type":"ContainerDied","Data":"eceb20fe407065110de59829235140d5b4eb9dca5d825c7c97f18254f2824c03"} Oct 30 17:39:07.368190 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:07.368014 4522 scope.go:122] "RemoveContainer" containerID="56fda00ed5e16c409cc69d65fb9063109ae58c0bd6c429b41977240d42a559c8" Oct 30 17:39:07.370849 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:07.370823 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" Oct 30 17:39:07.370962 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:07.370869 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p" event={"ID":"c9ad71b7-ad90-4c8c-bf27-0f10b23f1593","Type":"ContainerDied","Data":"4de15b3f22a48710206150e3f8bea0fecb80e5246799eb74c8f60dae7b3b7ee2"} Oct 30 17:39:07.378202 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:07.378185 4522 scope.go:122] "RemoveContainer" containerID="bf05f7463cc12ae7785f536115fbfa2a4ff775a268d2b6698e6b34725cdded74" Oct 30 17:39:07.439802 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:07.439775 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl"] Oct 30 17:39:07.442115 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:07.442093 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-096d75/distinct-w0-096d75-kserve-567644df56-8fnkl"] Oct 30 17:39:07.492529 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:07.492496 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p"] Oct 30 17:39:07.498949 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:07.498925 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-096d75/premium-w0-096d75-kserve-8477779bfc-qwp2p"] Oct 30 17:39:08.152382 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:08.152348 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="024635fe-4b1e-4366-82c6-8cb91b132a66" path="/var/lib/kubelet/pods/024635fe-4b1e-4366-82c6-8cb91b132a66/volumes" Oct 30 17:39:08.152787 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:08.152758 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e442fef-3f96-4523-8cae-814507c01bab" path="/var/lib/kubelet/pods/0e442fef-3f96-4523-8cae-814507c01bab/volumes" Oct 30 17:39:08.153136 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:08.153123 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f1e3bbb-ab1c-455d-a05d-d62f72c5d009" path="/var/lib/kubelet/pods/2f1e3bbb-ab1c-455d-a05d-d62f72c5d009/volumes" Oct 30 17:39:08.153782 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:08.153767 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bca92d15-cd50-43f0-bf26-e5603842657c" path="/var/lib/kubelet/pods/bca92d15-cd50-43f0-bf26-e5603842657c/volumes" Oct 30 17:39:08.154154 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:08.154142 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9ad71b7-ad90-4c8c-bf27-0f10b23f1593" path="/var/lib/kubelet/pods/c9ad71b7-ad90-4c8c-bf27-0f10b23f1593/volumes" Oct 30 17:39:11.733913 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:11.733874 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-8f92ef/ipp-route-8f92ef-kserve-7948cdcdf-r2dkv"] Oct 30 17:39:12.946938 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:39:12.946890 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:39:13.119867 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:13.119828 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-096d75-67ff8648-hmszh" podUID="0e716095-6ba5-49f8-a79b-5ec7a226336b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:39:15.594896 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:15.594858 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-096d75-hgnzk"] Oct 30 17:39:15.609204 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:15.609176 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-096d75-hgnzk"] Oct 30 17:39:16.144110 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:16.144076 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61b4a7c2-6306-4c7d-8968-a1fe3f753547" path="/var/lib/kubelet/pods/61b4a7c2-6306-4c7d-8968-a1fe3f753547/volumes" Oct 30 17:39:21.660431 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:21.660403 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-096d75-67ff8648-hmszh" Oct 30 17:39:21.753275 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:21.753241 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e716095-6ba5-49f8-a79b-5ec7a226336b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e716095-6ba5-49f8-a79b-5ec7a226336b-plugins-config-volume\") pod \"0e716095-6ba5-49f8-a79b-5ec7a226336b\" (UID: \"0e716095-6ba5-49f8-a79b-5ec7a226336b\") " Oct 30 17:39:21.753490 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:21.753369 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e716095-6ba5-49f8-a79b-5ec7a226336b-kube-api-access-lp7lf\" (UniqueName: \"kubernetes.io/projected/0e716095-6ba5-49f8-a79b-5ec7a226336b-kube-api-access-lp7lf\") pod \"0e716095-6ba5-49f8-a79b-5ec7a226336b\" (UID: \"0e716095-6ba5-49f8-a79b-5ec7a226336b\") " Oct 30 17:39:21.753761 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:21.753735 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e716095-6ba5-49f8-a79b-5ec7a226336b-plugins-config-volume" pod "0e716095-6ba5-49f8-a79b-5ec7a226336b" (UID: "0e716095-6ba5-49f8-a79b-5ec7a226336b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:39:21.755526 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:21.755501 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e716095-6ba5-49f8-a79b-5ec7a226336b-kube-api-access-lp7lf" pod "0e716095-6ba5-49f8-a79b-5ec7a226336b" (UID: "0e716095-6ba5-49f8-a79b-5ec7a226336b"). InnerVolumeSpecName "kube-api-access-lp7lf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:39:21.854749 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:21.854715 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e716095-6ba5-49f8-a79b-5ec7a226336b-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:39:21.854749 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:21.854744 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp7lf\" (UniqueName: \"kubernetes.io/projected/0e716095-6ba5-49f8-a79b-5ec7a226336b-kube-api-access-lp7lf\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:39:22.477932 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:22.477898 4522 generic.go:356] "Generic (PLEG): container finished" podID="0e716095-6ba5-49f8-a79b-5ec7a226336b" containerID="1bc673e20285e377f207342390515b137239392b2cfb111957e6a7214a22fbd8" exitCode=0 Oct 30 17:39:22.478142 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:22.477950 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-096d75-67ff8648-hmszh" Oct 30 17:39:22.478142 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:22.477979 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-096d75-67ff8648-hmszh" event={"ID":"0e716095-6ba5-49f8-a79b-5ec7a226336b","Type":"ContainerDied","Data":"1bc673e20285e377f207342390515b137239392b2cfb111957e6a7214a22fbd8"} Oct 30 17:39:22.478142 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:22.478031 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-096d75-67ff8648-hmszh" event={"ID":"0e716095-6ba5-49f8-a79b-5ec7a226336b","Type":"ContainerDied","Data":"1b3dfe4e6b9028d0a5dbeb284e32c332004fa1e558f73c618846e2f36b7adbd2"} Oct 30 17:39:22.478142 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:22.478056 4522 scope.go:122] "RemoveContainer" containerID="1bc673e20285e377f207342390515b137239392b2cfb111957e6a7214a22fbd8" Oct 30 17:39:22.487498 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:22.487472 4522 scope.go:122] "RemoveContainer" containerID="1bc673e20285e377f207342390515b137239392b2cfb111957e6a7214a22fbd8" Oct 30 17:39:22.487775 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:39:22.487751 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bc673e20285e377f207342390515b137239392b2cfb111957e6a7214a22fbd8\": container with ID starting with 1bc673e20285e377f207342390515b137239392b2cfb111957e6a7214a22fbd8 not found: ID does not exist" containerID="1bc673e20285e377f207342390515b137239392b2cfb111957e6a7214a22fbd8" Oct 30 17:39:22.487859 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:22.487789 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bc673e20285e377f207342390515b137239392b2cfb111957e6a7214a22fbd8"} err="failed to get container status \"1bc673e20285e377f207342390515b137239392b2cfb111957e6a7214a22fbd8\": rpc error: code = NotFound desc = could not find container \"1bc673e20285e377f207342390515b137239392b2cfb111957e6a7214a22fbd8\": container with ID starting with 1bc673e20285e377f207342390515b137239392b2cfb111957e6a7214a22fbd8 not found: ID does not exist" Oct 30 17:39:22.539679 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:22.539649 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-096d75-67ff8648-hmszh"] Oct 30 17:39:22.554376 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:22.554350 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-096d75-67ff8648-hmszh"] Oct 30 17:39:23.236403 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:23.236366 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk"] Oct 30 17:39:23.236846 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:23.236633 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk" podUID="93447f05-2d76-440f-b704-5e37d09eb5b9" containerName="istio-proxy" containerID="cri-o://1058a199527188cb4606777166e1a6ebfd898bec47e834876f4e7e58ac3d6126" gracePeriod=30 Oct 30 17:39:24.143903 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:24.143871 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e716095-6ba5-49f8-a79b-5ec7a226336b" path="/var/lib/kubelet/pods/0e716095-6ba5-49f8-a79b-5ec7a226336b/volumes" Oct 30 17:39:26.754664 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:26.754603 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-8f92ef/ipp-route-8f92ef-kserve-7948cdcdf-r2dkv" podUID="8a834b00-2f75-49d9-9624-ed1545b1a989" containerName="main" containerID="cri-o://cffac3184332cc22b788902b791e0d1500d4655e5c2e596e044353ac363ca097" gracePeriod=45 Oct 30 17:39:27.011443 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:27.011382 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-8f92ef/ipp-route-8f92ef-kserve-7948cdcdf-r2dkv" Oct 30 17:39:27.108574 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:27.108541 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a834b00-2f75-49d9-9624-ed1545b1a989-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a834b00-2f75-49d9-9624-ed1545b1a989-model-cache\") pod \"8a834b00-2f75-49d9-9624-ed1545b1a989\" (UID: \"8a834b00-2f75-49d9-9624-ed1545b1a989\") " Oct 30 17:39:27.108768 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:27.108595 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a834b00-2f75-49d9-9624-ed1545b1a989-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a834b00-2f75-49d9-9624-ed1545b1a989-tmp-dir\") pod \"8a834b00-2f75-49d9-9624-ed1545b1a989\" (UID: \"8a834b00-2f75-49d9-9624-ed1545b1a989\") " Oct 30 17:39:27.108768 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:27.108663 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a834b00-2f75-49d9-9624-ed1545b1a989-kube-api-access-6shdm\" (UniqueName: \"kubernetes.io/projected/8a834b00-2f75-49d9-9624-ed1545b1a989-kube-api-access-6shdm\") pod \"8a834b00-2f75-49d9-9624-ed1545b1a989\" (UID: \"8a834b00-2f75-49d9-9624-ed1545b1a989\") " Oct 30 17:39:27.108768 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:27.108720 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a834b00-2f75-49d9-9624-ed1545b1a989-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a834b00-2f75-49d9-9624-ed1545b1a989-dshm\") pod \"8a834b00-2f75-49d9-9624-ed1545b1a989\" (UID: \"8a834b00-2f75-49d9-9624-ed1545b1a989\") " Oct 30 17:39:27.108961 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:27.108784 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a834b00-2f75-49d9-9624-ed1545b1a989-tls-certs\" (UniqueName: \"kubernetes.io/secret/8a834b00-2f75-49d9-9624-ed1545b1a989-tls-certs\") pod \"8a834b00-2f75-49d9-9624-ed1545b1a989\" (UID: \"8a834b00-2f75-49d9-9624-ed1545b1a989\") " Oct 30 17:39:27.108961 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:27.108863 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a834b00-2f75-49d9-9624-ed1545b1a989-home\" (UniqueName: \"kubernetes.io/empty-dir/8a834b00-2f75-49d9-9624-ed1545b1a989-home\") pod \"8a834b00-2f75-49d9-9624-ed1545b1a989\" (UID: \"8a834b00-2f75-49d9-9624-ed1545b1a989\") " Oct 30 17:39:27.108961 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:27.108873 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a834b00-2f75-49d9-9624-ed1545b1a989-model-cache" pod "8a834b00-2f75-49d9-9624-ed1545b1a989" (UID: "8a834b00-2f75-49d9-9624-ed1545b1a989"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:27.108961 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:27.108924 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a834b00-2f75-49d9-9624-ed1545b1a989-tmp-dir" pod "8a834b00-2f75-49d9-9624-ed1545b1a989" (UID: "8a834b00-2f75-49d9-9624-ed1545b1a989"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:27.109186 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:27.109169 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a834b00-2f75-49d9-9624-ed1545b1a989-home" pod "8a834b00-2f75-49d9-9624-ed1545b1a989" (UID: "8a834b00-2f75-49d9-9624-ed1545b1a989"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:27.109243 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:27.109212 4522 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a834b00-2f75-49d9-9624-ed1545b1a989-model-cache\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:39:27.109243 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:27.109228 4522 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a834b00-2f75-49d9-9624-ed1545b1a989-tmp-dir\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:39:27.110963 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:27.110938 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a834b00-2f75-49d9-9624-ed1545b1a989-dshm" pod "8a834b00-2f75-49d9-9624-ed1545b1a989" (UID: "8a834b00-2f75-49d9-9624-ed1545b1a989"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:27.111108 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:27.111009 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a834b00-2f75-49d9-9624-ed1545b1a989-tls-certs" pod "8a834b00-2f75-49d9-9624-ed1545b1a989" (UID: "8a834b00-2f75-49d9-9624-ed1545b1a989"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:39:27.111108 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:27.111051 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a834b00-2f75-49d9-9624-ed1545b1a989-kube-api-access-6shdm" pod "8a834b00-2f75-49d9-9624-ed1545b1a989" (UID: "8a834b00-2f75-49d9-9624-ed1545b1a989"). InnerVolumeSpecName "kube-api-access-6shdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:39:27.209846 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:27.209815 4522 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a834b00-2f75-49d9-9624-ed1545b1a989-home\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:39:27.209846 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:27.209842 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6shdm\" (UniqueName: \"kubernetes.io/projected/8a834b00-2f75-49d9-9624-ed1545b1a989-kube-api-access-6shdm\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:39:27.209846 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:27.209853 4522 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a834b00-2f75-49d9-9624-ed1545b1a989-dshm\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:39:27.210096 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:27.209862 4522 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a834b00-2f75-49d9-9624-ed1545b1a989-tls-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:39:27.512505 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:27.512466 4522 generic.go:356] "Generic (PLEG): container finished" podID="8a834b00-2f75-49d9-9624-ed1545b1a989" containerID="cffac3184332cc22b788902b791e0d1500d4655e5c2e596e044353ac363ca097" exitCode=0 Oct 30 17:39:27.512697 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:27.512527 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-8f92ef/ipp-route-8f92ef-kserve-7948cdcdf-r2dkv" Oct 30 17:39:27.512697 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:27.512555 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-8f92ef/ipp-route-8f92ef-kserve-7948cdcdf-r2dkv" event={"ID":"8a834b00-2f75-49d9-9624-ed1545b1a989","Type":"ContainerDied","Data":"cffac3184332cc22b788902b791e0d1500d4655e5c2e596e044353ac363ca097"} Oct 30 17:39:27.512697 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:27.512597 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-8f92ef/ipp-route-8f92ef-kserve-7948cdcdf-r2dkv" event={"ID":"8a834b00-2f75-49d9-9624-ed1545b1a989","Type":"ContainerDied","Data":"6ea4b6b42cdc2cba1343fd9f088c1871dd2b931d5442723748ba8f4a5d465d0d"} Oct 30 17:39:27.512697 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:27.512614 4522 scope.go:122] "RemoveContainer" containerID="cffac3184332cc22b788902b791e0d1500d4655e5c2e596e044353ac363ca097" Oct 30 17:39:27.522581 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:27.522557 4522 scope.go:122] "RemoveContainer" containerID="cffac3184332cc22b788902b791e0d1500d4655e5c2e596e044353ac363ca097" Oct 30 17:39:27.522861 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:39:27.522840 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cffac3184332cc22b788902b791e0d1500d4655e5c2e596e044353ac363ca097\": container with ID starting with cffac3184332cc22b788902b791e0d1500d4655e5c2e596e044353ac363ca097 not found: ID does not exist" containerID="cffac3184332cc22b788902b791e0d1500d4655e5c2e596e044353ac363ca097" Oct 30 17:39:27.522917 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:27.522879 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cffac3184332cc22b788902b791e0d1500d4655e5c2e596e044353ac363ca097"} err="failed to get container status \"cffac3184332cc22b788902b791e0d1500d4655e5c2e596e044353ac363ca097\": rpc error: code = NotFound desc = could not find container \"cffac3184332cc22b788902b791e0d1500d4655e5c2e596e044353ac363ca097\": container with ID starting with cffac3184332cc22b788902b791e0d1500d4655e5c2e596e044353ac363ca097 not found: ID does not exist" Oct 30 17:39:27.573832 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:27.573800 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-8f92ef/ipp-route-8f92ef-kserve-7948cdcdf-r2dkv"] Oct 30 17:39:27.588508 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:27.588470 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-8f92ef/ipp-route-8f92ef-kserve-7948cdcdf-r2dkv"] Oct 30 17:39:27.955175 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:27.955134 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-8f92ef/ipp-route-8f92ef-kserve-7948cdcdf-r2dkv" podUID="8a834b00-2f75-49d9-9624-ed1545b1a989" containerName="main" probeResult="failure" output="Get \"https://10.134.1.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:39:28.144109 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:28.144078 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a834b00-2f75-49d9-9624-ed1545b1a989" path="/var/lib/kubelet/pods/8a834b00-2f75-49d9-9624-ed1545b1a989/volumes" Oct 30 17:39:28.510694 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:28.510630 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk" Oct 30 17:39:28.521035 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:28.520942 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93447f05-2d76-440f-b704-5e37d09eb5b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93447f05-2d76-440f-b704-5e37d09eb5b9-workload-socket\") pod \"93447f05-2d76-440f-b704-5e37d09eb5b9\" (UID: \"93447f05-2d76-440f-b704-5e37d09eb5b9\") " Oct 30 17:39:28.521035 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:28.520989 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93447f05-2d76-440f-b704-5e37d09eb5b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93447f05-2d76-440f-b704-5e37d09eb5b9-credential-socket\") pod \"93447f05-2d76-440f-b704-5e37d09eb5b9\" (UID: \"93447f05-2d76-440f-b704-5e37d09eb5b9\") " Oct 30 17:39:28.521035 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:28.521011 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93447f05-2d76-440f-b704-5e37d09eb5b9-kube-api-access-st6r5\" (UniqueName: \"kubernetes.io/projected/93447f05-2d76-440f-b704-5e37d09eb5b9-kube-api-access-st6r5\") pod \"93447f05-2d76-440f-b704-5e37d09eb5b9\" (UID: \"93447f05-2d76-440f-b704-5e37d09eb5b9\") " Oct 30 17:39:28.521374 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:28.521045 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93447f05-2d76-440f-b704-5e37d09eb5b9-istio-token\" (UniqueName: \"kubernetes.io/projected/93447f05-2d76-440f-b704-5e37d09eb5b9-istio-token\") pod \"93447f05-2d76-440f-b704-5e37d09eb5b9\" (UID: \"93447f05-2d76-440f-b704-5e37d09eb5b9\") " Oct 30 17:39:28.521374 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:28.521112 4522 generic.go:356] "Generic (PLEG): container finished" podID="93447f05-2d76-440f-b704-5e37d09eb5b9" containerID="1058a199527188cb4606777166e1a6ebfd898bec47e834876f4e7e58ac3d6126" exitCode=0 Oct 30 17:39:28.521374 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:28.521177 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk" event={"ID":"93447f05-2d76-440f-b704-5e37d09eb5b9","Type":"ContainerDied","Data":"1058a199527188cb4606777166e1a6ebfd898bec47e834876f4e7e58ac3d6126"} Oct 30 17:39:28.521374 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:28.521227 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk" event={"ID":"93447f05-2d76-440f-b704-5e37d09eb5b9","Type":"ContainerDied","Data":"78d7bab050147b824a53c070a69eca1e90f2d1574e9beaee8f864d06d90a6639"} Oct 30 17:39:28.521374 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:28.521237 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93447f05-2d76-440f-b704-5e37d09eb5b9-workload-socket" pod "93447f05-2d76-440f-b704-5e37d09eb5b9" (UID: "93447f05-2d76-440f-b704-5e37d09eb5b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:28.521374 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:28.521257 4522 scope.go:122] "RemoveContainer" containerID="1058a199527188cb4606777166e1a6ebfd898bec47e834876f4e7e58ac3d6126" Oct 30 17:39:28.521374 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:28.521259 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93447f05-2d76-440f-b704-5e37d09eb5b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93447f05-2d76-440f-b704-5e37d09eb5b9-istio-data\") pod \"93447f05-2d76-440f-b704-5e37d09eb5b9\" (UID: \"93447f05-2d76-440f-b704-5e37d09eb5b9\") " Oct 30 17:39:28.521374 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:28.521247 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk" Oct 30 17:39:28.521374 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:28.521364 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93447f05-2d76-440f-b704-5e37d09eb5b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93447f05-2d76-440f-b704-5e37d09eb5b9-istio-envoy\") pod \"93447f05-2d76-440f-b704-5e37d09eb5b9\" (UID: \"93447f05-2d76-440f-b704-5e37d09eb5b9\") " Oct 30 17:39:28.521374 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:28.521363 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93447f05-2d76-440f-b704-5e37d09eb5b9-credential-socket" pod "93447f05-2d76-440f-b704-5e37d09eb5b9" (UID: "93447f05-2d76-440f-b704-5e37d09eb5b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:28.521880 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:28.521439 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93447f05-2d76-440f-b704-5e37d09eb5b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93447f05-2d76-440f-b704-5e37d09eb5b9-istiod-ca-cert\") pod \"93447f05-2d76-440f-b704-5e37d09eb5b9\" (UID: \"93447f05-2d76-440f-b704-5e37d09eb5b9\") " Oct 30 17:39:28.521880 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:28.521512 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93447f05-2d76-440f-b704-5e37d09eb5b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93447f05-2d76-440f-b704-5e37d09eb5b9-workload-certs\") pod \"93447f05-2d76-440f-b704-5e37d09eb5b9\" (UID: \"93447f05-2d76-440f-b704-5e37d09eb5b9\") " Oct 30 17:39:28.521880 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:28.521528 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93447f05-2d76-440f-b704-5e37d09eb5b9-istio-data" pod "93447f05-2d76-440f-b704-5e37d09eb5b9" (UID: "93447f05-2d76-440f-b704-5e37d09eb5b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:28.521880 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:28.521547 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/93447f05-2d76-440f-b704-5e37d09eb5b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93447f05-2d76-440f-b704-5e37d09eb5b9-istio-podinfo\") pod \"93447f05-2d76-440f-b704-5e37d09eb5b9\" (UID: \"93447f05-2d76-440f-b704-5e37d09eb5b9\") " Oct 30 17:39:28.521880 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:28.521675 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93447f05-2d76-440f-b704-5e37d09eb5b9-workload-certs" pod "93447f05-2d76-440f-b704-5e37d09eb5b9" (UID: "93447f05-2d76-440f-b704-5e37d09eb5b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:28.521880 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:28.521736 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93447f05-2d76-440f-b704-5e37d09eb5b9-istiod-ca-cert" pod "93447f05-2d76-440f-b704-5e37d09eb5b9" (UID: "93447f05-2d76-440f-b704-5e37d09eb5b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:39:28.522132 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:28.521992 4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93447f05-2d76-440f-b704-5e37d09eb5b9-workload-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:39:28.522132 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:28.522014 4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93447f05-2d76-440f-b704-5e37d09eb5b9-credential-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:39:28.522132 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:28.522028 4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93447f05-2d76-440f-b704-5e37d09eb5b9-istio-data\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:39:28.522132 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:28.522042 4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93447f05-2d76-440f-b704-5e37d09eb5b9-istiod-ca-cert\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:39:28.522132 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:28.522056 4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93447f05-2d76-440f-b704-5e37d09eb5b9-workload-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:39:28.523870 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:28.523843 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93447f05-2d76-440f-b704-5e37d09eb5b9-istio-token" pod "93447f05-2d76-440f-b704-5e37d09eb5b9" (UID: "93447f05-2d76-440f-b704-5e37d09eb5b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:39:28.523992 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:28.523879 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93447f05-2d76-440f-b704-5e37d09eb5b9-kube-api-access-st6r5" pod "93447f05-2d76-440f-b704-5e37d09eb5b9" (UID: "93447f05-2d76-440f-b704-5e37d09eb5b9"). InnerVolumeSpecName "kube-api-access-st6r5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:39:28.523992 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:28.523921 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93447f05-2d76-440f-b704-5e37d09eb5b9-istio-envoy" pod "93447f05-2d76-440f-b704-5e37d09eb5b9" (UID: "93447f05-2d76-440f-b704-5e37d09eb5b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:28.523992 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:28.523966 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93447f05-2d76-440f-b704-5e37d09eb5b9-istio-podinfo" pod "93447f05-2d76-440f-b704-5e37d09eb5b9" (UID: "93447f05-2d76-440f-b704-5e37d09eb5b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:39:28.545776 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:28.545752 4522 scope.go:122] "RemoveContainer" containerID="388ee59b65c02b896fabf0d8713ff08db363affa92507f67f24e34bd21ed3bcb" Oct 30 17:39:28.555261 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:28.555233 4522 scope.go:122] "RemoveContainer" containerID="1058a199527188cb4606777166e1a6ebfd898bec47e834876f4e7e58ac3d6126" Oct 30 17:39:28.555569 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:39:28.555550 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1058a199527188cb4606777166e1a6ebfd898bec47e834876f4e7e58ac3d6126\": container with ID starting with 1058a199527188cb4606777166e1a6ebfd898bec47e834876f4e7e58ac3d6126 not found: ID does not exist" containerID="1058a199527188cb4606777166e1a6ebfd898bec47e834876f4e7e58ac3d6126" Oct 30 17:39:28.555625 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:28.555577 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1058a199527188cb4606777166e1a6ebfd898bec47e834876f4e7e58ac3d6126"} err="failed to get container status \"1058a199527188cb4606777166e1a6ebfd898bec47e834876f4e7e58ac3d6126\": rpc error: code = NotFound desc = could not find container \"1058a199527188cb4606777166e1a6ebfd898bec47e834876f4e7e58ac3d6126\": container with ID starting with 1058a199527188cb4606777166e1a6ebfd898bec47e834876f4e7e58ac3d6126 not found: ID does not exist" Oct 30 17:39:28.555625 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:28.555595 4522 scope.go:122] "RemoveContainer" containerID="388ee59b65c02b896fabf0d8713ff08db363affa92507f67f24e34bd21ed3bcb" Oct 30 17:39:28.555847 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:39:28.555829 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"388ee59b65c02b896fabf0d8713ff08db363affa92507f67f24e34bd21ed3bcb\": container with ID starting with 388ee59b65c02b896fabf0d8713ff08db363affa92507f67f24e34bd21ed3bcb not found: ID does not exist" containerID="388ee59b65c02b896fabf0d8713ff08db363affa92507f67f24e34bd21ed3bcb" Oct 30 17:39:28.555911 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:28.555853 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"388ee59b65c02b896fabf0d8713ff08db363affa92507f67f24e34bd21ed3bcb"} err="failed to get container status \"388ee59b65c02b896fabf0d8713ff08db363affa92507f67f24e34bd21ed3bcb\": rpc error: code = NotFound desc = could not find container \"388ee59b65c02b896fabf0d8713ff08db363affa92507f67f24e34bd21ed3bcb\": container with ID starting with 388ee59b65c02b896fabf0d8713ff08db363affa92507f67f24e34bd21ed3bcb not found: ID does not exist" Oct 30 17:39:28.622937 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:28.622903 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-st6r5\" (UniqueName: \"kubernetes.io/projected/93447f05-2d76-440f-b704-5e37d09eb5b9-kube-api-access-st6r5\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:39:28.622937 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:28.622933 4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93447f05-2d76-440f-b704-5e37d09eb5b9-istio-token\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:39:28.622937 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:28.622944 4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93447f05-2d76-440f-b704-5e37d09eb5b9-istio-envoy\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:39:28.623171 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:28.622952 4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93447f05-2d76-440f-b704-5e37d09eb5b9-istio-podinfo\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:39:28.882538 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:28.882506 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk"] Oct 30 17:39:28.887294 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:28.887266 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sznwk"] Oct 30 17:39:30.149606 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:30.149576 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93447f05-2d76-440f-b704-5e37d09eb5b9" path="/var/lib/kubelet/pods/93447f05-2d76-440f-b704-5e37d09eb5b9/volumes" Oct 30 17:39:42.947805 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:39:42.947766 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:39:57.119379 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:57.119345 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-054acd-nlx29"] Oct 30 17:39:57.121748 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:57.119796 4522 cpu_manager.go:414] "containerMap: removing container" podUID="123ecbf4-9360-4086-8c6f-6fbad0ae75d2" containerName="revoke-keys" Oct 30 17:39:57.121748 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:57.119808 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="123ecbf4-9360-4086-8c6f-6fbad0ae75d2" containerName="revoke-keys" Oct 30 17:39:57.121748 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:57.119821 4522 cpu_manager.go:414] "containerMap: removing container" podUID="0e442fef-3f96-4523-8cae-814507c01bab" containerName="payload-pre-processing" Oct 30 17:39:57.121748 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:57.119827 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e442fef-3f96-4523-8cae-814507c01bab" containerName="payload-pre-processing" Oct 30 17:39:57.121748 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:57.119839 4522 cpu_manager.go:414] "containerMap: removing container" podUID="93447f05-2d76-440f-b704-5e37d09eb5b9" containerName="istio-proxy" Oct 30 17:39:57.121748 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:57.119844 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93447f05-2d76-440f-b704-5e37d09eb5b9" containerName="istio-proxy" Oct 30 17:39:57.121748 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:57.119855 4522 cpu_manager.go:414] "containerMap: removing container" podUID="8a834b00-2f75-49d9-9624-ed1545b1a989" containerName="main" Oct 30 17:39:57.121748 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:57.119859 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a834b00-2f75-49d9-9624-ed1545b1a989" containerName="main" Oct 30 17:39:57.121748 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:57.119866 4522 cpu_manager.go:414] "containerMap: removing container" podUID="bca92d15-cd50-43f0-bf26-e5603842657c" containerName="payload-processing" Oct 30 17:39:57.121748 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:57.119870 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bca92d15-cd50-43f0-bf26-e5603842657c" containerName="payload-processing" Oct 30 17:39:57.121748 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:57.119877 4522 cpu_manager.go:414] "containerMap: removing container" podUID="41a11823-1100-41a5-bd01-b67ec1ed5191" containerName="revoke-keys" Oct 30 17:39:57.121748 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:57.119881 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41a11823-1100-41a5-bd01-b67ec1ed5191" containerName="revoke-keys" Oct 30 17:39:57.121748 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:57.119895 4522 cpu_manager.go:414] "containerMap: removing container" podUID="2f1e3bbb-ab1c-455d-a05d-d62f72c5d009" containerName="main" Oct 30 17:39:57.121748 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:57.119899 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f1e3bbb-ab1c-455d-a05d-d62f72c5d009" containerName="main" Oct 30 17:39:57.121748 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:57.119907 4522 cpu_manager.go:414] "containerMap: removing container" podUID="93447f05-2d76-440f-b704-5e37d09eb5b9" containerName="istio-proxy" Oct 30 17:39:57.121748 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:57.119912 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93447f05-2d76-440f-b704-5e37d09eb5b9" containerName="istio-proxy" Oct 30 17:39:57.121748 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:57.119923 4522 cpu_manager.go:414] "containerMap: removing container" podUID="e25622db-17ca-433c-bd0a-94589349d821" containerName="payload-pre-processing" Oct 30 17:39:57.121748 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:57.119927 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e25622db-17ca-433c-bd0a-94589349d821" containerName="payload-pre-processing" Oct 30 17:39:57.121748 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:57.119936 4522 cpu_manager.go:414] "containerMap: removing container" podUID="0e716095-6ba5-49f8-a79b-5ec7a226336b" containerName="payload-processing" Oct 30 17:39:57.121748 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:57.119941 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e716095-6ba5-49f8-a79b-5ec7a226336b" containerName="payload-processing" Oct 30 17:39:57.121748 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:57.119948 4522 cpu_manager.go:414] "containerMap: removing container" podUID="c9ad71b7-ad90-4c8c-bf27-0f10b23f1593" containerName="main" Oct 30 17:39:57.121748 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:57.119954 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9ad71b7-ad90-4c8c-bf27-0f10b23f1593" containerName="main" Oct 30 17:39:57.121748 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:57.120010 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a834b00-2f75-49d9-9624-ed1545b1a989" containerName="main" Oct 30 17:39:57.121748 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:57.120019 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="bca92d15-cd50-43f0-bf26-e5603842657c" containerName="payload-processing" Oct 30 17:39:57.121748 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:57.120026 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="41a11823-1100-41a5-bd01-b67ec1ed5191" containerName="revoke-keys" Oct 30 17:39:57.121748 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:57.120034 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f1e3bbb-ab1c-455d-a05d-d62f72c5d009" containerName="main" Oct 30 17:39:57.121748 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:57.120040 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="93447f05-2d76-440f-b704-5e37d09eb5b9" containerName="istio-proxy" Oct 30 17:39:57.121748 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:57.120047 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="e25622db-17ca-433c-bd0a-94589349d821" containerName="payload-pre-processing" Oct 30 17:39:57.121748 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:57.120054 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e716095-6ba5-49f8-a79b-5ec7a226336b" containerName="payload-processing" Oct 30 17:39:57.121748 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:57.120061 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9ad71b7-ad90-4c8c-bf27-0f10b23f1593" containerName="main" Oct 30 17:39:57.121748 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:57.120067 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="123ecbf4-9360-4086-8c6f-6fbad0ae75d2" containerName="revoke-keys" Oct 30 17:39:57.121748 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:57.120075 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e442fef-3f96-4523-8cae-814507c01bab" containerName="payload-pre-processing" Oct 30 17:39:57.122992 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:57.122976 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-054acd-nlx29" Oct 30 17:39:57.152078 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:57.152048 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-054acd-nlx29"] Oct 30 17:39:57.172364 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:57.172333 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:39:57.172364 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:57.172340 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:39:57.205725 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:57.205688 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dd75d39-c37c-4bed-ab32-e804b6de0204-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-054acd-nlx29\" (UID: \"0dd75d39-c37c-4bed-ab32-e804b6de0204\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-054acd-nlx29" Oct 30 17:39:57.205898 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:57.205743 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l5kd\" (UniqueName: \"kubernetes.io/projected/0dd75d39-c37c-4bed-ab32-e804b6de0204-kube-api-access-5l5kd\") pod \"maas-api-revoke-keys-e2e-ipp-b-054acd-nlx29\" (UID: \"0dd75d39-c37c-4bed-ab32-e804b6de0204\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-054acd-nlx29" Oct 30 17:39:57.205898 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:57.205861 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dd75d39-c37c-4bed-ab32-e804b6de0204-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-054acd-nlx29\" (UID: \"0dd75d39-c37c-4bed-ab32-e804b6de0204\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-054acd-nlx29" Oct 30 17:39:57.307203 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:57.307167 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dd75d39-c37c-4bed-ab32-e804b6de0204-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-054acd-nlx29\" (UID: \"0dd75d39-c37c-4bed-ab32-e804b6de0204\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-054acd-nlx29" Oct 30 17:39:57.307411 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:57.307233 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dd75d39-c37c-4bed-ab32-e804b6de0204-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-054acd-nlx29\" (UID: \"0dd75d39-c37c-4bed-ab32-e804b6de0204\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-054acd-nlx29" Oct 30 17:39:57.307411 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:57.307274 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l5kd\" (UniqueName: \"kubernetes.io/projected/0dd75d39-c37c-4bed-ab32-e804b6de0204-kube-api-access-5l5kd\") pod \"maas-api-revoke-keys-e2e-ipp-b-054acd-nlx29\" (UID: \"0dd75d39-c37c-4bed-ab32-e804b6de0204\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-054acd-nlx29" Oct 30 17:39:57.308035 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:57.308010 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dd75d39-c37c-4bed-ab32-e804b6de0204-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-054acd-nlx29\" (UID: \"0dd75d39-c37c-4bed-ab32-e804b6de0204\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-054acd-nlx29" Oct 30 17:39:57.309821 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:57.309782 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dd75d39-c37c-4bed-ab32-e804b6de0204-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-054acd-nlx29\" (UID: \"0dd75d39-c37c-4bed-ab32-e804b6de0204\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-054acd-nlx29" Oct 30 17:39:57.333957 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:57.333927 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l5kd\" (UniqueName: \"kubernetes.io/projected/0dd75d39-c37c-4bed-ab32-e804b6de0204-kube-api-access-5l5kd\") pod \"maas-api-revoke-keys-e2e-ipp-b-054acd-nlx29\" (UID: \"0dd75d39-c37c-4bed-ab32-e804b6de0204\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-054acd-nlx29" Oct 30 17:39:57.436282 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:57.436194 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-054acd-nlx29" Oct 30 17:39:57.741458 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:39:57.741419 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dd75d39_c37c_4bed_ab32_e804b6de0204.slice/crio-5f97cf25c0b6e4c279a6bbb6760d938eb9eb34c22f2d47785655e3a15b23f1c2 WatchSource:0}: Error finding container 5f97cf25c0b6e4c279a6bbb6760d938eb9eb34c22f2d47785655e3a15b23f1c2: Status 404 returned error can't find the container with id 5f97cf25c0b6e4c279a6bbb6760d938eb9eb34c22f2d47785655e3a15b23f1c2 Oct 30 17:39:57.742447 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:57.742424 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-054acd-nlx29"] Oct 30 17:39:58.716347 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:58.716299 4522 generic.go:356] "Generic (PLEG): container finished" podID="0dd75d39-c37c-4bed-ab32-e804b6de0204" containerID="621b562886c0054886b9d8230d30205f9bbe70176dbf1d227d63267799d52972" exitCode=0 Oct 30 17:39:58.716347 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:58.716333 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-054acd-nlx29" event={"ID":"0dd75d39-c37c-4bed-ab32-e804b6de0204","Type":"ContainerDied","Data":"621b562886c0054886b9d8230d30205f9bbe70176dbf1d227d63267799d52972"} Oct 30 17:39:58.716801 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:58.716384 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-054acd-nlx29" event={"ID":"0dd75d39-c37c-4bed-ab32-e804b6de0204","Type":"ContainerStarted","Data":"5f97cf25c0b6e4c279a6bbb6760d938eb9eb34c22f2d47785655e3a15b23f1c2"} Oct 30 17:39:59.870266 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:39:59.870239 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-054acd-nlx29" Oct 30 17:40:00.034184 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:00.034102 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0dd75d39-c37c-4bed-ab32-e804b6de0204-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dd75d39-c37c-4bed-ab32-e804b6de0204-maas-api-xks-ca\") pod \"0dd75d39-c37c-4bed-ab32-e804b6de0204\" (UID: \"0dd75d39-c37c-4bed-ab32-e804b6de0204\") " Oct 30 17:40:00.034394 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:00.034216 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dd75d39-c37c-4bed-ab32-e804b6de0204-kube-api-access-5l5kd\" (UniqueName: \"kubernetes.io/projected/0dd75d39-c37c-4bed-ab32-e804b6de0204-kube-api-access-5l5kd\") pod \"0dd75d39-c37c-4bed-ab32-e804b6de0204\" (UID: \"0dd75d39-c37c-4bed-ab32-e804b6de0204\") " Oct 30 17:40:00.034394 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:00.034249 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dd75d39-c37c-4bed-ab32-e804b6de0204-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dd75d39-c37c-4bed-ab32-e804b6de0204-maas-api-openshift-ca\") pod \"0dd75d39-c37c-4bed-ab32-e804b6de0204\" (UID: \"0dd75d39-c37c-4bed-ab32-e804b6de0204\") " Oct 30 17:40:00.034667 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:00.034636 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dd75d39-c37c-4bed-ab32-e804b6de0204-maas-api-openshift-ca" pod "0dd75d39-c37c-4bed-ab32-e804b6de0204" (UID: "0dd75d39-c37c-4bed-ab32-e804b6de0204"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:40:00.036271 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:00.036244 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dd75d39-c37c-4bed-ab32-e804b6de0204-maas-api-xks-ca" pod "0dd75d39-c37c-4bed-ab32-e804b6de0204" (UID: "0dd75d39-c37c-4bed-ab32-e804b6de0204"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:40:00.036401 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:00.036379 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dd75d39-c37c-4bed-ab32-e804b6de0204-kube-api-access-5l5kd" pod "0dd75d39-c37c-4bed-ab32-e804b6de0204" (UID: "0dd75d39-c37c-4bed-ab32-e804b6de0204"). InnerVolumeSpecName "kube-api-access-5l5kd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:40:00.135604 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:00.135565 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l5kd\" (UniqueName: \"kubernetes.io/projected/0dd75d39-c37c-4bed-ab32-e804b6de0204-kube-api-access-5l5kd\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:40:00.135604 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:00.135599 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dd75d39-c37c-4bed-ab32-e804b6de0204-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:40:00.135604 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:00.135610 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dd75d39-c37c-4bed-ab32-e804b6de0204-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:40:00.731862 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:00.731837 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-054acd-nlx29" Oct 30 17:40:00.731862 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:00.731837 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-054acd-nlx29" event={"ID":"0dd75d39-c37c-4bed-ab32-e804b6de0204","Type":"ContainerDied","Data":"5f97cf25c0b6e4c279a6bbb6760d938eb9eb34c22f2d47785655e3a15b23f1c2"} Oct 30 17:40:00.732136 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:00.731905 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f97cf25c0b6e4c279a6bbb6760d938eb9eb34c22f2d47785655e3a15b23f1c2" Oct 30 17:40:08.542108 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:08.542070 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-054acd-5bd75fbb7d-b8qks"] Oct 30 17:40:08.542582 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:08.542376 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-054acd-5bd75fbb7d-b8qks" podUID="d2d64419-530d-4740-affc-9730206d31a1" containerName="payload-processing" containerID="cri-o://3c8e45b4a07105dc7a9e1b7620db5f2b861d339b2777c646f469f12db79e3834" gracePeriod=30 Oct 30 17:40:08.556895 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:08.556862 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-054acd-8fb78649f-2tvjd"] Oct 30 17:40:08.557121 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:08.557095 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-054acd-8fb78649f-2tvjd" podUID="0f17482c-7c2f-441a-9a58-fcfae062d99d" containerName="payload-pre-processing" containerID="cri-o://19801787dda5be12957a876aaee44aeef1e0075976408a357c4e58c0f07274c4" gracePeriod=30 Oct 30 17:40:08.736756 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:08.736710 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-054acd-8fb78649f-2tvjd" podUID="0f17482c-7c2f-441a-9a58-fcfae062d99d" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.1.111:9004: connect: connection refused" Oct 30 17:40:08.786500 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:08.786416 4522 generic.go:356] "Generic (PLEG): container finished" podID="0f17482c-7c2f-441a-9a58-fcfae062d99d" containerID="19801787dda5be12957a876aaee44aeef1e0075976408a357c4e58c0f07274c4" exitCode=0 Oct 30 17:40:08.786639 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:08.786504 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-054acd-8fb78649f-2tvjd" event={"ID":"0f17482c-7c2f-441a-9a58-fcfae062d99d","Type":"ContainerDied","Data":"19801787dda5be12957a876aaee44aeef1e0075976408a357c4e58c0f07274c4"} Oct 30 17:40:08.788598 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:08.788560 4522 generic.go:356] "Generic (PLEG): container finished" podID="d2d64419-530d-4740-affc-9730206d31a1" containerID="3c8e45b4a07105dc7a9e1b7620db5f2b861d339b2777c646f469f12db79e3834" exitCode=0 Oct 30 17:40:08.788734 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:08.788682 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-054acd-5bd75fbb7d-b8qks" event={"ID":"d2d64419-530d-4740-affc-9730206d31a1","Type":"ContainerDied","Data":"3c8e45b4a07105dc7a9e1b7620db5f2b861d339b2777c646f469f12db79e3834"} Oct 30 17:40:08.837455 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:08.837398 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-054acd-5bd75fbb7d-b8qks" Oct 30 17:40:08.866594 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:08.866570 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-054acd-8fb78649f-2tvjd" Oct 30 17:40:08.919886 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:08.919847 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2d64419-530d-4740-affc-9730206d31a1-kube-api-access-78f9z\" (UniqueName: \"kubernetes.io/projected/d2d64419-530d-4740-affc-9730206d31a1-kube-api-access-78f9z\") pod \"d2d64419-530d-4740-affc-9730206d31a1\" (UID: \"d2d64419-530d-4740-affc-9730206d31a1\") " Oct 30 17:40:08.920051 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:08.919966 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f17482c-7c2f-441a-9a58-fcfae062d99d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f17482c-7c2f-441a-9a58-fcfae062d99d-plugins-config-volume\") pod \"0f17482c-7c2f-441a-9a58-fcfae062d99d\" (UID: \"0f17482c-7c2f-441a-9a58-fcfae062d99d\") " Oct 30 17:40:08.920051 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:08.920011 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2d64419-530d-4740-affc-9730206d31a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2d64419-530d-4740-affc-9730206d31a1-plugins-config-volume\") pod \"d2d64419-530d-4740-affc-9730206d31a1\" (UID: \"d2d64419-530d-4740-affc-9730206d31a1\") " Oct 30 17:40:08.920186 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:08.920137 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f17482c-7c2f-441a-9a58-fcfae062d99d-kube-api-access-5x8vt\" (UniqueName: \"kubernetes.io/projected/0f17482c-7c2f-441a-9a58-fcfae062d99d-kube-api-access-5x8vt\") pod \"0f17482c-7c2f-441a-9a58-fcfae062d99d\" (UID: \"0f17482c-7c2f-441a-9a58-fcfae062d99d\") " Oct 30 17:40:08.920453 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:08.920427 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f17482c-7c2f-441a-9a58-fcfae062d99d-plugins-config-volume" pod "0f17482c-7c2f-441a-9a58-fcfae062d99d" (UID: "0f17482c-7c2f-441a-9a58-fcfae062d99d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:40:08.920554 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:08.920450 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2d64419-530d-4740-affc-9730206d31a1-plugins-config-volume" pod "d2d64419-530d-4740-affc-9730206d31a1" (UID: "d2d64419-530d-4740-affc-9730206d31a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:40:08.922131 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:08.922106 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f17482c-7c2f-441a-9a58-fcfae062d99d-kube-api-access-5x8vt" pod "0f17482c-7c2f-441a-9a58-fcfae062d99d" (UID: "0f17482c-7c2f-441a-9a58-fcfae062d99d"). InnerVolumeSpecName "kube-api-access-5x8vt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:40:08.922215 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:08.922180 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2d64419-530d-4740-affc-9730206d31a1-kube-api-access-78f9z" pod "d2d64419-530d-4740-affc-9730206d31a1" (UID: "d2d64419-530d-4740-affc-9730206d31a1"). InnerVolumeSpecName "kube-api-access-78f9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:40:09.020975 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:09.020937 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78f9z\" (UniqueName: \"kubernetes.io/projected/d2d64419-530d-4740-affc-9730206d31a1-kube-api-access-78f9z\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:40:09.020975 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:09.020968 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f17482c-7c2f-441a-9a58-fcfae062d99d-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:40:09.020975 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:09.020977 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2d64419-530d-4740-affc-9730206d31a1-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:40:09.021221 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:09.020987 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5x8vt\" (UniqueName: \"kubernetes.io/projected/0f17482c-7c2f-441a-9a58-fcfae062d99d-kube-api-access-5x8vt\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:40:09.113656 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:09.113562 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-054acd-nlx29"] Oct 30 17:40:09.117677 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:09.117650 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-054acd-nlx29"] Oct 30 17:40:09.736534 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:09.736479 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-b-054acd-5bd75fbb7d-b8qks" podUID="d2d64419-530d-4740-affc-9730206d31a1" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.1.112:9004: i/o timeout" Oct 30 17:40:09.796733 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:09.796702 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-054acd-8fb78649f-2tvjd" Oct 30 17:40:09.796733 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:09.796726 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-054acd-8fb78649f-2tvjd" event={"ID":"0f17482c-7c2f-441a-9a58-fcfae062d99d","Type":"ContainerDied","Data":"31ada452cb838e99cebaa0fcfded306e5fc7f2b3fceaea4a0db5b9462e4eb57c"} Oct 30 17:40:09.797024 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:09.796795 4522 scope.go:122] "RemoveContainer" containerID="19801787dda5be12957a876aaee44aeef1e0075976408a357c4e58c0f07274c4" Oct 30 17:40:09.798188 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:09.798176 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-054acd-5bd75fbb7d-b8qks" Oct 30 17:40:09.798254 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:09.798207 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-054acd-5bd75fbb7d-b8qks" event={"ID":"d2d64419-530d-4740-affc-9730206d31a1","Type":"ContainerDied","Data":"57bfa4df1c9ff42f61e4f1128f52cc07128841427b7b3d4d339c4a14cb8adda1"} Oct 30 17:40:09.806983 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:09.806966 4522 scope.go:122] "RemoveContainer" containerID="3c8e45b4a07105dc7a9e1b7620db5f2b861d339b2777c646f469f12db79e3834" Oct 30 17:40:09.855440 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:09.855411 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-054acd-8fb78649f-2tvjd"] Oct 30 17:40:09.870009 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:09.869977 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-054acd-8fb78649f-2tvjd"] Oct 30 17:40:09.900831 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:09.900799 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-054acd-5bd75fbb7d-b8qks"] Oct 30 17:40:09.905255 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:09.905230 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-054acd-5bd75fbb7d-b8qks"] Oct 30 17:40:10.143849 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:10.143771 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dd75d39-c37c-4bed-ab32-e804b6de0204" path="/var/lib/kubelet/pods/0dd75d39-c37c-4bed-ab32-e804b6de0204/volumes" Oct 30 17:40:10.144167 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:10.144153 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f17482c-7c2f-441a-9a58-fcfae062d99d" path="/var/lib/kubelet/pods/0f17482c-7c2f-441a-9a58-fcfae062d99d/volumes" Oct 30 17:40:10.144560 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:10.144547 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2d64419-530d-4740-affc-9730206d31a1" path="/var/lib/kubelet/pods/d2d64419-530d-4740-affc-9730206d31a1/volumes" Oct 30 17:40:11.163663 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:11.163633 4522 scope.go:122] "RemoveContainer" containerID="fda6cfd8cdade92e9c49fa92721df406af53f5aaea9f8f525ec2a0e288e6e582" Oct 30 17:40:11.175235 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:11.175212 4522 scope.go:122] "RemoveContainer" containerID="bde599dc3027ee90e60fa454f6f62a3e7546fad9b132f1c576d6afbc7812a3d3" Oct 30 17:40:11.184264 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:11.184241 4522 scope.go:122] "RemoveContainer" containerID="7a19f6cbc73c2b4a8348b60670c5ee1637fa625b7259f4f4caf90e726feb4286" Oct 30 17:40:11.192498 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:11.192470 4522 scope.go:122] "RemoveContainer" containerID="6f38426a25b5f146e8f2eac0b0e85935e9a1f3a40b8ba2b80345d94031215b56" Oct 30 17:40:12.947461 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:40:12.947426 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:40:27.006062 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:27.005980 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-b87f3c-f97f9ffc4-f5qb8"] Oct 30 17:40:27.035710 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:27.035671 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b87f3c-85b7f76ff9-mm6sf"] Oct 30 17:40:32.007106 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:32.007053 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b87f3c-f97f9ffc4-f5qb8" podUID="f26c6749-26c9-44cc-81bf-0e49d20ddc1a" containerName="payload-processing" containerID="cri-o://592d1ad5a47798de16b10641da42a340a8f5671970c0e1a316331b4b1703b744" gracePeriod=25 Oct 30 17:40:32.036202 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:32.036138 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b87f3c-85b7f76ff9-mm6sf" podUID="d718a4ec-8b4d-4d81-bd11-bc1e6c704b1f" containerName="payload-pre-processing" containerID="cri-o://1df72c2f845671c180833129ae22615d1b2ca295ebe8ba7050e658fa56a0b3a2" gracePeriod=25 Oct 30 17:40:38.398279 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:38.398242 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b87f3c-f97f9ffc4-f5qb8" podUID="f26c6749-26c9-44cc-81bf-0e49d20ddc1a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:40:38.398781 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:38.398278 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b87f3c-85b7f76ff9-mm6sf" podUID="d718a4ec-8b4d-4d81-bd11-bc1e6c704b1f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:40:42.946677 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:40:42.946638 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:40:46.538813 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:46.538780 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-e13a51-b5f84f694-7tz8j"] Oct 30 17:40:46.577655 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:46.577620 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e13a51-7f74f8bfcf-5wstg"] Oct 30 17:40:47.631446 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:47.631414 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e13a51-75c9c5946-dcfnn"] Oct 30 17:40:47.631965 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:47.631633 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e13a51-75c9c5946-dcfnn" podUID="5451bfad-41b1-4c30-bdbb-6821a3ca6727" containerName="maas-api" containerID="cri-o://72b50ec3728559dc6e41c6af2cb0646490d1a929d0b6648ed71bb6bfca65db51" gracePeriod=30 Oct 30 17:40:47.681771 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:47.681731 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e13a51-75c9c5946-dcfnn" podUID="5451bfad-41b1-4c30-bdbb-6821a3ca6727" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.1.41:8443/health\": dial tcp 10.134.1.41:8443: connect: connection refused" Oct 30 17:40:47.882609 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:47.882541 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e13a51-75c9c5946-dcfnn" Oct 30 17:40:47.913544 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:47.913510 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5451bfad-41b1-4c30-bdbb-6821a3ca6727-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5451bfad-41b1-4c30-bdbb-6821a3ca6727-openshift-service-ca\") pod \"5451bfad-41b1-4c30-bdbb-6821a3ca6727\" (UID: \"5451bfad-41b1-4c30-bdbb-6821a3ca6727\") " Oct 30 17:40:47.913728 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:47.913607 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5451bfad-41b1-4c30-bdbb-6821a3ca6727-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5451bfad-41b1-4c30-bdbb-6821a3ca6727-maas-api-tls\") pod \"5451bfad-41b1-4c30-bdbb-6821a3ca6727\" (UID: \"5451bfad-41b1-4c30-bdbb-6821a3ca6727\") " Oct 30 17:40:47.913728 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:47.913670 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5451bfad-41b1-4c30-bdbb-6821a3ca6727-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5451bfad-41b1-4c30-bdbb-6821a3ca6727-metrics-certs\") pod \"5451bfad-41b1-4c30-bdbb-6821a3ca6727\" (UID: \"5451bfad-41b1-4c30-bdbb-6821a3ca6727\") " Oct 30 17:40:47.913819 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:47.913736 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5451bfad-41b1-4c30-bdbb-6821a3ca6727-kube-api-access-mn6xg\" (UniqueName: \"kubernetes.io/projected/5451bfad-41b1-4c30-bdbb-6821a3ca6727-kube-api-access-mn6xg\") pod \"5451bfad-41b1-4c30-bdbb-6821a3ca6727\" (UID: \"5451bfad-41b1-4c30-bdbb-6821a3ca6727\") " Oct 30 17:40:47.913974 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:47.913952 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5451bfad-41b1-4c30-bdbb-6821a3ca6727-openshift-service-ca" pod "5451bfad-41b1-4c30-bdbb-6821a3ca6727" (UID: "5451bfad-41b1-4c30-bdbb-6821a3ca6727"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:40:47.914100 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:47.914082 4522 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5451bfad-41b1-4c30-bdbb-6821a3ca6727-openshift-service-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:40:47.915938 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:47.915909 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5451bfad-41b1-4c30-bdbb-6821a3ca6727-kube-api-access-mn6xg" pod "5451bfad-41b1-4c30-bdbb-6821a3ca6727" (UID: "5451bfad-41b1-4c30-bdbb-6821a3ca6727"). InnerVolumeSpecName "kube-api-access-mn6xg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:40:47.915938 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:47.915920 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5451bfad-41b1-4c30-bdbb-6821a3ca6727-metrics-certs" pod "5451bfad-41b1-4c30-bdbb-6821a3ca6727" (UID: "5451bfad-41b1-4c30-bdbb-6821a3ca6727"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:40:47.916085 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:47.915941 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5451bfad-41b1-4c30-bdbb-6821a3ca6727-maas-api-tls" pod "5451bfad-41b1-4c30-bdbb-6821a3ca6727" (UID: "5451bfad-41b1-4c30-bdbb-6821a3ca6727"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:40:48.014969 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:48.014932 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5451bfad-41b1-4c30-bdbb-6821a3ca6727-maas-api-tls\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:40:48.014969 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:48.014963 4522 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5451bfad-41b1-4c30-bdbb-6821a3ca6727-metrics-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:40:48.014969 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:48.014972 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mn6xg\" (UniqueName: \"kubernetes.io/projected/5451bfad-41b1-4c30-bdbb-6821a3ca6727-kube-api-access-mn6xg\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:40:48.033250 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:48.033212 4522 generic.go:356] "Generic (PLEG): container finished" podID="5451bfad-41b1-4c30-bdbb-6821a3ca6727" containerID="72b50ec3728559dc6e41c6af2cb0646490d1a929d0b6648ed71bb6bfca65db51" exitCode=0 Oct 30 17:40:48.033444 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:48.033287 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e13a51-75c9c5946-dcfnn" Oct 30 17:40:48.033444 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:48.033300 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e13a51-75c9c5946-dcfnn" event={"ID":"5451bfad-41b1-4c30-bdbb-6821a3ca6727","Type":"ContainerDied","Data":"72b50ec3728559dc6e41c6af2cb0646490d1a929d0b6648ed71bb6bfca65db51"} Oct 30 17:40:48.033540 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:48.033466 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e13a51-75c9c5946-dcfnn" event={"ID":"5451bfad-41b1-4c30-bdbb-6821a3ca6727","Type":"ContainerDied","Data":"c571eff37b4c10208dd6e116a47dfd01350a232a013727a66fa0aab28e97a3b1"} Oct 30 17:40:48.033540 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:48.033490 4522 scope.go:122] "RemoveContainer" containerID="72b50ec3728559dc6e41c6af2cb0646490d1a929d0b6648ed71bb6bfca65db51" Oct 30 17:40:48.045038 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:48.045020 4522 scope.go:122] "RemoveContainer" containerID="72b50ec3728559dc6e41c6af2cb0646490d1a929d0b6648ed71bb6bfca65db51" Oct 30 17:40:48.045351 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:40:48.045295 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72b50ec3728559dc6e41c6af2cb0646490d1a929d0b6648ed71bb6bfca65db51\": container with ID starting with 72b50ec3728559dc6e41c6af2cb0646490d1a929d0b6648ed71bb6bfca65db51 not found: ID does not exist" containerID="72b50ec3728559dc6e41c6af2cb0646490d1a929d0b6648ed71bb6bfca65db51" Oct 30 17:40:48.045447 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:48.045358 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72b50ec3728559dc6e41c6af2cb0646490d1a929d0b6648ed71bb6bfca65db51"} err="failed to get container status \"72b50ec3728559dc6e41c6af2cb0646490d1a929d0b6648ed71bb6bfca65db51\": rpc error: code = NotFound desc = could not find container \"72b50ec3728559dc6e41c6af2cb0646490d1a929d0b6648ed71bb6bfca65db51\": container with ID starting with 72b50ec3728559dc6e41c6af2cb0646490d1a929d0b6648ed71bb6bfca65db51 not found: ID does not exist" Oct 30 17:40:48.092211 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:48.092180 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e13a51-75c9c5946-dcfnn"] Oct 30 17:40:48.096725 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:48.096702 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e13a51-75c9c5946-dcfnn"] Oct 30 17:40:48.143779 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:48.143708 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5451bfad-41b1-4c30-bdbb-6821a3ca6727" path="/var/lib/kubelet/pods/5451bfad-41b1-4c30-bdbb-6821a3ca6727/volumes" Oct 30 17:40:48.398558 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:48.398460 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b87f3c-85b7f76ff9-mm6sf" podUID="d718a4ec-8b4d-4d81-bd11-bc1e6c704b1f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:40:48.398558 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:48.398479 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b87f3c-f97f9ffc4-f5qb8" podUID="f26c6749-26c9-44cc-81bf-0e49d20ddc1a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:40:51.539317 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:51.539269 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e13a51-b5f84f694-7tz8j" podUID="4135bd88-8ef0-44c0-8679-9f6719ef3a18" containerName="payload-processing" containerID="cri-o://9152f62938dfa61b26630e9b83ed63c9536a27d663e9ad339b936567ab797007" gracePeriod=25 Oct 30 17:40:51.577961 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:51.577903 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e13a51-7f74f8bfcf-5wstg" podUID="eef74243-b455-4c59-baf2-d4ae87e3e5c4" containerName="payload-pre-processing" containerID="cri-o://9df31bc48b79b7872a08ab9617b8719fe068abc5cfa06a45c27eddb926eeb5e7" gracePeriod=25 Oct 30 17:40:52.276482 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:52.276245 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b87f3c-f97f9ffc4-f5qb8" Oct 30 17:40:52.315344 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:52.315291 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b87f3c-85b7f76ff9-mm6sf" Oct 30 17:40:52.360993 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:52.360961 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d718a4ec-8b4d-4d81-bd11-bc1e6c704b1f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d718a4ec-8b4d-4d81-bd11-bc1e6c704b1f-plugins-config-volume\") pod \"d718a4ec-8b4d-4d81-bd11-bc1e6c704b1f\" (UID: \"d718a4ec-8b4d-4d81-bd11-bc1e6c704b1f\") " Oct 30 17:40:52.361213 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:52.361076 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d718a4ec-8b4d-4d81-bd11-bc1e6c704b1f-kube-api-access-6c9jb\" (UniqueName: \"kubernetes.io/projected/d718a4ec-8b4d-4d81-bd11-bc1e6c704b1f-kube-api-access-6c9jb\") pod \"d718a4ec-8b4d-4d81-bd11-bc1e6c704b1f\" (UID: \"d718a4ec-8b4d-4d81-bd11-bc1e6c704b1f\") " Oct 30 17:40:52.361213 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:52.361111 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f26c6749-26c9-44cc-81bf-0e49d20ddc1a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f26c6749-26c9-44cc-81bf-0e49d20ddc1a-plugins-config-volume\") pod \"f26c6749-26c9-44cc-81bf-0e49d20ddc1a\" (UID: \"f26c6749-26c9-44cc-81bf-0e49d20ddc1a\") " Oct 30 17:40:52.361213 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:52.361163 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f26c6749-26c9-44cc-81bf-0e49d20ddc1a-kube-api-access-6szwn\" (UniqueName: \"kubernetes.io/projected/f26c6749-26c9-44cc-81bf-0e49d20ddc1a-kube-api-access-6szwn\") pod \"f26c6749-26c9-44cc-81bf-0e49d20ddc1a\" (UID: \"f26c6749-26c9-44cc-81bf-0e49d20ddc1a\") " Oct 30 17:40:52.361568 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:52.361540 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d718a4ec-8b4d-4d81-bd11-bc1e6c704b1f-plugins-config-volume" pod "d718a4ec-8b4d-4d81-bd11-bc1e6c704b1f" (UID: "d718a4ec-8b4d-4d81-bd11-bc1e6c704b1f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:40:52.361642 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:52.361550 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f26c6749-26c9-44cc-81bf-0e49d20ddc1a-plugins-config-volume" pod "f26c6749-26c9-44cc-81bf-0e49d20ddc1a" (UID: "f26c6749-26c9-44cc-81bf-0e49d20ddc1a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:40:52.363328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:52.363288 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d718a4ec-8b4d-4d81-bd11-bc1e6c704b1f-kube-api-access-6c9jb" pod "d718a4ec-8b4d-4d81-bd11-bc1e6c704b1f" (UID: "d718a4ec-8b4d-4d81-bd11-bc1e6c704b1f"). InnerVolumeSpecName "kube-api-access-6c9jb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:40:52.363434 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:52.363344 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f26c6749-26c9-44cc-81bf-0e49d20ddc1a-kube-api-access-6szwn" pod "f26c6749-26c9-44cc-81bf-0e49d20ddc1a" (UID: "f26c6749-26c9-44cc-81bf-0e49d20ddc1a"). InnerVolumeSpecName "kube-api-access-6szwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:40:52.462563 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:52.462526 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d718a4ec-8b4d-4d81-bd11-bc1e6c704b1f-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:40:52.462563 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:52.462555 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c9jb\" (UniqueName: \"kubernetes.io/projected/d718a4ec-8b4d-4d81-bd11-bc1e6c704b1f-kube-api-access-6c9jb\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:40:52.462563 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:52.462565 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f26c6749-26c9-44cc-81bf-0e49d20ddc1a-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:40:52.462816 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:52.462575 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6szwn\" (UniqueName: \"kubernetes.io/projected/f26c6749-26c9-44cc-81bf-0e49d20ddc1a-kube-api-access-6szwn\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:40:53.064031 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:53.063992 4522 generic.go:356] "Generic (PLEG): container finished" podID="f26c6749-26c9-44cc-81bf-0e49d20ddc1a" containerID="592d1ad5a47798de16b10641da42a340a8f5671970c0e1a316331b4b1703b744" exitCode=0 Oct 30 17:40:53.064579 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:53.064041 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b87f3c-f97f9ffc4-f5qb8" Oct 30 17:40:53.064579 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:53.064077 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b87f3c-f97f9ffc4-f5qb8" event={"ID":"f26c6749-26c9-44cc-81bf-0e49d20ddc1a","Type":"ContainerDied","Data":"592d1ad5a47798de16b10641da42a340a8f5671970c0e1a316331b4b1703b744"} Oct 30 17:40:53.064579 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:53.064132 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b87f3c-f97f9ffc4-f5qb8" event={"ID":"f26c6749-26c9-44cc-81bf-0e49d20ddc1a","Type":"ContainerDied","Data":"d2ea113d79d94255e77d323cac5ef71d57703dd28c6804a21770d45247ee84c8"} Oct 30 17:40:53.064579 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:53.064161 4522 scope.go:122] "RemoveContainer" containerID="592d1ad5a47798de16b10641da42a340a8f5671970c0e1a316331b4b1703b744" Oct 30 17:40:53.065637 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:53.065618 4522 generic.go:356] "Generic (PLEG): container finished" podID="d718a4ec-8b4d-4d81-bd11-bc1e6c704b1f" containerID="1df72c2f845671c180833129ae22615d1b2ca295ebe8ba7050e658fa56a0b3a2" exitCode=0 Oct 30 17:40:53.065739 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:53.065686 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b87f3c-85b7f76ff9-mm6sf" event={"ID":"d718a4ec-8b4d-4d81-bd11-bc1e6c704b1f","Type":"ContainerDied","Data":"1df72c2f845671c180833129ae22615d1b2ca295ebe8ba7050e658fa56a0b3a2"} Oct 30 17:40:53.065739 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:53.065716 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b87f3c-85b7f76ff9-mm6sf" Oct 30 17:40:53.065867 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:53.065722 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b87f3c-85b7f76ff9-mm6sf" event={"ID":"d718a4ec-8b4d-4d81-bd11-bc1e6c704b1f","Type":"ContainerDied","Data":"3410c19db18aca985a0fc4f3a0f14474144cc6c3d55597998eec28a53d93bdff"} Oct 30 17:40:53.085082 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:53.085061 4522 scope.go:122] "RemoveContainer" containerID="592d1ad5a47798de16b10641da42a340a8f5671970c0e1a316331b4b1703b744" Oct 30 17:40:53.085400 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:40:53.085380 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"592d1ad5a47798de16b10641da42a340a8f5671970c0e1a316331b4b1703b744\": container with ID starting with 592d1ad5a47798de16b10641da42a340a8f5671970c0e1a316331b4b1703b744 not found: ID does not exist" containerID="592d1ad5a47798de16b10641da42a340a8f5671970c0e1a316331b4b1703b744" Oct 30 17:40:53.085472 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:53.085408 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"592d1ad5a47798de16b10641da42a340a8f5671970c0e1a316331b4b1703b744"} err="failed to get container status \"592d1ad5a47798de16b10641da42a340a8f5671970c0e1a316331b4b1703b744\": rpc error: code = NotFound desc = could not find container \"592d1ad5a47798de16b10641da42a340a8f5671970c0e1a316331b4b1703b744\": container with ID starting with 592d1ad5a47798de16b10641da42a340a8f5671970c0e1a316331b4b1703b744 not found: ID does not exist" Oct 30 17:40:53.085472 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:53.085423 4522 scope.go:122] "RemoveContainer" containerID="1df72c2f845671c180833129ae22615d1b2ca295ebe8ba7050e658fa56a0b3a2" Oct 30 17:40:53.094174 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:53.094157 4522 scope.go:122] "RemoveContainer" containerID="1df72c2f845671c180833129ae22615d1b2ca295ebe8ba7050e658fa56a0b3a2" Oct 30 17:40:53.094453 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:40:53.094434 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1df72c2f845671c180833129ae22615d1b2ca295ebe8ba7050e658fa56a0b3a2\": container with ID starting with 1df72c2f845671c180833129ae22615d1b2ca295ebe8ba7050e658fa56a0b3a2 not found: ID does not exist" containerID="1df72c2f845671c180833129ae22615d1b2ca295ebe8ba7050e658fa56a0b3a2" Oct 30 17:40:53.094505 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:53.094460 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1df72c2f845671c180833129ae22615d1b2ca295ebe8ba7050e658fa56a0b3a2"} err="failed to get container status \"1df72c2f845671c180833129ae22615d1b2ca295ebe8ba7050e658fa56a0b3a2\": rpc error: code = NotFound desc = could not find container \"1df72c2f845671c180833129ae22615d1b2ca295ebe8ba7050e658fa56a0b3a2\": container with ID starting with 1df72c2f845671c180833129ae22615d1b2ca295ebe8ba7050e658fa56a0b3a2 not found: ID does not exist" Oct 30 17:40:53.141208 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:53.141173 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b87f3c-85b7f76ff9-mm6sf"] Oct 30 17:40:53.155274 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:53.155242 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b87f3c-85b7f76ff9-mm6sf"] Oct 30 17:40:53.184507 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:53.184466 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-b87f3c-f97f9ffc4-f5qb8"] Oct 30 17:40:53.189067 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:53.189044 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-b87f3c-f97f9ffc4-f5qb8"] Oct 30 17:40:54.143165 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:54.143131 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d718a4ec-8b4d-4d81-bd11-bc1e6c704b1f" path="/var/lib/kubelet/pods/d718a4ec-8b4d-4d81-bd11-bc1e6c704b1f/volumes" Oct 30 17:40:54.143571 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:54.143543 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f26c6749-26c9-44cc-81bf-0e49d20ddc1a" path="/var/lib/kubelet/pods/f26c6749-26c9-44cc-81bf-0e49d20ddc1a/volumes" Oct 30 17:40:55.924350 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:55.924290 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e13a51-7f74f8bfcf-5wstg" podUID="eef74243-b455-4c59-baf2-d4ae87e3e5c4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:40:56.202665 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:56.202633 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb"] Oct 30 17:40:56.202936 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:56.202907 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb" podUID="2d64632e-0973-4589-9c43-b7b3b7bb95b3" containerName="istio-proxy" containerID="cri-o://29ce2393b5b40d235814699534f9ba13a83b59f1162f52c9a9951d845d970924" gracePeriod=30 Oct 30 17:40:56.902903 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:56.902871 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d384dc-hwmmf"] Oct 30 17:40:56.903701 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:56.903674 4522 cpu_manager.go:414] "containerMap: removing container" podUID="0f17482c-7c2f-441a-9a58-fcfae062d99d" containerName="payload-pre-processing" Oct 30 17:40:56.903845 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:56.903832 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f17482c-7c2f-441a-9a58-fcfae062d99d" containerName="payload-pre-processing" Oct 30 17:40:56.903924 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:56.903914 4522 cpu_manager.go:414] "containerMap: removing container" podUID="0dd75d39-c37c-4bed-ab32-e804b6de0204" containerName="revoke-keys" Oct 30 17:40:56.903993 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:56.903984 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dd75d39-c37c-4bed-ab32-e804b6de0204" containerName="revoke-keys" Oct 30 17:40:56.904075 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:56.904066 4522 cpu_manager.go:414] "containerMap: removing container" podUID="d2d64419-530d-4740-affc-9730206d31a1" containerName="payload-processing" Oct 30 17:40:56.904150 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:56.904141 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2d64419-530d-4740-affc-9730206d31a1" containerName="payload-processing" Oct 30 17:40:56.904236 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:56.904228 4522 cpu_manager.go:414] "containerMap: removing container" podUID="f26c6749-26c9-44cc-81bf-0e49d20ddc1a" containerName="payload-processing" Oct 30 17:40:56.904318 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:56.904294 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f26c6749-26c9-44cc-81bf-0e49d20ddc1a" containerName="payload-processing" Oct 30 17:40:56.904410 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:56.904401 4522 cpu_manager.go:414] "containerMap: removing container" podUID="5451bfad-41b1-4c30-bdbb-6821a3ca6727" containerName="maas-api" Oct 30 17:40:56.904474 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:56.904464 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5451bfad-41b1-4c30-bdbb-6821a3ca6727" containerName="maas-api" Oct 30 17:40:56.904552 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:56.904543 4522 cpu_manager.go:414] "containerMap: removing container" podUID="d718a4ec-8b4d-4d81-bd11-bc1e6c704b1f" containerName="payload-pre-processing" Oct 30 17:40:56.904675 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:56.904665 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d718a4ec-8b4d-4d81-bd11-bc1e6c704b1f" containerName="payload-pre-processing" Oct 30 17:40:56.904848 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:56.904838 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="5451bfad-41b1-4c30-bdbb-6821a3ca6727" containerName="maas-api" Oct 30 17:40:56.904925 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:56.904917 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="d718a4ec-8b4d-4d81-bd11-bc1e6c704b1f" containerName="payload-pre-processing" Oct 30 17:40:56.905005 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:56.904996 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f17482c-7c2f-441a-9a58-fcfae062d99d" containerName="payload-pre-processing" Oct 30 17:40:56.905073 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:56.905065 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dd75d39-c37c-4bed-ab32-e804b6de0204" containerName="revoke-keys" Oct 30 17:40:56.905141 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:56.905133 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2d64419-530d-4740-affc-9730206d31a1" containerName="payload-processing" Oct 30 17:40:56.905217 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:56.905207 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="93447f05-2d76-440f-b704-5e37d09eb5b9" containerName="istio-proxy" Oct 30 17:40:56.905285 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:56.905277 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="f26c6749-26c9-44cc-81bf-0e49d20ddc1a" containerName="payload-processing" Oct 30 17:40:56.922842 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:56.922819 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d384dc-hwmmf" Oct 30 17:40:56.929129 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:56.929102 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d384dc-hwmmf"] Oct 30 17:40:56.940070 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:56.940042 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:40:56.940207 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:56.940059 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:40:57.003210 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:57.003182 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:40:57.003210 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:57.003187 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:40:57.021740 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:57.021716 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fd43dd1-318c-4a75-9516-afc296f8dbca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-d384dc-hwmmf\" (UID: \"8fd43dd1-318c-4a75-9516-afc296f8dbca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d384dc-hwmmf" Oct 30 17:40:57.021813 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:57.021761 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nbqd\" (UniqueName: \"kubernetes.io/projected/8fd43dd1-318c-4a75-9516-afc296f8dbca-kube-api-access-6nbqd\") pod \"maas-api-revoke-keys-e2e-worker-w6-d384dc-hwmmf\" (UID: \"8fd43dd1-318c-4a75-9516-afc296f8dbca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d384dc-hwmmf" Oct 30 17:40:57.021813 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:57.021784 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fd43dd1-318c-4a75-9516-afc296f8dbca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-d384dc-hwmmf\" (UID: \"8fd43dd1-318c-4a75-9516-afc296f8dbca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d384dc-hwmmf" Oct 30 17:40:57.123379 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:57.123345 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fd43dd1-318c-4a75-9516-afc296f8dbca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-d384dc-hwmmf\" (UID: \"8fd43dd1-318c-4a75-9516-afc296f8dbca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d384dc-hwmmf" Oct 30 17:40:57.123540 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:57.123396 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nbqd\" (UniqueName: \"kubernetes.io/projected/8fd43dd1-318c-4a75-9516-afc296f8dbca-kube-api-access-6nbqd\") pod \"maas-api-revoke-keys-e2e-worker-w6-d384dc-hwmmf\" (UID: \"8fd43dd1-318c-4a75-9516-afc296f8dbca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d384dc-hwmmf" Oct 30 17:40:57.123540 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:57.123419 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fd43dd1-318c-4a75-9516-afc296f8dbca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-d384dc-hwmmf\" (UID: \"8fd43dd1-318c-4a75-9516-afc296f8dbca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d384dc-hwmmf" Oct 30 17:40:57.124011 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:57.123992 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fd43dd1-318c-4a75-9516-afc296f8dbca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-d384dc-hwmmf\" (UID: \"8fd43dd1-318c-4a75-9516-afc296f8dbca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d384dc-hwmmf" Oct 30 17:40:57.125827 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:57.125806 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fd43dd1-318c-4a75-9516-afc296f8dbca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-d384dc-hwmmf\" (UID: \"8fd43dd1-318c-4a75-9516-afc296f8dbca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d384dc-hwmmf" Oct 30 17:40:57.149146 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:57.149120 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nbqd\" (UniqueName: \"kubernetes.io/projected/8fd43dd1-318c-4a75-9516-afc296f8dbca-kube-api-access-6nbqd\") pod \"maas-api-revoke-keys-e2e-worker-w6-d384dc-hwmmf\" (UID: \"8fd43dd1-318c-4a75-9516-afc296f8dbca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d384dc-hwmmf" Oct 30 17:40:57.236903 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:57.236873 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d384dc-hwmmf" Oct 30 17:40:57.538916 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:57.538888 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d384dc-hwmmf"] Oct 30 17:40:57.539039 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:40:57.538926 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fd43dd1_318c_4a75_9516_afc296f8dbca.slice/crio-c8487eadf39ec5221624e34dda6a182d8fdbc1526cdb6f19372327fbfc37c17d WatchSource:0}: Error finding container c8487eadf39ec5221624e34dda6a182d8fdbc1526cdb6f19372327fbfc37c17d: Status 404 returned error can't find the container with id c8487eadf39ec5221624e34dda6a182d8fdbc1526cdb6f19372327fbfc37c17d Oct 30 17:40:57.944990 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:57.944958 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e13a51-b5f84f694-7tz8j" podUID="4135bd88-8ef0-44c0-8679-9f6719ef3a18" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:40:58.098250 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:58.098156 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d384dc-hwmmf" event={"ID":"8fd43dd1-318c-4a75-9516-afc296f8dbca","Type":"ContainerStarted","Data":"b71b17ad69ffb5a53c9233564c39a7172a539bd2237712e9f225eb12a3980220"} Oct 30 17:40:58.098250 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:58.098213 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d384dc-hwmmf" event={"ID":"8fd43dd1-318c-4a75-9516-afc296f8dbca","Type":"ContainerStarted","Data":"c8487eadf39ec5221624e34dda6a182d8fdbc1526cdb6f19372327fbfc37c17d"} Oct 30 17:40:58.144102 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:58.144058 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d384dc-hwmmf" podStartSLOduration=2.144045311 podStartE2EDuration="2.144045311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:40:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:40:58.143565042 +0000 UTC m=+3232.144319956" watchObservedRunningTime="2026-10-30 17:40:58.144045311 +0000 UTC m=+3232.144800212" Oct 30 17:40:59.105399 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:59.105360 4522 generic.go:356] "Generic (PLEG): container finished" podID="8fd43dd1-318c-4a75-9516-afc296f8dbca" containerID="b71b17ad69ffb5a53c9233564c39a7172a539bd2237712e9f225eb12a3980220" exitCode=0 Oct 30 17:40:59.105874 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:40:59.105443 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d384dc-hwmmf" event={"ID":"8fd43dd1-318c-4a75-9516-afc296f8dbca","Type":"ContainerDied","Data":"b71b17ad69ffb5a53c9233564c39a7172a539bd2237712e9f225eb12a3980220"} Oct 30 17:41:00.247209 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:00.247182 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d384dc-hwmmf" Oct 30 17:41:00.355708 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:00.355669 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fd43dd1-318c-4a75-9516-afc296f8dbca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fd43dd1-318c-4a75-9516-afc296f8dbca-maas-api-openshift-ca\") pod \"8fd43dd1-318c-4a75-9516-afc296f8dbca\" (UID: \"8fd43dd1-318c-4a75-9516-afc296f8dbca\") " Oct 30 17:41:00.355920 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:00.355815 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fd43dd1-318c-4a75-9516-afc296f8dbca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fd43dd1-318c-4a75-9516-afc296f8dbca-maas-api-xks-ca\") pod \"8fd43dd1-318c-4a75-9516-afc296f8dbca\" (UID: \"8fd43dd1-318c-4a75-9516-afc296f8dbca\") " Oct 30 17:41:00.355920 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:00.355883 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fd43dd1-318c-4a75-9516-afc296f8dbca-kube-api-access-6nbqd\" (UniqueName: \"kubernetes.io/projected/8fd43dd1-318c-4a75-9516-afc296f8dbca-kube-api-access-6nbqd\") pod \"8fd43dd1-318c-4a75-9516-afc296f8dbca\" (UID: \"8fd43dd1-318c-4a75-9516-afc296f8dbca\") " Oct 30 17:41:00.356061 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:00.356031 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fd43dd1-318c-4a75-9516-afc296f8dbca-maas-api-openshift-ca" pod "8fd43dd1-318c-4a75-9516-afc296f8dbca" (UID: "8fd43dd1-318c-4a75-9516-afc296f8dbca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:41:00.356198 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:00.356183 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fd43dd1-318c-4a75-9516-afc296f8dbca-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:41:00.357987 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:00.357964 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fd43dd1-318c-4a75-9516-afc296f8dbca-maas-api-xks-ca" pod "8fd43dd1-318c-4a75-9516-afc296f8dbca" (UID: "8fd43dd1-318c-4a75-9516-afc296f8dbca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:41:00.358111 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:00.358053 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fd43dd1-318c-4a75-9516-afc296f8dbca-kube-api-access-6nbqd" pod "8fd43dd1-318c-4a75-9516-afc296f8dbca" (UID: "8fd43dd1-318c-4a75-9516-afc296f8dbca"). InnerVolumeSpecName "kube-api-access-6nbqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:41:00.457128 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:00.457096 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nbqd\" (UniqueName: \"kubernetes.io/projected/8fd43dd1-318c-4a75-9516-afc296f8dbca-kube-api-access-6nbqd\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:41:00.457128 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:00.457124 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fd43dd1-318c-4a75-9516-afc296f8dbca-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:41:01.122980 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:01.122942 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d384dc-hwmmf" event={"ID":"8fd43dd1-318c-4a75-9516-afc296f8dbca","Type":"ContainerDied","Data":"c8487eadf39ec5221624e34dda6a182d8fdbc1526cdb6f19372327fbfc37c17d"} Oct 30 17:41:01.122980 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:01.122965 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d384dc-hwmmf" Oct 30 17:41:01.123204 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:01.122988 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8487eadf39ec5221624e34dda6a182d8fdbc1526cdb6f19372327fbfc37c17d" Oct 30 17:41:01.447474 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:01.447446 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb" Oct 30 17:41:01.567531 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:01.567500 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d64632e-0973-4589-9c43-b7b3b7bb95b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d64632e-0973-4589-9c43-b7b3b7bb95b3-istio-podinfo\") pod \"2d64632e-0973-4589-9c43-b7b3b7bb95b3\" (UID: \"2d64632e-0973-4589-9c43-b7b3b7bb95b3\") " Oct 30 17:41:01.567754 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:01.567544 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d64632e-0973-4589-9c43-b7b3b7bb95b3-istio-token\" (UniqueName: \"kubernetes.io/projected/2d64632e-0973-4589-9c43-b7b3b7bb95b3-istio-token\") pod \"2d64632e-0973-4589-9c43-b7b3b7bb95b3\" (UID: \"2d64632e-0973-4589-9c43-b7b3b7bb95b3\") " Oct 30 17:41:01.567754 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:01.567578 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d64632e-0973-4589-9c43-b7b3b7bb95b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d64632e-0973-4589-9c43-b7b3b7bb95b3-workload-certs\") pod \"2d64632e-0973-4589-9c43-b7b3b7bb95b3\" (UID: \"2d64632e-0973-4589-9c43-b7b3b7bb95b3\") " Oct 30 17:41:01.567754 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:01.567627 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d64632e-0973-4589-9c43-b7b3b7bb95b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d64632e-0973-4589-9c43-b7b3b7bb95b3-istio-envoy\") pod \"2d64632e-0973-4589-9c43-b7b3b7bb95b3\" (UID: \"2d64632e-0973-4589-9c43-b7b3b7bb95b3\") " Oct 30 17:41:01.567754 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:01.567682 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d64632e-0973-4589-9c43-b7b3b7bb95b3-kube-api-access-m4lvq\" (UniqueName: \"kubernetes.io/projected/2d64632e-0973-4589-9c43-b7b3b7bb95b3-kube-api-access-m4lvq\") pod \"2d64632e-0973-4589-9c43-b7b3b7bb95b3\" (UID: \"2d64632e-0973-4589-9c43-b7b3b7bb95b3\") " Oct 30 17:41:01.567754 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:01.567707 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d64632e-0973-4589-9c43-b7b3b7bb95b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d64632e-0973-4589-9c43-b7b3b7bb95b3-credential-socket\") pod \"2d64632e-0973-4589-9c43-b7b3b7bb95b3\" (UID: \"2d64632e-0973-4589-9c43-b7b3b7bb95b3\") " Oct 30 17:41:01.567754 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:01.567755 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d64632e-0973-4589-9c43-b7b3b7bb95b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d64632e-0973-4589-9c43-b7b3b7bb95b3-istio-data\") pod \"2d64632e-0973-4589-9c43-b7b3b7bb95b3\" (UID: \"2d64632e-0973-4589-9c43-b7b3b7bb95b3\") " Oct 30 17:41:01.568106 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:01.567817 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d64632e-0973-4589-9c43-b7b3b7bb95b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d64632e-0973-4589-9c43-b7b3b7bb95b3-workload-socket\") pod \"2d64632e-0973-4589-9c43-b7b3b7bb95b3\" (UID: \"2d64632e-0973-4589-9c43-b7b3b7bb95b3\") " Oct 30 17:41:01.568106 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:01.567862 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d64632e-0973-4589-9c43-b7b3b7bb95b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d64632e-0973-4589-9c43-b7b3b7bb95b3-istiod-ca-cert\") pod \"2d64632e-0973-4589-9c43-b7b3b7bb95b3\" (UID: \"2d64632e-0973-4589-9c43-b7b3b7bb95b3\") " Oct 30 17:41:01.568106 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:01.567950 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d64632e-0973-4589-9c43-b7b3b7bb95b3-workload-certs" pod "2d64632e-0973-4589-9c43-b7b3b7bb95b3" (UID: "2d64632e-0973-4589-9c43-b7b3b7bb95b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:41:01.568279 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:01.568098 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d64632e-0973-4589-9c43-b7b3b7bb95b3-credential-socket" pod "2d64632e-0973-4589-9c43-b7b3b7bb95b3" (UID: "2d64632e-0973-4589-9c43-b7b3b7bb95b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:41:01.568356 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:01.568274 4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d64632e-0973-4589-9c43-b7b3b7bb95b3-workload-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:41:01.568356 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:01.568295 4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d64632e-0973-4589-9c43-b7b3b7bb95b3-credential-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:41:01.568356 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:01.568281 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d64632e-0973-4589-9c43-b7b3b7bb95b3-istio-data" pod "2d64632e-0973-4589-9c43-b7b3b7bb95b3" (UID: "2d64632e-0973-4589-9c43-b7b3b7bb95b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:41:01.568356 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:01.568333 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d64632e-0973-4589-9c43-b7b3b7bb95b3-istiod-ca-cert" pod "2d64632e-0973-4589-9c43-b7b3b7bb95b3" (UID: "2d64632e-0973-4589-9c43-b7b3b7bb95b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:41:01.568554 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:01.568364 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d64632e-0973-4589-9c43-b7b3b7bb95b3-workload-socket" pod "2d64632e-0973-4589-9c43-b7b3b7bb95b3" (UID: "2d64632e-0973-4589-9c43-b7b3b7bb95b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:41:01.569916 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:01.569889 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d64632e-0973-4589-9c43-b7b3b7bb95b3-istio-token" pod "2d64632e-0973-4589-9c43-b7b3b7bb95b3" (UID: "2d64632e-0973-4589-9c43-b7b3b7bb95b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:41:01.570035 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:01.569924 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d64632e-0973-4589-9c43-b7b3b7bb95b3-istio-envoy" pod "2d64632e-0973-4589-9c43-b7b3b7bb95b3" (UID: "2d64632e-0973-4589-9c43-b7b3b7bb95b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:41:01.570035 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:01.569955 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d64632e-0973-4589-9c43-b7b3b7bb95b3-istio-podinfo" pod "2d64632e-0973-4589-9c43-b7b3b7bb95b3" (UID: "2d64632e-0973-4589-9c43-b7b3b7bb95b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:41:01.570204 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:01.570190 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d64632e-0973-4589-9c43-b7b3b7bb95b3-kube-api-access-m4lvq" pod "2d64632e-0973-4589-9c43-b7b3b7bb95b3" (UID: "2d64632e-0973-4589-9c43-b7b3b7bb95b3"). InnerVolumeSpecName "kube-api-access-m4lvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:41:01.669206 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:01.669166 4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d64632e-0973-4589-9c43-b7b3b7bb95b3-istio-envoy\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:41:01.669206 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:01.669195 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4lvq\" (UniqueName: \"kubernetes.io/projected/2d64632e-0973-4589-9c43-b7b3b7bb95b3-kube-api-access-m4lvq\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:41:01.669206 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:01.669206 4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d64632e-0973-4589-9c43-b7b3b7bb95b3-istio-data\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:41:01.669206 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:01.669215 4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d64632e-0973-4589-9c43-b7b3b7bb95b3-workload-socket\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:41:01.669522 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:01.669223 4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d64632e-0973-4589-9c43-b7b3b7bb95b3-istiod-ca-cert\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:41:01.669522 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:01.669232 4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d64632e-0973-4589-9c43-b7b3b7bb95b3-istio-podinfo\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:41:01.669522 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:01.669240 4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d64632e-0973-4589-9c43-b7b3b7bb95b3-istio-token\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:41:01.708949 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:01.708918 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f"] Oct 30 17:41:01.719073 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:01.719041 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv"] Oct 30 17:41:02.130780 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:02.130691 4522 generic.go:356] "Generic (PLEG): container finished" podID="2d64632e-0973-4589-9c43-b7b3b7bb95b3" containerID="29ce2393b5b40d235814699534f9ba13a83b59f1162f52c9a9951d845d970924" exitCode=0 Oct 30 17:41:02.130780 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:02.130749 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb" Oct 30 17:41:02.131029 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:02.130780 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb" event={"ID":"2d64632e-0973-4589-9c43-b7b3b7bb95b3","Type":"ContainerDied","Data":"29ce2393b5b40d235814699534f9ba13a83b59f1162f52c9a9951d845d970924"} Oct 30 17:41:02.131029 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:02.130839 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb" event={"ID":"2d64632e-0973-4589-9c43-b7b3b7bb95b3","Type":"ContainerDied","Data":"d6c26fc2dc39bc21b85df0e2d00830a69043d800db559721e7cd13a50457211a"} Oct 30 17:41:02.131029 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:02.130863 4522 scope.go:122] "RemoveContainer" containerID="29ce2393b5b40d235814699534f9ba13a83b59f1162f52c9a9951d845d970924" Oct 30 17:41:02.141548 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:02.141528 4522 scope.go:122] "RemoveContainer" containerID="af69effeda096f045d40224956e652f26bcd057525521bd203c2fbe7fe259de0" Oct 30 17:41:02.151507 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:02.151489 4522 scope.go:122] "RemoveContainer" containerID="29ce2393b5b40d235814699534f9ba13a83b59f1162f52c9a9951d845d970924" Oct 30 17:41:02.151780 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:41:02.151757 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29ce2393b5b40d235814699534f9ba13a83b59f1162f52c9a9951d845d970924\": container with ID starting with 29ce2393b5b40d235814699534f9ba13a83b59f1162f52c9a9951d845d970924 not found: ID does not exist" containerID="29ce2393b5b40d235814699534f9ba13a83b59f1162f52c9a9951d845d970924" Oct 30 17:41:02.151859 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:02.151796 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29ce2393b5b40d235814699534f9ba13a83b59f1162f52c9a9951d845d970924"} err="failed to get container status \"29ce2393b5b40d235814699534f9ba13a83b59f1162f52c9a9951d845d970924\": rpc error: code = NotFound desc = could not find container \"29ce2393b5b40d235814699534f9ba13a83b59f1162f52c9a9951d845d970924\": container with ID starting with 29ce2393b5b40d235814699534f9ba13a83b59f1162f52c9a9951d845d970924 not found: ID does not exist" Oct 30 17:41:02.151859 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:02.151823 4522 scope.go:122] "RemoveContainer" containerID="af69effeda096f045d40224956e652f26bcd057525521bd203c2fbe7fe259de0" Oct 30 17:41:02.152065 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:41:02.152047 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af69effeda096f045d40224956e652f26bcd057525521bd203c2fbe7fe259de0\": container with ID starting with af69effeda096f045d40224956e652f26bcd057525521bd203c2fbe7fe259de0 not found: ID does not exist" containerID="af69effeda096f045d40224956e652f26bcd057525521bd203c2fbe7fe259de0" Oct 30 17:41:02.152150 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:02.152071 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af69effeda096f045d40224956e652f26bcd057525521bd203c2fbe7fe259de0"} err="failed to get container status \"af69effeda096f045d40224956e652f26bcd057525521bd203c2fbe7fe259de0\": rpc error: code = NotFound desc = could not find container \"af69effeda096f045d40224956e652f26bcd057525521bd203c2fbe7fe259de0\": container with ID starting with af69effeda096f045d40224956e652f26bcd057525521bd203c2fbe7fe259de0 not found: ID does not exist" Oct 30 17:41:02.192124 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:02.192090 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb"] Oct 30 17:41:02.206575 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:02.206541 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e13a51-openshift-default-5c64c4bfb5-gg4zb"] Oct 30 17:41:04.144148 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:04.144110 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d64632e-0973-4589-9c43-b7b3b7bb95b3" path="/var/lib/kubelet/pods/2d64632e-0973-4589-9c43-b7b3b7bb95b3/volumes" Oct 30 17:41:05.924838 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:05.924799 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e13a51-7f74f8bfcf-5wstg" podUID="eef74243-b455-4c59-baf2-d4ae87e3e5c4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:41:07.006433 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:07.006399 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-d384dc-7f78968dd6-2w6cp"] Oct 30 17:41:07.945192 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:07.945155 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e13a51-b5f84f694-7tz8j" podUID="4135bd88-8ef0-44c0-8679-9f6719ef3a18" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:41:09.659346 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:09.659293 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7678bc72-vdl8x"] Oct 30 17:41:09.659771 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:09.659683 4522 cpu_manager.go:414] "containerMap: removing container" podUID="2d64632e-0973-4589-9c43-b7b3b7bb95b3" containerName="istio-proxy" Oct 30 17:41:09.659771 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:09.659693 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d64632e-0973-4589-9c43-b7b3b7bb95b3" containerName="istio-proxy" Oct 30 17:41:09.659771 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:09.659704 4522 cpu_manager.go:414] "containerMap: removing container" podUID="8fd43dd1-318c-4a75-9516-afc296f8dbca" containerName="revoke-keys" Oct 30 17:41:09.659771 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:09.659710 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fd43dd1-318c-4a75-9516-afc296f8dbca" containerName="revoke-keys" Oct 30 17:41:09.659771 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:09.659728 4522 cpu_manager.go:414] "containerMap: removing container" podUID="2d64632e-0973-4589-9c43-b7b3b7bb95b3" containerName="istio-proxy" Oct 30 17:41:09.659771 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:09.659733 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d64632e-0973-4589-9c43-b7b3b7bb95b3" containerName="istio-proxy" Oct 30 17:41:09.660025 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:09.659803 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d64632e-0973-4589-9c43-b7b3b7bb95b3" containerName="istio-proxy" Oct 30 17:41:09.660025 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:09.659817 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fd43dd1-318c-4a75-9516-afc296f8dbca" containerName="revoke-keys" Oct 30 17:41:09.662875 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:09.662859 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7678bc72-vdl8x" Oct 30 17:41:09.683831 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:09.683805 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:41:09.687127 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:09.687099 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7678bc72-vdl8x"] Oct 30 17:41:09.698506 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:09.698477 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:41:09.698506 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:09.698484 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:41:09.698680 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:09.698514 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:41:09.743254 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:09.743222 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4s2v\" (UniqueName: \"kubernetes.io/projected/66176bec-e6cb-48ca-8a4d-dde77870ecdc-kube-api-access-t4s2v\") pod \"maas-api-revoke-sub-7678bc72-vdl8x\" (UID: \"66176bec-e6cb-48ca-8a4d-dde77870ecdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7678bc72-vdl8x" Oct 30 17:41:09.743445 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:09.743274 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66176bec-e6cb-48ca-8a4d-dde77870ecdc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7678bc72-vdl8x\" (UID: \"66176bec-e6cb-48ca-8a4d-dde77870ecdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7678bc72-vdl8x" Oct 30 17:41:09.743492 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:09.743454 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66176bec-e6cb-48ca-8a4d-dde77870ecdc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7678bc72-vdl8x\" (UID: \"66176bec-e6cb-48ca-8a4d-dde77870ecdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7678bc72-vdl8x" Oct 30 17:41:09.844746 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:09.844711 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4s2v\" (UniqueName: \"kubernetes.io/projected/66176bec-e6cb-48ca-8a4d-dde77870ecdc-kube-api-access-t4s2v\") pod \"maas-api-revoke-sub-7678bc72-vdl8x\" (UID: \"66176bec-e6cb-48ca-8a4d-dde77870ecdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7678bc72-vdl8x" Oct 30 17:41:09.844909 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:09.844761 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66176bec-e6cb-48ca-8a4d-dde77870ecdc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7678bc72-vdl8x\" (UID: \"66176bec-e6cb-48ca-8a4d-dde77870ecdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7678bc72-vdl8x" Oct 30 17:41:09.844909 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:09.844824 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66176bec-e6cb-48ca-8a4d-dde77870ecdc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7678bc72-vdl8x\" (UID: \"66176bec-e6cb-48ca-8a4d-dde77870ecdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7678bc72-vdl8x" Oct 30 17:41:09.845621 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:09.845600 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66176bec-e6cb-48ca-8a4d-dde77870ecdc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7678bc72-vdl8x\" (UID: \"66176bec-e6cb-48ca-8a4d-dde77870ecdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7678bc72-vdl8x" Oct 30 17:41:09.847127 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:09.847104 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66176bec-e6cb-48ca-8a4d-dde77870ecdc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7678bc72-vdl8x\" (UID: \"66176bec-e6cb-48ca-8a4d-dde77870ecdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7678bc72-vdl8x" Oct 30 17:41:09.870695 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:09.870668 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4s2v\" (UniqueName: \"kubernetes.io/projected/66176bec-e6cb-48ca-8a4d-dde77870ecdc-kube-api-access-t4s2v\") pod \"maas-api-revoke-sub-7678bc72-vdl8x\" (UID: \"66176bec-e6cb-48ca-8a4d-dde77870ecdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7678bc72-vdl8x" Oct 30 17:41:09.976721 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:09.976685 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7678bc72-vdl8x" Oct 30 17:41:10.279900 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:10.279864 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7678bc72-vdl8x"] Oct 30 17:41:10.280678 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:41:10.280653 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66176bec_e6cb_48ca_8a4d_dde77870ecdc.slice/crio-4dc7736caee69b1c4f4998d9c7e69f0fd08d83a49b370603fb0a5adba9a4126d WatchSource:0}: Error finding container 4dc7736caee69b1c4f4998d9c7e69f0fd08d83a49b370603fb0a5adba9a4126d: Status 404 returned error can't find the container with id 4dc7736caee69b1c4f4998d9c7e69f0fd08d83a49b370603fb0a5adba9a4126d Oct 30 17:41:11.192183 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:11.192146 4522 generic.go:356] "Generic (PLEG): container finished" podID="66176bec-e6cb-48ca-8a4d-dde77870ecdc" containerID="584bc63908c867c5299c4a622dd3ed50db2392ac936276405003e77279d82cf8" exitCode=0 Oct 30 17:41:11.192624 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:11.192230 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7678bc72-vdl8x" event={"ID":"66176bec-e6cb-48ca-8a4d-dde77870ecdc","Type":"ContainerDied","Data":"584bc63908c867c5299c4a622dd3ed50db2392ac936276405003e77279d82cf8"} Oct 30 17:41:11.192624 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:11.192281 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7678bc72-vdl8x" event={"ID":"66176bec-e6cb-48ca-8a4d-dde77870ecdc","Type":"ContainerStarted","Data":"4dc7736caee69b1c4f4998d9c7e69f0fd08d83a49b370603fb0a5adba9a4126d"} Oct 30 17:41:11.247530 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:11.247502 4522 scope.go:122] "RemoveContainer" containerID="22a310e48f7ba0a5e0a894f9974d8aaa48aaa3ddf1ed6ea53890940718d6b019" Oct 30 17:41:11.257400 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:11.257368 4522 scope.go:122] "RemoveContainer" containerID="88c4d3c2ab6164876b1e6185e120ff62aff4d63fe3cc56a2894876386aaceef3" Oct 30 17:41:11.265777 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:11.265756 4522 scope.go:122] "RemoveContainer" containerID="d68ecf395852633b0f0efe526aec7bdb11348fc5b3016510484b195a6d78ecdd" Oct 30 17:41:11.274548 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:11.274526 4522 scope.go:122] "RemoveContainer" containerID="4593c955277d68961d927e44192f9f7414aba43357ed2fe6dbd7b096e8d40d8a" Oct 30 17:41:11.283390 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:11.283366 4522 scope.go:122] "RemoveContainer" containerID="707999ee8800ff92209c079b94eebf7134160cb768a4f398ad37dcda39ca04d8" Oct 30 17:41:11.294054 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:11.294033 4522 scope.go:122] "RemoveContainer" containerID="4a28b6d54658fe1829bec3f964c180b1566a416c6b3ea2862d1a10cc1fa1bb89" Oct 30 17:41:11.817427 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:11.817405 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e13a51-b5f84f694-7tz8j" Oct 30 17:41:11.840637 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:11.840613 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e13a51-7f74f8bfcf-5wstg" Oct 30 17:41:11.964082 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:11.964049 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eef74243-b455-4c59-baf2-d4ae87e3e5c4-kube-api-access-zrtzf\" (UniqueName: \"kubernetes.io/projected/eef74243-b455-4c59-baf2-d4ae87e3e5c4-kube-api-access-zrtzf\") pod \"eef74243-b455-4c59-baf2-d4ae87e3e5c4\" (UID: \"eef74243-b455-4c59-baf2-d4ae87e3e5c4\") " Oct 30 17:41:11.964299 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:11.964106 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eef74243-b455-4c59-baf2-d4ae87e3e5c4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eef74243-b455-4c59-baf2-d4ae87e3e5c4-plugins-config-volume\") pod \"eef74243-b455-4c59-baf2-d4ae87e3e5c4\" (UID: \"eef74243-b455-4c59-baf2-d4ae87e3e5c4\") " Oct 30 17:41:11.964299 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:11.964178 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4135bd88-8ef0-44c0-8679-9f6719ef3a18-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4135bd88-8ef0-44c0-8679-9f6719ef3a18-plugins-config-volume\") pod \"4135bd88-8ef0-44c0-8679-9f6719ef3a18\" (UID: \"4135bd88-8ef0-44c0-8679-9f6719ef3a18\") " Oct 30 17:41:11.964299 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:11.964217 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4135bd88-8ef0-44c0-8679-9f6719ef3a18-kube-api-access-m6hjs\" (UniqueName: \"kubernetes.io/projected/4135bd88-8ef0-44c0-8679-9f6719ef3a18-kube-api-access-m6hjs\") pod \"4135bd88-8ef0-44c0-8679-9f6719ef3a18\" (UID: \"4135bd88-8ef0-44c0-8679-9f6719ef3a18\") " Oct 30 17:41:11.964686 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:11.964651 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4135bd88-8ef0-44c0-8679-9f6719ef3a18-plugins-config-volume" pod "4135bd88-8ef0-44c0-8679-9f6719ef3a18" (UID: "4135bd88-8ef0-44c0-8679-9f6719ef3a18"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:41:11.964686 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:11.964653 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eef74243-b455-4c59-baf2-d4ae87e3e5c4-plugins-config-volume" pod "eef74243-b455-4c59-baf2-d4ae87e3e5c4" (UID: "eef74243-b455-4c59-baf2-d4ae87e3e5c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:41:11.966456 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:11.966436 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eef74243-b455-4c59-baf2-d4ae87e3e5c4-kube-api-access-zrtzf" pod "eef74243-b455-4c59-baf2-d4ae87e3e5c4" (UID: "eef74243-b455-4c59-baf2-d4ae87e3e5c4"). InnerVolumeSpecName "kube-api-access-zrtzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:41:11.966528 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:11.966509 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4135bd88-8ef0-44c0-8679-9f6719ef3a18-kube-api-access-m6hjs" pod "4135bd88-8ef0-44c0-8679-9f6719ef3a18" (UID: "4135bd88-8ef0-44c0-8679-9f6719ef3a18"). InnerVolumeSpecName "kube-api-access-m6hjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:41:12.007702 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:12.007659 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-d384dc-7f78968dd6-2w6cp" podUID="acfdfc44-26dc-4fb0-a04a-1e15d28939ee" containerName="payload-processing" containerID="cri-o://58feeccd2dd1800d66537dfb9c04583ebddc4ba7ad9e347347fea19d27946e55" gracePeriod=25 Oct 30 17:41:12.065475 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:12.065375 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrtzf\" (UniqueName: \"kubernetes.io/projected/eef74243-b455-4c59-baf2-d4ae87e3e5c4-kube-api-access-zrtzf\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:41:12.065475 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:12.065411 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eef74243-b455-4c59-baf2-d4ae87e3e5c4-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:41:12.065475 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:12.065425 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4135bd88-8ef0-44c0-8679-9f6719ef3a18-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:41:12.065475 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:12.065436 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6hjs\" (UniqueName: \"kubernetes.io/projected/4135bd88-8ef0-44c0-8679-9f6719ef3a18-kube-api-access-m6hjs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:41:12.199563 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:12.199530 4522 generic.go:356] "Generic (PLEG): container finished" podID="eef74243-b455-4c59-baf2-d4ae87e3e5c4" containerID="9df31bc48b79b7872a08ab9617b8719fe068abc5cfa06a45c27eddb926eeb5e7" exitCode=0 Oct 30 17:41:12.200054 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:12.199575 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e13a51-7f74f8bfcf-5wstg" Oct 30 17:41:12.200054 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:12.199614 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e13a51-7f74f8bfcf-5wstg" event={"ID":"eef74243-b455-4c59-baf2-d4ae87e3e5c4","Type":"ContainerDied","Data":"9df31bc48b79b7872a08ab9617b8719fe068abc5cfa06a45c27eddb926eeb5e7"} Oct 30 17:41:12.200054 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:12.199673 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e13a51-7f74f8bfcf-5wstg" event={"ID":"eef74243-b455-4c59-baf2-d4ae87e3e5c4","Type":"ContainerDied","Data":"900dda9a2f4c724b299142f6ce42c3c236341ac5514f25b8c3185d916e76683a"} Oct 30 17:41:12.200054 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:12.199710 4522 scope.go:122] "RemoveContainer" containerID="9df31bc48b79b7872a08ab9617b8719fe068abc5cfa06a45c27eddb926eeb5e7" Oct 30 17:41:12.201189 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:12.201168 4522 generic.go:356] "Generic (PLEG): container finished" podID="4135bd88-8ef0-44c0-8679-9f6719ef3a18" containerID="9152f62938dfa61b26630e9b83ed63c9536a27d663e9ad339b936567ab797007" exitCode=0 Oct 30 17:41:12.201292 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:12.201208 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e13a51-b5f84f694-7tz8j" Oct 30 17:41:12.201292 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:12.201258 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e13a51-b5f84f694-7tz8j" event={"ID":"4135bd88-8ef0-44c0-8679-9f6719ef3a18","Type":"ContainerDied","Data":"9152f62938dfa61b26630e9b83ed63c9536a27d663e9ad339b936567ab797007"} Oct 30 17:41:12.201445 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:12.201343 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e13a51-b5f84f694-7tz8j" event={"ID":"4135bd88-8ef0-44c0-8679-9f6719ef3a18","Type":"ContainerDied","Data":"bc66c6dedeebd3acdafdbcbf1d439099805e46f9debea09f2732cfbf997b0c46"} Oct 30 17:41:12.210351 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:12.210334 4522 scope.go:122] "RemoveContainer" containerID="9df31bc48b79b7872a08ab9617b8719fe068abc5cfa06a45c27eddb926eeb5e7" Oct 30 17:41:12.210671 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:41:12.210648 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9df31bc48b79b7872a08ab9617b8719fe068abc5cfa06a45c27eddb926eeb5e7\": container with ID starting with 9df31bc48b79b7872a08ab9617b8719fe068abc5cfa06a45c27eddb926eeb5e7 not found: ID does not exist" containerID="9df31bc48b79b7872a08ab9617b8719fe068abc5cfa06a45c27eddb926eeb5e7" Oct 30 17:41:12.210726 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:12.210674 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9df31bc48b79b7872a08ab9617b8719fe068abc5cfa06a45c27eddb926eeb5e7"} err="failed to get container status \"9df31bc48b79b7872a08ab9617b8719fe068abc5cfa06a45c27eddb926eeb5e7\": rpc error: code = NotFound desc = could not find container \"9df31bc48b79b7872a08ab9617b8719fe068abc5cfa06a45c27eddb926eeb5e7\": container with ID starting with 9df31bc48b79b7872a08ab9617b8719fe068abc5cfa06a45c27eddb926eeb5e7 not found: ID does not exist" Oct 30 17:41:12.210726 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:12.210690 4522 scope.go:122] "RemoveContainer" containerID="9152f62938dfa61b26630e9b83ed63c9536a27d663e9ad339b936567ab797007" Oct 30 17:41:12.220383 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:12.220362 4522 scope.go:122] "RemoveContainer" containerID="9152f62938dfa61b26630e9b83ed63c9536a27d663e9ad339b936567ab797007" Oct 30 17:41:12.220690 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:41:12.220672 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9152f62938dfa61b26630e9b83ed63c9536a27d663e9ad339b936567ab797007\": container with ID starting with 9152f62938dfa61b26630e9b83ed63c9536a27d663e9ad339b936567ab797007 not found: ID does not exist" containerID="9152f62938dfa61b26630e9b83ed63c9536a27d663e9ad339b936567ab797007" Oct 30 17:41:12.220773 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:12.220698 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9152f62938dfa61b26630e9b83ed63c9536a27d663e9ad339b936567ab797007"} err="failed to get container status \"9152f62938dfa61b26630e9b83ed63c9536a27d663e9ad339b936567ab797007\": rpc error: code = NotFound desc = could not find container \"9152f62938dfa61b26630e9b83ed63c9536a27d663e9ad339b936567ab797007\": container with ID starting with 9152f62938dfa61b26630e9b83ed63c9536a27d663e9ad339b936567ab797007 not found: ID does not exist" Oct 30 17:41:12.262199 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:12.262164 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-e13a51-b5f84f694-7tz8j"] Oct 30 17:41:12.276714 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:12.276683 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-e13a51-b5f84f694-7tz8j"] Oct 30 17:41:12.314884 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:12.314856 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e13a51-7f74f8bfcf-5wstg"] Oct 30 17:41:12.319562 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:12.319536 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e13a51-7f74f8bfcf-5wstg"] Oct 30 17:41:12.335956 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:12.335935 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7678bc72-vdl8x" Oct 30 17:41:12.468988 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:12.468958 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66176bec-e6cb-48ca-8a4d-dde77870ecdc-kube-api-access-t4s2v\" (UniqueName: \"kubernetes.io/projected/66176bec-e6cb-48ca-8a4d-dde77870ecdc-kube-api-access-t4s2v\") pod \"66176bec-e6cb-48ca-8a4d-dde77870ecdc\" (UID: \"66176bec-e6cb-48ca-8a4d-dde77870ecdc\") " Oct 30 17:41:12.469217 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:12.469013 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66176bec-e6cb-48ca-8a4d-dde77870ecdc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66176bec-e6cb-48ca-8a4d-dde77870ecdc-maas-api-openshift-ca\") pod \"66176bec-e6cb-48ca-8a4d-dde77870ecdc\" (UID: \"66176bec-e6cb-48ca-8a4d-dde77870ecdc\") " Oct 30 17:41:12.469217 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:12.469052 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66176bec-e6cb-48ca-8a4d-dde77870ecdc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66176bec-e6cb-48ca-8a4d-dde77870ecdc-maas-api-xks-ca\") pod \"66176bec-e6cb-48ca-8a4d-dde77870ecdc\" (UID: \"66176bec-e6cb-48ca-8a4d-dde77870ecdc\") " Oct 30 17:41:12.469494 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:12.469469 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66176bec-e6cb-48ca-8a4d-dde77870ecdc-maas-api-openshift-ca" pod "66176bec-e6cb-48ca-8a4d-dde77870ecdc" (UID: "66176bec-e6cb-48ca-8a4d-dde77870ecdc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:41:12.471232 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:12.471208 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66176bec-e6cb-48ca-8a4d-dde77870ecdc-maas-api-xks-ca" pod "66176bec-e6cb-48ca-8a4d-dde77870ecdc" (UID: "66176bec-e6cb-48ca-8a4d-dde77870ecdc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:41:12.471357 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:12.471285 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66176bec-e6cb-48ca-8a4d-dde77870ecdc-kube-api-access-t4s2v" pod "66176bec-e6cb-48ca-8a4d-dde77870ecdc" (UID: "66176bec-e6cb-48ca-8a4d-dde77870ecdc"). InnerVolumeSpecName "kube-api-access-t4s2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:41:12.569957 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:12.569868 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4s2v\" (UniqueName: \"kubernetes.io/projected/66176bec-e6cb-48ca-8a4d-dde77870ecdc-kube-api-access-t4s2v\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:41:12.569957 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:12.569909 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66176bec-e6cb-48ca-8a4d-dde77870ecdc-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:41:12.569957 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:12.569925 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66176bec-e6cb-48ca-8a4d-dde77870ecdc-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:41:12.951350 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:41:12.951302 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:41:13.209677 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:13.209642 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7678bc72-vdl8x" event={"ID":"66176bec-e6cb-48ca-8a4d-dde77870ecdc","Type":"ContainerDied","Data":"4dc7736caee69b1c4f4998d9c7e69f0fd08d83a49b370603fb0a5adba9a4126d"} Oct 30 17:41:13.209677 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:13.209677 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7678bc72-vdl8x" Oct 30 17:41:13.210124 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:13.209692 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4dc7736caee69b1c4f4998d9c7e69f0fd08d83a49b370603fb0a5adba9a4126d" Oct 30 17:41:14.128386 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:14.128347 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-d384dc-7f78968dd6-2w6cp" podUID="acfdfc44-26dc-4fb0-a04a-1e15d28939ee" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:41:14.143504 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:14.143473 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4135bd88-8ef0-44c0-8679-9f6719ef3a18" path="/var/lib/kubelet/pods/4135bd88-8ef0-44c0-8679-9f6719ef3a18/volumes" Oct 30 17:41:14.143870 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:14.143857 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eef74243-b455-4c59-baf2-d4ae87e3e5c4" path="/var/lib/kubelet/pods/eef74243-b455-4c59-baf2-d4ae87e3e5c4/volumes" Oct 30 17:41:16.729464 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:16.729404 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" podUID="8a6a85ad-c944-450f-b0bb-e3b64b0f56f0" containerName="main" containerID="cri-o://a2b7cbb389bdee3d842c9043f5abd81df99eac77a4b68b502e2fb699c8fbe56b" gracePeriod=45 Oct 30 17:41:16.742764 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:16.742712 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" podUID="a3e5e732-2c67-4377-9755-e2178bf33772" containerName="main" containerID="cri-o://4169820d4cf8e16e4fb503ca82588005c1c4d08d1195d62025bf66b21c1a6763" gracePeriod=45 Oct 30 17:41:16.990545 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:16.990476 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" Oct 30 17:41:17.016894 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:17.016868 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" Oct 30 17:41:17.115565 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:17.115525 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3e5e732-2c67-4377-9755-e2178bf33772-kube-api-access-qd5pj\" (UniqueName: \"kubernetes.io/projected/a3e5e732-2c67-4377-9755-e2178bf33772-kube-api-access-qd5pj\") pod \"a3e5e732-2c67-4377-9755-e2178bf33772\" (UID: \"a3e5e732-2c67-4377-9755-e2178bf33772\") " Oct 30 17:41:17.115773 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:17.115596 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3e5e732-2c67-4377-9755-e2178bf33772-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3e5e732-2c67-4377-9755-e2178bf33772-tmp-dir\") pod \"a3e5e732-2c67-4377-9755-e2178bf33772\" (UID: \"a3e5e732-2c67-4377-9755-e2178bf33772\") " Oct 30 17:41:17.115773 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:17.115632 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a6a85ad-c944-450f-b0bb-e3b64b0f56f0-kube-api-access-hvprb\" (UniqueName: \"kubernetes.io/projected/8a6a85ad-c944-450f-b0bb-e3b64b0f56f0-kube-api-access-hvprb\") pod \"8a6a85ad-c944-450f-b0bb-e3b64b0f56f0\" (UID: \"8a6a85ad-c944-450f-b0bb-e3b64b0f56f0\") " Oct 30 17:41:17.115773 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:17.115701 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3e5e732-2c67-4377-9755-e2178bf33772-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3e5e732-2c67-4377-9755-e2178bf33772-model-cache\") pod \"a3e5e732-2c67-4377-9755-e2178bf33772\" (UID: \"a3e5e732-2c67-4377-9755-e2178bf33772\") " Oct 30 17:41:17.115773 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:17.115764 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a6a85ad-c944-450f-b0bb-e3b64b0f56f0-tls-certs\" (UniqueName: \"kubernetes.io/secret/8a6a85ad-c944-450f-b0bb-e3b64b0f56f0-tls-certs\") pod \"8a6a85ad-c944-450f-b0bb-e3b64b0f56f0\" (UID: \"8a6a85ad-c944-450f-b0bb-e3b64b0f56f0\") " Oct 30 17:41:17.116018 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:17.115793 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a6a85ad-c944-450f-b0bb-e3b64b0f56f0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a6a85ad-c944-450f-b0bb-e3b64b0f56f0-model-cache\") pod \"8a6a85ad-c944-450f-b0bb-e3b64b0f56f0\" (UID: \"8a6a85ad-c944-450f-b0bb-e3b64b0f56f0\") " Oct 30 17:41:17.116018 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:17.115845 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3e5e732-2c67-4377-9755-e2178bf33772-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3e5e732-2c67-4377-9755-e2178bf33772-dshm\") pod \"a3e5e732-2c67-4377-9755-e2178bf33772\" (UID: \"a3e5e732-2c67-4377-9755-e2178bf33772\") " Oct 30 17:41:17.116018 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:17.115878 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3e5e732-2c67-4377-9755-e2178bf33772-home\" (UniqueName: \"kubernetes.io/empty-dir/a3e5e732-2c67-4377-9755-e2178bf33772-home\") pod \"a3e5e732-2c67-4377-9755-e2178bf33772\" (UID: \"a3e5e732-2c67-4377-9755-e2178bf33772\") " Oct 30 17:41:17.116192 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:17.116036 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3e5e732-2c67-4377-9755-e2178bf33772-model-cache" pod "a3e5e732-2c67-4377-9755-e2178bf33772" (UID: "a3e5e732-2c67-4377-9755-e2178bf33772"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:41:17.116192 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:17.116098 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3e5e732-2c67-4377-9755-e2178bf33772-home" pod "a3e5e732-2c67-4377-9755-e2178bf33772" (UID: "a3e5e732-2c67-4377-9755-e2178bf33772"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:41:17.116192 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:17.116141 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a6a85ad-c944-450f-b0bb-e3b64b0f56f0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a6a85ad-c944-450f-b0bb-e3b64b0f56f0-dshm\") pod \"8a6a85ad-c944-450f-b0bb-e3b64b0f56f0\" (UID: \"8a6a85ad-c944-450f-b0bb-e3b64b0f56f0\") " Oct 30 17:41:17.116192 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:17.116188 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a6a85ad-c944-450f-b0bb-e3b64b0f56f0-home\" (UniqueName: \"kubernetes.io/empty-dir/8a6a85ad-c944-450f-b0bb-e3b64b0f56f0-home\") pod \"8a6a85ad-c944-450f-b0bb-e3b64b0f56f0\" (UID: \"8a6a85ad-c944-450f-b0bb-e3b64b0f56f0\") " Oct 30 17:41:17.116464 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:17.116253 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3e5e732-2c67-4377-9755-e2178bf33772-tls-certs\" (UniqueName: \"kubernetes.io/secret/a3e5e732-2c67-4377-9755-e2178bf33772-tls-certs\") pod \"a3e5e732-2c67-4377-9755-e2178bf33772\" (UID: \"a3e5e732-2c67-4377-9755-e2178bf33772\") " Oct 30 17:41:17.116464 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:17.116287 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a6a85ad-c944-450f-b0bb-e3b64b0f56f0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a6a85ad-c944-450f-b0bb-e3b64b0f56f0-tmp-dir\") pod \"8a6a85ad-c944-450f-b0bb-e3b64b0f56f0\" (UID: \"8a6a85ad-c944-450f-b0bb-e3b64b0f56f0\") " Oct 30 17:41:17.116464 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:17.116353 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a6a85ad-c944-450f-b0bb-e3b64b0f56f0-model-cache" pod "8a6a85ad-c944-450f-b0bb-e3b64b0f56f0" (UID: "8a6a85ad-c944-450f-b0bb-e3b64b0f56f0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:41:17.116654 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:17.116589 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a6a85ad-c944-450f-b0bb-e3b64b0f56f0-home" pod "8a6a85ad-c944-450f-b0bb-e3b64b0f56f0" (UID: "8a6a85ad-c944-450f-b0bb-e3b64b0f56f0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:41:17.116708 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:17.116696 4522 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3e5e732-2c67-4377-9755-e2178bf33772-home\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:41:17.116766 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:17.116716 4522 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a6a85ad-c944-450f-b0bb-e3b64b0f56f0-home\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:41:17.116822 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:17.116731 4522 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3e5e732-2c67-4377-9755-e2178bf33772-model-cache\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:41:17.116822 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:17.116795 4522 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a6a85ad-c944-450f-b0bb-e3b64b0f56f0-model-cache\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:41:17.116822 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:17.116794 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a6a85ad-c944-450f-b0bb-e3b64b0f56f0-tmp-dir" pod "8a6a85ad-c944-450f-b0bb-e3b64b0f56f0" (UID: "8a6a85ad-c944-450f-b0bb-e3b64b0f56f0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:41:17.117106 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:17.117073 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3e5e732-2c67-4377-9755-e2178bf33772-tmp-dir" pod "a3e5e732-2c67-4377-9755-e2178bf33772" (UID: "a3e5e732-2c67-4377-9755-e2178bf33772"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:41:17.118664 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:17.118624 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3e5e732-2c67-4377-9755-e2178bf33772-kube-api-access-qd5pj" pod "a3e5e732-2c67-4377-9755-e2178bf33772" (UID: "a3e5e732-2c67-4377-9755-e2178bf33772"). InnerVolumeSpecName "kube-api-access-qd5pj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:41:17.118935 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:17.118905 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a6a85ad-c944-450f-b0bb-e3b64b0f56f0-tls-certs" pod "8a6a85ad-c944-450f-b0bb-e3b64b0f56f0" (UID: "8a6a85ad-c944-450f-b0bb-e3b64b0f56f0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:41:17.118935 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:17.118926 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a6a85ad-c944-450f-b0bb-e3b64b0f56f0-kube-api-access-hvprb" pod "8a6a85ad-c944-450f-b0bb-e3b64b0f56f0" (UID: "8a6a85ad-c944-450f-b0bb-e3b64b0f56f0"). InnerVolumeSpecName "kube-api-access-hvprb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:41:17.119202 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:17.119173 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3e5e732-2c67-4377-9755-e2178bf33772-dshm" pod "a3e5e732-2c67-4377-9755-e2178bf33772" (UID: "a3e5e732-2c67-4377-9755-e2178bf33772"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:41:17.119297 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:17.119209 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a6a85ad-c944-450f-b0bb-e3b64b0f56f0-dshm" pod "8a6a85ad-c944-450f-b0bb-e3b64b0f56f0" (UID: "8a6a85ad-c944-450f-b0bb-e3b64b0f56f0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:41:17.119625 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:17.119599 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3e5e732-2c67-4377-9755-e2178bf33772-tls-certs" pod "a3e5e732-2c67-4377-9755-e2178bf33772" (UID: "a3e5e732-2c67-4377-9755-e2178bf33772"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:41:17.217695 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:17.217659 4522 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a6a85ad-c944-450f-b0bb-e3b64b0f56f0-tls-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:41:17.217695 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:17.217687 4522 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3e5e732-2c67-4377-9755-e2178bf33772-dshm\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:41:17.217695 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:17.217695 4522 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a6a85ad-c944-450f-b0bb-e3b64b0f56f0-dshm\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:41:17.217695 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:17.217702 4522 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3e5e732-2c67-4377-9755-e2178bf33772-tls-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:41:17.217980 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:17.217710 4522 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a6a85ad-c944-450f-b0bb-e3b64b0f56f0-tmp-dir\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:41:17.217980 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:17.217719 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qd5pj\" (UniqueName: \"kubernetes.io/projected/a3e5e732-2c67-4377-9755-e2178bf33772-kube-api-access-qd5pj\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:41:17.217980 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:17.217727 4522 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3e5e732-2c67-4377-9755-e2178bf33772-tmp-dir\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:41:17.217980 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:17.217737 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvprb\" (UniqueName: \"kubernetes.io/projected/8a6a85ad-c944-450f-b0bb-e3b64b0f56f0-kube-api-access-hvprb\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:41:17.235615 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:17.235582 4522 generic.go:356] "Generic (PLEG): container finished" podID="8a6a85ad-c944-450f-b0bb-e3b64b0f56f0" containerID="a2b7cbb389bdee3d842c9043f5abd81df99eac77a4b68b502e2fb699c8fbe56b" exitCode=0 Oct 30 17:41:17.235777 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:17.235627 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" Oct 30 17:41:17.235777 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:17.235671 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" event={"ID":"8a6a85ad-c944-450f-b0bb-e3b64b0f56f0","Type":"ContainerDied","Data":"a2b7cbb389bdee3d842c9043f5abd81df99eac77a4b68b502e2fb699c8fbe56b"} Oct 30 17:41:17.235777 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:17.235720 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f" event={"ID":"8a6a85ad-c944-450f-b0bb-e3b64b0f56f0","Type":"ContainerDied","Data":"5cf12ca50dcd0c769d232c8a09e7a0bc54790beed990a25aff34a2647edcf351"} Oct 30 17:41:17.235777 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:17.235744 4522 scope.go:122] "RemoveContainer" containerID="a2b7cbb389bdee3d842c9043f5abd81df99eac77a4b68b502e2fb699c8fbe56b" Oct 30 17:41:17.237200 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:17.237175 4522 generic.go:356] "Generic (PLEG): container finished" podID="a3e5e732-2c67-4377-9755-e2178bf33772" containerID="4169820d4cf8e16e4fb503ca82588005c1c4d08d1195d62025bf66b21c1a6763" exitCode=0 Oct 30 17:41:17.237326 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:17.237200 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" event={"ID":"a3e5e732-2c67-4377-9755-e2178bf33772","Type":"ContainerDied","Data":"4169820d4cf8e16e4fb503ca82588005c1c4d08d1195d62025bf66b21c1a6763"} Oct 30 17:41:17.237326 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:17.237245 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" event={"ID":"a3e5e732-2c67-4377-9755-e2178bf33772","Type":"ContainerDied","Data":"e174670ab51a83264656d1bfc7a95367c71748298b5e99d9073943b7075cf7cc"} Oct 30 17:41:17.237326 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:17.237213 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv" Oct 30 17:41:17.246842 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:17.246825 4522 scope.go:122] "RemoveContainer" containerID="a2b7cbb389bdee3d842c9043f5abd81df99eac77a4b68b502e2fb699c8fbe56b" Oct 30 17:41:17.247160 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:41:17.247140 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2b7cbb389bdee3d842c9043f5abd81df99eac77a4b68b502e2fb699c8fbe56b\": container with ID starting with a2b7cbb389bdee3d842c9043f5abd81df99eac77a4b68b502e2fb699c8fbe56b not found: ID does not exist" containerID="a2b7cbb389bdee3d842c9043f5abd81df99eac77a4b68b502e2fb699c8fbe56b" Oct 30 17:41:17.247272 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:17.247167 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2b7cbb389bdee3d842c9043f5abd81df99eac77a4b68b502e2fb699c8fbe56b"} err="failed to get container status \"a2b7cbb389bdee3d842c9043f5abd81df99eac77a4b68b502e2fb699c8fbe56b\": rpc error: code = NotFound desc = could not find container \"a2b7cbb389bdee3d842c9043f5abd81df99eac77a4b68b502e2fb699c8fbe56b\": container with ID starting with a2b7cbb389bdee3d842c9043f5abd81df99eac77a4b68b502e2fb699c8fbe56b not found: ID does not exist" Oct 30 17:41:17.247272 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:17.247187 4522 scope.go:122] "RemoveContainer" containerID="4169820d4cf8e16e4fb503ca82588005c1c4d08d1195d62025bf66b21c1a6763" Oct 30 17:41:17.255723 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:17.255705 4522 scope.go:122] "RemoveContainer" containerID="4169820d4cf8e16e4fb503ca82588005c1c4d08d1195d62025bf66b21c1a6763" Oct 30 17:41:17.255962 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:41:17.255945 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4169820d4cf8e16e4fb503ca82588005c1c4d08d1195d62025bf66b21c1a6763\": container with ID starting with 4169820d4cf8e16e4fb503ca82588005c1c4d08d1195d62025bf66b21c1a6763 not found: ID does not exist" containerID="4169820d4cf8e16e4fb503ca82588005c1c4d08d1195d62025bf66b21c1a6763" Oct 30 17:41:17.256010 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:17.255969 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4169820d4cf8e16e4fb503ca82588005c1c4d08d1195d62025bf66b21c1a6763"} err="failed to get container status \"4169820d4cf8e16e4fb503ca82588005c1c4d08d1195d62025bf66b21c1a6763\": rpc error: code = NotFound desc = could not find container \"4169820d4cf8e16e4fb503ca82588005c1c4d08d1195d62025bf66b21c1a6763\": container with ID starting with 4169820d4cf8e16e4fb503ca82588005c1c4d08d1195d62025bf66b21c1a6763 not found: ID does not exist" Oct 30 17:41:17.295728 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:17.295695 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f"] Oct 30 17:41:17.297823 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:17.297803 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-d384dc/premium-w6-d384dc-kserve-7fcb98cb96-8956f"] Oct 30 17:41:17.336262 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:17.336232 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv"] Oct 30 17:41:17.338214 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:17.338189 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-d384dc/simulator-w6-d384dc-kserve-546b95bbb9-vc9bv"] Oct 30 17:41:18.143595 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:18.143517 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a6a85ad-c944-450f-b0bb-e3b64b0f56f0" path="/var/lib/kubelet/pods/8a6a85ad-c944-450f-b0bb-e3b64b0f56f0/volumes" Oct 30 17:41:18.143961 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:18.143926 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3e5e732-2c67-4377-9755-e2178bf33772" path="/var/lib/kubelet/pods/a3e5e732-2c67-4377-9755-e2178bf33772/volumes" Oct 30 17:41:24.128226 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:24.128192 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-d384dc-7f78968dd6-2w6cp" podUID="acfdfc44-26dc-4fb0-a04a-1e15d28939ee" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:41:25.483464 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:25.483429 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d384dc-hwmmf"] Oct 30 17:41:25.498191 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:25.498163 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d384dc-hwmmf"] Oct 30 17:41:26.143592 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:26.143559 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fd43dd1-318c-4a75-9516-afc296f8dbca" path="/var/lib/kubelet/pods/8fd43dd1-318c-4a75-9516-afc296f8dbca/volumes" Oct 30 17:41:32.264283 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:32.264217 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-d384dc-7f78968dd6-2w6cp" Oct 30 17:41:32.320698 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:32.320658 4522 generic.go:356] "Generic (PLEG): container finished" podID="acfdfc44-26dc-4fb0-a04a-1e15d28939ee" containerID="58feeccd2dd1800d66537dfb9c04583ebddc4ba7ad9e347347fea19d27946e55" exitCode=0 Oct 30 17:41:32.320890 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:32.320714 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-d384dc-7f78968dd6-2w6cp" Oct 30 17:41:32.320890 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:32.320727 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-d384dc-7f78968dd6-2w6cp" event={"ID":"acfdfc44-26dc-4fb0-a04a-1e15d28939ee","Type":"ContainerDied","Data":"58feeccd2dd1800d66537dfb9c04583ebddc4ba7ad9e347347fea19d27946e55"} Oct 30 17:41:32.320890 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:32.320780 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-d384dc-7f78968dd6-2w6cp" event={"ID":"acfdfc44-26dc-4fb0-a04a-1e15d28939ee","Type":"ContainerDied","Data":"edce3f648687fc5ef79e948f4fe6a3d9a2cd7d25d0f8ae073c432dda5f80b280"} Oct 30 17:41:32.320890 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:32.320802 4522 scope.go:122] "RemoveContainer" containerID="58feeccd2dd1800d66537dfb9c04583ebddc4ba7ad9e347347fea19d27946e55" Oct 30 17:41:32.329837 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:32.329810 4522 scope.go:122] "RemoveContainer" containerID="58feeccd2dd1800d66537dfb9c04583ebddc4ba7ad9e347347fea19d27946e55" Oct 30 17:41:32.330136 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:41:32.330111 4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58feeccd2dd1800d66537dfb9c04583ebddc4ba7ad9e347347fea19d27946e55\": container with ID starting with 58feeccd2dd1800d66537dfb9c04583ebddc4ba7ad9e347347fea19d27946e55 not found: ID does not exist" containerID="58feeccd2dd1800d66537dfb9c04583ebddc4ba7ad9e347347fea19d27946e55" Oct 30 17:41:32.330238 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:32.330150 4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58feeccd2dd1800d66537dfb9c04583ebddc4ba7ad9e347347fea19d27946e55"} err="failed to get container status \"58feeccd2dd1800d66537dfb9c04583ebddc4ba7ad9e347347fea19d27946e55\": rpc error: code = NotFound desc = could not find container \"58feeccd2dd1800d66537dfb9c04583ebddc4ba7ad9e347347fea19d27946e55\": container with ID starting with 58feeccd2dd1800d66537dfb9c04583ebddc4ba7ad9e347347fea19d27946e55 not found: ID does not exist" Oct 30 17:41:32.354464 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:32.354440 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acfdfc44-26dc-4fb0-a04a-1e15d28939ee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acfdfc44-26dc-4fb0-a04a-1e15d28939ee-plugins-config-volume\") pod \"acfdfc44-26dc-4fb0-a04a-1e15d28939ee\" (UID: \"acfdfc44-26dc-4fb0-a04a-1e15d28939ee\") " Oct 30 17:41:32.354604 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:32.354587 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acfdfc44-26dc-4fb0-a04a-1e15d28939ee-kube-api-access-j6wr9\" (UniqueName: \"kubernetes.io/projected/acfdfc44-26dc-4fb0-a04a-1e15d28939ee-kube-api-access-j6wr9\") pod \"acfdfc44-26dc-4fb0-a04a-1e15d28939ee\" (UID: \"acfdfc44-26dc-4fb0-a04a-1e15d28939ee\") " Oct 30 17:41:32.354866 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:32.354836 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acfdfc44-26dc-4fb0-a04a-1e15d28939ee-plugins-config-volume" pod "acfdfc44-26dc-4fb0-a04a-1e15d28939ee" (UID: "acfdfc44-26dc-4fb0-a04a-1e15d28939ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:41:32.355013 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:32.354908 4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acfdfc44-26dc-4fb0-a04a-1e15d28939ee-plugins-config-volume\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:41:32.356580 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:32.356553 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acfdfc44-26dc-4fb0-a04a-1e15d28939ee-kube-api-access-j6wr9" pod "acfdfc44-26dc-4fb0-a04a-1e15d28939ee" (UID: "acfdfc44-26dc-4fb0-a04a-1e15d28939ee"). InnerVolumeSpecName "kube-api-access-j6wr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:41:32.456065 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:32.456026 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6wr9\" (UniqueName: \"kubernetes.io/projected/acfdfc44-26dc-4fb0-a04a-1e15d28939ee-kube-api-access-j6wr9\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:41:32.670541 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:32.670510 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-d384dc-7f78968dd6-2w6cp"] Oct 30 17:41:32.684981 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:32.684947 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-d384dc-7f78968dd6-2w6cp"] Oct 30 17:41:33.033389 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:33.033353 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cb7367c-m8xfm"] Oct 30 17:41:33.036122 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:33.036099 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cb7367c-m8xfm"] Oct 30 17:41:34.143326 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:34.143282 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="444cd488-a651-413f-afc9-907bf73a1879" path="/var/lib/kubelet/pods/444cd488-a651-413f-afc9-907bf73a1879/volumes" Oct 30 17:41:34.143717 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:34.143679 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acfdfc44-26dc-4fb0-a04a-1e15d28939ee" path="/var/lib/kubelet/pods/acfdfc44-26dc-4fb0-a04a-1e15d28939ee/volumes" Oct 30 17:41:42.946975 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:41:42.946928 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:41:54.590387 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:54.590291 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ljkmh"] Oct 30 17:41:54.590772 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:54.590662 4522 cpu_manager.go:414] "containerMap: removing container" podUID="4135bd88-8ef0-44c0-8679-9f6719ef3a18" containerName="payload-processing" Oct 30 17:41:54.590772 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:54.590673 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4135bd88-8ef0-44c0-8679-9f6719ef3a18" containerName="payload-processing" Oct 30 17:41:54.590772 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:54.590690 4522 cpu_manager.go:414] "containerMap: removing container" podUID="8a6a85ad-c944-450f-b0bb-e3b64b0f56f0" containerName="main" Oct 30 17:41:54.590772 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:54.590696 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a6a85ad-c944-450f-b0bb-e3b64b0f56f0" containerName="main" Oct 30 17:41:54.590772 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:54.590706 4522 cpu_manager.go:414] "containerMap: removing container" podUID="66176bec-e6cb-48ca-8a4d-dde77870ecdc" containerName="revoke-keys" Oct 30 17:41:54.590772 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:54.590712 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66176bec-e6cb-48ca-8a4d-dde77870ecdc" containerName="revoke-keys" Oct 30 17:41:54.590772 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:54.590720 4522 cpu_manager.go:414] "containerMap: removing container" podUID="acfdfc44-26dc-4fb0-a04a-1e15d28939ee" containerName="payload-processing" Oct 30 17:41:54.590772 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:54.590724 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acfdfc44-26dc-4fb0-a04a-1e15d28939ee" containerName="payload-processing" Oct 30 17:41:54.590772 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:54.590733 4522 cpu_manager.go:414] "containerMap: removing container" podUID="a3e5e732-2c67-4377-9755-e2178bf33772" containerName="main" Oct 30 17:41:54.590772 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:54.590738 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3e5e732-2c67-4377-9755-e2178bf33772" containerName="main" Oct 30 17:41:54.590772 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:54.590744 4522 cpu_manager.go:414] "containerMap: removing container" podUID="eef74243-b455-4c59-baf2-d4ae87e3e5c4" containerName="payload-pre-processing" Oct 30 17:41:54.590772 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:54.590749 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eef74243-b455-4c59-baf2-d4ae87e3e5c4" containerName="payload-pre-processing" Oct 30 17:41:54.591178 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:54.590819 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a6a85ad-c944-450f-b0bb-e3b64b0f56f0" containerName="main" Oct 30 17:41:54.591178 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:54.590828 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="66176bec-e6cb-48ca-8a4d-dde77870ecdc" containerName="revoke-keys" Oct 30 17:41:54.591178 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:54.590835 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="acfdfc44-26dc-4fb0-a04a-1e15d28939ee" containerName="payload-processing" Oct 30 17:41:54.591178 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:54.590842 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3e5e732-2c67-4377-9755-e2178bf33772" containerName="main" Oct 30 17:41:54.591178 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:54.590848 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="eef74243-b455-4c59-baf2-d4ae87e3e5c4" containerName="payload-pre-processing" Oct 30 17:41:54.591178 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:54.590857 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="4135bd88-8ef0-44c0-8679-9f6719ef3a18" containerName="payload-processing" Oct 30 17:41:54.591178 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:54.590863 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d64632e-0973-4589-9c43-b7b3b7bb95b3" containerName="istio-proxy" Oct 30 17:41:54.593926 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:54.593909 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ljkmh" Oct 30 17:41:54.628855 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:54.628822 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ljkmh"] Oct 30 17:41:54.760546 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:54.760505 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41ac1e04-01e0-4a54-940a-5b2eb66f1e2c-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ljkmh\" (UID: \"41ac1e04-01e0-4a54-940a-5b2eb66f1e2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ljkmh" Oct 30 17:41:54.760546 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:54.760548 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41ac1e04-01e0-4a54-940a-5b2eb66f1e2c-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ljkmh\" (UID: \"41ac1e04-01e0-4a54-940a-5b2eb66f1e2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ljkmh" Oct 30 17:41:54.760794 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:54.760650 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41ac1e04-01e0-4a54-940a-5b2eb66f1e2c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ljkmh\" (UID: \"41ac1e04-01e0-4a54-940a-5b2eb66f1e2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ljkmh" Oct 30 17:41:54.760794 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:54.760684 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg27r\" (UniqueName: \"kubernetes.io/projected/41ac1e04-01e0-4a54-940a-5b2eb66f1e2c-kube-api-access-pg27r\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ljkmh\" (UID: \"41ac1e04-01e0-4a54-940a-5b2eb66f1e2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ljkmh" Oct 30 17:41:54.760794 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:54.760724 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41ac1e04-01e0-4a54-940a-5b2eb66f1e2c-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ljkmh\" (UID: \"41ac1e04-01e0-4a54-940a-5b2eb66f1e2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ljkmh" Oct 30 17:41:54.760976 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:54.760803 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41ac1e04-01e0-4a54-940a-5b2eb66f1e2c-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ljkmh\" (UID: \"41ac1e04-01e0-4a54-940a-5b2eb66f1e2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ljkmh" Oct 30 17:41:54.760976 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:54.760841 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41ac1e04-01e0-4a54-940a-5b2eb66f1e2c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ljkmh\" (UID: \"41ac1e04-01e0-4a54-940a-5b2eb66f1e2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ljkmh" Oct 30 17:41:54.760976 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:54.760872 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41ac1e04-01e0-4a54-940a-5b2eb66f1e2c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ljkmh\" (UID: \"41ac1e04-01e0-4a54-940a-5b2eb66f1e2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ljkmh" Oct 30 17:41:54.760976 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:54.760923 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41ac1e04-01e0-4a54-940a-5b2eb66f1e2c-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ljkmh\" (UID: \"41ac1e04-01e0-4a54-940a-5b2eb66f1e2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ljkmh" Oct 30 17:41:54.861566 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:54.861485 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41ac1e04-01e0-4a54-940a-5b2eb66f1e2c-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ljkmh\" (UID: \"41ac1e04-01e0-4a54-940a-5b2eb66f1e2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ljkmh" Oct 30 17:41:54.861566 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:54.861526 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41ac1e04-01e0-4a54-940a-5b2eb66f1e2c-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ljkmh\" (UID: \"41ac1e04-01e0-4a54-940a-5b2eb66f1e2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ljkmh" Oct 30 17:41:54.861566 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:54.861559 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41ac1e04-01e0-4a54-940a-5b2eb66f1e2c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ljkmh\" (UID: \"41ac1e04-01e0-4a54-940a-5b2eb66f1e2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ljkmh" Oct 30 17:41:54.861831 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:54.861588 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg27r\" (UniqueName: \"kubernetes.io/projected/41ac1e04-01e0-4a54-940a-5b2eb66f1e2c-kube-api-access-pg27r\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ljkmh\" (UID: \"41ac1e04-01e0-4a54-940a-5b2eb66f1e2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ljkmh" Oct 30 17:41:54.861831 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:54.861615 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41ac1e04-01e0-4a54-940a-5b2eb66f1e2c-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ljkmh\" (UID: \"41ac1e04-01e0-4a54-940a-5b2eb66f1e2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ljkmh" Oct 30 17:41:54.861831 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:54.861650 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41ac1e04-01e0-4a54-940a-5b2eb66f1e2c-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ljkmh\" (UID: \"41ac1e04-01e0-4a54-940a-5b2eb66f1e2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ljkmh" Oct 30 17:41:54.861831 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:54.861677 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41ac1e04-01e0-4a54-940a-5b2eb66f1e2c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ljkmh\" (UID: \"41ac1e04-01e0-4a54-940a-5b2eb66f1e2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ljkmh" Oct 30 17:41:54.861831 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:54.861706 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41ac1e04-01e0-4a54-940a-5b2eb66f1e2c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ljkmh\" (UID: \"41ac1e04-01e0-4a54-940a-5b2eb66f1e2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ljkmh" Oct 30 17:41:54.861831 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:54.861733 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41ac1e04-01e0-4a54-940a-5b2eb66f1e2c-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ljkmh\" (UID: \"41ac1e04-01e0-4a54-940a-5b2eb66f1e2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ljkmh" Oct 30 17:41:54.862178 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:54.862065 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41ac1e04-01e0-4a54-940a-5b2eb66f1e2c-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ljkmh\" (UID: \"41ac1e04-01e0-4a54-940a-5b2eb66f1e2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ljkmh" Oct 30 17:41:54.862178 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:54.862128 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41ac1e04-01e0-4a54-940a-5b2eb66f1e2c-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ljkmh\" (UID: \"41ac1e04-01e0-4a54-940a-5b2eb66f1e2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ljkmh" Oct 30 17:41:54.862178 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:54.862171 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41ac1e04-01e0-4a54-940a-5b2eb66f1e2c-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ljkmh\" (UID: \"41ac1e04-01e0-4a54-940a-5b2eb66f1e2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ljkmh" Oct 30 17:41:54.862400 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:54.862201 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41ac1e04-01e0-4a54-940a-5b2eb66f1e2c-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ljkmh\" (UID: \"41ac1e04-01e0-4a54-940a-5b2eb66f1e2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ljkmh" Oct 30 17:41:54.862542 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:54.862520 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41ac1e04-01e0-4a54-940a-5b2eb66f1e2c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ljkmh\" (UID: \"41ac1e04-01e0-4a54-940a-5b2eb66f1e2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ljkmh" Oct 30 17:41:54.863938 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:54.863918 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41ac1e04-01e0-4a54-940a-5b2eb66f1e2c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ljkmh\" (UID: \"41ac1e04-01e0-4a54-940a-5b2eb66f1e2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ljkmh" Oct 30 17:41:54.864394 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:54.864371 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41ac1e04-01e0-4a54-940a-5b2eb66f1e2c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ljkmh\" (UID: \"41ac1e04-01e0-4a54-940a-5b2eb66f1e2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ljkmh" Oct 30 17:41:54.887904 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:54.887869 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41ac1e04-01e0-4a54-940a-5b2eb66f1e2c-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ljkmh\" (UID: \"41ac1e04-01e0-4a54-940a-5b2eb66f1e2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ljkmh" Oct 30 17:41:54.888011 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:54.887934 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg27r\" (UniqueName: \"kubernetes.io/projected/41ac1e04-01e0-4a54-940a-5b2eb66f1e2c-kube-api-access-pg27r\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ljkmh\" (UID: \"41ac1e04-01e0-4a54-940a-5b2eb66f1e2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ljkmh" Oct 30 17:41:54.906790 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:54.906767 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ljkmh" Oct 30 17:41:55.229295 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:41:55.229265 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41ac1e04_01e0_4a54_940a_5b2eb66f1e2c.slice/crio-445607e6516bf11abfc9f9944f6568a0ce66cbd8af8204a77acb3f99afaa9bc7 WatchSource:0}: Error finding container 445607e6516bf11abfc9f9944f6568a0ce66cbd8af8204a77acb3f99afaa9bc7: Status 404 returned error can't find the container with id 445607e6516bf11abfc9f9944f6568a0ce66cbd8af8204a77acb3f99afaa9bc7 Oct 30 17:41:55.229443 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:55.229348 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ljkmh"] Oct 30 17:41:55.238774 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:55.231838 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:41:55.238774 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:55.231991 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:41:55.238774 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:55.232083 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:41:55.463149 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:55.463108 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ljkmh" event={"ID":"41ac1e04-01e0-4a54-940a-5b2eb66f1e2c","Type":"ContainerStarted","Data":"14a88bf1af1850d2a662a98c2794b19c815d80a3153d9ef0db0868fe70f2e56c"} Oct 30 17:41:55.463359 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:55.463206 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ljkmh" event={"ID":"41ac1e04-01e0-4a54-940a-5b2eb66f1e2c","Type":"ContainerStarted","Data":"445607e6516bf11abfc9f9944f6568a0ce66cbd8af8204a77acb3f99afaa9bc7"} Oct 30 17:41:55.515119 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:55.515010 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ljkmh" podStartSLOduration=1.514998382 podStartE2EDuration="1.514998382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:41:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:41:55.514270417 +0000 UTC m=+3289.515025339" watchObservedRunningTime="2026-10-30 17:41:55.514998382 +0000 UTC m=+3289.515753273" Oct 30 17:41:55.912391 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:55.910700 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ljkmh" Oct 30 17:41:56.036322 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:56.036281 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daacd80b-7mkw4"] Oct 30 17:41:56.039069 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:56.039051 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daacd80b-7mkw4"] Oct 30 17:41:56.143418 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:56.143384 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d80fae1e-5c7b-4cb4-9b50-eae3ffc175ff" path="/var/lib/kubelet/pods/d80fae1e-5c7b-4cb4-9b50-eae3ffc175ff/volumes" Oct 30 17:41:56.913750 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:56.913378 4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-ljkmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.124:15021/healthz/ready\": dial tcp 10.134.1.124:15021: connect: connection refused" start-of-body= Oct 30 17:41:56.913750 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:56.913448 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ljkmh" podUID="41ac1e04-01e0-4a54-940a-5b2eb66f1e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.124:15021/healthz/ready\": dial tcp 10.134.1.124:15021: connect: connection refused" Oct 30 17:41:57.910331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:57.909734 4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-ljkmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.124:15021/healthz/ready\": dial tcp 10.134.1.124:15021: connect: connection refused" start-of-body= Oct 30 17:41:57.910331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:57.909800 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ljkmh" podUID="41ac1e04-01e0-4a54-940a-5b2eb66f1e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.124:15021/healthz/ready\": dial tcp 10.134.1.124:15021: connect: connection refused" Oct 30 17:41:58.910330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:58.909716 4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-ljkmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.124:15021/healthz/ready\": dial tcp 10.134.1.124:15021: connect: connection refused" start-of-body= Oct 30 17:41:58.910330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:58.909801 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ljkmh" podUID="41ac1e04-01e0-4a54-940a-5b2eb66f1e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.124:15021/healthz/ready\": dial tcp 10.134.1.124:15021: connect: connection refused" Oct 30 17:41:59.037143 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:59.037106 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a13a40c3-wjvk5"] Oct 30 17:41:59.051371 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:59.051338 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a13a40c3-wjvk5"] Oct 30 17:41:59.914756 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:59.914714 4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-ljkmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.124:15021/healthz/ready\": dial tcp 10.134.1.124:15021: connect: connection refused" start-of-body= Oct 30 17:41:59.915370 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:41:59.915338 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ljkmh" podUID="41ac1e04-01e0-4a54-940a-5b2eb66f1e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.124:15021/healthz/ready\": dial tcp 10.134.1.124:15021: connect: connection refused" Oct 30 17:42:00.144293 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:00.144212 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78ab131d-5658-4d13-be06-4943fcdd609b" path="/var/lib/kubelet/pods/78ab131d-5658-4d13-be06-4943fcdd609b/volumes" Oct 30 17:42:00.872426 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:00.872392 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4qmbk"] Oct 30 17:42:00.877375 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:00.877351 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4qmbk" Oct 30 17:42:00.920333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:00.903178 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4qmbk"] Oct 30 17:42:00.921447 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:00.908686 4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-ljkmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.124:15021/healthz/ready\": dial tcp 10.134.1.124:15021: connect: connection refused" start-of-body= Oct 30 17:42:00.921447 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:00.921002 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ljkmh" podUID="41ac1e04-01e0-4a54-940a-5b2eb66f1e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.124:15021/healthz/ready\": dial tcp 10.134.1.124:15021: connect: connection refused" Oct 30 17:42:00.921447 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:00.917684 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:42:00.921447 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:00.917633 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:42:00.921447 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:00.917816 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:42:00.921447 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:00.917896 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:42:01.044803 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:01.044766 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8p2r\" (UniqueName: \"kubernetes.io/projected/141d6736-d8dc-4fd0-a387-75241011c8fe-kube-api-access-l8p2r\") pod \"maas-api-revoke-sub-85237e45-4qmbk\" (UID: \"141d6736-d8dc-4fd0-a387-75241011c8fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4qmbk" Oct 30 17:42:01.045027 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:01.044899 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/141d6736-d8dc-4fd0-a387-75241011c8fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-4qmbk\" (UID: \"141d6736-d8dc-4fd0-a387-75241011c8fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4qmbk" Oct 30 17:42:01.045027 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:01.044984 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/141d6736-d8dc-4fd0-a387-75241011c8fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-4qmbk\" (UID: \"141d6736-d8dc-4fd0-a387-75241011c8fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4qmbk" Oct 30 17:42:01.146664 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:01.146564 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/141d6736-d8dc-4fd0-a387-75241011c8fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-4qmbk\" (UID: \"141d6736-d8dc-4fd0-a387-75241011c8fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4qmbk" Oct 30 17:42:01.146849 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:01.146683 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/141d6736-d8dc-4fd0-a387-75241011c8fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-4qmbk\" (UID: \"141d6736-d8dc-4fd0-a387-75241011c8fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4qmbk" Oct 30 17:42:01.146849 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:01.146761 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8p2r\" (UniqueName: \"kubernetes.io/projected/141d6736-d8dc-4fd0-a387-75241011c8fe-kube-api-access-l8p2r\") pod \"maas-api-revoke-sub-85237e45-4qmbk\" (UID: \"141d6736-d8dc-4fd0-a387-75241011c8fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4qmbk" Oct 30 17:42:01.147414 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:01.147385 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/141d6736-d8dc-4fd0-a387-75241011c8fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-4qmbk\" (UID: \"141d6736-d8dc-4fd0-a387-75241011c8fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4qmbk" Oct 30 17:42:01.148997 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:01.148978 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/141d6736-d8dc-4fd0-a387-75241011c8fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-4qmbk\" (UID: \"141d6736-d8dc-4fd0-a387-75241011c8fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4qmbk" Oct 30 17:42:01.172828 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:01.172794 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8p2r\" (UniqueName: \"kubernetes.io/projected/141d6736-d8dc-4fd0-a387-75241011c8fe-kube-api-access-l8p2r\") pod \"maas-api-revoke-sub-85237e45-4qmbk\" (UID: \"141d6736-d8dc-4fd0-a387-75241011c8fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4qmbk" Oct 30 17:42:01.190611 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:01.190586 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4qmbk" Oct 30 17:42:01.495389 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:01.495341 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4qmbk"] Oct 30 17:42:01.495658 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:42:01.495617 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod141d6736_d8dc_4fd0_a387_75241011c8fe.slice/crio-1b51ae97c941a51733420213ad859c5620ea1b88107deb59425452a7f71a459f WatchSource:0}: Error finding container 1b51ae97c941a51733420213ad859c5620ea1b88107deb59425452a7f71a459f: Status 404 returned error can't find the container with id 1b51ae97c941a51733420213ad859c5620ea1b88107deb59425452a7f71a459f Oct 30 17:42:01.907640 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:01.907601 4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-ljkmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.124:15021/healthz/ready\": dial tcp 10.134.1.124:15021: connect: connection refused" start-of-body= Oct 30 17:42:01.907834 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:01.907656 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ljkmh" podUID="41ac1e04-01e0-4a54-940a-5b2eb66f1e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.124:15021/healthz/ready\": dial tcp 10.134.1.124:15021: connect: connection refused" Oct 30 17:42:02.515290 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:02.515259 4522 generic.go:356] "Generic (PLEG): container finished" podID="141d6736-d8dc-4fd0-a387-75241011c8fe" containerID="a12deec508270e5c1e543f415d7bd65b3891179e12047fd40b7806b8ba147814" exitCode=0 Oct 30 17:42:02.515713 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:02.515296 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4qmbk" event={"ID":"141d6736-d8dc-4fd0-a387-75241011c8fe","Type":"ContainerDied","Data":"a12deec508270e5c1e543f415d7bd65b3891179e12047fd40b7806b8ba147814"} Oct 30 17:42:02.515713 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:02.515356 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4qmbk" event={"ID":"141d6736-d8dc-4fd0-a387-75241011c8fe","Type":"ContainerStarted","Data":"1b51ae97c941a51733420213ad859c5620ea1b88107deb59425452a7f71a459f"} Oct 30 17:42:03.665122 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:03.665092 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4qmbk" Oct 30 17:42:03.671229 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:03.671199 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/141d6736-d8dc-4fd0-a387-75241011c8fe-kube-api-access-l8p2r\" (UniqueName: \"kubernetes.io/projected/141d6736-d8dc-4fd0-a387-75241011c8fe-kube-api-access-l8p2r\") pod \"141d6736-d8dc-4fd0-a387-75241011c8fe\" (UID: \"141d6736-d8dc-4fd0-a387-75241011c8fe\") " Oct 30 17:42:03.671415 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:03.671259 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/141d6736-d8dc-4fd0-a387-75241011c8fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/141d6736-d8dc-4fd0-a387-75241011c8fe-maas-api-xks-ca\") pod \"141d6736-d8dc-4fd0-a387-75241011c8fe\" (UID: \"141d6736-d8dc-4fd0-a387-75241011c8fe\") " Oct 30 17:42:03.671415 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:03.671339 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/141d6736-d8dc-4fd0-a387-75241011c8fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/141d6736-d8dc-4fd0-a387-75241011c8fe-maas-api-openshift-ca\") pod \"141d6736-d8dc-4fd0-a387-75241011c8fe\" (UID: \"141d6736-d8dc-4fd0-a387-75241011c8fe\") " Oct 30 17:42:03.671701 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:03.671679 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/141d6736-d8dc-4fd0-a387-75241011c8fe-maas-api-openshift-ca" pod "141d6736-d8dc-4fd0-a387-75241011c8fe" (UID: "141d6736-d8dc-4fd0-a387-75241011c8fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:42:03.673424 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:03.673399 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/141d6736-d8dc-4fd0-a387-75241011c8fe-maas-api-xks-ca" pod "141d6736-d8dc-4fd0-a387-75241011c8fe" (UID: "141d6736-d8dc-4fd0-a387-75241011c8fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:42:03.673524 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:03.673504 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/141d6736-d8dc-4fd0-a387-75241011c8fe-kube-api-access-l8p2r" pod "141d6736-d8dc-4fd0-a387-75241011c8fe" (UID: "141d6736-d8dc-4fd0-a387-75241011c8fe"). InnerVolumeSpecName "kube-api-access-l8p2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:42:03.772113 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:03.772071 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8p2r\" (UniqueName: \"kubernetes.io/projected/141d6736-d8dc-4fd0-a387-75241011c8fe-kube-api-access-l8p2r\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:42:03.772113 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:03.772107 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/141d6736-d8dc-4fd0-a387-75241011c8fe-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:42:03.772113 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:03.772120 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/141d6736-d8dc-4fd0-a387-75241011c8fe-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:42:03.907774 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:03.907669 4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-ljkmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:42:03.907774 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:03.907733 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ljkmh" podUID="41ac1e04-01e0-4a54-940a-5b2eb66f1e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:42:04.532543 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:04.532499 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4qmbk" event={"ID":"141d6736-d8dc-4fd0-a387-75241011c8fe","Type":"ContainerDied","Data":"1b51ae97c941a51733420213ad859c5620ea1b88107deb59425452a7f71a459f"} Oct 30 17:42:04.532543 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:04.532531 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4qmbk" Oct 30 17:42:04.532783 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:04.532564 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b51ae97c941a51733420213ad859c5620ea1b88107deb59425452a7f71a459f" Oct 30 17:42:04.949619 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:04.949465 4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-ljkmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:42:04.949619 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:04.949515 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ljkmh" podUID="41ac1e04-01e0-4a54-940a-5b2eb66f1e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:42:05.624717 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:05.624686 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ljkmh" Oct 30 17:42:05.624949 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:05.624927 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ljkmh" Oct 30 17:42:05.625626 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:05.625605 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ljkmh" Oct 30 17:42:11.349373 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:11.349343 4522 scope.go:122] "RemoveContainer" containerID="5dea86fde6ced17f0ff315d1655a40987d2209a106ce22d20ce4dc244e84ff3f" Oct 30 17:42:11.358475 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:11.358448 4522 scope.go:122] "RemoveContainer" containerID="44e48c57f6dd1bf710e820062e3994d0cfc0257f972d23653b44ab3fc6071f27" Oct 30 17:42:11.367101 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:11.367080 4522 scope.go:122] "RemoveContainer" containerID="84c9e63948375892cfd014cd46bef80e01f2ea2054bebebb746eeae3a738b597" Oct 30 17:42:11.375314 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:11.375285 4522 scope.go:122] "RemoveContainer" containerID="3d5820a1611860355bf8cd7f388dbe3ad6e6045cd47d56d5041b78c74cf9c9d3" Oct 30 17:42:11.383290 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:11.383271 4522 scope.go:122] "RemoveContainer" containerID="83c02e96ba2d051cfe1c4ecc0e0c23803afca77d0ce59b2f5a7b9902ba4870b7" Oct 30 17:42:12.950047 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:42:12.950006 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:42:32.036539 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:32.036499 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da10aa3f-zkrjd"] Oct 30 17:42:32.051130 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:32.051099 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da10aa3f-zkrjd"] Oct 30 17:42:32.143109 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:32.143073 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4831a6b7-f606-47ee-82bc-d31a0b0288df" path="/var/lib/kubelet/pods/4831a6b7-f606-47ee-82bc-d31a0b0288df/volumes" Oct 30 17:42:42.946837 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:42:42.946798 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:42:52.102408 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:52.102369 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mw7n5"] Oct 30 17:42:52.102821 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:52.102750 4522 cpu_manager.go:414] "containerMap: removing container" podUID="141d6736-d8dc-4fd0-a387-75241011c8fe" containerName="revoke-keys" Oct 30 17:42:52.102821 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:52.102762 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="141d6736-d8dc-4fd0-a387-75241011c8fe" containerName="revoke-keys" Oct 30 17:42:52.102821 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:52.102815 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="141d6736-d8dc-4fd0-a387-75241011c8fe" containerName="revoke-keys" Oct 30 17:42:52.105962 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:52.105946 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mw7n5" Oct 30 17:42:52.126541 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:52.126507 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mw7n5"] Oct 30 17:42:52.138756 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:52.138726 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:42:52.138948 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:52.138759 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:42:52.138948 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:52.138824 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:42:52.138948 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:52.138935 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:42:52.148931 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:52.148893 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8tmz\" (UniqueName: \"kubernetes.io/projected/33d4dc43-08f4-49a2-88da-0910939e5254-kube-api-access-r8tmz\") pod \"maas-api-revoke-sub-c1e271a9-mw7n5\" (UID: \"33d4dc43-08f4-49a2-88da-0910939e5254\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mw7n5" Oct 30 17:42:52.149098 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:52.148972 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33d4dc43-08f4-49a2-88da-0910939e5254-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-mw7n5\" (UID: \"33d4dc43-08f4-49a2-88da-0910939e5254\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mw7n5" Oct 30 17:42:52.149098 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:52.149046 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33d4dc43-08f4-49a2-88da-0910939e5254-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-mw7n5\" (UID: \"33d4dc43-08f4-49a2-88da-0910939e5254\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mw7n5" Oct 30 17:42:52.250536 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:52.250499 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33d4dc43-08f4-49a2-88da-0910939e5254-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-mw7n5\" (UID: \"33d4dc43-08f4-49a2-88da-0910939e5254\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mw7n5" Oct 30 17:42:52.250721 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:52.250556 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8tmz\" (UniqueName: \"kubernetes.io/projected/33d4dc43-08f4-49a2-88da-0910939e5254-kube-api-access-r8tmz\") pod \"maas-api-revoke-sub-c1e271a9-mw7n5\" (UID: \"33d4dc43-08f4-49a2-88da-0910939e5254\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mw7n5" Oct 30 17:42:52.250721 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:52.250601 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33d4dc43-08f4-49a2-88da-0910939e5254-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-mw7n5\" (UID: \"33d4dc43-08f4-49a2-88da-0910939e5254\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mw7n5" Oct 30 17:42:52.251221 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:52.251202 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33d4dc43-08f4-49a2-88da-0910939e5254-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-mw7n5\" (UID: \"33d4dc43-08f4-49a2-88da-0910939e5254\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mw7n5" Oct 30 17:42:52.252919 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:52.252898 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33d4dc43-08f4-49a2-88da-0910939e5254-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-mw7n5\" (UID: \"33d4dc43-08f4-49a2-88da-0910939e5254\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mw7n5" Oct 30 17:42:52.276767 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:52.276740 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8tmz\" (UniqueName: \"kubernetes.io/projected/33d4dc43-08f4-49a2-88da-0910939e5254-kube-api-access-r8tmz\") pod \"maas-api-revoke-sub-c1e271a9-mw7n5\" (UID: \"33d4dc43-08f4-49a2-88da-0910939e5254\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mw7n5" Oct 30 17:42:52.418727 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:52.418635 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mw7n5" Oct 30 17:42:52.718615 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:52.718583 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mw7n5"] Oct 30 17:42:52.718794 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:42:52.718763 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33d4dc43_08f4_49a2_88da_0910939e5254.slice/crio-c70620eddc2d0f6705578527ec5018d2a90e6cdb16fff4a020a6fef3a8a3cd80 WatchSource:0}: Error finding container c70620eddc2d0f6705578527ec5018d2a90e6cdb16fff4a020a6fef3a8a3cd80: Status 404 returned error can't find the container with id c70620eddc2d0f6705578527ec5018d2a90e6cdb16fff4a020a6fef3a8a3cd80 Oct 30 17:42:52.806744 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:52.806711 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mw7n5" event={"ID":"33d4dc43-08f4-49a2-88da-0910939e5254","Type":"ContainerStarted","Data":"c70620eddc2d0f6705578527ec5018d2a90e6cdb16fff4a020a6fef3a8a3cd80"} Oct 30 17:42:53.813599 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:53.813566 4522 generic.go:356] "Generic (PLEG): container finished" podID="33d4dc43-08f4-49a2-88da-0910939e5254" containerID="6ae86e7aa2780f4ec530a806560022ffc33366146bd99e0730e8eea049301e34" exitCode=0 Oct 30 17:42:53.814014 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:53.813643 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mw7n5" event={"ID":"33d4dc43-08f4-49a2-88da-0910939e5254","Type":"ContainerDied","Data":"6ae86e7aa2780f4ec530a806560022ffc33366146bd99e0730e8eea049301e34"} Oct 30 17:42:54.951174 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:54.951149 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mw7n5" Oct 30 17:42:55.076273 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:55.076179 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33d4dc43-08f4-49a2-88da-0910939e5254-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33d4dc43-08f4-49a2-88da-0910939e5254-maas-api-xks-ca\") pod \"33d4dc43-08f4-49a2-88da-0910939e5254\" (UID: \"33d4dc43-08f4-49a2-88da-0910939e5254\") " Oct 30 17:42:55.076273 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:55.076261 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33d4dc43-08f4-49a2-88da-0910939e5254-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33d4dc43-08f4-49a2-88da-0910939e5254-maas-api-openshift-ca\") pod \"33d4dc43-08f4-49a2-88da-0910939e5254\" (UID: \"33d4dc43-08f4-49a2-88da-0910939e5254\") " Oct 30 17:42:55.076496 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:55.076367 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33d4dc43-08f4-49a2-88da-0910939e5254-kube-api-access-r8tmz\" (UniqueName: \"kubernetes.io/projected/33d4dc43-08f4-49a2-88da-0910939e5254-kube-api-access-r8tmz\") pod \"33d4dc43-08f4-49a2-88da-0910939e5254\" (UID: \"33d4dc43-08f4-49a2-88da-0910939e5254\") " Oct 30 17:42:55.076698 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:55.076667 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33d4dc43-08f4-49a2-88da-0910939e5254-maas-api-openshift-ca" pod "33d4dc43-08f4-49a2-88da-0910939e5254" (UID: "33d4dc43-08f4-49a2-88da-0910939e5254"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:42:55.078369 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:55.078351 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33d4dc43-08f4-49a2-88da-0910939e5254-maas-api-xks-ca" pod "33d4dc43-08f4-49a2-88da-0910939e5254" (UID: "33d4dc43-08f4-49a2-88da-0910939e5254"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:42:55.078470 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:55.078455 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33d4dc43-08f4-49a2-88da-0910939e5254-kube-api-access-r8tmz" pod "33d4dc43-08f4-49a2-88da-0910939e5254" (UID: "33d4dc43-08f4-49a2-88da-0910939e5254"). InnerVolumeSpecName "kube-api-access-r8tmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:42:55.177782 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:55.177746 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8tmz\" (UniqueName: \"kubernetes.io/projected/33d4dc43-08f4-49a2-88da-0910939e5254-kube-api-access-r8tmz\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:42:55.177782 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:55.177778 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33d4dc43-08f4-49a2-88da-0910939e5254-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:42:55.177988 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:55.177794 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33d4dc43-08f4-49a2-88da-0910939e5254-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:42:55.825929 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:55.825896 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mw7n5" Oct 30 17:42:55.826133 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:55.825933 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mw7n5" event={"ID":"33d4dc43-08f4-49a2-88da-0910939e5254","Type":"ContainerDied","Data":"c70620eddc2d0f6705578527ec5018d2a90e6cdb16fff4a020a6fef3a8a3cd80"} Oct 30 17:42:55.826133 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:42:55.826015 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c70620eddc2d0f6705578527ec5018d2a90e6cdb16fff4a020a6fef3a8a3cd80" Oct 30 17:43:09.878849 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:09.878813 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hgtsq"] Oct 30 17:43:09.879594 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:09.879329 4522 cpu_manager.go:414] "containerMap: removing container" podUID="33d4dc43-08f4-49a2-88da-0910939e5254" containerName="revoke-keys" Oct 30 17:43:09.879594 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:09.879349 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33d4dc43-08f4-49a2-88da-0910939e5254" containerName="revoke-keys" Oct 30 17:43:09.879594 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:09.879459 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="33d4dc43-08f4-49a2-88da-0910939e5254" containerName="revoke-keys" Oct 30 17:43:09.882607 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:09.882589 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hgtsq" Oct 30 17:43:09.966000 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:09.965953 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hgtsq"] Oct 30 17:43:10.049222 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:10.049163 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d445c57-2efa-4ae2-8a16-9e4dcdac8544-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hgtsq\" (UID: \"9d445c57-2efa-4ae2-8a16-9e4dcdac8544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hgtsq" Oct 30 17:43:10.049473 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:10.049237 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d445c57-2efa-4ae2-8a16-9e4dcdac8544-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hgtsq\" (UID: \"9d445c57-2efa-4ae2-8a16-9e4dcdac8544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hgtsq" Oct 30 17:43:10.049473 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:10.049299 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d445c57-2efa-4ae2-8a16-9e4dcdac8544-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hgtsq\" (UID: \"9d445c57-2efa-4ae2-8a16-9e4dcdac8544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hgtsq" Oct 30 17:43:10.049473 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:10.049369 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d445c57-2efa-4ae2-8a16-9e4dcdac8544-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hgtsq\" (UID: \"9d445c57-2efa-4ae2-8a16-9e4dcdac8544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hgtsq" Oct 30 17:43:10.049473 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:10.049418 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d445c57-2efa-4ae2-8a16-9e4dcdac8544-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hgtsq\" (UID: \"9d445c57-2efa-4ae2-8a16-9e4dcdac8544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hgtsq" Oct 30 17:43:10.049473 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:10.049471 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d445c57-2efa-4ae2-8a16-9e4dcdac8544-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hgtsq\" (UID: \"9d445c57-2efa-4ae2-8a16-9e4dcdac8544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hgtsq" Oct 30 17:43:10.049811 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:10.049505 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d445c57-2efa-4ae2-8a16-9e4dcdac8544-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hgtsq\" (UID: \"9d445c57-2efa-4ae2-8a16-9e4dcdac8544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hgtsq" Oct 30 17:43:10.049811 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:10.049536 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d445c57-2efa-4ae2-8a16-9e4dcdac8544-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hgtsq\" (UID: \"9d445c57-2efa-4ae2-8a16-9e4dcdac8544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hgtsq" Oct 30 17:43:10.049811 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:10.049580 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfzzh\" (UniqueName: \"kubernetes.io/projected/9d445c57-2efa-4ae2-8a16-9e4dcdac8544-kube-api-access-jfzzh\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hgtsq\" (UID: \"9d445c57-2efa-4ae2-8a16-9e4dcdac8544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hgtsq" Oct 30 17:43:10.150640 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:10.150538 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfzzh\" (UniqueName: \"kubernetes.io/projected/9d445c57-2efa-4ae2-8a16-9e4dcdac8544-kube-api-access-jfzzh\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hgtsq\" (UID: \"9d445c57-2efa-4ae2-8a16-9e4dcdac8544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hgtsq" Oct 30 17:43:10.150640 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:10.150619 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d445c57-2efa-4ae2-8a16-9e4dcdac8544-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hgtsq\" (UID: \"9d445c57-2efa-4ae2-8a16-9e4dcdac8544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hgtsq" Oct 30 17:43:10.150909 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:10.150653 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d445c57-2efa-4ae2-8a16-9e4dcdac8544-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hgtsq\" (UID: \"9d445c57-2efa-4ae2-8a16-9e4dcdac8544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hgtsq" Oct 30 17:43:10.150909 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:10.150704 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d445c57-2efa-4ae2-8a16-9e4dcdac8544-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hgtsq\" (UID: \"9d445c57-2efa-4ae2-8a16-9e4dcdac8544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hgtsq" Oct 30 17:43:10.150909 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:10.150740 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d445c57-2efa-4ae2-8a16-9e4dcdac8544-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hgtsq\" (UID: \"9d445c57-2efa-4ae2-8a16-9e4dcdac8544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hgtsq" Oct 30 17:43:10.150909 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:10.150780 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d445c57-2efa-4ae2-8a16-9e4dcdac8544-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hgtsq\" (UID: \"9d445c57-2efa-4ae2-8a16-9e4dcdac8544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hgtsq" Oct 30 17:43:10.150909 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:10.150827 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d445c57-2efa-4ae2-8a16-9e4dcdac8544-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hgtsq\" (UID: \"9d445c57-2efa-4ae2-8a16-9e4dcdac8544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hgtsq" Oct 30 17:43:10.150909 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:10.150857 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d445c57-2efa-4ae2-8a16-9e4dcdac8544-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hgtsq\" (UID: \"9d445c57-2efa-4ae2-8a16-9e4dcdac8544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hgtsq" Oct 30 17:43:10.150909 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:10.150884 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d445c57-2efa-4ae2-8a16-9e4dcdac8544-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hgtsq\" (UID: \"9d445c57-2efa-4ae2-8a16-9e4dcdac8544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hgtsq" Oct 30 17:43:10.151383 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:10.151354 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d445c57-2efa-4ae2-8a16-9e4dcdac8544-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hgtsq\" (UID: \"9d445c57-2efa-4ae2-8a16-9e4dcdac8544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hgtsq" Oct 30 17:43:10.152156 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:10.152129 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d445c57-2efa-4ae2-8a16-9e4dcdac8544-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hgtsq\" (UID: \"9d445c57-2efa-4ae2-8a16-9e4dcdac8544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hgtsq" Oct 30 17:43:10.152448 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:10.152426 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d445c57-2efa-4ae2-8a16-9e4dcdac8544-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hgtsq\" (UID: \"9d445c57-2efa-4ae2-8a16-9e4dcdac8544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hgtsq" Oct 30 17:43:10.153540 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:10.153512 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d445c57-2efa-4ae2-8a16-9e4dcdac8544-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hgtsq\" (UID: \"9d445c57-2efa-4ae2-8a16-9e4dcdac8544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hgtsq" Oct 30 17:43:10.154081 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:10.154051 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d445c57-2efa-4ae2-8a16-9e4dcdac8544-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hgtsq\" (UID: \"9d445c57-2efa-4ae2-8a16-9e4dcdac8544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hgtsq" Oct 30 17:43:10.158346 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:10.157273 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d445c57-2efa-4ae2-8a16-9e4dcdac8544-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hgtsq\" (UID: \"9d445c57-2efa-4ae2-8a16-9e4dcdac8544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hgtsq" Oct 30 17:43:10.158988 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:10.158958 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d445c57-2efa-4ae2-8a16-9e4dcdac8544-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hgtsq\" (UID: \"9d445c57-2efa-4ae2-8a16-9e4dcdac8544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hgtsq" Oct 30 17:43:10.180064 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:10.180043 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d445c57-2efa-4ae2-8a16-9e4dcdac8544-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hgtsq\" (UID: \"9d445c57-2efa-4ae2-8a16-9e4dcdac8544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hgtsq" Oct 30 17:43:10.180506 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:10.180480 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfzzh\" (UniqueName: \"kubernetes.io/projected/9d445c57-2efa-4ae2-8a16-9e4dcdac8544-kube-api-access-jfzzh\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hgtsq\" (UID: \"9d445c57-2efa-4ae2-8a16-9e4dcdac8544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hgtsq" Oct 30 17:43:10.196838 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:10.196806 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hgtsq" Oct 30 17:43:10.541739 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:10.541705 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hgtsq"] Oct 30 17:43:10.542987 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:43:10.542940 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d445c57_2efa_4ae2_8a16_9e4dcdac8544.slice/crio-a3934d1b4e15a0f3fa9559606b5cde726cfdead0fb97c65fdd293eb5bab0a807 WatchSource:0}: Error finding container a3934d1b4e15a0f3fa9559606b5cde726cfdead0fb97c65fdd293eb5bab0a807: Status 404 returned error can't find the container with id a3934d1b4e15a0f3fa9559606b5cde726cfdead0fb97c65fdd293eb5bab0a807 Oct 30 17:43:10.546006 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:10.545966 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:43:10.546134 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:10.546057 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:43:10.546134 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:10.546107 4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:43:10.949460 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:10.949368 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hgtsq" event={"ID":"9d445c57-2efa-4ae2-8a16-9e4dcdac8544","Type":"ContainerStarted","Data":"8fae83c090dae466bd9e01bef40330fb143def3d5db6d782b9e5fd5fbedbb68a"} Oct 30 17:43:10.949460 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:10.949434 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hgtsq" event={"ID":"9d445c57-2efa-4ae2-8a16-9e4dcdac8544","Type":"ContainerStarted","Data":"a3934d1b4e15a0f3fa9559606b5cde726cfdead0fb97c65fdd293eb5bab0a807"} Oct 30 17:43:11.005819 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:11.005742 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hgtsq" podStartSLOduration=2.005723173 podStartE2EDuration="2.005723173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:43:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:43:11.004021917 +0000 UTC m=+3365.004776819" watchObservedRunningTime="2026-10-30 17:43:11.005723173 +0000 UTC m=+3365.006478077" Oct 30 17:43:11.209391 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:11.199891 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hgtsq" Oct 30 17:43:11.209391 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:11.200019 4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-hgtsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.127:15021/healthz/ready\": dial tcp 10.134.1.127:15021: connect: connection refused" start-of-body= Oct 30 17:43:11.209391 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:11.200078 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hgtsq" podUID="9d445c57-2efa-4ae2-8a16-9e4dcdac8544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.127:15021/healthz/ready\": dial tcp 10.134.1.127:15021: connect: connection refused" Oct 30 17:43:11.445515 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:11.445485 4522 scope.go:122] "RemoveContainer" containerID="1a711f43fd4fb0e883aa107f076c4164cd79a410b4888159eb8c32820d067121" Oct 30 17:43:11.454430 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:11.454408 4522 scope.go:122] "RemoveContainer" containerID="65cae631033de8d75467565ad77e58739fc7eaeb3908f1a7ae0d4ebc1be0d5b4" Oct 30 17:43:11.462385 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:11.462359 4522 scope.go:122] "RemoveContainer" containerID="7da3be0e62b7aae34cc5414c014d5608325d722b4a6e469c5d750068348c7df9" Oct 30 17:43:11.470053 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:11.470035 4522 scope.go:122] "RemoveContainer" containerID="ce6c5675593741b541f1e00d0109a5bd0fda93dcb0d3fbf9ddecf639ab057b35" Oct 30 17:43:11.478072 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:11.478054 4522 scope.go:122] "RemoveContainer" containerID="411d2d43af71dcc876bde3ca58f4e4358edc601f34b00fef5ff0f932fb3ebffd" Oct 30 17:43:11.485427 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:11.485409 4522 scope.go:122] "RemoveContainer" containerID="a73ec6e06360f6b4f2b9899b33b1b598dbb3ab6e751daf14c3c055b5a1aee4f5" Oct 30 17:43:11.516782 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:11.516745 4522 scope.go:122] "RemoveContainer" containerID="257e7f139feea5598bad5b77784789b47f9b1789737cd077734e6e422e36e960" Oct 30 17:43:11.527866 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:11.527837 4522 scope.go:122] "RemoveContainer" containerID="72b286b72405fbbfeb40ffad83bf89f6f248c0e19c2429a4ab99a646a2a15957" Oct 30 17:43:12.212331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:12.205746 4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-hgtsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.127:15021/healthz/ready\": dial tcp 10.134.1.127:15021: connect: connection refused" start-of-body= Oct 30 17:43:12.212331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:12.205808 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hgtsq" podUID="9d445c57-2efa-4ae2-8a16-9e4dcdac8544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.127:15021/healthz/ready\": dial tcp 10.134.1.127:15021: connect: connection refused" Oct 30 17:43:12.269690 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:12.269648 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-f6x7w"] Oct 30 17:43:12.274913 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:12.274886 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-f6x7w" Oct 30 17:43:12.309803 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:12.309606 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:43:12.309803 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:12.309736 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:43:12.314617 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:12.314448 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:43:12.314617 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:12.314583 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:43:12.322332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:12.318907 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-f6x7w"] Oct 30 17:43:12.376046 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:12.376014 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwgqt\" (UniqueName: \"kubernetes.io/projected/9fe5072f-e703-49ca-9d89-4279b34e66dc-kube-api-access-dwgqt\") pod \"maas-api-revoke-sub-2e1f5b5f-f6x7w\" (UID: \"9fe5072f-e703-49ca-9d89-4279b34e66dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-f6x7w" Oct 30 17:43:12.376239 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:12.376087 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fe5072f-e703-49ca-9d89-4279b34e66dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-f6x7w\" (UID: \"9fe5072f-e703-49ca-9d89-4279b34e66dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-f6x7w" Oct 30 17:43:12.376286 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:12.376225 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fe5072f-e703-49ca-9d89-4279b34e66dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-f6x7w\" (UID: \"9fe5072f-e703-49ca-9d89-4279b34e66dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-f6x7w" Oct 30 17:43:12.477076 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:12.476983 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fe5072f-e703-49ca-9d89-4279b34e66dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-f6x7w\" (UID: \"9fe5072f-e703-49ca-9d89-4279b34e66dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-f6x7w" Oct 30 17:43:12.477076 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:12.477042 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fe5072f-e703-49ca-9d89-4279b34e66dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-f6x7w\" (UID: \"9fe5072f-e703-49ca-9d89-4279b34e66dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-f6x7w" Oct 30 17:43:12.477280 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:12.477096 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwgqt\" (UniqueName: \"kubernetes.io/projected/9fe5072f-e703-49ca-9d89-4279b34e66dc-kube-api-access-dwgqt\") pod \"maas-api-revoke-sub-2e1f5b5f-f6x7w\" (UID: \"9fe5072f-e703-49ca-9d89-4279b34e66dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-f6x7w" Oct 30 17:43:12.477867 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:12.477840 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fe5072f-e703-49ca-9d89-4279b34e66dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-f6x7w\" (UID: \"9fe5072f-e703-49ca-9d89-4279b34e66dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-f6x7w" Oct 30 17:43:12.479463 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:12.479437 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fe5072f-e703-49ca-9d89-4279b34e66dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-f6x7w\" (UID: \"9fe5072f-e703-49ca-9d89-4279b34e66dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-f6x7w" Oct 30 17:43:12.516331 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:12.514440 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwgqt\" (UniqueName: \"kubernetes.io/projected/9fe5072f-e703-49ca-9d89-4279b34e66dc-kube-api-access-dwgqt\") pod \"maas-api-revoke-sub-2e1f5b5f-f6x7w\" (UID: \"9fe5072f-e703-49ca-9d89-4279b34e66dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-f6x7w" Oct 30 17:43:12.588245 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:12.588209 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-f6x7w" Oct 30 17:43:12.920704 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:12.919286 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-f6x7w"] Oct 30 17:43:12.951509 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:43:12.951479 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:43:12.960416 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:12.960389 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-f6x7w" event={"ID":"9fe5072f-e703-49ca-9d89-4279b34e66dc","Type":"ContainerStarted","Data":"a791a6dfeb2cc922599de441bde32a2815f1287d4ed870e9446bf2994ce5192b"} Oct 30 17:43:13.213388 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:13.204719 4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-hgtsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.127:15021/healthz/ready\": dial tcp 10.134.1.127:15021: connect: connection refused" start-of-body= Oct 30 17:43:13.213388 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:13.204775 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hgtsq" podUID="9d445c57-2efa-4ae2-8a16-9e4dcdac8544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.127:15021/healthz/ready\": dial tcp 10.134.1.127:15021: connect: connection refused" Oct 30 17:43:13.966632 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:13.966603 4522 generic.go:356] "Generic (PLEG): container finished" podID="9fe5072f-e703-49ca-9d89-4279b34e66dc" containerID="4ba155bedd974ba5d7f091d6d114cc6c44c2779231411409d4df68c789484e49" exitCode=0 Oct 30 17:43:13.966801 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:13.966688 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-f6x7w" event={"ID":"9fe5072f-e703-49ca-9d89-4279b34e66dc","Type":"ContainerDied","Data":"4ba155bedd974ba5d7f091d6d114cc6c44c2779231411409d4df68c789484e49"} Oct 30 17:43:14.216347 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:14.212372 4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-hgtsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.127:15021/healthz/ready\": dial tcp 10.134.1.127:15021: connect: connection refused" start-of-body= Oct 30 17:43:14.216347 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:14.212452 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hgtsq" podUID="9d445c57-2efa-4ae2-8a16-9e4dcdac8544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.127:15021/healthz/ready\": dial tcp 10.134.1.127:15021: connect: connection refused" Oct 30 17:43:15.143259 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:15.143230 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-f6x7w" Oct 30 17:43:15.216333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:15.213716 4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-hgtsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.127:15021/healthz/ready\": dial tcp 10.134.1.127:15021: connect: connection refused" start-of-body= Oct 30 17:43:15.216333 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:15.213776 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hgtsq" podUID="9d445c57-2efa-4ae2-8a16-9e4dcdac8544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.127:15021/healthz/ready\": dial tcp 10.134.1.127:15021: connect: connection refused" Oct 30 17:43:15.323529 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:15.323439 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fe5072f-e703-49ca-9d89-4279b34e66dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fe5072f-e703-49ca-9d89-4279b34e66dc-maas-api-xks-ca\") pod \"9fe5072f-e703-49ca-9d89-4279b34e66dc\" (UID: \"9fe5072f-e703-49ca-9d89-4279b34e66dc\") " Oct 30 17:43:15.323689 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:15.323536 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fe5072f-e703-49ca-9d89-4279b34e66dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fe5072f-e703-49ca-9d89-4279b34e66dc-maas-api-openshift-ca\") pod \"9fe5072f-e703-49ca-9d89-4279b34e66dc\" (UID: \"9fe5072f-e703-49ca-9d89-4279b34e66dc\") " Oct 30 17:43:15.323689 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:15.323561 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fe5072f-e703-49ca-9d89-4279b34e66dc-kube-api-access-dwgqt\" (UniqueName: \"kubernetes.io/projected/9fe5072f-e703-49ca-9d89-4279b34e66dc-kube-api-access-dwgqt\") pod \"9fe5072f-e703-49ca-9d89-4279b34e66dc\" (UID: \"9fe5072f-e703-49ca-9d89-4279b34e66dc\") " Oct 30 17:43:15.324099 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:15.323869 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fe5072f-e703-49ca-9d89-4279b34e66dc-maas-api-openshift-ca" pod "9fe5072f-e703-49ca-9d89-4279b34e66dc" (UID: "9fe5072f-e703-49ca-9d89-4279b34e66dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:43:15.324099 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:15.324023 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fe5072f-e703-49ca-9d89-4279b34e66dc-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:43:15.325835 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:15.325814 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fe5072f-e703-49ca-9d89-4279b34e66dc-maas-api-xks-ca" pod "9fe5072f-e703-49ca-9d89-4279b34e66dc" (UID: "9fe5072f-e703-49ca-9d89-4279b34e66dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:43:15.325943 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:15.325873 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fe5072f-e703-49ca-9d89-4279b34e66dc-kube-api-access-dwgqt" pod "9fe5072f-e703-49ca-9d89-4279b34e66dc" (UID: "9fe5072f-e703-49ca-9d89-4279b34e66dc"). InnerVolumeSpecName "kube-api-access-dwgqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:43:15.424973 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:15.424937 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fe5072f-e703-49ca-9d89-4279b34e66dc-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:43:15.424973 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:15.424968 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwgqt\" (UniqueName: \"kubernetes.io/projected/9fe5072f-e703-49ca-9d89-4279b34e66dc-kube-api-access-dwgqt\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:43:15.978663 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:15.978633 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-f6x7w" Oct 30 17:43:15.978847 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:15.978673 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-f6x7w" event={"ID":"9fe5072f-e703-49ca-9d89-4279b34e66dc","Type":"ContainerDied","Data":"a791a6dfeb2cc922599de441bde32a2815f1287d4ed870e9446bf2994ce5192b"} Oct 30 17:43:15.978847 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:15.978720 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a791a6dfeb2cc922599de441bde32a2815f1287d4ed870e9446bf2994ce5192b" Oct 30 17:43:16.210343 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:16.209677 4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-hgtsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.127:15021/healthz/ready\": dial tcp 10.134.1.127:15021: connect: connection refused" start-of-body= Oct 30 17:43:16.210343 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:16.209737 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hgtsq" podUID="9d445c57-2efa-4ae2-8a16-9e4dcdac8544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.127:15021/healthz/ready\": dial tcp 10.134.1.127:15021: connect: connection refused" Oct 30 17:43:17.198067 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:17.198034 4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-hgtsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.127:15021/healthz/ready\": dial tcp 10.134.1.127:15021: connect: connection refused" start-of-body= Oct 30 17:43:17.198498 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:17.198086 4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hgtsq" podUID="9d445c57-2efa-4ae2-8a16-9e4dcdac8544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.127:15021/healthz/ready\": dial tcp 10.134.1.127:15021: connect: connection refused" Oct 30 17:43:18.716704 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:18.716669 4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hgtsq" Oct 30 17:43:18.717227 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:18.716972 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hgtsq" Oct 30 17:43:18.717871 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:18.717848 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hgtsq" Oct 30 17:43:23.727011 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:23.726962 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r7zb5"] Oct 30 17:43:23.727857 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:23.727832 4522 cpu_manager.go:414] "containerMap: removing container" podUID="9fe5072f-e703-49ca-9d89-4279b34e66dc" containerName="revoke-keys" Oct 30 17:43:23.727909 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:23.727862 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fe5072f-e703-49ca-9d89-4279b34e66dc" containerName="revoke-keys" Oct 30 17:43:23.727966 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:23.727955 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fe5072f-e703-49ca-9d89-4279b34e66dc" containerName="revoke-keys" Oct 30 17:43:23.731726 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:23.731710 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r7zb5" Oct 30 17:43:23.766542 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:23.766499 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:43:23.766542 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:23.766508 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:43:23.766542 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:23.766539 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:43:23.766819 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:23.766590 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:43:23.769954 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:23.769926 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r7zb5"] Oct 30 17:43:23.807099 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:23.807067 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjq6h\" (UniqueName: \"kubernetes.io/projected/82e47fa5-9c29-44d5-97c8-0c41230b1f0f-kube-api-access-wjq6h\") pod \"maas-api-revoke-sub-ec843018-r7zb5\" (UID: \"82e47fa5-9c29-44d5-97c8-0c41230b1f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r7zb5" Oct 30 17:43:23.807270 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:23.807114 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82e47fa5-9c29-44d5-97c8-0c41230b1f0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-r7zb5\" (UID: \"82e47fa5-9c29-44d5-97c8-0c41230b1f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r7zb5" Oct 30 17:43:23.807270 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:23.807252 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82e47fa5-9c29-44d5-97c8-0c41230b1f0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-r7zb5\" (UID: \"82e47fa5-9c29-44d5-97c8-0c41230b1f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r7zb5" Oct 30 17:43:23.908596 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:23.908556 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82e47fa5-9c29-44d5-97c8-0c41230b1f0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-r7zb5\" (UID: \"82e47fa5-9c29-44d5-97c8-0c41230b1f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r7zb5" Oct 30 17:43:23.908812 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:23.908643 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjq6h\" (UniqueName: \"kubernetes.io/projected/82e47fa5-9c29-44d5-97c8-0c41230b1f0f-kube-api-access-wjq6h\") pod \"maas-api-revoke-sub-ec843018-r7zb5\" (UID: \"82e47fa5-9c29-44d5-97c8-0c41230b1f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r7zb5" Oct 30 17:43:23.908812 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:23.908684 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82e47fa5-9c29-44d5-97c8-0c41230b1f0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-r7zb5\" (UID: \"82e47fa5-9c29-44d5-97c8-0c41230b1f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r7zb5" Oct 30 17:43:23.909249 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:23.909223 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82e47fa5-9c29-44d5-97c8-0c41230b1f0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-r7zb5\" (UID: \"82e47fa5-9c29-44d5-97c8-0c41230b1f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r7zb5" Oct 30 17:43:23.911096 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:23.911064 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82e47fa5-9c29-44d5-97c8-0c41230b1f0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-r7zb5\" (UID: \"82e47fa5-9c29-44d5-97c8-0c41230b1f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r7zb5" Oct 30 17:43:23.935291 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:23.935256 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjq6h\" (UniqueName: \"kubernetes.io/projected/82e47fa5-9c29-44d5-97c8-0c41230b1f0f-kube-api-access-wjq6h\") pod \"maas-api-revoke-sub-ec843018-r7zb5\" (UID: \"82e47fa5-9c29-44d5-97c8-0c41230b1f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r7zb5" Oct 30 17:43:24.044115 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:24.044028 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r7zb5" Oct 30 17:43:24.361651 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:24.361622 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r7zb5"] Oct 30 17:43:24.363209 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:43:24.363181 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82e47fa5_9c29_44d5_97c8_0c41230b1f0f.slice/crio-a15226da57b4b4f994bc0d3625afbdb043e638047ad58035be0ed809b268aaa6 WatchSource:0}: Error finding container a15226da57b4b4f994bc0d3625afbdb043e638047ad58035be0ed809b268aaa6: Status 404 returned error can't find the container with id a15226da57b4b4f994bc0d3625afbdb043e638047ad58035be0ed809b268aaa6 Oct 30 17:43:25.033389 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:25.033349 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r7zb5" event={"ID":"82e47fa5-9c29-44d5-97c8-0c41230b1f0f","Type":"ContainerStarted","Data":"b9ffc797f7b104821669cf3d470505e6820eb8e4f44f5f83492af99ea341427d"} Oct 30 17:43:25.033929 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:25.033412 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r7zb5" event={"ID":"82e47fa5-9c29-44d5-97c8-0c41230b1f0f","Type":"ContainerStarted","Data":"a15226da57b4b4f994bc0d3625afbdb043e638047ad58035be0ed809b268aaa6"} Oct 30 17:43:25.079146 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:25.079099 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r7zb5" podStartSLOduration=2.079086076 podStartE2EDuration="2.079086076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:43:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:43:25.078916764 +0000 UTC m=+3379.079671665" watchObservedRunningTime="2026-10-30 17:43:25.079086076 +0000 UTC m=+3379.079840977" Oct 30 17:43:26.042320 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:26.042271 4522 generic.go:356] "Generic (PLEG): container finished" podID="82e47fa5-9c29-44d5-97c8-0c41230b1f0f" containerID="b9ffc797f7b104821669cf3d470505e6820eb8e4f44f5f83492af99ea341427d" exitCode=0 Oct 30 17:43:26.042994 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:26.042369 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r7zb5" event={"ID":"82e47fa5-9c29-44d5-97c8-0c41230b1f0f","Type":"ContainerDied","Data":"b9ffc797f7b104821669cf3d470505e6820eb8e4f44f5f83492af99ea341427d"} Oct 30 17:43:27.214202 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:27.214173 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r7zb5" Oct 30 17:43:27.344260 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:27.344159 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82e47fa5-9c29-44d5-97c8-0c41230b1f0f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82e47fa5-9c29-44d5-97c8-0c41230b1f0f-maas-api-xks-ca\") pod \"82e47fa5-9c29-44d5-97c8-0c41230b1f0f\" (UID: \"82e47fa5-9c29-44d5-97c8-0c41230b1f0f\") " Oct 30 17:43:27.344260 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:27.344249 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82e47fa5-9c29-44d5-97c8-0c41230b1f0f-kube-api-access-wjq6h\" (UniqueName: \"kubernetes.io/projected/82e47fa5-9c29-44d5-97c8-0c41230b1f0f-kube-api-access-wjq6h\") pod \"82e47fa5-9c29-44d5-97c8-0c41230b1f0f\" (UID: \"82e47fa5-9c29-44d5-97c8-0c41230b1f0f\") " Oct 30 17:43:27.344557 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:27.344298 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82e47fa5-9c29-44d5-97c8-0c41230b1f0f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82e47fa5-9c29-44d5-97c8-0c41230b1f0f-maas-api-openshift-ca\") pod \"82e47fa5-9c29-44d5-97c8-0c41230b1f0f\" (UID: \"82e47fa5-9c29-44d5-97c8-0c41230b1f0f\") " Oct 30 17:43:27.345016 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:27.344982 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82e47fa5-9c29-44d5-97c8-0c41230b1f0f-maas-api-openshift-ca" pod "82e47fa5-9c29-44d5-97c8-0c41230b1f0f" (UID: "82e47fa5-9c29-44d5-97c8-0c41230b1f0f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:43:27.346495 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:27.346467 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82e47fa5-9c29-44d5-97c8-0c41230b1f0f-kube-api-access-wjq6h" pod "82e47fa5-9c29-44d5-97c8-0c41230b1f0f" (UID: "82e47fa5-9c29-44d5-97c8-0c41230b1f0f"). InnerVolumeSpecName "kube-api-access-wjq6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:43:27.346889 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:27.346858 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82e47fa5-9c29-44d5-97c8-0c41230b1f0f-maas-api-xks-ca" pod "82e47fa5-9c29-44d5-97c8-0c41230b1f0f" (UID: "82e47fa5-9c29-44d5-97c8-0c41230b1f0f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:43:27.445949 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:27.445896 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82e47fa5-9c29-44d5-97c8-0c41230b1f0f-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:43:27.445949 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:27.445941 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjq6h\" (UniqueName: \"kubernetes.io/projected/82e47fa5-9c29-44d5-97c8-0c41230b1f0f-kube-api-access-wjq6h\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:43:27.445949 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:27.445959 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82e47fa5-9c29-44d5-97c8-0c41230b1f0f-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:43:28.064294 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:28.064262 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r7zb5" Oct 30 17:43:28.064652 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:28.064357 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r7zb5" event={"ID":"82e47fa5-9c29-44d5-97c8-0c41230b1f0f","Type":"ContainerDied","Data":"a15226da57b4b4f994bc0d3625afbdb043e638047ad58035be0ed809b268aaa6"} Oct 30 17:43:28.064774 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:28.064704 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a15226da57b4b4f994bc0d3625afbdb043e638047ad58035be0ed809b268aaa6" Oct 30 17:43:42.946441 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:43:42.946399 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:43:51.795416 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:51.795381 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-rsqhg"] Oct 30 17:43:51.795845 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:51.795797 4522 cpu_manager.go:414] "containerMap: removing container" podUID="82e47fa5-9c29-44d5-97c8-0c41230b1f0f" containerName="revoke-keys" Oct 30 17:43:51.795845 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:51.795808 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82e47fa5-9c29-44d5-97c8-0c41230b1f0f" containerName="revoke-keys" Oct 30 17:43:51.795942 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:51.795875 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="82e47fa5-9c29-44d5-97c8-0c41230b1f0f" containerName="revoke-keys" Oct 30 17:43:51.799217 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:51.799200 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-rsqhg" Oct 30 17:43:51.816641 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:51.816614 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:43:51.830965 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:51.830930 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:43:51.831124 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:51.830939 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:43:51.831124 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:51.831006 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:43:51.833872 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:51.833845 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-rsqhg"] Oct 30 17:43:51.846137 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:51.846110 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7pgj\" (UniqueName: \"kubernetes.io/projected/05ba384e-c0d8-4559-bb29-40ae912bd8b3-kube-api-access-d7pgj\") pod \"maas-api-revoke-sub-046e1343-rsqhg\" (UID: \"05ba384e-c0d8-4559-bb29-40ae912bd8b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-rsqhg" Oct 30 17:43:51.846293 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:51.846159 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05ba384e-c0d8-4559-bb29-40ae912bd8b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-rsqhg\" (UID: \"05ba384e-c0d8-4559-bb29-40ae912bd8b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-rsqhg" Oct 30 17:43:51.846387 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:51.846359 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05ba384e-c0d8-4559-bb29-40ae912bd8b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-rsqhg\" (UID: \"05ba384e-c0d8-4559-bb29-40ae912bd8b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-rsqhg" Oct 30 17:43:51.947255 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:51.947219 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7pgj\" (UniqueName: \"kubernetes.io/projected/05ba384e-c0d8-4559-bb29-40ae912bd8b3-kube-api-access-d7pgj\") pod \"maas-api-revoke-sub-046e1343-rsqhg\" (UID: \"05ba384e-c0d8-4559-bb29-40ae912bd8b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-rsqhg" Oct 30 17:43:51.947488 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:51.947272 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05ba384e-c0d8-4559-bb29-40ae912bd8b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-rsqhg\" (UID: \"05ba384e-c0d8-4559-bb29-40ae912bd8b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-rsqhg" Oct 30 17:43:51.947488 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:51.947337 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05ba384e-c0d8-4559-bb29-40ae912bd8b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-rsqhg\" (UID: \"05ba384e-c0d8-4559-bb29-40ae912bd8b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-rsqhg" Oct 30 17:43:51.947945 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:51.947924 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05ba384e-c0d8-4559-bb29-40ae912bd8b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-rsqhg\" (UID: \"05ba384e-c0d8-4559-bb29-40ae912bd8b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-rsqhg" Oct 30 17:43:51.949721 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:51.949695 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05ba384e-c0d8-4559-bb29-40ae912bd8b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-rsqhg\" (UID: \"05ba384e-c0d8-4559-bb29-40ae912bd8b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-rsqhg" Oct 30 17:43:51.973385 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:51.973361 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7pgj\" (UniqueName: \"kubernetes.io/projected/05ba384e-c0d8-4559-bb29-40ae912bd8b3-kube-api-access-d7pgj\") pod \"maas-api-revoke-sub-046e1343-rsqhg\" (UID: \"05ba384e-c0d8-4559-bb29-40ae912bd8b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-rsqhg" Oct 30 17:43:52.111884 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:52.111796 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-rsqhg" Oct 30 17:43:52.413400 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:52.413374 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-rsqhg"] Oct 30 17:43:52.413912 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:43:52.413890 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05ba384e_c0d8_4559_bb29_40ae912bd8b3.slice/crio-a3f1bcb90679de0efbd254a8568be20501a35f838c1e7e96f1bc9825137d4c1f WatchSource:0}: Error finding container a3f1bcb90679de0efbd254a8568be20501a35f838c1e7e96f1bc9825137d4c1f: Status 404 returned error can't find the container with id a3f1bcb90679de0efbd254a8568be20501a35f838c1e7e96f1bc9825137d4c1f Oct 30 17:43:53.244978 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:53.244951 4522 generic.go:356] "Generic (PLEG): container finished" podID="05ba384e-c0d8-4559-bb29-40ae912bd8b3" containerID="f88ec1e80954436be11013f14a157ac414b4fdd0ff87aa1d8b1f1236cbcd4e9d" exitCode=0 Oct 30 17:43:53.245375 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:53.245029 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-rsqhg" event={"ID":"05ba384e-c0d8-4559-bb29-40ae912bd8b3","Type":"ContainerDied","Data":"f88ec1e80954436be11013f14a157ac414b4fdd0ff87aa1d8b1f1236cbcd4e9d"} Oct 30 17:43:53.245375 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:53.245081 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-rsqhg" event={"ID":"05ba384e-c0d8-4559-bb29-40ae912bd8b3","Type":"ContainerStarted","Data":"a3f1bcb90679de0efbd254a8568be20501a35f838c1e7e96f1bc9825137d4c1f"} Oct 30 17:43:54.385086 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:54.385065 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-rsqhg" Oct 30 17:43:54.472166 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:54.472134 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05ba384e-c0d8-4559-bb29-40ae912bd8b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05ba384e-c0d8-4559-bb29-40ae912bd8b3-maas-api-openshift-ca\") pod \"05ba384e-c0d8-4559-bb29-40ae912bd8b3\" (UID: \"05ba384e-c0d8-4559-bb29-40ae912bd8b3\") " Oct 30 17:43:54.472346 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:54.472178 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05ba384e-c0d8-4559-bb29-40ae912bd8b3-kube-api-access-d7pgj\" (UniqueName: \"kubernetes.io/projected/05ba384e-c0d8-4559-bb29-40ae912bd8b3-kube-api-access-d7pgj\") pod \"05ba384e-c0d8-4559-bb29-40ae912bd8b3\" (UID: \"05ba384e-c0d8-4559-bb29-40ae912bd8b3\") " Oct 30 17:43:54.472346 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:54.472241 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05ba384e-c0d8-4559-bb29-40ae912bd8b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05ba384e-c0d8-4559-bb29-40ae912bd8b3-maas-api-xks-ca\") pod \"05ba384e-c0d8-4559-bb29-40ae912bd8b3\" (UID: \"05ba384e-c0d8-4559-bb29-40ae912bd8b3\") " Oct 30 17:43:54.472561 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:54.472533 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05ba384e-c0d8-4559-bb29-40ae912bd8b3-maas-api-openshift-ca" pod "05ba384e-c0d8-4559-bb29-40ae912bd8b3" (UID: "05ba384e-c0d8-4559-bb29-40ae912bd8b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:43:54.474413 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:54.474368 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05ba384e-c0d8-4559-bb29-40ae912bd8b3-kube-api-access-d7pgj" pod "05ba384e-c0d8-4559-bb29-40ae912bd8b3" (UID: "05ba384e-c0d8-4559-bb29-40ae912bd8b3"). InnerVolumeSpecName "kube-api-access-d7pgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:43:54.474527 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:54.474444 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05ba384e-c0d8-4559-bb29-40ae912bd8b3-maas-api-xks-ca" pod "05ba384e-c0d8-4559-bb29-40ae912bd8b3" (UID: "05ba384e-c0d8-4559-bb29-40ae912bd8b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:43:54.573625 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:54.573541 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05ba384e-c0d8-4559-bb29-40ae912bd8b3-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:43:54.573625 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:54.573571 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7pgj\" (UniqueName: \"kubernetes.io/projected/05ba384e-c0d8-4559-bb29-40ae912bd8b3-kube-api-access-d7pgj\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:43:54.573625 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:54.573593 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05ba384e-c0d8-4559-bb29-40ae912bd8b3-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:43:55.031409 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:55.031375 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc1e4760-bm4jl"] Oct 30 17:43:55.045817 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:55.045790 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc1e4760-bm4jl"] Oct 30 17:43:55.255823 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:55.255795 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-rsqhg" Oct 30 17:43:55.256000 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:55.255841 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-rsqhg" event={"ID":"05ba384e-c0d8-4559-bb29-40ae912bd8b3","Type":"ContainerDied","Data":"a3f1bcb90679de0efbd254a8568be20501a35f838c1e7e96f1bc9825137d4c1f"} Oct 30 17:43:55.256000 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:55.255886 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3f1bcb90679de0efbd254a8568be20501a35f838c1e7e96f1bc9825137d4c1f" Oct 30 17:43:56.143104 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:43:56.143068 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82b766ef-96a7-4c73-886f-56e20fc1422c" path="/var/lib/kubelet/pods/82b766ef-96a7-4c73-886f-56e20fc1422c/volumes" Oct 30 17:44:02.136652 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:02.136609 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-c7xm2"] Oct 30 17:44:02.137064 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:02.137004 4522 cpu_manager.go:414] "containerMap: removing container" podUID="05ba384e-c0d8-4559-bb29-40ae912bd8b3" containerName="revoke-keys" Oct 30 17:44:02.137064 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:02.137015 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05ba384e-c0d8-4559-bb29-40ae912bd8b3" containerName="revoke-keys" Oct 30 17:44:02.137146 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:02.137085 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="05ba384e-c0d8-4559-bb29-40ae912bd8b3" containerName="revoke-keys" Oct 30 17:44:02.141535 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:02.141507 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-c7xm2" Oct 30 17:44:02.160836 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:02.160808 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-c7xm2"] Oct 30 17:44:02.172725 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:02.172694 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:44:02.172862 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:02.172775 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:44:02.172862 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:02.172777 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:44:02.173793 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:02.173773 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:44:02.247827 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:02.247789 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77612860-9c4c-4eac-95c8-c42db1b411e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-c7xm2\" (UID: \"77612860-9c4c-4eac-95c8-c42db1b411e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-c7xm2" Oct 30 17:44:02.247827 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:02.247829 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77612860-9c4c-4eac-95c8-c42db1b411e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-c7xm2\" (UID: \"77612860-9c4c-4eac-95c8-c42db1b411e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-c7xm2" Oct 30 17:44:02.248067 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:02.247951 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8dxl\" (UniqueName: \"kubernetes.io/projected/77612860-9c4c-4eac-95c8-c42db1b411e0-kube-api-access-p8dxl\") pod \"maas-api-revoke-sub-d3fba056-c7xm2\" (UID: \"77612860-9c4c-4eac-95c8-c42db1b411e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-c7xm2" Oct 30 17:44:02.348524 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:02.348491 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77612860-9c4c-4eac-95c8-c42db1b411e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-c7xm2\" (UID: \"77612860-9c4c-4eac-95c8-c42db1b411e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-c7xm2" Oct 30 17:44:02.348707 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:02.348545 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77612860-9c4c-4eac-95c8-c42db1b411e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-c7xm2\" (UID: \"77612860-9c4c-4eac-95c8-c42db1b411e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-c7xm2" Oct 30 17:44:02.348707 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:02.348600 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8dxl\" (UniqueName: \"kubernetes.io/projected/77612860-9c4c-4eac-95c8-c42db1b411e0-kube-api-access-p8dxl\") pod \"maas-api-revoke-sub-d3fba056-c7xm2\" (UID: \"77612860-9c4c-4eac-95c8-c42db1b411e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-c7xm2" Oct 30 17:44:02.349294 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:02.349271 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77612860-9c4c-4eac-95c8-c42db1b411e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-c7xm2\" (UID: \"77612860-9c4c-4eac-95c8-c42db1b411e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-c7xm2" Oct 30 17:44:02.350929 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:02.350910 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77612860-9c4c-4eac-95c8-c42db1b411e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-c7xm2\" (UID: \"77612860-9c4c-4eac-95c8-c42db1b411e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-c7xm2" Oct 30 17:44:02.375323 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:02.375279 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8dxl\" (UniqueName: \"kubernetes.io/projected/77612860-9c4c-4eac-95c8-c42db1b411e0-kube-api-access-p8dxl\") pod \"maas-api-revoke-sub-d3fba056-c7xm2\" (UID: \"77612860-9c4c-4eac-95c8-c42db1b411e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-c7xm2" Oct 30 17:44:02.454367 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:02.454339 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-c7xm2" Oct 30 17:44:02.761053 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:02.761023 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-c7xm2"] Oct 30 17:44:02.762257 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:44:02.762229 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77612860_9c4c_4eac_95c8_c42db1b411e0.slice/crio-4cfd38020cd509fdb7e25fbed217a44f93e4072b863aa757713ad08dbd12f6a9 WatchSource:0}: Error finding container 4cfd38020cd509fdb7e25fbed217a44f93e4072b863aa757713ad08dbd12f6a9: Status 404 returned error can't find the container with id 4cfd38020cd509fdb7e25fbed217a44f93e4072b863aa757713ad08dbd12f6a9 Oct 30 17:44:03.307064 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:03.306961 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-c7xm2" event={"ID":"77612860-9c4c-4eac-95c8-c42db1b411e0","Type":"ContainerStarted","Data":"7612af96ed60f6d1f975588732a6f12635f84ba765166c21aa18c9f0628cfc94"} Oct 30 17:44:03.307064 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:03.307023 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-c7xm2" event={"ID":"77612860-9c4c-4eac-95c8-c42db1b411e0","Type":"ContainerStarted","Data":"4cfd38020cd509fdb7e25fbed217a44f93e4072b863aa757713ad08dbd12f6a9"} Oct 30 17:44:03.354833 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:03.354780 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-c7xm2" podStartSLOduration=1.354764559 podStartE2EDuration="1.354764559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:44:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:44:03.352822492 +0000 UTC m=+3417.353577405" watchObservedRunningTime="2026-10-30 17:44:03.354764559 +0000 UTC m=+3417.355519465" Oct 30 17:44:04.032151 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:04.032114 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bc5d398-p6pgs"] Oct 30 17:44:04.046625 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:04.046590 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bc5d398-p6pgs"] Oct 30 17:44:04.049681 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:04.049655 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a3d7477-hv4h8"] Oct 30 17:44:04.052523 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:04.052501 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a3d7477-hv4h8"] Oct 30 17:44:04.146704 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:04.146661 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="123ecbf4-9360-4086-8c6f-6fbad0ae75d2" path="/var/lib/kubelet/pods/123ecbf4-9360-4086-8c6f-6fbad0ae75d2/volumes" Oct 30 17:44:04.147316 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:04.147274 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41a11823-1100-41a5-bd01-b67ec1ed5191" path="/var/lib/kubelet/pods/41a11823-1100-41a5-bd01-b67ec1ed5191/volumes" Oct 30 17:44:04.319143 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:04.318601 4522 generic.go:356] "Generic (PLEG): container finished" podID="77612860-9c4c-4eac-95c8-c42db1b411e0" containerID="7612af96ed60f6d1f975588732a6f12635f84ba765166c21aa18c9f0628cfc94" exitCode=0 Oct 30 17:44:04.319143 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:04.318663 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-c7xm2" event={"ID":"77612860-9c4c-4eac-95c8-c42db1b411e0","Type":"ContainerDied","Data":"7612af96ed60f6d1f975588732a6f12635f84ba765166c21aa18c9f0628cfc94"} Oct 30 17:44:05.689956 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:05.689931 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-c7xm2" Oct 30 17:44:05.784645 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:05.784612 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77612860-9c4c-4eac-95c8-c42db1b411e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77612860-9c4c-4eac-95c8-c42db1b411e0-maas-api-openshift-ca\") pod \"77612860-9c4c-4eac-95c8-c42db1b411e0\" (UID: \"77612860-9c4c-4eac-95c8-c42db1b411e0\") " Oct 30 17:44:05.784844 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:05.784680 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77612860-9c4c-4eac-95c8-c42db1b411e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77612860-9c4c-4eac-95c8-c42db1b411e0-maas-api-xks-ca\") pod \"77612860-9c4c-4eac-95c8-c42db1b411e0\" (UID: \"77612860-9c4c-4eac-95c8-c42db1b411e0\") " Oct 30 17:44:05.784844 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:05.784776 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77612860-9c4c-4eac-95c8-c42db1b411e0-kube-api-access-p8dxl\" (UniqueName: \"kubernetes.io/projected/77612860-9c4c-4eac-95c8-c42db1b411e0-kube-api-access-p8dxl\") pod \"77612860-9c4c-4eac-95c8-c42db1b411e0\" (UID: \"77612860-9c4c-4eac-95c8-c42db1b411e0\") " Oct 30 17:44:05.785075 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:05.785050 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77612860-9c4c-4eac-95c8-c42db1b411e0-maas-api-openshift-ca" pod "77612860-9c4c-4eac-95c8-c42db1b411e0" (UID: "77612860-9c4c-4eac-95c8-c42db1b411e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:44:05.786972 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:05.786942 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77612860-9c4c-4eac-95c8-c42db1b411e0-maas-api-xks-ca" pod "77612860-9c4c-4eac-95c8-c42db1b411e0" (UID: "77612860-9c4c-4eac-95c8-c42db1b411e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:44:05.787102 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:05.787089 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77612860-9c4c-4eac-95c8-c42db1b411e0-kube-api-access-p8dxl" pod "77612860-9c4c-4eac-95c8-c42db1b411e0" (UID: "77612860-9c4c-4eac-95c8-c42db1b411e0"). InnerVolumeSpecName "kube-api-access-p8dxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:44:05.886248 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:05.886206 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8dxl\" (UniqueName: \"kubernetes.io/projected/77612860-9c4c-4eac-95c8-c42db1b411e0-kube-api-access-p8dxl\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:44:05.886248 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:05.886239 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77612860-9c4c-4eac-95c8-c42db1b411e0-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:44:05.886248 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:05.886249 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77612860-9c4c-4eac-95c8-c42db1b411e0-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:44:06.344689 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:06.344655 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-c7xm2" Oct 30 17:44:06.344868 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:06.344692 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-c7xm2" event={"ID":"77612860-9c4c-4eac-95c8-c42db1b411e0","Type":"ContainerDied","Data":"4cfd38020cd509fdb7e25fbed217a44f93e4072b863aa757713ad08dbd12f6a9"} Oct 30 17:44:06.344868 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:06.344742 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4cfd38020cd509fdb7e25fbed217a44f93e4072b863aa757713ad08dbd12f6a9" Oct 30 17:44:11.589032 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:11.588995 4522 scope.go:122] "RemoveContainer" containerID="161d1894517498a2c81f823624697cccca7454eae5c550d7a3adb492a08a7f6e" Oct 30 17:44:11.597971 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:11.597949 4522 scope.go:122] "RemoveContainer" containerID="555856a636bd042252bd7fb3eab6ea56ef05ffde618561785f046cb2b7a51a32" Oct 30 17:44:11.606166 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:11.606146 4522 scope.go:122] "RemoveContainer" containerID="90c473fffbbc1486f63cb4c5d77dc271409b5da0395994c7b051963269ba05fb" Oct 30 17:44:12.947025 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:44:12.946991 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:44:23.769088 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:23.769049 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p5k5v"] Oct 30 17:44:23.769760 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:23.769638 4522 cpu_manager.go:414] "containerMap: removing container" podUID="77612860-9c4c-4eac-95c8-c42db1b411e0" containerName="revoke-keys" Oct 30 17:44:23.769760 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:23.769659 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77612860-9c4c-4eac-95c8-c42db1b411e0" containerName="revoke-keys" Oct 30 17:44:23.769896 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:23.769788 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="77612860-9c4c-4eac-95c8-c42db1b411e0" containerName="revoke-keys" Oct 30 17:44:23.774474 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:23.774452 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p5k5v" Oct 30 17:44:23.793128 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:23.793094 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p5k5v"] Oct 30 17:44:23.804847 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:23.804819 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:44:23.805071 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:23.804947 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:44:23.805071 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:23.804967 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:44:23.807166 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:23.807137 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:44:23.893369 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:23.893327 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5s48\" (UniqueName: \"kubernetes.io/projected/92483a74-647f-4a31-8d32-022ddb3bff47-kube-api-access-s5s48\") pod \"maas-api-revoke-sub-ae594979-p5k5v\" (UID: \"92483a74-647f-4a31-8d32-022ddb3bff47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p5k5v" Oct 30 17:44:23.893551 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:23.893414 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92483a74-647f-4a31-8d32-022ddb3bff47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-p5k5v\" (UID: \"92483a74-647f-4a31-8d32-022ddb3bff47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p5k5v" Oct 30 17:44:23.893551 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:23.893474 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92483a74-647f-4a31-8d32-022ddb3bff47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-p5k5v\" (UID: \"92483a74-647f-4a31-8d32-022ddb3bff47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p5k5v" Oct 30 17:44:23.994214 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:23.994175 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5s48\" (UniqueName: \"kubernetes.io/projected/92483a74-647f-4a31-8d32-022ddb3bff47-kube-api-access-s5s48\") pod \"maas-api-revoke-sub-ae594979-p5k5v\" (UID: \"92483a74-647f-4a31-8d32-022ddb3bff47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p5k5v" Oct 30 17:44:23.994214 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:23.994222 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92483a74-647f-4a31-8d32-022ddb3bff47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-p5k5v\" (UID: \"92483a74-647f-4a31-8d32-022ddb3bff47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p5k5v" Oct 30 17:44:23.994508 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:23.994247 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92483a74-647f-4a31-8d32-022ddb3bff47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-p5k5v\" (UID: \"92483a74-647f-4a31-8d32-022ddb3bff47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p5k5v" Oct 30 17:44:23.994936 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:23.994912 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92483a74-647f-4a31-8d32-022ddb3bff47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-p5k5v\" (UID: \"92483a74-647f-4a31-8d32-022ddb3bff47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p5k5v" Oct 30 17:44:23.996670 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:23.996649 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92483a74-647f-4a31-8d32-022ddb3bff47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-p5k5v\" (UID: \"92483a74-647f-4a31-8d32-022ddb3bff47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p5k5v" Oct 30 17:44:24.020039 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:24.019979 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5s48\" (UniqueName: \"kubernetes.io/projected/92483a74-647f-4a31-8d32-022ddb3bff47-kube-api-access-s5s48\") pod \"maas-api-revoke-sub-ae594979-p5k5v\" (UID: \"92483a74-647f-4a31-8d32-022ddb3bff47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p5k5v" Oct 30 17:44:24.091169 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:24.091140 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p5k5v" Oct 30 17:44:24.392684 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:24.392653 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p5k5v"] Oct 30 17:44:24.392836 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:44:24.392702 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92483a74_647f_4a31_8d32_022ddb3bff47.slice/crio-87e454e3a13bc8c7a0da81c8cd6ad5d52b934a455bc814a15fedd2fe4586da56 WatchSource:0}: Error finding container 87e454e3a13bc8c7a0da81c8cd6ad5d52b934a455bc814a15fedd2fe4586da56: Status 404 returned error can't find the container with id 87e454e3a13bc8c7a0da81c8cd6ad5d52b934a455bc814a15fedd2fe4586da56 Oct 30 17:44:24.490169 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:24.490135 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p5k5v" event={"ID":"92483a74-647f-4a31-8d32-022ddb3bff47","Type":"ContainerStarted","Data":"87e454e3a13bc8c7a0da81c8cd6ad5d52b934a455bc814a15fedd2fe4586da56"} Oct 30 17:44:25.497484 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:25.497401 4522 generic.go:356] "Generic (PLEG): container finished" podID="92483a74-647f-4a31-8d32-022ddb3bff47" containerID="44fc1eb5d4f77810c87b45cb54b834630e2abe4ba6500a1ebef66efdd12302dd" exitCode=0 Oct 30 17:44:25.497850 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:25.497486 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p5k5v" event={"ID":"92483a74-647f-4a31-8d32-022ddb3bff47","Type":"ContainerDied","Data":"44fc1eb5d4f77810c87b45cb54b834630e2abe4ba6500a1ebef66efdd12302dd"} Oct 30 17:44:26.635171 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:26.635146 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p5k5v" Oct 30 17:44:26.822472 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:26.822388 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92483a74-647f-4a31-8d32-022ddb3bff47-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92483a74-647f-4a31-8d32-022ddb3bff47-maas-api-openshift-ca\") pod \"92483a74-647f-4a31-8d32-022ddb3bff47\" (UID: \"92483a74-647f-4a31-8d32-022ddb3bff47\") " Oct 30 17:44:26.822642 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:26.822521 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92483a74-647f-4a31-8d32-022ddb3bff47-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92483a74-647f-4a31-8d32-022ddb3bff47-maas-api-xks-ca\") pod \"92483a74-647f-4a31-8d32-022ddb3bff47\" (UID: \"92483a74-647f-4a31-8d32-022ddb3bff47\") " Oct 30 17:44:26.822642 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:26.822588 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92483a74-647f-4a31-8d32-022ddb3bff47-kube-api-access-s5s48\" (UniqueName: \"kubernetes.io/projected/92483a74-647f-4a31-8d32-022ddb3bff47-kube-api-access-s5s48\") pod \"92483a74-647f-4a31-8d32-022ddb3bff47\" (UID: \"92483a74-647f-4a31-8d32-022ddb3bff47\") " Oct 30 17:44:26.822795 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:26.822770 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92483a74-647f-4a31-8d32-022ddb3bff47-maas-api-openshift-ca" pod "92483a74-647f-4a31-8d32-022ddb3bff47" (UID: "92483a74-647f-4a31-8d32-022ddb3bff47"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:44:26.822938 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:26.822920 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92483a74-647f-4a31-8d32-022ddb3bff47-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:44:26.824734 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:26.824708 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92483a74-647f-4a31-8d32-022ddb3bff47-maas-api-xks-ca" pod "92483a74-647f-4a31-8d32-022ddb3bff47" (UID: "92483a74-647f-4a31-8d32-022ddb3bff47"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:44:26.824827 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:26.824771 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92483a74-647f-4a31-8d32-022ddb3bff47-kube-api-access-s5s48" pod "92483a74-647f-4a31-8d32-022ddb3bff47" (UID: "92483a74-647f-4a31-8d32-022ddb3bff47"). InnerVolumeSpecName "kube-api-access-s5s48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:44:26.924268 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:26.924227 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92483a74-647f-4a31-8d32-022ddb3bff47-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:44:26.924268 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:26.924262 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5s48\" (UniqueName: \"kubernetes.io/projected/92483a74-647f-4a31-8d32-022ddb3bff47-kube-api-access-s5s48\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:44:27.509602 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:27.509566 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p5k5v" Oct 30 17:44:27.509797 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:27.509601 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p5k5v" event={"ID":"92483a74-647f-4a31-8d32-022ddb3bff47","Type":"ContainerDied","Data":"87e454e3a13bc8c7a0da81c8cd6ad5d52b934a455bc814a15fedd2fe4586da56"} Oct 30 17:44:27.509797 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:27.509650 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87e454e3a13bc8c7a0da81c8cd6ad5d52b934a455bc814a15fedd2fe4586da56" Oct 30 17:44:42.946732 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:44:42.946692 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:44:45.332348 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:45.332299 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gl5bx"] Oct 30 17:44:45.332776 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:45.332705 4522 cpu_manager.go:414] "containerMap: removing container" podUID="92483a74-647f-4a31-8d32-022ddb3bff47" containerName="revoke-keys" Oct 30 17:44:45.332776 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:45.332720 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92483a74-647f-4a31-8d32-022ddb3bff47" containerName="revoke-keys" Oct 30 17:44:45.332853 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:45.332808 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="92483a74-647f-4a31-8d32-022ddb3bff47" containerName="revoke-keys" Oct 30 17:44:45.335989 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:45.335970 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gl5bx" Oct 30 17:44:45.353946 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:45.353885 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:44:45.354160 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:45.353904 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:44:45.368517 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:45.368473 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:44:45.368517 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:45.368496 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:44:45.372094 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:45.372063 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gl5bx"] Oct 30 17:44:45.421220 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:45.421188 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/302d5493-1efb-4725-bbc1-beeb41f49048-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-gl5bx\" (UID: \"302d5493-1efb-4725-bbc1-beeb41f49048\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gl5bx" Oct 30 17:44:45.421220 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:45.421225 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/302d5493-1efb-4725-bbc1-beeb41f49048-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-gl5bx\" (UID: \"302d5493-1efb-4725-bbc1-beeb41f49048\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gl5bx" Oct 30 17:44:45.421455 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:45.421243 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkrfw\" (UniqueName: \"kubernetes.io/projected/302d5493-1efb-4725-bbc1-beeb41f49048-kube-api-access-nkrfw\") pod \"maas-api-revoke-sub-a4aef7d7-gl5bx\" (UID: \"302d5493-1efb-4725-bbc1-beeb41f49048\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gl5bx" Oct 30 17:44:45.521934 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:45.521900 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/302d5493-1efb-4725-bbc1-beeb41f49048-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-gl5bx\" (UID: \"302d5493-1efb-4725-bbc1-beeb41f49048\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gl5bx" Oct 30 17:44:45.521934 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:45.521941 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/302d5493-1efb-4725-bbc1-beeb41f49048-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-gl5bx\" (UID: \"302d5493-1efb-4725-bbc1-beeb41f49048\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gl5bx" Oct 30 17:44:45.522153 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:45.521963 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkrfw\" (UniqueName: \"kubernetes.io/projected/302d5493-1efb-4725-bbc1-beeb41f49048-kube-api-access-nkrfw\") pod \"maas-api-revoke-sub-a4aef7d7-gl5bx\" (UID: \"302d5493-1efb-4725-bbc1-beeb41f49048\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gl5bx" Oct 30 17:44:45.522695 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:45.522672 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/302d5493-1efb-4725-bbc1-beeb41f49048-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-gl5bx\" (UID: \"302d5493-1efb-4725-bbc1-beeb41f49048\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gl5bx" Oct 30 17:44:45.524387 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:45.524360 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/302d5493-1efb-4725-bbc1-beeb41f49048-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-gl5bx\" (UID: \"302d5493-1efb-4725-bbc1-beeb41f49048\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gl5bx" Oct 30 17:44:45.546826 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:45.546795 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkrfw\" (UniqueName: \"kubernetes.io/projected/302d5493-1efb-4725-bbc1-beeb41f49048-kube-api-access-nkrfw\") pod \"maas-api-revoke-sub-a4aef7d7-gl5bx\" (UID: \"302d5493-1efb-4725-bbc1-beeb41f49048\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gl5bx" Oct 30 17:44:45.648768 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:45.648691 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gl5bx" Oct 30 17:44:45.954759 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:44:45.954727 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod302d5493_1efb_4725_bbc1_beeb41f49048.slice/crio-91271e0c11924d3ff522de6acbdd27a7b4ba16ed27d2f2e4c41071a52f685df6 WatchSource:0}: Error finding container 91271e0c11924d3ff522de6acbdd27a7b4ba16ed27d2f2e4c41071a52f685df6: Status 404 returned error can't find the container with id 91271e0c11924d3ff522de6acbdd27a7b4ba16ed27d2f2e4c41071a52f685df6 Oct 30 17:44:45.954879 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:45.954789 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gl5bx"] Oct 30 17:44:46.612819 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:46.612731 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gl5bx" event={"ID":"302d5493-1efb-4725-bbc1-beeb41f49048","Type":"ContainerStarted","Data":"a48e48c7ea1b212354a05a03ac6553a072ad27d03e5ae0859d7e7bf9ef20b700"} Oct 30 17:44:46.612819 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:46.612786 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gl5bx" event={"ID":"302d5493-1efb-4725-bbc1-beeb41f49048","Type":"ContainerStarted","Data":"91271e0c11924d3ff522de6acbdd27a7b4ba16ed27d2f2e4c41071a52f685df6"} Oct 30 17:44:46.663114 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:46.663057 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gl5bx" podStartSLOduration=1.663043987 podStartE2EDuration="1.663043987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:44:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:44:46.660677374 +0000 UTC m=+3460.661432276" watchObservedRunningTime="2026-10-30 17:44:46.663043987 +0000 UTC m=+3460.663798887" Oct 30 17:44:47.618832 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:47.618799 4522 generic.go:356] "Generic (PLEG): container finished" podID="302d5493-1efb-4725-bbc1-beeb41f49048" containerID="a48e48c7ea1b212354a05a03ac6553a072ad27d03e5ae0859d7e7bf9ef20b700" exitCode=0 Oct 30 17:44:47.619236 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:47.618880 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gl5bx" event={"ID":"302d5493-1efb-4725-bbc1-beeb41f49048","Type":"ContainerDied","Data":"a48e48c7ea1b212354a05a03ac6553a072ad27d03e5ae0859d7e7bf9ef20b700"} Oct 30 17:44:48.757961 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:48.757935 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gl5bx" Oct 30 17:44:48.854859 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:48.854821 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/302d5493-1efb-4725-bbc1-beeb41f49048-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/302d5493-1efb-4725-bbc1-beeb41f49048-maas-api-xks-ca\") pod \"302d5493-1efb-4725-bbc1-beeb41f49048\" (UID: \"302d5493-1efb-4725-bbc1-beeb41f49048\") " Oct 30 17:44:48.855039 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:48.854958 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/302d5493-1efb-4725-bbc1-beeb41f49048-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/302d5493-1efb-4725-bbc1-beeb41f49048-maas-api-openshift-ca\") pod \"302d5493-1efb-4725-bbc1-beeb41f49048\" (UID: \"302d5493-1efb-4725-bbc1-beeb41f49048\") " Oct 30 17:44:48.855039 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:48.854997 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/302d5493-1efb-4725-bbc1-beeb41f49048-kube-api-access-nkrfw\" (UniqueName: \"kubernetes.io/projected/302d5493-1efb-4725-bbc1-beeb41f49048-kube-api-access-nkrfw\") pod \"302d5493-1efb-4725-bbc1-beeb41f49048\" (UID: \"302d5493-1efb-4725-bbc1-beeb41f49048\") " Oct 30 17:44:48.855385 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:48.855354 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/302d5493-1efb-4725-bbc1-beeb41f49048-maas-api-openshift-ca" pod "302d5493-1efb-4725-bbc1-beeb41f49048" (UID: "302d5493-1efb-4725-bbc1-beeb41f49048"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:44:48.857076 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:48.857048 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/302d5493-1efb-4725-bbc1-beeb41f49048-maas-api-xks-ca" pod "302d5493-1efb-4725-bbc1-beeb41f49048" (UID: "302d5493-1efb-4725-bbc1-beeb41f49048"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:44:48.857191 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:48.857099 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/302d5493-1efb-4725-bbc1-beeb41f49048-kube-api-access-nkrfw" pod "302d5493-1efb-4725-bbc1-beeb41f49048" (UID: "302d5493-1efb-4725-bbc1-beeb41f49048"). InnerVolumeSpecName "kube-api-access-nkrfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:44:48.955801 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:48.955769 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/302d5493-1efb-4725-bbc1-beeb41f49048-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:44:48.955801 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:48.955799 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/302d5493-1efb-4725-bbc1-beeb41f49048-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:44:48.955998 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:48.955810 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkrfw\" (UniqueName: \"kubernetes.io/projected/302d5493-1efb-4725-bbc1-beeb41f49048-kube-api-access-nkrfw\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:44:49.629465 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:49.629434 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gl5bx" Oct 30 17:44:49.629630 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:49.629470 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gl5bx" event={"ID":"302d5493-1efb-4725-bbc1-beeb41f49048","Type":"ContainerDied","Data":"91271e0c11924d3ff522de6acbdd27a7b4ba16ed27d2f2e4c41071a52f685df6"} Oct 30 17:44:49.629630 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:44:49.629518 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91271e0c11924d3ff522de6acbdd27a7b4ba16ed27d2f2e4c41071a52f685df6" Oct 30 17:45:00.145671 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:00.145591 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889705-kt24j"] Oct 30 17:45:00.146089 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:00.145959 4522 cpu_manager.go:414] "containerMap: removing container" podUID="302d5493-1efb-4725-bbc1-beeb41f49048" containerName="revoke-keys" Oct 30 17:45:00.146089 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:00.145971 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="302d5493-1efb-4725-bbc1-beeb41f49048" containerName="revoke-keys" Oct 30 17:45:00.146089 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:00.146047 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="302d5493-1efb-4725-bbc1-beeb41f49048" containerName="revoke-keys" Oct 30 17:45:00.149123 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:00.149105 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889705-kt24j" Oct 30 17:45:00.181613 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:00.181476 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:45:00.181613 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:00.181568 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:45:00.181871 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:00.181722 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:45:00.183896 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:00.183870 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889705-kt24j"] Oct 30 17:45:00.263735 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:00.263680 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15227e43-3a4c-49d5-a026-d741b2e69165-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889705-kt24j\" (UID: \"15227e43-3a4c-49d5-a026-d741b2e69165\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889705-kt24j" Oct 30 17:45:00.263735 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:00.263739 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpwm7\" (UniqueName: \"kubernetes.io/projected/15227e43-3a4c-49d5-a026-d741b2e69165-kube-api-access-lpwm7\") pod \"maas-api-key-cleanup-29889705-kt24j\" (UID: \"15227e43-3a4c-49d5-a026-d741b2e69165\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889705-kt24j" Oct 30 17:45:00.365002 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:00.364947 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15227e43-3a4c-49d5-a026-d741b2e69165-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889705-kt24j\" (UID: \"15227e43-3a4c-49d5-a026-d741b2e69165\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889705-kt24j" Oct 30 17:45:00.365002 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:00.365008 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpwm7\" (UniqueName: \"kubernetes.io/projected/15227e43-3a4c-49d5-a026-d741b2e69165-kube-api-access-lpwm7\") pod \"maas-api-key-cleanup-29889705-kt24j\" (UID: \"15227e43-3a4c-49d5-a026-d741b2e69165\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889705-kt24j" Oct 30 17:45:00.365658 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:00.365637 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15227e43-3a4c-49d5-a026-d741b2e69165-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889705-kt24j\" (UID: \"15227e43-3a4c-49d5-a026-d741b2e69165\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889705-kt24j" Oct 30 17:45:00.389430 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:00.389400 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpwm7\" (UniqueName: \"kubernetes.io/projected/15227e43-3a4c-49d5-a026-d741b2e69165-kube-api-access-lpwm7\") pod \"maas-api-key-cleanup-29889705-kt24j\" (UID: \"15227e43-3a4c-49d5-a026-d741b2e69165\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889705-kt24j" Oct 30 17:45:00.462206 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:00.462171 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889705-kt24j" Oct 30 17:45:00.772360 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:45:00.772327 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15227e43_3a4c_49d5_a026_d741b2e69165.slice/crio-06b3da73fb3abd75dd494079228f6a6186b92479341dbf9bb5ebb87222889aa4 WatchSource:0}: Error finding container 06b3da73fb3abd75dd494079228f6a6186b92479341dbf9bb5ebb87222889aa4: Status 404 returned error can't find the container with id 06b3da73fb3abd75dd494079228f6a6186b92479341dbf9bb5ebb87222889aa4 Oct 30 17:45:00.772704 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:00.772680 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889705-kt24j"] Oct 30 17:45:01.709579 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:01.706427 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889705-kt24j" event={"ID":"15227e43-3a4c-49d5-a026-d741b2e69165","Type":"ContainerStarted","Data":"55dd2ef2e8b45989af2ad1ea212455fa82b9ad1851cc676b5f0b10a4f8be51e0"} Oct 30 17:45:01.709579 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:01.706495 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889705-kt24j" event={"ID":"15227e43-3a4c-49d5-a026-d741b2e69165","Type":"ContainerStarted","Data":"06b3da73fb3abd75dd494079228f6a6186b92479341dbf9bb5ebb87222889aa4"} Oct 30 17:45:01.750575 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:01.750516 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889705-kt24j" podStartSLOduration=1.7505010410000001 podStartE2EDuration="1.750501041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:45:01.749241895 +0000 UTC m=+3475.749996817" watchObservedRunningTime="2026-10-30 17:45:01.750501041 +0000 UTC m=+3475.751255946" Oct 30 17:45:02.739211 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:02.739156 4522 generic.go:356] "Generic (PLEG): container finished" podID="15227e43-3a4c-49d5-a026-d741b2e69165" containerID="55dd2ef2e8b45989af2ad1ea212455fa82b9ad1851cc676b5f0b10a4f8be51e0" exitCode=60 Oct 30 17:45:02.739985 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:02.739947 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889705-kt24j" event={"ID":"15227e43-3a4c-49d5-a026-d741b2e69165","Type":"ContainerDied","Data":"55dd2ef2e8b45989af2ad1ea212455fa82b9ad1851cc676b5f0b10a4f8be51e0"} Oct 30 17:45:02.740747 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:02.740728 4522 scope.go:122] "RemoveContainer" containerID="55dd2ef2e8b45989af2ad1ea212455fa82b9ad1851cc676b5f0b10a4f8be51e0" Oct 30 17:45:03.773119 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:03.773077 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889705-kt24j" event={"ID":"15227e43-3a4c-49d5-a026-d741b2e69165","Type":"ContainerStarted","Data":"e1ae51211eb3c1ac86d7938187b6f7bdf5698b5900d54e3433fe6062116a10cd"} Oct 30 17:45:04.779639 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:04.779604 4522 generic.go:356] "Generic (PLEG): container finished" podID="15227e43-3a4c-49d5-a026-d741b2e69165" containerID="e1ae51211eb3c1ac86d7938187b6f7bdf5698b5900d54e3433fe6062116a10cd" exitCode=60 Oct 30 17:45:04.780142 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:04.779697 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889705-kt24j" event={"ID":"15227e43-3a4c-49d5-a026-d741b2e69165","Type":"ContainerDied","Data":"e1ae51211eb3c1ac86d7938187b6f7bdf5698b5900d54e3433fe6062116a10cd"} Oct 30 17:45:04.780142 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:04.779767 4522 scope.go:122] "RemoveContainer" containerID="55dd2ef2e8b45989af2ad1ea212455fa82b9ad1851cc676b5f0b10a4f8be51e0" Oct 30 17:45:04.780142 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:04.779989 4522 scope.go:122] "RemoveContainer" containerID="e1ae51211eb3c1ac86d7938187b6f7bdf5698b5900d54e3433fe6062116a10cd" Oct 30 17:45:04.780332 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:45:04.780238 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889705-kt24j_odh-ai-gateway-infra(15227e43-3a4c-49d5-a026-d741b2e69165)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889705-kt24j" podUID="15227e43-3a4c-49d5-a026-d741b2e69165" Oct 30 17:45:05.787278 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:05.787251 4522 scope.go:122] "RemoveContainer" containerID="e1ae51211eb3c1ac86d7938187b6f7bdf5698b5900d54e3433fe6062116a10cd" Oct 30 17:45:05.787710 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:45:05.787493 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889705-kt24j_odh-ai-gateway-infra(15227e43-3a4c-49d5-a026-d741b2e69165)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889705-kt24j" podUID="15227e43-3a4c-49d5-a026-d741b2e69165" Oct 30 17:45:11.634174 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:11.634142 4522 scope.go:122] "RemoveContainer" containerID="3b62497adc43b67740b7180160088bb8e64027da57fe2c3936a3acb514504366" Oct 30 17:45:11.643177 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:11.643155 4522 scope.go:122] "RemoveContainer" containerID="fb25fd622a003cb825a415fa30d9ccc0fb9a0052fce1f3eafebf76e55ee27112" Oct 30 17:45:11.651606 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:11.651588 4522 scope.go:122] "RemoveContainer" containerID="95902780cf4b2b8bb8a8a8f94487cfa8f3b0c4e5da332faae4fb502e7efbe5c1" Oct 30 17:45:11.660421 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:11.660398 4522 scope.go:122] "RemoveContainer" containerID="02debccec32f9af2cc35d5f20849b899e16491db40cdb4980d66cc4bb41a0ca2" Oct 30 17:45:11.668519 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:11.668497 4522 scope.go:122] "RemoveContainer" containerID="c88856ddce22cc0a772dc518b80070d64c925d9fad1371e345563ddd74961fab" Oct 30 17:45:11.676166 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:11.676150 4522 scope.go:122] "RemoveContainer" containerID="82c71da55d9bb43a180922abadad2ccfb26c105bc2f347cd5c857a00c673612d" Oct 30 17:45:12.947887 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:45:12.947852 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:45:17.138061 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:17.138021 4522 scope.go:122] "RemoveContainer" containerID="e1ae51211eb3c1ac86d7938187b6f7bdf5698b5900d54e3433fe6062116a10cd" Oct 30 17:45:17.850580 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:17.850541 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889705-kt24j" event={"ID":"15227e43-3a4c-49d5-a026-d741b2e69165","Type":"ContainerStarted","Data":"756393595af71ef2ebb752ec5df03f7ca43f05ab27e85a99e3c7bdd097ede1ef"} Oct 30 17:45:18.859914 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:18.859864 4522 generic.go:356] "Generic (PLEG): container finished" podID="15227e43-3a4c-49d5-a026-d741b2e69165" containerID="756393595af71ef2ebb752ec5df03f7ca43f05ab27e85a99e3c7bdd097ede1ef" exitCode=60 Oct 30 17:45:18.860394 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:18.859948 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889705-kt24j" event={"ID":"15227e43-3a4c-49d5-a026-d741b2e69165","Type":"ContainerDied","Data":"756393595af71ef2ebb752ec5df03f7ca43f05ab27e85a99e3c7bdd097ede1ef"} Oct 30 17:45:18.860394 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:18.860030 4522 scope.go:122] "RemoveContainer" containerID="e1ae51211eb3c1ac86d7938187b6f7bdf5698b5900d54e3433fe6062116a10cd" Oct 30 17:45:18.860547 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:18.860449 4522 scope.go:122] "RemoveContainer" containerID="756393595af71ef2ebb752ec5df03f7ca43f05ab27e85a99e3c7bdd097ede1ef" Oct 30 17:45:18.860801 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:45:18.860778 4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889705-kt24j_odh-ai-gateway-infra(15227e43-3a4c-49d5-a026-d741b2e69165)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889705-kt24j" podUID="15227e43-3a4c-49d5-a026-d741b2e69165" Oct 30 17:45:18.907799 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:18.907758 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889705-kt24j"] Oct 30 17:45:20.203328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:20.200666 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889705-kt24j" Oct 30 17:45:20.306781 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:20.306738 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15227e43-3a4c-49d5-a026-d741b2e69165-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15227e43-3a4c-49d5-a026-d741b2e69165-maas-api-service-ca\") pod \"15227e43-3a4c-49d5-a026-d741b2e69165\" (UID: \"15227e43-3a4c-49d5-a026-d741b2e69165\") " Oct 30 17:45:20.307099 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:20.307077 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15227e43-3a4c-49d5-a026-d741b2e69165-kube-api-access-lpwm7\" (UniqueName: \"kubernetes.io/projected/15227e43-3a4c-49d5-a026-d741b2e69165-kube-api-access-lpwm7\") pod \"15227e43-3a4c-49d5-a026-d741b2e69165\" (UID: \"15227e43-3a4c-49d5-a026-d741b2e69165\") " Oct 30 17:45:20.308403 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:20.308379 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15227e43-3a4c-49d5-a026-d741b2e69165-maas-api-service-ca" pod "15227e43-3a4c-49d5-a026-d741b2e69165" (UID: "15227e43-3a4c-49d5-a026-d741b2e69165"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:45:20.358581 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:20.358538 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15227e43-3a4c-49d5-a026-d741b2e69165-kube-api-access-lpwm7" pod "15227e43-3a4c-49d5-a026-d741b2e69165" (UID: "15227e43-3a4c-49d5-a026-d741b2e69165"). InnerVolumeSpecName "kube-api-access-lpwm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:45:20.416136 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:20.416094 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpwm7\" (UniqueName: \"kubernetes.io/projected/15227e43-3a4c-49d5-a026-d741b2e69165-kube-api-access-lpwm7\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:45:20.430510 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:20.430426 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15227e43-3a4c-49d5-a026-d741b2e69165-maas-api-service-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:45:20.880185 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:20.880149 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889705-kt24j" Oct 30 17:45:20.880185 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:20.880164 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889705-kt24j" event={"ID":"15227e43-3a4c-49d5-a026-d741b2e69165","Type":"ContainerDied","Data":"06b3da73fb3abd75dd494079228f6a6186b92479341dbf9bb5ebb87222889aa4"} Oct 30 17:45:20.880459 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:20.880279 4522 scope.go:122] "RemoveContainer" containerID="756393595af71ef2ebb752ec5df03f7ca43f05ab27e85a99e3c7bdd097ede1ef" Oct 30 17:45:20.944402 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:20.944362 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889705-kt24j"] Oct 30 17:45:20.958813 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:20.958780 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889705-kt24j"] Oct 30 17:45:22.143357 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:22.143325 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15227e43-3a4c-49d5-a026-d741b2e69165" path="/var/lib/kubelet/pods/15227e43-3a4c-49d5-a026-d741b2e69165/volumes" Oct 30 17:45:22.467575 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:22.467544 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ms2k9"] Oct 30 17:45:22.467931 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:22.467919 4522 cpu_manager.go:414] "containerMap: removing container" podUID="15227e43-3a4c-49d5-a026-d741b2e69165" containerName="cleanup" Oct 30 17:45:22.467984 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:22.467932 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15227e43-3a4c-49d5-a026-d741b2e69165" containerName="cleanup" Oct 30 17:45:22.467984 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:22.467942 4522 cpu_manager.go:414] "containerMap: removing container" podUID="15227e43-3a4c-49d5-a026-d741b2e69165" containerName="cleanup" Oct 30 17:45:22.467984 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:22.467947 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15227e43-3a4c-49d5-a026-d741b2e69165" containerName="cleanup" Oct 30 17:45:22.467984 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:22.467965 4522 cpu_manager.go:414] "containerMap: removing container" podUID="15227e43-3a4c-49d5-a026-d741b2e69165" containerName="cleanup" Oct 30 17:45:22.467984 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:22.467970 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15227e43-3a4c-49d5-a026-d741b2e69165" containerName="cleanup" Oct 30 17:45:22.468151 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:22.468033 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="15227e43-3a4c-49d5-a026-d741b2e69165" containerName="cleanup" Oct 30 17:45:22.468151 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:22.468041 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="15227e43-3a4c-49d5-a026-d741b2e69165" containerName="cleanup" Oct 30 17:45:22.468151 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:22.468049 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="15227e43-3a4c-49d5-a026-d741b2e69165" containerName="cleanup" Oct 30 17:45:22.472768 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:22.472751 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ms2k9" Oct 30 17:45:22.490533 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:22.490507 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:45:22.490685 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:22.490553 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:45:22.492443 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:22.492421 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ms2k9"] Oct 30 17:45:22.504042 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:22.504018 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:45:22.504743 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:22.504728 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:45:22.579074 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:22.579040 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/343b8093-c8a1-4dd2-b187-731295b0bc89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-ms2k9\" (UID: \"343b8093-c8a1-4dd2-b187-731295b0bc89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ms2k9" Oct 30 17:45:22.579265 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:22.579189 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j5j7\" (UniqueName: \"kubernetes.io/projected/343b8093-c8a1-4dd2-b187-731295b0bc89-kube-api-access-2j5j7\") pod \"maas-api-revoke-sub-0b7e8edb-ms2k9\" (UID: \"343b8093-c8a1-4dd2-b187-731295b0bc89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ms2k9" Oct 30 17:45:22.579265 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:22.579217 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/343b8093-c8a1-4dd2-b187-731295b0bc89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-ms2k9\" (UID: \"343b8093-c8a1-4dd2-b187-731295b0bc89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ms2k9" Oct 30 17:45:22.679739 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:22.679705 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/343b8093-c8a1-4dd2-b187-731295b0bc89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-ms2k9\" (UID: \"343b8093-c8a1-4dd2-b187-731295b0bc89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ms2k9" Oct 30 17:45:22.679944 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:22.679810 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j5j7\" (UniqueName: \"kubernetes.io/projected/343b8093-c8a1-4dd2-b187-731295b0bc89-kube-api-access-2j5j7\") pod \"maas-api-revoke-sub-0b7e8edb-ms2k9\" (UID: \"343b8093-c8a1-4dd2-b187-731295b0bc89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ms2k9" Oct 30 17:45:22.679944 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:22.679834 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/343b8093-c8a1-4dd2-b187-731295b0bc89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-ms2k9\" (UID: \"343b8093-c8a1-4dd2-b187-731295b0bc89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ms2k9" Oct 30 17:45:22.680503 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:22.680466 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/343b8093-c8a1-4dd2-b187-731295b0bc89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-ms2k9\" (UID: \"343b8093-c8a1-4dd2-b187-731295b0bc89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ms2k9" Oct 30 17:45:22.682150 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:22.682125 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/343b8093-c8a1-4dd2-b187-731295b0bc89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-ms2k9\" (UID: \"343b8093-c8a1-4dd2-b187-731295b0bc89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ms2k9" Oct 30 17:45:22.705714 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:22.705687 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j5j7\" (UniqueName: \"kubernetes.io/projected/343b8093-c8a1-4dd2-b187-731295b0bc89-kube-api-access-2j5j7\") pod \"maas-api-revoke-sub-0b7e8edb-ms2k9\" (UID: \"343b8093-c8a1-4dd2-b187-731295b0bc89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ms2k9" Oct 30 17:45:22.785054 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:22.784968 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ms2k9" Oct 30 17:45:23.084339 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:45:23.084280 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod343b8093_c8a1_4dd2_b187_731295b0bc89.slice/crio-4487db910f36f84baec0403232f0c29b9d21f00b2370794a567adc3b23d788d0 WatchSource:0}: Error finding container 4487db910f36f84baec0403232f0c29b9d21f00b2370794a567adc3b23d788d0: Status 404 returned error can't find the container with id 4487db910f36f84baec0403232f0c29b9d21f00b2370794a567adc3b23d788d0 Oct 30 17:45:23.084725 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:23.084701 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ms2k9"] Oct 30 17:45:23.899318 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:23.899270 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ms2k9" event={"ID":"343b8093-c8a1-4dd2-b187-731295b0bc89","Type":"ContainerStarted","Data":"deaad1d15800d0dd63f9a2a0d350a921c42e910a3780f779a5812c113984ef7d"} Oct 30 17:45:23.899797 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:23.899341 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ms2k9" event={"ID":"343b8093-c8a1-4dd2-b187-731295b0bc89","Type":"ContainerStarted","Data":"4487db910f36f84baec0403232f0c29b9d21f00b2370794a567adc3b23d788d0"} Oct 30 17:45:23.945895 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:23.945842 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ms2k9" podStartSLOduration=1.945827535 podStartE2EDuration="1.945827535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:45:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:45:23.944578503 +0000 UTC m=+3497.945333417" watchObservedRunningTime="2026-10-30 17:45:23.945827535 +0000 UTC m=+3497.946582434" Oct 30 17:45:24.905213 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:24.905179 4522 generic.go:356] "Generic (PLEG): container finished" podID="343b8093-c8a1-4dd2-b187-731295b0bc89" containerID="deaad1d15800d0dd63f9a2a0d350a921c42e910a3780f779a5812c113984ef7d" exitCode=0 Oct 30 17:45:24.905649 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:24.905267 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ms2k9" event={"ID":"343b8093-c8a1-4dd2-b187-731295b0bc89","Type":"ContainerDied","Data":"deaad1d15800d0dd63f9a2a0d350a921c42e910a3780f779a5812c113984ef7d"} Oct 30 17:45:26.044524 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:26.044496 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ms2k9" Oct 30 17:45:26.212285 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:26.212248 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/343b8093-c8a1-4dd2-b187-731295b0bc89-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/343b8093-c8a1-4dd2-b187-731295b0bc89-maas-api-openshift-ca\") pod \"343b8093-c8a1-4dd2-b187-731295b0bc89\" (UID: \"343b8093-c8a1-4dd2-b187-731295b0bc89\") " Oct 30 17:45:26.212519 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:26.212366 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/343b8093-c8a1-4dd2-b187-731295b0bc89-kube-api-access-2j5j7\" (UniqueName: \"kubernetes.io/projected/343b8093-c8a1-4dd2-b187-731295b0bc89-kube-api-access-2j5j7\") pod \"343b8093-c8a1-4dd2-b187-731295b0bc89\" (UID: \"343b8093-c8a1-4dd2-b187-731295b0bc89\") " Oct 30 17:45:26.212519 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:26.212409 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/343b8093-c8a1-4dd2-b187-731295b0bc89-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/343b8093-c8a1-4dd2-b187-731295b0bc89-maas-api-xks-ca\") pod \"343b8093-c8a1-4dd2-b187-731295b0bc89\" (UID: \"343b8093-c8a1-4dd2-b187-731295b0bc89\") " Oct 30 17:45:26.212713 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:26.212689 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/343b8093-c8a1-4dd2-b187-731295b0bc89-maas-api-openshift-ca" pod "343b8093-c8a1-4dd2-b187-731295b0bc89" (UID: "343b8093-c8a1-4dd2-b187-731295b0bc89"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:45:26.214520 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:26.214493 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/343b8093-c8a1-4dd2-b187-731295b0bc89-maas-api-xks-ca" pod "343b8093-c8a1-4dd2-b187-731295b0bc89" (UID: "343b8093-c8a1-4dd2-b187-731295b0bc89"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:45:26.214520 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:26.214492 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/343b8093-c8a1-4dd2-b187-731295b0bc89-kube-api-access-2j5j7" pod "343b8093-c8a1-4dd2-b187-731295b0bc89" (UID: "343b8093-c8a1-4dd2-b187-731295b0bc89"). InnerVolumeSpecName "kube-api-access-2j5j7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:45:26.313126 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:26.313086 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/343b8093-c8a1-4dd2-b187-731295b0bc89-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:45:26.313126 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:26.313118 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2j5j7\" (UniqueName: \"kubernetes.io/projected/343b8093-c8a1-4dd2-b187-731295b0bc89-kube-api-access-2j5j7\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:45:26.313126 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:26.313130 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/343b8093-c8a1-4dd2-b187-731295b0bc89-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:45:26.916710 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:26.916671 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ms2k9" event={"ID":"343b8093-c8a1-4dd2-b187-731295b0bc89","Type":"ContainerDied","Data":"4487db910f36f84baec0403232f0c29b9d21f00b2370794a567adc3b23d788d0"} Oct 30 17:45:26.916710 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:26.916693 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ms2k9" Oct 30 17:45:26.916981 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:26.916721 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4487db910f36f84baec0403232f0c29b9d21f00b2370794a567adc3b23d788d0" Oct 30 17:45:33.237369 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:33.237330 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zh64d"] Oct 30 17:45:33.237939 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:33.237728 4522 cpu_manager.go:414] "containerMap: removing container" podUID="343b8093-c8a1-4dd2-b187-731295b0bc89" containerName="revoke-keys" Oct 30 17:45:33.237939 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:33.237740 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="343b8093-c8a1-4dd2-b187-731295b0bc89" containerName="revoke-keys" Oct 30 17:45:33.237939 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:33.237837 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="343b8093-c8a1-4dd2-b187-731295b0bc89" containerName="revoke-keys" Oct 30 17:45:33.241382 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:33.241361 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zh64d" Oct 30 17:45:33.262919 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:33.262884 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zh64d"] Oct 30 17:45:33.274856 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:33.274823 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:45:33.274856 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:33.274853 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:45:33.279637 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:33.279609 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4002e5f-5f10-450b-8289-4437e6d16511-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-zh64d\" (UID: \"d4002e5f-5f10-450b-8289-4437e6d16511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zh64d" Oct 30 17:45:33.279803 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:33.279642 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwvzw\" (UniqueName: \"kubernetes.io/projected/d4002e5f-5f10-450b-8289-4437e6d16511-kube-api-access-mwvzw\") pod \"maas-api-revoke-sub-3d4156f1-zh64d\" (UID: \"d4002e5f-5f10-450b-8289-4437e6d16511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zh64d" Oct 30 17:45:33.279803 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:33.279680 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4002e5f-5f10-450b-8289-4437e6d16511-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-zh64d\" (UID: \"d4002e5f-5f10-450b-8289-4437e6d16511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zh64d" Oct 30 17:45:33.350795 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:33.350760 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:45:33.350795 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:33.350780 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:45:33.380371 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:33.380334 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4002e5f-5f10-450b-8289-4437e6d16511-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-zh64d\" (UID: \"d4002e5f-5f10-450b-8289-4437e6d16511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zh64d" Oct 30 17:45:33.380371 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:33.380374 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwvzw\" (UniqueName: \"kubernetes.io/projected/d4002e5f-5f10-450b-8289-4437e6d16511-kube-api-access-mwvzw\") pod \"maas-api-revoke-sub-3d4156f1-zh64d\" (UID: \"d4002e5f-5f10-450b-8289-4437e6d16511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zh64d" Oct 30 17:45:33.380570 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:33.380408 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4002e5f-5f10-450b-8289-4437e6d16511-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-zh64d\" (UID: \"d4002e5f-5f10-450b-8289-4437e6d16511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zh64d" Oct 30 17:45:33.381079 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:33.381055 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4002e5f-5f10-450b-8289-4437e6d16511-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-zh64d\" (UID: \"d4002e5f-5f10-450b-8289-4437e6d16511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zh64d" Oct 30 17:45:33.382734 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:33.382714 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4002e5f-5f10-450b-8289-4437e6d16511-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-zh64d\" (UID: \"d4002e5f-5f10-450b-8289-4437e6d16511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zh64d" Oct 30 17:45:33.407437 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:33.407408 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwvzw\" (UniqueName: \"kubernetes.io/projected/d4002e5f-5f10-450b-8289-4437e6d16511-kube-api-access-mwvzw\") pod \"maas-api-revoke-sub-3d4156f1-zh64d\" (UID: \"d4002e5f-5f10-450b-8289-4437e6d16511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zh64d" Oct 30 17:45:33.554063 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:33.553981 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zh64d" Oct 30 17:45:33.861622 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:33.861592 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zh64d"] Oct 30 17:45:33.862238 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:45:33.862212 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4002e5f_5f10_450b_8289_4437e6d16511.slice/crio-36e5ca44a6137ff4a90144e70248a9bd7bea5349c6194bd39a4fccbd26bab197 WatchSource:0}: Error finding container 36e5ca44a6137ff4a90144e70248a9bd7bea5349c6194bd39a4fccbd26bab197: Status 404 returned error can't find the container with id 36e5ca44a6137ff4a90144e70248a9bd7bea5349c6194bd39a4fccbd26bab197 Oct 30 17:45:33.955118 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:33.955080 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zh64d" event={"ID":"d4002e5f-5f10-450b-8289-4437e6d16511","Type":"ContainerStarted","Data":"36e5ca44a6137ff4a90144e70248a9bd7bea5349c6194bd39a4fccbd26bab197"} Oct 30 17:45:34.961676 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:34.961640 4522 generic.go:356] "Generic (PLEG): container finished" podID="d4002e5f-5f10-450b-8289-4437e6d16511" containerID="bac1cd701e4a86011a433d26dae4ed27a610c94a80526d7dc19ac15b7ac2e792" exitCode=0 Oct 30 17:45:34.962064 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:34.961725 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zh64d" event={"ID":"d4002e5f-5f10-450b-8289-4437e6d16511","Type":"ContainerDied","Data":"bac1cd701e4a86011a433d26dae4ed27a610c94a80526d7dc19ac15b7ac2e792"} Oct 30 17:45:36.098706 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:36.098675 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zh64d" Oct 30 17:45:36.104637 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:36.104614 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4002e5f-5f10-450b-8289-4437e6d16511-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4002e5f-5f10-450b-8289-4437e6d16511-maas-api-openshift-ca\") pod \"d4002e5f-5f10-450b-8289-4437e6d16511\" (UID: \"d4002e5f-5f10-450b-8289-4437e6d16511\") " Oct 30 17:45:36.104722 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:36.104667 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4002e5f-5f10-450b-8289-4437e6d16511-kube-api-access-mwvzw\" (UniqueName: \"kubernetes.io/projected/d4002e5f-5f10-450b-8289-4437e6d16511-kube-api-access-mwvzw\") pod \"d4002e5f-5f10-450b-8289-4437e6d16511\" (UID: \"d4002e5f-5f10-450b-8289-4437e6d16511\") " Oct 30 17:45:36.104766 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:36.104745 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4002e5f-5f10-450b-8289-4437e6d16511-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4002e5f-5f10-450b-8289-4437e6d16511-maas-api-xks-ca\") pod \"d4002e5f-5f10-450b-8289-4437e6d16511\" (UID: \"d4002e5f-5f10-450b-8289-4437e6d16511\") " Oct 30 17:45:36.105002 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:36.104975 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4002e5f-5f10-450b-8289-4437e6d16511-maas-api-openshift-ca" pod "d4002e5f-5f10-450b-8289-4437e6d16511" (UID: "d4002e5f-5f10-450b-8289-4437e6d16511"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:45:36.106728 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:36.106703 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4002e5f-5f10-450b-8289-4437e6d16511-maas-api-xks-ca" pod "d4002e5f-5f10-450b-8289-4437e6d16511" (UID: "d4002e5f-5f10-450b-8289-4437e6d16511"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:45:36.106846 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:36.106733 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4002e5f-5f10-450b-8289-4437e6d16511-kube-api-access-mwvzw" pod "d4002e5f-5f10-450b-8289-4437e6d16511" (UID: "d4002e5f-5f10-450b-8289-4437e6d16511"). InnerVolumeSpecName "kube-api-access-mwvzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:45:36.205812 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:36.205771 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4002e5f-5f10-450b-8289-4437e6d16511-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:45:36.205812 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:36.205809 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwvzw\" (UniqueName: \"kubernetes.io/projected/d4002e5f-5f10-450b-8289-4437e6d16511-kube-api-access-mwvzw\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:45:36.206025 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:36.205823 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4002e5f-5f10-450b-8289-4437e6d16511-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:45:36.973634 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:36.973606 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zh64d" Oct 30 17:45:36.973827 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:36.973640 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zh64d" event={"ID":"d4002e5f-5f10-450b-8289-4437e6d16511","Type":"ContainerDied","Data":"36e5ca44a6137ff4a90144e70248a9bd7bea5349c6194bd39a4fccbd26bab197"} Oct 30 17:45:36.973827 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:36.973687 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36e5ca44a6137ff4a90144e70248a9bd7bea5349c6194bd39a4fccbd26bab197" Oct 30 17:45:42.947086 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:45:42.947048 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:45:54.623067 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:54.623026 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zh64d"] Oct 30 17:45:54.637405 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:54.637372 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zh64d"] Oct 30 17:45:54.666082 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:54.666049 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bfkfw"] Oct 30 17:45:54.666528 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:54.666508 4522 cpu_manager.go:414] "containerMap: removing container" podUID="d4002e5f-5f10-450b-8289-4437e6d16511" containerName="revoke-keys" Oct 30 17:45:54.666528 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:54.666526 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4002e5f-5f10-450b-8289-4437e6d16511" containerName="revoke-keys" Oct 30 17:45:54.666731 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:54.666640 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4002e5f-5f10-450b-8289-4437e6d16511" containerName="revoke-keys" Oct 30 17:45:54.670148 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:54.670124 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bfkfw" Oct 30 17:45:54.690111 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:54.690084 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:45:54.690270 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:54.690243 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:45:54.690371 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:54.690301 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:45:54.690894 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:54.690874 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:45:54.692886 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:54.692863 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bfkfw"] Oct 30 17:45:54.819544 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:54.819509 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f18f1fd-4060-4853-961a-e733cf80b70c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-bfkfw\" (UID: \"3f18f1fd-4060-4853-961a-e733cf80b70c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bfkfw" Oct 30 17:45:54.819732 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:54.819555 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q2gd\" (UniqueName: \"kubernetes.io/projected/3f18f1fd-4060-4853-961a-e733cf80b70c-kube-api-access-4q2gd\") pod \"maas-api-revoke-sub-3d4156f1-bfkfw\" (UID: \"3f18f1fd-4060-4853-961a-e733cf80b70c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bfkfw" Oct 30 17:45:54.819732 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:54.819652 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f18f1fd-4060-4853-961a-e733cf80b70c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-bfkfw\" (UID: \"3f18f1fd-4060-4853-961a-e733cf80b70c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bfkfw" Oct 30 17:45:54.920438 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:54.920355 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f18f1fd-4060-4853-961a-e733cf80b70c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-bfkfw\" (UID: \"3f18f1fd-4060-4853-961a-e733cf80b70c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bfkfw" Oct 30 17:45:54.920438 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:54.920396 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q2gd\" (UniqueName: \"kubernetes.io/projected/3f18f1fd-4060-4853-961a-e733cf80b70c-kube-api-access-4q2gd\") pod \"maas-api-revoke-sub-3d4156f1-bfkfw\" (UID: \"3f18f1fd-4060-4853-961a-e733cf80b70c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bfkfw" Oct 30 17:45:54.920438 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:54.920431 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f18f1fd-4060-4853-961a-e733cf80b70c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-bfkfw\" (UID: \"3f18f1fd-4060-4853-961a-e733cf80b70c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bfkfw" Oct 30 17:45:54.921004 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:54.920984 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f18f1fd-4060-4853-961a-e733cf80b70c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-bfkfw\" (UID: \"3f18f1fd-4060-4853-961a-e733cf80b70c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bfkfw" Oct 30 17:45:54.922758 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:54.922739 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f18f1fd-4060-4853-961a-e733cf80b70c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-bfkfw\" (UID: \"3f18f1fd-4060-4853-961a-e733cf80b70c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bfkfw" Oct 30 17:45:54.947245 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:54.947220 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q2gd\" (UniqueName: \"kubernetes.io/projected/3f18f1fd-4060-4853-961a-e733cf80b70c-kube-api-access-4q2gd\") pod \"maas-api-revoke-sub-3d4156f1-bfkfw\" (UID: \"3f18f1fd-4060-4853-961a-e733cf80b70c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bfkfw" Oct 30 17:45:54.983160 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:54.983138 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bfkfw" Oct 30 17:45:55.278474 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:55.278435 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bfkfw"] Oct 30 17:45:55.278612 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:45:55.278535 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f18f1fd_4060_4853_961a_e733cf80b70c.slice/crio-24d600d9069e0fe1622d60d392081169bb78211fd6e5b2ddfe36cf392f35561d WatchSource:0}: Error finding container 24d600d9069e0fe1622d60d392081169bb78211fd6e5b2ddfe36cf392f35561d: Status 404 returned error can't find the container with id 24d600d9069e0fe1622d60d392081169bb78211fd6e5b2ddfe36cf392f35561d Oct 30 17:45:56.096245 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:56.096208 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bfkfw" event={"ID":"3f18f1fd-4060-4853-961a-e733cf80b70c","Type":"ContainerStarted","Data":"8e249bfd4d9f4bf7d0630effb1d9c579c8eb19a661d9f88d64a66463419d4612"} Oct 30 17:45:56.096655 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:56.096278 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bfkfw" event={"ID":"3f18f1fd-4060-4853-961a-e733cf80b70c","Type":"ContainerStarted","Data":"24d600d9069e0fe1622d60d392081169bb78211fd6e5b2ddfe36cf392f35561d"} Oct 30 17:45:56.145364 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:56.145319 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bfkfw" podStartSLOduration=2.145294063 podStartE2EDuration="2.145294063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:45:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:45:56.141616871 +0000 UTC m=+3530.142371798" watchObservedRunningTime="2026-10-30 17:45:56.145294063 +0000 UTC m=+3530.146048963" Oct 30 17:45:56.145615 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:56.145596 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4002e5f-5f10-450b-8289-4437e6d16511" path="/var/lib/kubelet/pods/d4002e5f-5f10-450b-8289-4437e6d16511/volumes" Oct 30 17:45:57.102124 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:57.102092 4522 generic.go:356] "Generic (PLEG): container finished" podID="3f18f1fd-4060-4853-961a-e733cf80b70c" containerID="8e249bfd4d9f4bf7d0630effb1d9c579c8eb19a661d9f88d64a66463419d4612" exitCode=0 Oct 30 17:45:57.102546 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:57.102131 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bfkfw" event={"ID":"3f18f1fd-4060-4853-961a-e733cf80b70c","Type":"ContainerDied","Data":"8e249bfd4d9f4bf7d0630effb1d9c579c8eb19a661d9f88d64a66463419d4612"} Oct 30 17:45:58.240136 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:58.240109 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bfkfw" Oct 30 17:45:58.356502 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:58.356462 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f18f1fd-4060-4853-961a-e733cf80b70c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f18f1fd-4060-4853-961a-e733cf80b70c-maas-api-openshift-ca\") pod \"3f18f1fd-4060-4853-961a-e733cf80b70c\" (UID: \"3f18f1fd-4060-4853-961a-e733cf80b70c\") " Oct 30 17:45:58.356692 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:58.356555 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f18f1fd-4060-4853-961a-e733cf80b70c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f18f1fd-4060-4853-961a-e733cf80b70c-maas-api-xks-ca\") pod \"3f18f1fd-4060-4853-961a-e733cf80b70c\" (UID: \"3f18f1fd-4060-4853-961a-e733cf80b70c\") " Oct 30 17:45:58.356692 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:58.356594 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f18f1fd-4060-4853-961a-e733cf80b70c-kube-api-access-4q2gd\" (UniqueName: \"kubernetes.io/projected/3f18f1fd-4060-4853-961a-e733cf80b70c-kube-api-access-4q2gd\") pod \"3f18f1fd-4060-4853-961a-e733cf80b70c\" (UID: \"3f18f1fd-4060-4853-961a-e733cf80b70c\") " Oct 30 17:45:58.356890 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:58.356860 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f18f1fd-4060-4853-961a-e733cf80b70c-maas-api-openshift-ca" pod "3f18f1fd-4060-4853-961a-e733cf80b70c" (UID: "3f18f1fd-4060-4853-961a-e733cf80b70c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:45:58.358798 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:58.358743 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f18f1fd-4060-4853-961a-e733cf80b70c-kube-api-access-4q2gd" pod "3f18f1fd-4060-4853-961a-e733cf80b70c" (UID: "3f18f1fd-4060-4853-961a-e733cf80b70c"). InnerVolumeSpecName "kube-api-access-4q2gd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:45:58.358798 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:58.358743 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f18f1fd-4060-4853-961a-e733cf80b70c-maas-api-xks-ca" pod "3f18f1fd-4060-4853-961a-e733cf80b70c" (UID: "3f18f1fd-4060-4853-961a-e733cf80b70c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:45:58.458218 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:58.458181 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4q2gd\" (UniqueName: \"kubernetes.io/projected/3f18f1fd-4060-4853-961a-e733cf80b70c-kube-api-access-4q2gd\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:45:58.458218 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:58.458212 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f18f1fd-4060-4853-961a-e733cf80b70c-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:45:58.458218 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:58.458222 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f18f1fd-4060-4853-961a-e733cf80b70c-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:45:59.113110 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:59.113078 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bfkfw" Oct 30 17:45:59.113110 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:59.113094 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bfkfw" event={"ID":"3f18f1fd-4060-4853-961a-e733cf80b70c","Type":"ContainerDied","Data":"24d600d9069e0fe1622d60d392081169bb78211fd6e5b2ddfe36cf392f35561d"} Oct 30 17:45:59.113405 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:45:59.113144 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24d600d9069e0fe1622d60d392081169bb78211fd6e5b2ddfe36cf392f35561d" Oct 30 17:46:11.715502 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:46:11.715468 4522 scope.go:122] "RemoveContainer" containerID="621b562886c0054886b9d8230d30205f9bbe70176dbf1d227d63267799d52972" Oct 30 17:46:12.950279 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:46:12.950240 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:46:13.036174 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:46:13.036141 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7678bc72-vdl8x"] Oct 30 17:46:13.050601 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:46:13.050571 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7678bc72-vdl8x"] Oct 30 17:46:14.144675 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:46:14.144640 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66176bec-e6cb-48ca-8a4d-dde77870ecdc" path="/var/lib/kubelet/pods/66176bec-e6cb-48ca-8a4d-dde77870ecdc/volumes" Oct 30 17:46:18.559017 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:46:18.558981 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bfkfw"] Oct 30 17:46:18.573271 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:46:18.573237 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bfkfw"] Oct 30 17:46:18.607332 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:46:18.607294 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmp64"] Oct 30 17:46:18.607679 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:46:18.607667 4522 cpu_manager.go:414] "containerMap: removing container" podUID="3f18f1fd-4060-4853-961a-e733cf80b70c" containerName="revoke-keys" Oct 30 17:46:18.607726 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:46:18.607681 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f18f1fd-4060-4853-961a-e733cf80b70c" containerName="revoke-keys" Oct 30 17:46:18.607774 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:46:18.607765 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f18f1fd-4060-4853-961a-e733cf80b70c" containerName="revoke-keys" Oct 30 17:46:18.613298 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:46:18.613262 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmp64" Oct 30 17:46:18.631213 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:46:18.631183 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:46:18.631388 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:46:18.631257 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:46:18.631388 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:46:18.631275 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:46:18.631388 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:46:18.631297 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:46:18.632020 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:46:18.631993 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmp64"] Oct 30 17:46:18.652827 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:46:18.652795 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e912380-f070-48c3-8e10-f27aadd27621-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-cmp64\" (UID: \"4e912380-f070-48c3-8e10-f27aadd27621\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmp64" Oct 30 17:46:18.652973 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:46:18.652858 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lntn\" (UniqueName: \"kubernetes.io/projected/4e912380-f070-48c3-8e10-f27aadd27621-kube-api-access-2lntn\") pod \"maas-api-revoke-sub-3d4156f1-cmp64\" (UID: \"4e912380-f070-48c3-8e10-f27aadd27621\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmp64" Oct 30 17:46:18.653018 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:46:18.652980 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e912380-f070-48c3-8e10-f27aadd27621-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-cmp64\" (UID: \"4e912380-f070-48c3-8e10-f27aadd27621\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmp64" Oct 30 17:46:18.754230 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:46:18.754195 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e912380-f070-48c3-8e10-f27aadd27621-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-cmp64\" (UID: \"4e912380-f070-48c3-8e10-f27aadd27621\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmp64" Oct 30 17:46:18.754443 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:46:18.754301 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e912380-f070-48c3-8e10-f27aadd27621-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-cmp64\" (UID: \"4e912380-f070-48c3-8e10-f27aadd27621\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmp64" Oct 30 17:46:18.754443 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:46:18.754388 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lntn\" (UniqueName: \"kubernetes.io/projected/4e912380-f070-48c3-8e10-f27aadd27621-kube-api-access-2lntn\") pod \"maas-api-revoke-sub-3d4156f1-cmp64\" (UID: \"4e912380-f070-48c3-8e10-f27aadd27621\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmp64" Oct 30 17:46:18.755015 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:46:18.754993 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e912380-f070-48c3-8e10-f27aadd27621-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-cmp64\" (UID: \"4e912380-f070-48c3-8e10-f27aadd27621\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmp64" Oct 30 17:46:18.756654 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:46:18.756633 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e912380-f070-48c3-8e10-f27aadd27621-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-cmp64\" (UID: \"4e912380-f070-48c3-8e10-f27aadd27621\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmp64" Oct 30 17:46:18.780547 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:46:18.780519 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lntn\" (UniqueName: \"kubernetes.io/projected/4e912380-f070-48c3-8e10-f27aadd27621-kube-api-access-2lntn\") pod \"maas-api-revoke-sub-3d4156f1-cmp64\" (UID: \"4e912380-f070-48c3-8e10-f27aadd27621\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmp64" Oct 30 17:46:18.929737 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:46:18.929618 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmp64" Oct 30 17:46:19.224664 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:46:19.224631 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e912380_f070_48c3_8e10_f27aadd27621.slice/crio-3628f475d9f01226d9a4fc609f33075975b999f17491ea25b58596252e245212 WatchSource:0}: Error finding container 3628f475d9f01226d9a4fc609f33075975b999f17491ea25b58596252e245212: Status 404 returned error can't find the container with id 3628f475d9f01226d9a4fc609f33075975b999f17491ea25b58596252e245212 Oct 30 17:46:19.224981 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:46:19.224959 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmp64"] Oct 30 17:46:20.143953 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:46:20.143924 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f18f1fd-4060-4853-961a-e733cf80b70c" path="/var/lib/kubelet/pods/3f18f1fd-4060-4853-961a-e733cf80b70c/volumes" Oct 30 17:46:20.218214 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:46:20.218186 4522 generic.go:356] "Generic (PLEG): container finished" podID="4e912380-f070-48c3-8e10-f27aadd27621" containerID="7f6939961bc601126f9159f7598534a30ed5ffbceabca55abbe768b3e473cecd" exitCode=0 Oct 30 17:46:20.218422 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:46:20.218277 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmp64" event={"ID":"4e912380-f070-48c3-8e10-f27aadd27621","Type":"ContainerDied","Data":"7f6939961bc601126f9159f7598534a30ed5ffbceabca55abbe768b3e473cecd"} Oct 30 17:46:20.218422 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:46:20.218340 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmp64" event={"ID":"4e912380-f070-48c3-8e10-f27aadd27621","Type":"ContainerStarted","Data":"3628f475d9f01226d9a4fc609f33075975b999f17491ea25b58596252e245212"} Oct 30 17:46:21.358552 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:46:21.358521 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmp64" Oct 30 17:46:21.481709 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:46:21.481605 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e912380-f070-48c3-8e10-f27aadd27621-kube-api-access-2lntn\" (UniqueName: \"kubernetes.io/projected/4e912380-f070-48c3-8e10-f27aadd27621-kube-api-access-2lntn\") pod \"4e912380-f070-48c3-8e10-f27aadd27621\" (UID: \"4e912380-f070-48c3-8e10-f27aadd27621\") " Oct 30 17:46:21.481709 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:46:21.481672 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e912380-f070-48c3-8e10-f27aadd27621-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e912380-f070-48c3-8e10-f27aadd27621-maas-api-openshift-ca\") pod \"4e912380-f070-48c3-8e10-f27aadd27621\" (UID: \"4e912380-f070-48c3-8e10-f27aadd27621\") " Oct 30 17:46:21.481934 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:46:21.481876 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e912380-f070-48c3-8e10-f27aadd27621-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e912380-f070-48c3-8e10-f27aadd27621-maas-api-xks-ca\") pod \"4e912380-f070-48c3-8e10-f27aadd27621\" (UID: \"4e912380-f070-48c3-8e10-f27aadd27621\") " Oct 30 17:46:21.482103 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:46:21.482065 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e912380-f070-48c3-8e10-f27aadd27621-maas-api-openshift-ca" pod "4e912380-f070-48c3-8e10-f27aadd27621" (UID: "4e912380-f070-48c3-8e10-f27aadd27621"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:46:21.482238 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:46:21.482198 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e912380-f070-48c3-8e10-f27aadd27621-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:46:21.483896 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:46:21.483873 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e912380-f070-48c3-8e10-f27aadd27621-maas-api-xks-ca" pod "4e912380-f070-48c3-8e10-f27aadd27621" (UID: "4e912380-f070-48c3-8e10-f27aadd27621"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:46:21.483896 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:46:21.483884 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e912380-f070-48c3-8e10-f27aadd27621-kube-api-access-2lntn" pod "4e912380-f070-48c3-8e10-f27aadd27621" (UID: "4e912380-f070-48c3-8e10-f27aadd27621"). InnerVolumeSpecName "kube-api-access-2lntn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:46:21.583559 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:46:21.583523 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lntn\" (UniqueName: \"kubernetes.io/projected/4e912380-f070-48c3-8e10-f27aadd27621-kube-api-access-2lntn\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:46:21.583559 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:46:21.583553 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e912380-f070-48c3-8e10-f27aadd27621-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:46:22.228712 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:46:22.228685 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmp64" Oct 30 17:46:22.228883 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:46:22.228717 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmp64" event={"ID":"4e912380-f070-48c3-8e10-f27aadd27621","Type":"ContainerDied","Data":"3628f475d9f01226d9a4fc609f33075975b999f17491ea25b58596252e245212"} Oct 30 17:46:22.228883 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:46:22.228772 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3628f475d9f01226d9a4fc609f33075975b999f17491ea25b58596252e245212" Oct 30 17:46:42.947183 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:46:42.947146 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:47:04.031632 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:04.031589 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4qmbk"] Oct 30 17:47:04.046040 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:04.046006 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4qmbk"] Oct 30 17:47:04.143229 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:04.143190 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="141d6736-d8dc-4fd0-a387-75241011c8fe" path="/var/lib/kubelet/pods/141d6736-d8dc-4fd0-a387-75241011c8fe/volumes" Oct 30 17:47:11.740048 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:11.740018 4522 scope.go:122] "RemoveContainer" containerID="584bc63908c867c5299c4a622dd3ed50db2392ac936276405003e77279d82cf8" Oct 30 17:47:11.749226 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:11.749203 4522 scope.go:122] "RemoveContainer" containerID="b71b17ad69ffb5a53c9233564c39a7172a539bd2237712e9f225eb12a3980220" Oct 30 17:47:12.948072 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:47:12.948034 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:47:25.793982 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:25.793950 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff8a412d-nd6g9"] Oct 30 17:47:25.794407 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:25.794365 4522 cpu_manager.go:414] "containerMap: removing container" podUID="4e912380-f070-48c3-8e10-f27aadd27621" containerName="revoke-keys" Oct 30 17:47:25.794407 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:25.794377 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e912380-f070-48c3-8e10-f27aadd27621" containerName="revoke-keys" Oct 30 17:47:25.794492 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:25.794458 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e912380-f070-48c3-8e10-f27aadd27621" containerName="revoke-keys" Oct 30 17:47:25.799191 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:25.799173 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff8a412d-nd6g9" Oct 30 17:47:25.802281 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:25.802257 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/202abe8e-fbc7-4d3d-994e-e5fa7aea6657-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff8a412d-nd6g9\" (UID: \"202abe8e-fbc7-4d3d-994e-e5fa7aea6657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff8a412d-nd6g9" Oct 30 17:47:25.802411 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:25.802326 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fscp\" (UniqueName: \"kubernetes.io/projected/202abe8e-fbc7-4d3d-994e-e5fa7aea6657-kube-api-access-2fscp\") pod \"maas-api-revoke-sub-ff8a412d-nd6g9\" (UID: \"202abe8e-fbc7-4d3d-994e-e5fa7aea6657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff8a412d-nd6g9" Oct 30 17:47:25.802411 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:25.802394 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/202abe8e-fbc7-4d3d-994e-e5fa7aea6657-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff8a412d-nd6g9\" (UID: \"202abe8e-fbc7-4d3d-994e-e5fa7aea6657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff8a412d-nd6g9" Oct 30 17:47:25.819668 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:25.819642 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff8a412d-nd6g9"] Oct 30 17:47:25.830773 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:25.830742 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:47:25.830913 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:25.830791 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:47:25.830913 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:25.830840 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:47:25.831044 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:25.830949 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:47:25.903617 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:25.903580 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/202abe8e-fbc7-4d3d-994e-e5fa7aea6657-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff8a412d-nd6g9\" (UID: \"202abe8e-fbc7-4d3d-994e-e5fa7aea6657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff8a412d-nd6g9" Oct 30 17:47:25.903807 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:25.903627 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fscp\" (UniqueName: \"kubernetes.io/projected/202abe8e-fbc7-4d3d-994e-e5fa7aea6657-kube-api-access-2fscp\") pod \"maas-api-revoke-sub-ff8a412d-nd6g9\" (UID: \"202abe8e-fbc7-4d3d-994e-e5fa7aea6657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff8a412d-nd6g9" Oct 30 17:47:25.903807 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:25.903652 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/202abe8e-fbc7-4d3d-994e-e5fa7aea6657-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff8a412d-nd6g9\" (UID: \"202abe8e-fbc7-4d3d-994e-e5fa7aea6657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff8a412d-nd6g9" Oct 30 17:47:25.904336 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:25.904283 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/202abe8e-fbc7-4d3d-994e-e5fa7aea6657-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff8a412d-nd6g9\" (UID: \"202abe8e-fbc7-4d3d-994e-e5fa7aea6657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff8a412d-nd6g9" Oct 30 17:47:25.906083 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:25.906061 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/202abe8e-fbc7-4d3d-994e-e5fa7aea6657-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff8a412d-nd6g9\" (UID: \"202abe8e-fbc7-4d3d-994e-e5fa7aea6657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff8a412d-nd6g9" Oct 30 17:47:25.929287 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:25.929255 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fscp\" (UniqueName: \"kubernetes.io/projected/202abe8e-fbc7-4d3d-994e-e5fa7aea6657-kube-api-access-2fscp\") pod \"maas-api-revoke-sub-ff8a412d-nd6g9\" (UID: \"202abe8e-fbc7-4d3d-994e-e5fa7aea6657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff8a412d-nd6g9" Oct 30 17:47:26.112225 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:26.112136 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff8a412d-nd6g9" Oct 30 17:47:26.414790 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:47:26.414757 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod202abe8e_fbc7_4d3d_994e_e5fa7aea6657.slice/crio-fd953e4644d09ac76411c337ed2676e28f743c141fa007d7205a8b81a5d91104 WatchSource:0}: Error finding container fd953e4644d09ac76411c337ed2676e28f743c141fa007d7205a8b81a5d91104: Status 404 returned error can't find the container with id fd953e4644d09ac76411c337ed2676e28f743c141fa007d7205a8b81a5d91104 Oct 30 17:47:26.414924 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:26.414828 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff8a412d-nd6g9"] Oct 30 17:47:26.554140 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:26.554105 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff8a412d-nd6g9" event={"ID":"202abe8e-fbc7-4d3d-994e-e5fa7aea6657","Type":"ContainerStarted","Data":"fd953e4644d09ac76411c337ed2676e28f743c141fa007d7205a8b81a5d91104"} Oct 30 17:47:27.561753 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:27.561654 4522 generic.go:356] "Generic (PLEG): container finished" podID="202abe8e-fbc7-4d3d-994e-e5fa7aea6657" containerID="5804ec0c3d4a1c3180797846f5e9a0e1ea16373e4b3e0f82289c941ad841b600" exitCode=0 Oct 30 17:47:27.561753 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:27.561695 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff8a412d-nd6g9" event={"ID":"202abe8e-fbc7-4d3d-994e-e5fa7aea6657","Type":"ContainerDied","Data":"5804ec0c3d4a1c3180797846f5e9a0e1ea16373e4b3e0f82289c941ad841b600"} Oct 30 17:47:28.713226 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:28.713197 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff8a412d-nd6g9" Oct 30 17:47:28.730260 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:28.730232 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/202abe8e-fbc7-4d3d-994e-e5fa7aea6657-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/202abe8e-fbc7-4d3d-994e-e5fa7aea6657-maas-api-openshift-ca\") pod \"202abe8e-fbc7-4d3d-994e-e5fa7aea6657\" (UID: \"202abe8e-fbc7-4d3d-994e-e5fa7aea6657\") " Oct 30 17:47:28.730444 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:28.730292 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/202abe8e-fbc7-4d3d-994e-e5fa7aea6657-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/202abe8e-fbc7-4d3d-994e-e5fa7aea6657-maas-api-xks-ca\") pod \"202abe8e-fbc7-4d3d-994e-e5fa7aea6657\" (UID: \"202abe8e-fbc7-4d3d-994e-e5fa7aea6657\") " Oct 30 17:47:28.730444 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:28.730396 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/202abe8e-fbc7-4d3d-994e-e5fa7aea6657-kube-api-access-2fscp\" (UniqueName: \"kubernetes.io/projected/202abe8e-fbc7-4d3d-994e-e5fa7aea6657-kube-api-access-2fscp\") pod \"202abe8e-fbc7-4d3d-994e-e5fa7aea6657\" (UID: \"202abe8e-fbc7-4d3d-994e-e5fa7aea6657\") " Oct 30 17:47:28.730628 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:28.730606 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/202abe8e-fbc7-4d3d-994e-e5fa7aea6657-maas-api-openshift-ca" pod "202abe8e-fbc7-4d3d-994e-e5fa7aea6657" (UID: "202abe8e-fbc7-4d3d-994e-e5fa7aea6657"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:47:28.732484 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:28.732449 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/202abe8e-fbc7-4d3d-994e-e5fa7aea6657-maas-api-xks-ca" pod "202abe8e-fbc7-4d3d-994e-e5fa7aea6657" (UID: "202abe8e-fbc7-4d3d-994e-e5fa7aea6657"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:47:28.732587 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:28.732511 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/202abe8e-fbc7-4d3d-994e-e5fa7aea6657-kube-api-access-2fscp" pod "202abe8e-fbc7-4d3d-994e-e5fa7aea6657" (UID: "202abe8e-fbc7-4d3d-994e-e5fa7aea6657"). InnerVolumeSpecName "kube-api-access-2fscp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:47:28.831592 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:28.831508 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/202abe8e-fbc7-4d3d-994e-e5fa7aea6657-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:47:28.831592 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:28.831541 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/202abe8e-fbc7-4d3d-994e-e5fa7aea6657-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:47:28.831592 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:28.831550 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fscp\" (UniqueName: \"kubernetes.io/projected/202abe8e-fbc7-4d3d-994e-e5fa7aea6657-kube-api-access-2fscp\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:47:29.572817 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:29.572782 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff8a412d-nd6g9" Oct 30 17:47:29.573010 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:29.572811 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff8a412d-nd6g9" event={"ID":"202abe8e-fbc7-4d3d-994e-e5fa7aea6657","Type":"ContainerDied","Data":"fd953e4644d09ac76411c337ed2676e28f743c141fa007d7205a8b81a5d91104"} Oct 30 17:47:29.573010 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:29.572869 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd953e4644d09ac76411c337ed2676e28f743c141fa007d7205a8b81a5d91104" Oct 30 17:47:37.056854 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:37.056819 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c118c3-b6fqp"] Oct 30 17:47:37.057373 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:37.057213 4522 cpu_manager.go:414] "containerMap: removing container" podUID="202abe8e-fbc7-4d3d-994e-e5fa7aea6657" containerName="revoke-keys" Oct 30 17:47:37.057373 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:37.057223 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="202abe8e-fbc7-4d3d-994e-e5fa7aea6657" containerName="revoke-keys" Oct 30 17:47:37.057373 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:37.057287 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="202abe8e-fbc7-4d3d-994e-e5fa7aea6657" containerName="revoke-keys" Oct 30 17:47:37.060451 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:37.060434 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c118c3-b6fqp" Oct 30 17:47:37.077882 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:37.077858 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:47:37.092373 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:37.092351 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:47:37.092373 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:37.092357 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:47:37.092608 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:37.092412 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:47:37.094677 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:37.094658 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c118c3-b6fqp"] Oct 30 17:47:37.112428 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:37.112403 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23f03b36-2aa1-45a7-9b74-29864eda9db4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c118c3-b6fqp\" (UID: \"23f03b36-2aa1-45a7-9b74-29864eda9db4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c118c3-b6fqp" Oct 30 17:47:37.112583 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:37.112522 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58pft\" (UniqueName: \"kubernetes.io/projected/23f03b36-2aa1-45a7-9b74-29864eda9db4-kube-api-access-58pft\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c118c3-b6fqp\" (UID: \"23f03b36-2aa1-45a7-9b74-29864eda9db4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c118c3-b6fqp" Oct 30 17:47:37.112632 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:37.112590 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23f03b36-2aa1-45a7-9b74-29864eda9db4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c118c3-b6fqp\" (UID: \"23f03b36-2aa1-45a7-9b74-29864eda9db4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c118c3-b6fqp" Oct 30 17:47:37.214026 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:37.213991 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58pft\" (UniqueName: \"kubernetes.io/projected/23f03b36-2aa1-45a7-9b74-29864eda9db4-kube-api-access-58pft\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c118c3-b6fqp\" (UID: \"23f03b36-2aa1-45a7-9b74-29864eda9db4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c118c3-b6fqp" Oct 30 17:47:37.214228 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:37.214040 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23f03b36-2aa1-45a7-9b74-29864eda9db4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c118c3-b6fqp\" (UID: \"23f03b36-2aa1-45a7-9b74-29864eda9db4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c118c3-b6fqp" Oct 30 17:47:37.214228 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:37.214144 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23f03b36-2aa1-45a7-9b74-29864eda9db4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c118c3-b6fqp\" (UID: \"23f03b36-2aa1-45a7-9b74-29864eda9db4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c118c3-b6fqp" Oct 30 17:47:37.214777 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:37.214749 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23f03b36-2aa1-45a7-9b74-29864eda9db4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c118c3-b6fqp\" (UID: \"23f03b36-2aa1-45a7-9b74-29864eda9db4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c118c3-b6fqp" Oct 30 17:47:37.216579 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:37.216561 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23f03b36-2aa1-45a7-9b74-29864eda9db4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c118c3-b6fqp\" (UID: \"23f03b36-2aa1-45a7-9b74-29864eda9db4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c118c3-b6fqp" Oct 30 17:47:37.239671 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:37.239643 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58pft\" (UniqueName: \"kubernetes.io/projected/23f03b36-2aa1-45a7-9b74-29864eda9db4-kube-api-access-58pft\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c118c3-b6fqp\" (UID: \"23f03b36-2aa1-45a7-9b74-29864eda9db4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c118c3-b6fqp" Oct 30 17:47:37.373626 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:37.373543 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c118c3-b6fqp" Oct 30 17:47:37.672067 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:37.672030 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c118c3-b6fqp"] Oct 30 17:47:37.672657 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:47:37.672629 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23f03b36_2aa1_45a7_9b74_29864eda9db4.slice/crio-e68c2e106243ce2891730ab1c16867207dc6ff405fded3a6766eb4973b3cd560 WatchSource:0}: Error finding container e68c2e106243ce2891730ab1c16867207dc6ff405fded3a6766eb4973b3cd560: Status 404 returned error can't find the container with id e68c2e106243ce2891730ab1c16867207dc6ff405fded3a6766eb4973b3cd560 Oct 30 17:47:38.618972 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:38.618943 4522 generic.go:356] "Generic (PLEG): container finished" podID="23f03b36-2aa1-45a7-9b74-29864eda9db4" containerID="555c19ed0cd13a835cb1709ca5a0f9f4fb21dcf46e999229fdc8da0c782a4d81" exitCode=0 Oct 30 17:47:38.619401 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:38.618975 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c118c3-b6fqp" event={"ID":"23f03b36-2aa1-45a7-9b74-29864eda9db4","Type":"ContainerDied","Data":"555c19ed0cd13a835cb1709ca5a0f9f4fb21dcf46e999229fdc8da0c782a4d81"} Oct 30 17:47:38.619401 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:38.619032 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c118c3-b6fqp" event={"ID":"23f03b36-2aa1-45a7-9b74-29864eda9db4","Type":"ContainerStarted","Data":"e68c2e106243ce2891730ab1c16867207dc6ff405fded3a6766eb4973b3cd560"} Oct 30 17:47:39.757378 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:39.757351 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c118c3-b6fqp" Oct 30 17:47:39.839007 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:39.838974 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23f03b36-2aa1-45a7-9b74-29864eda9db4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23f03b36-2aa1-45a7-9b74-29864eda9db4-maas-api-openshift-ca\") pod \"23f03b36-2aa1-45a7-9b74-29864eda9db4\" (UID: \"23f03b36-2aa1-45a7-9b74-29864eda9db4\") " Oct 30 17:47:39.839183 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:39.839103 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23f03b36-2aa1-45a7-9b74-29864eda9db4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23f03b36-2aa1-45a7-9b74-29864eda9db4-maas-api-xks-ca\") pod \"23f03b36-2aa1-45a7-9b74-29864eda9db4\" (UID: \"23f03b36-2aa1-45a7-9b74-29864eda9db4\") " Oct 30 17:47:39.839250 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:39.839183 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23f03b36-2aa1-45a7-9b74-29864eda9db4-kube-api-access-58pft\" (UniqueName: \"kubernetes.io/projected/23f03b36-2aa1-45a7-9b74-29864eda9db4-kube-api-access-58pft\") pod \"23f03b36-2aa1-45a7-9b74-29864eda9db4\" (UID: \"23f03b36-2aa1-45a7-9b74-29864eda9db4\") " Oct 30 17:47:39.839476 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:39.839447 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23f03b36-2aa1-45a7-9b74-29864eda9db4-maas-api-openshift-ca" pod "23f03b36-2aa1-45a7-9b74-29864eda9db4" (UID: "23f03b36-2aa1-45a7-9b74-29864eda9db4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:47:39.841249 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:39.841220 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23f03b36-2aa1-45a7-9b74-29864eda9db4-kube-api-access-58pft" pod "23f03b36-2aa1-45a7-9b74-29864eda9db4" (UID: "23f03b36-2aa1-45a7-9b74-29864eda9db4"). InnerVolumeSpecName "kube-api-access-58pft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:47:39.841393 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:39.841333 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23f03b36-2aa1-45a7-9b74-29864eda9db4-maas-api-xks-ca" pod "23f03b36-2aa1-45a7-9b74-29864eda9db4" (UID: "23f03b36-2aa1-45a7-9b74-29864eda9db4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:47:39.940242 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:39.940158 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23f03b36-2aa1-45a7-9b74-29864eda9db4-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:47:39.940242 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:39.940192 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58pft\" (UniqueName: \"kubernetes.io/projected/23f03b36-2aa1-45a7-9b74-29864eda9db4-kube-api-access-58pft\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:47:39.940242 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:39.940202 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23f03b36-2aa1-45a7-9b74-29864eda9db4-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:47:40.630193 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:40.630165 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c118c3-b6fqp" Oct 30 17:47:40.630385 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:40.630169 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c118c3-b6fqp" event={"ID":"23f03b36-2aa1-45a7-9b74-29864eda9db4","Type":"ContainerDied","Data":"e68c2e106243ce2891730ab1c16867207dc6ff405fded3a6766eb4973b3cd560"} Oct 30 17:47:40.630385 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:40.630290 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e68c2e106243ce2891730ab1c16867207dc6ff405fded3a6766eb4973b3cd560" Oct 30 17:47:42.945904 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:47:42.945867 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:47:50.045896 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:50.045677 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c118c3-b6fqp"] Oct 30 17:47:50.060178 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:50.060141 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c118c3-b6fqp"] Oct 30 17:47:50.143402 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:50.143362 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23f03b36-2aa1-45a7-9b74-29864eda9db4" path="/var/lib/kubelet/pods/23f03b36-2aa1-45a7-9b74-29864eda9db4/volumes" Oct 30 17:47:56.037743 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:56.037709 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mw7n5"] Oct 30 17:47:56.052015 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:56.051984 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mw7n5"] Oct 30 17:47:56.143105 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:56.143069 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33d4dc43-08f4-49a2-88da-0910939e5254" path="/var/lib/kubelet/pods/33d4dc43-08f4-49a2-88da-0910939e5254/volumes" Oct 30 17:47:58.579473 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:58.579438 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmp64"] Oct 30 17:47:58.593719 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:58.593687 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmp64"] Oct 30 17:47:58.630611 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:58.630581 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gg6r8"] Oct 30 17:47:58.630975 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:58.630963 4522 cpu_manager.go:414] "containerMap: removing container" podUID="23f03b36-2aa1-45a7-9b74-29864eda9db4" containerName="revoke-keys" Oct 30 17:47:58.631026 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:58.630977 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23f03b36-2aa1-45a7-9b74-29864eda9db4" containerName="revoke-keys" Oct 30 17:47:58.631063 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:58.631034 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="23f03b36-2aa1-45a7-9b74-29864eda9db4" containerName="revoke-keys" Oct 30 17:47:58.635266 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:58.635248 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gg6r8" Oct 30 17:47:58.652828 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:58.652789 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:47:58.652828 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:58.652820 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:47:58.653062 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:58.652875 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:47:58.653062 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:58.652926 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:47:58.654213 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:58.654192 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gg6r8"] Oct 30 17:47:58.728994 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:58.728962 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5v69\" (UniqueName: \"kubernetes.io/projected/54fb12af-bc22-4ed2-a3a0-d5930744641f-kube-api-access-b5v69\") pod \"maas-api-revoke-sub-3d4156f1-gg6r8\" (UID: \"54fb12af-bc22-4ed2-a3a0-d5930744641f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gg6r8" Oct 30 17:47:58.728994 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:58.728999 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54fb12af-bc22-4ed2-a3a0-d5930744641f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-gg6r8\" (UID: \"54fb12af-bc22-4ed2-a3a0-d5930744641f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gg6r8" Oct 30 17:47:58.729229 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:58.729107 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54fb12af-bc22-4ed2-a3a0-d5930744641f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-gg6r8\" (UID: \"54fb12af-bc22-4ed2-a3a0-d5930744641f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gg6r8" Oct 30 17:47:58.830262 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:58.830165 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54fb12af-bc22-4ed2-a3a0-d5930744641f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-gg6r8\" (UID: \"54fb12af-bc22-4ed2-a3a0-d5930744641f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gg6r8" Oct 30 17:47:58.830437 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:58.830278 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5v69\" (UniqueName: \"kubernetes.io/projected/54fb12af-bc22-4ed2-a3a0-d5930744641f-kube-api-access-b5v69\") pod \"maas-api-revoke-sub-3d4156f1-gg6r8\" (UID: \"54fb12af-bc22-4ed2-a3a0-d5930744641f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gg6r8" Oct 30 17:47:58.830437 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:58.830326 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54fb12af-bc22-4ed2-a3a0-d5930744641f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-gg6r8\" (UID: \"54fb12af-bc22-4ed2-a3a0-d5930744641f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gg6r8" Oct 30 17:47:58.830850 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:58.830829 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54fb12af-bc22-4ed2-a3a0-d5930744641f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-gg6r8\" (UID: \"54fb12af-bc22-4ed2-a3a0-d5930744641f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gg6r8" Oct 30 17:47:58.832696 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:58.832674 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54fb12af-bc22-4ed2-a3a0-d5930744641f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-gg6r8\" (UID: \"54fb12af-bc22-4ed2-a3a0-d5930744641f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gg6r8" Oct 30 17:47:58.856398 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:58.856366 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5v69\" (UniqueName: \"kubernetes.io/projected/54fb12af-bc22-4ed2-a3a0-d5930744641f-kube-api-access-b5v69\") pod \"maas-api-revoke-sub-3d4156f1-gg6r8\" (UID: \"54fb12af-bc22-4ed2-a3a0-d5930744641f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gg6r8" Oct 30 17:47:58.948914 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:58.948881 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gg6r8" Oct 30 17:47:59.248474 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:47:59.248432 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54fb12af_bc22_4ed2_a3a0_d5930744641f.slice/crio-79c4640cdea31e25b563602da1b946a607d210401bcc1322a053a2473e6bafde WatchSource:0}: Error finding container 79c4640cdea31e25b563602da1b946a607d210401bcc1322a053a2473e6bafde: Status 404 returned error can't find the container with id 79c4640cdea31e25b563602da1b946a607d210401bcc1322a053a2473e6bafde Oct 30 17:47:59.248585 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:59.248466 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gg6r8"] Oct 30 17:47:59.734816 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:59.734774 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gg6r8" event={"ID":"54fb12af-bc22-4ed2-a3a0-d5930744641f","Type":"ContainerStarted","Data":"2aae67debaf80a9f23bd63dee9c59a2af10884faccf8457cc9311e8942b3f5ec"} Oct 30 17:47:59.735202 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:59.734859 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gg6r8" event={"ID":"54fb12af-bc22-4ed2-a3a0-d5930744641f","Type":"ContainerStarted","Data":"79c4640cdea31e25b563602da1b946a607d210401bcc1322a053a2473e6bafde"} Oct 30 17:47:59.782682 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:47:59.782587 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gg6r8" podStartSLOduration=1.78257421 podStartE2EDuration="1.78257421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:47:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:47:59.780327336 +0000 UTC m=+3653.781082251" watchObservedRunningTime="2026-10-30 17:47:59.78257421 +0000 UTC m=+3653.783329111" Oct 30 17:48:00.142926 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:48:00.142893 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e912380-f070-48c3-8e10-f27aadd27621" path="/var/lib/kubelet/pods/4e912380-f070-48c3-8e10-f27aadd27621/volumes" Oct 30 17:48:00.741403 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:48:00.741372 4522 generic.go:356] "Generic (PLEG): container finished" podID="54fb12af-bc22-4ed2-a3a0-d5930744641f" containerID="2aae67debaf80a9f23bd63dee9c59a2af10884faccf8457cc9311e8942b3f5ec" exitCode=0 Oct 30 17:48:00.741822 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:48:00.741454 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gg6r8" event={"ID":"54fb12af-bc22-4ed2-a3a0-d5930744641f","Type":"ContainerDied","Data":"2aae67debaf80a9f23bd63dee9c59a2af10884faccf8457cc9311e8942b3f5ec"} Oct 30 17:48:01.884526 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:48:01.884501 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gg6r8" Oct 30 17:48:01.957178 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:48:01.957138 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54fb12af-bc22-4ed2-a3a0-d5930744641f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54fb12af-bc22-4ed2-a3a0-d5930744641f-maas-api-openshift-ca\") pod \"54fb12af-bc22-4ed2-a3a0-d5930744641f\" (UID: \"54fb12af-bc22-4ed2-a3a0-d5930744641f\") " Oct 30 17:48:01.957411 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:48:01.957215 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54fb12af-bc22-4ed2-a3a0-d5930744641f-kube-api-access-b5v69\" (UniqueName: \"kubernetes.io/projected/54fb12af-bc22-4ed2-a3a0-d5930744641f-kube-api-access-b5v69\") pod \"54fb12af-bc22-4ed2-a3a0-d5930744641f\" (UID: \"54fb12af-bc22-4ed2-a3a0-d5930744641f\") " Oct 30 17:48:01.957411 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:48:01.957252 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54fb12af-bc22-4ed2-a3a0-d5930744641f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54fb12af-bc22-4ed2-a3a0-d5930744641f-maas-api-xks-ca\") pod \"54fb12af-bc22-4ed2-a3a0-d5930744641f\" (UID: \"54fb12af-bc22-4ed2-a3a0-d5930744641f\") " Oct 30 17:48:01.957643 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:48:01.957610 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54fb12af-bc22-4ed2-a3a0-d5930744641f-maas-api-openshift-ca" pod "54fb12af-bc22-4ed2-a3a0-d5930744641f" (UID: "54fb12af-bc22-4ed2-a3a0-d5930744641f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:48:01.959484 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:48:01.959455 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54fb12af-bc22-4ed2-a3a0-d5930744641f-maas-api-xks-ca" pod "54fb12af-bc22-4ed2-a3a0-d5930744641f" (UID: "54fb12af-bc22-4ed2-a3a0-d5930744641f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:48:01.959484 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:48:01.959474 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54fb12af-bc22-4ed2-a3a0-d5930744641f-kube-api-access-b5v69" pod "54fb12af-bc22-4ed2-a3a0-d5930744641f" (UID: "54fb12af-bc22-4ed2-a3a0-d5930744641f"). InnerVolumeSpecName "kube-api-access-b5v69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:48:02.058939 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:48:02.058829 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54fb12af-bc22-4ed2-a3a0-d5930744641f-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:48:02.058939 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:48:02.058876 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5v69\" (UniqueName: \"kubernetes.io/projected/54fb12af-bc22-4ed2-a3a0-d5930744641f-kube-api-access-b5v69\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:48:02.058939 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:48:02.058887 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54fb12af-bc22-4ed2-a3a0-d5930744641f-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:48:02.752884 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:48:02.752856 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gg6r8" Oct 30 17:48:02.753044 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:48:02.752845 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gg6r8" event={"ID":"54fb12af-bc22-4ed2-a3a0-d5930744641f","Type":"ContainerDied","Data":"79c4640cdea31e25b563602da1b946a607d210401bcc1322a053a2473e6bafde"} Oct 30 17:48:02.753044 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:48:02.752979 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79c4640cdea31e25b563602da1b946a607d210401bcc1322a053a2473e6bafde" Oct 30 17:48:11.772731 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:48:11.772692 4522 scope.go:122] "RemoveContainer" containerID="a12deec508270e5c1e543f415d7bd65b3891179e12047fd40b7806b8ba147814" Oct 30 17:48:13.019108 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:48:13.017956 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:48:16.031601 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:48:16.031565 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-f6x7w"] Oct 30 17:48:16.047374 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:48:16.047338 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-f6x7w"] Oct 30 17:48:16.143475 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:48:16.143445 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fe5072f-e703-49ca-9d89-4279b34e66dc" path="/var/lib/kubelet/pods/9fe5072f-e703-49ca-9d89-4279b34e66dc/volumes" Oct 30 17:48:28.031076 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:48:28.031042 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r7zb5"] Oct 30 17:48:28.045382 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:48:28.045343 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r7zb5"] Oct 30 17:48:28.143007 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:48:28.142974 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82e47fa5-9c29-44d5-97c8-0c41230b1f0f" path="/var/lib/kubelet/pods/82e47fa5-9c29-44d5-97c8-0c41230b1f0f/volumes" Oct 30 17:48:42.947473 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:48:42.947435 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:48:55.042589 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:48:55.042552 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-rsqhg"] Oct 30 17:48:55.056867 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:48:55.056841 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-rsqhg"] Oct 30 17:48:56.143057 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:48:56.143014 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05ba384e-c0d8-4559-bb29-40ae912bd8b3" path="/var/lib/kubelet/pods/05ba384e-c0d8-4559-bb29-40ae912bd8b3/volumes" Oct 30 17:49:06.038940 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:49:06.038905 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-c7xm2"] Oct 30 17:49:06.053391 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:49:06.053363 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-c7xm2"] Oct 30 17:49:06.143149 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:49:06.143111 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77612860-9c4c-4eac-95c8-c42db1b411e0" path="/var/lib/kubelet/pods/77612860-9c4c-4eac-95c8-c42db1b411e0/volumes" Oct 30 17:49:11.849063 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:49:11.849028 4522 scope.go:122] "RemoveContainer" containerID="6ae86e7aa2780f4ec530a806560022ffc33366146bd99e0730e8eea049301e34" Oct 30 17:49:12.947003 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:49:12.946959 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:49:27.030868 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:49:27.030786 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p5k5v"] Oct 30 17:49:27.045022 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:49:27.044995 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p5k5v"] Oct 30 17:49:28.142692 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:49:28.142660 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92483a74-647f-4a31-8d32-022ddb3bff47" path="/var/lib/kubelet/pods/92483a74-647f-4a31-8d32-022ddb3bff47/volumes" Oct 30 17:49:42.947384 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:49:42.947347 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:49:49.032900 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:49:49.032864 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gl5bx"] Oct 30 17:49:49.047267 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:49:49.047237 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gl5bx"] Oct 30 17:49:50.146275 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:49:50.146235 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="302d5493-1efb-4725-bbc1-beeb41f49048" path="/var/lib/kubelet/pods/302d5493-1efb-4725-bbc1-beeb41f49048/volumes" Oct 30 17:49:53.424345 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:49:53.424297 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-x7zf9"] Oct 30 17:49:53.424747 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:49:53.424714 4522 cpu_manager.go:414] "containerMap: removing container" podUID="54fb12af-bc22-4ed2-a3a0-d5930744641f" containerName="revoke-keys" Oct 30 17:49:53.424747 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:49:53.424726 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54fb12af-bc22-4ed2-a3a0-d5930744641f" containerName="revoke-keys" Oct 30 17:49:53.424828 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:49:53.424780 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="54fb12af-bc22-4ed2-a3a0-d5930744641f" containerName="revoke-keys" Oct 30 17:49:53.427957 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:49:53.427938 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-x7zf9" Oct 30 17:49:53.445288 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:49:53.445263 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:49:53.459821 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:49:53.459788 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:49:53.461562 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:49:53.461540 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-x7zf9"] Oct 30 17:49:53.483601 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:49:53.483569 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:49:53.483739 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:49:53.483615 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:49:53.572240 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:49:53.572202 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/111f192f-5ccc-454a-90de-d94b2f283bb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-x7zf9\" (UID: \"111f192f-5ccc-454a-90de-d94b2f283bb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-x7zf9" Oct 30 17:49:53.572453 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:49:53.572256 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/111f192f-5ccc-454a-90de-d94b2f283bb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-x7zf9\" (UID: \"111f192f-5ccc-454a-90de-d94b2f283bb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-x7zf9" Oct 30 17:49:53.572453 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:49:53.572338 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7ffg\" (UniqueName: \"kubernetes.io/projected/111f192f-5ccc-454a-90de-d94b2f283bb0-kube-api-access-n7ffg\") pod \"maas-api-revoke-sub-d576e68b-x7zf9\" (UID: \"111f192f-5ccc-454a-90de-d94b2f283bb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-x7zf9" Oct 30 17:49:53.673727 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:49:53.673694 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/111f192f-5ccc-454a-90de-d94b2f283bb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-x7zf9\" (UID: \"111f192f-5ccc-454a-90de-d94b2f283bb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-x7zf9" Oct 30 17:49:53.673727 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:49:53.673730 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7ffg\" (UniqueName: \"kubernetes.io/projected/111f192f-5ccc-454a-90de-d94b2f283bb0-kube-api-access-n7ffg\") pod \"maas-api-revoke-sub-d576e68b-x7zf9\" (UID: \"111f192f-5ccc-454a-90de-d94b2f283bb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-x7zf9" Oct 30 17:49:53.674073 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:49:53.673821 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/111f192f-5ccc-454a-90de-d94b2f283bb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-x7zf9\" (UID: \"111f192f-5ccc-454a-90de-d94b2f283bb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-x7zf9" Oct 30 17:49:53.674546 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:49:53.674502 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/111f192f-5ccc-454a-90de-d94b2f283bb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-x7zf9\" (UID: \"111f192f-5ccc-454a-90de-d94b2f283bb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-x7zf9" Oct 30 17:49:53.676073 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:49:53.676056 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/111f192f-5ccc-454a-90de-d94b2f283bb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-x7zf9\" (UID: \"111f192f-5ccc-454a-90de-d94b2f283bb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-x7zf9" Oct 30 17:49:53.699561 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:49:53.699534 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7ffg\" (UniqueName: \"kubernetes.io/projected/111f192f-5ccc-454a-90de-d94b2f283bb0-kube-api-access-n7ffg\") pod \"maas-api-revoke-sub-d576e68b-x7zf9\" (UID: \"111f192f-5ccc-454a-90de-d94b2f283bb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-x7zf9" Oct 30 17:49:53.740526 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:49:53.740491 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-x7zf9" Oct 30 17:49:54.038437 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:49:54.038403 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-x7zf9"] Oct 30 17:49:54.398053 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:49:54.398019 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-x7zf9" event={"ID":"111f192f-5ccc-454a-90de-d94b2f283bb0","Type":"ContainerStarted","Data":"236c933883a37d7a1ebc2fa701ebf280491b78e97c43cb2889ebb787db06b4cf"} Oct 30 17:49:54.398234 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:49:54.398067 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-x7zf9" event={"ID":"111f192f-5ccc-454a-90de-d94b2f283bb0","Type":"ContainerStarted","Data":"7ab6005db8ae836bb7684cc887c671737fd69fd51a1026bee8d871da0a3352af"} Oct 30 17:49:54.445287 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:49:54.445231 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-x7zf9" podStartSLOduration=1.4452168429999999 podStartE2EDuration="1.445216843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:49:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:49:54.444005997 +0000 UTC m=+3768.444760898" watchObservedRunningTime="2026-10-30 17:49:54.445216843 +0000 UTC m=+3768.445971743" Oct 30 17:49:55.404449 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:49:55.404412 4522 generic.go:356] "Generic (PLEG): container finished" podID="111f192f-5ccc-454a-90de-d94b2f283bb0" containerID="236c933883a37d7a1ebc2fa701ebf280491b78e97c43cb2889ebb787db06b4cf" exitCode=0 Oct 30 17:49:55.404671 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:49:55.404501 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-x7zf9" event={"ID":"111f192f-5ccc-454a-90de-d94b2f283bb0","Type":"ContainerDied","Data":"236c933883a37d7a1ebc2fa701ebf280491b78e97c43cb2889ebb787db06b4cf"} Oct 30 17:49:56.540609 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:49:56.540579 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-x7zf9" Oct 30 17:49:56.604861 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:49:56.604828 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/111f192f-5ccc-454a-90de-d94b2f283bb0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/111f192f-5ccc-454a-90de-d94b2f283bb0-maas-api-xks-ca\") pod \"111f192f-5ccc-454a-90de-d94b2f283bb0\" (UID: \"111f192f-5ccc-454a-90de-d94b2f283bb0\") " Oct 30 17:49:56.605041 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:49:56.604877 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/111f192f-5ccc-454a-90de-d94b2f283bb0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/111f192f-5ccc-454a-90de-d94b2f283bb0-maas-api-openshift-ca\") pod \"111f192f-5ccc-454a-90de-d94b2f283bb0\" (UID: \"111f192f-5ccc-454a-90de-d94b2f283bb0\") " Oct 30 17:49:56.605041 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:49:56.604918 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/111f192f-5ccc-454a-90de-d94b2f283bb0-kube-api-access-n7ffg\" (UniqueName: \"kubernetes.io/projected/111f192f-5ccc-454a-90de-d94b2f283bb0-kube-api-access-n7ffg\") pod \"111f192f-5ccc-454a-90de-d94b2f283bb0\" (UID: \"111f192f-5ccc-454a-90de-d94b2f283bb0\") " Oct 30 17:49:56.605287 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:49:56.605258 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/111f192f-5ccc-454a-90de-d94b2f283bb0-maas-api-openshift-ca" pod "111f192f-5ccc-454a-90de-d94b2f283bb0" (UID: "111f192f-5ccc-454a-90de-d94b2f283bb0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:49:56.607085 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:49:56.607055 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/111f192f-5ccc-454a-90de-d94b2f283bb0-maas-api-xks-ca" pod "111f192f-5ccc-454a-90de-d94b2f283bb0" (UID: "111f192f-5ccc-454a-90de-d94b2f283bb0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:49:56.607214 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:49:56.607091 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/111f192f-5ccc-454a-90de-d94b2f283bb0-kube-api-access-n7ffg" pod "111f192f-5ccc-454a-90de-d94b2f283bb0" (UID: "111f192f-5ccc-454a-90de-d94b2f283bb0"). InnerVolumeSpecName "kube-api-access-n7ffg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:49:56.705835 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:49:56.705802 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/111f192f-5ccc-454a-90de-d94b2f283bb0-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:49:56.705835 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:49:56.705836 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/111f192f-5ccc-454a-90de-d94b2f283bb0-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:49:56.706063 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:49:56.705849 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7ffg\" (UniqueName: \"kubernetes.io/projected/111f192f-5ccc-454a-90de-d94b2f283bb0-kube-api-access-n7ffg\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:49:57.415348 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:49:57.415318 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-x7zf9" Oct 30 17:49:57.415348 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:49:57.415328 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-x7zf9" event={"ID":"111f192f-5ccc-454a-90de-d94b2f283bb0","Type":"ContainerDied","Data":"7ab6005db8ae836bb7684cc887c671737fd69fd51a1026bee8d871da0a3352af"} Oct 30 17:49:57.415566 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:49:57.415370 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ab6005db8ae836bb7684cc887c671737fd69fd51a1026bee8d871da0a3352af" Oct 30 17:50:10.596423 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:50:10.596389 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lnn2v"] Oct 30 17:50:10.598768 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:50:10.596776 4522 cpu_manager.go:414] "containerMap: removing container" podUID="111f192f-5ccc-454a-90de-d94b2f283bb0" containerName="revoke-keys" Oct 30 17:50:10.598768 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:50:10.596787 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="111f192f-5ccc-454a-90de-d94b2f283bb0" containerName="revoke-keys" Oct 30 17:50:10.598768 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:50:10.596851 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="111f192f-5ccc-454a-90de-d94b2f283bb0" containerName="revoke-keys" Oct 30 17:50:10.599657 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:50:10.599640 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lnn2v" Oct 30 17:50:10.619076 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:50:10.619051 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:50:10.621157 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:50:10.621131 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lnn2v"] Oct 30 17:50:10.633525 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:50:10.633505 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:50:10.633652 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:50:10.633540 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:50:10.633722 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:50:10.633695 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:50:10.673483 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:50:10.673453 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6tz9\" (UniqueName: \"kubernetes.io/projected/3e9cbcae-2d9d-44f0-8531-d49b9f335a01-kube-api-access-d6tz9\") pod \"maas-api-revoke-sub-d2bc93bc-lnn2v\" (UID: \"3e9cbcae-2d9d-44f0-8531-d49b9f335a01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lnn2v" Oct 30 17:50:10.673634 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:50:10.673487 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e9cbcae-2d9d-44f0-8531-d49b9f335a01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-lnn2v\" (UID: \"3e9cbcae-2d9d-44f0-8531-d49b9f335a01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lnn2v" Oct 30 17:50:10.673678 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:50:10.673649 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e9cbcae-2d9d-44f0-8531-d49b9f335a01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-lnn2v\" (UID: \"3e9cbcae-2d9d-44f0-8531-d49b9f335a01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lnn2v" Oct 30 17:50:10.775066 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:50:10.775021 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e9cbcae-2d9d-44f0-8531-d49b9f335a01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-lnn2v\" (UID: \"3e9cbcae-2d9d-44f0-8531-d49b9f335a01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lnn2v" Oct 30 17:50:10.775287 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:50:10.775096 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6tz9\" (UniqueName: \"kubernetes.io/projected/3e9cbcae-2d9d-44f0-8531-d49b9f335a01-kube-api-access-d6tz9\") pod \"maas-api-revoke-sub-d2bc93bc-lnn2v\" (UID: \"3e9cbcae-2d9d-44f0-8531-d49b9f335a01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lnn2v" Oct 30 17:50:10.775287 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:50:10.775122 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e9cbcae-2d9d-44f0-8531-d49b9f335a01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-lnn2v\" (UID: \"3e9cbcae-2d9d-44f0-8531-d49b9f335a01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lnn2v" Oct 30 17:50:10.775752 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:50:10.775727 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e9cbcae-2d9d-44f0-8531-d49b9f335a01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-lnn2v\" (UID: \"3e9cbcae-2d9d-44f0-8531-d49b9f335a01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lnn2v" Oct 30 17:50:10.777645 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:50:10.777621 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e9cbcae-2d9d-44f0-8531-d49b9f335a01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-lnn2v\" (UID: \"3e9cbcae-2d9d-44f0-8531-d49b9f335a01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lnn2v" Oct 30 17:50:10.800952 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:50:10.800920 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6tz9\" (UniqueName: \"kubernetes.io/projected/3e9cbcae-2d9d-44f0-8531-d49b9f335a01-kube-api-access-d6tz9\") pod \"maas-api-revoke-sub-d2bc93bc-lnn2v\" (UID: \"3e9cbcae-2d9d-44f0-8531-d49b9f335a01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lnn2v" Oct 30 17:50:10.911788 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:50:10.911684 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lnn2v" Oct 30 17:50:11.211114 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:50:11.211075 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e9cbcae_2d9d_44f0_8531_d49b9f335a01.slice/crio-f5d50f884a669864d81147abebe9bc207b5ac1e71a5b2e2eb66cbcc2e6f63f08 WatchSource:0}: Error finding container f5d50f884a669864d81147abebe9bc207b5ac1e71a5b2e2eb66cbcc2e6f63f08: Status 404 returned error can't find the container with id f5d50f884a669864d81147abebe9bc207b5ac1e71a5b2e2eb66cbcc2e6f63f08 Oct 30 17:50:11.212494 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:50:11.212462 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lnn2v"] Oct 30 17:50:11.492915 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:50:11.492826 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lnn2v" event={"ID":"3e9cbcae-2d9d-44f0-8531-d49b9f335a01","Type":"ContainerStarted","Data":"f5d50f884a669864d81147abebe9bc207b5ac1e71a5b2e2eb66cbcc2e6f63f08"} Oct 30 17:50:11.872593 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:50:11.872508 4522 scope.go:122] "RemoveContainer" containerID="f88ec1e80954436be11013f14a157ac414b4fdd0ff87aa1d8b1f1236cbcd4e9d" Oct 30 17:50:11.880980 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:50:11.880959 4522 scope.go:122] "RemoveContainer" containerID="b9ffc797f7b104821669cf3d470505e6820eb8e4f44f5f83492af99ea341427d" Oct 30 17:50:11.888893 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:50:11.888869 4522 scope.go:122] "RemoveContainer" containerID="4ba155bedd974ba5d7f091d6d114cc6c44c2779231411409d4df68c789484e49" Oct 30 17:50:11.896507 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:50:11.896485 4522 scope.go:122] "RemoveContainer" containerID="7612af96ed60f6d1f975588732a6f12635f84ba765166c21aa18c9f0628cfc94" Oct 30 17:50:12.499408 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:50:12.499375 4522 generic.go:356] "Generic (PLEG): container finished" podID="3e9cbcae-2d9d-44f0-8531-d49b9f335a01" containerID="8a8d588a4f5dc022324776359975de229ed31251a9695d95fd595e068ea909e8" exitCode=0 Oct 30 17:50:12.499617 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:50:12.499465 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lnn2v" event={"ID":"3e9cbcae-2d9d-44f0-8531-d49b9f335a01","Type":"ContainerDied","Data":"8a8d588a4f5dc022324776359975de229ed31251a9695d95fd595e068ea909e8"} Oct 30 17:50:12.948732 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:50:12.948642 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:50:13.629650 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:50:13.629622 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lnn2v" Oct 30 17:50:13.705060 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:50:13.705030 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e9cbcae-2d9d-44f0-8531-d49b9f335a01-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e9cbcae-2d9d-44f0-8531-d49b9f335a01-maas-api-openshift-ca\") pod \"3e9cbcae-2d9d-44f0-8531-d49b9f335a01\" (UID: \"3e9cbcae-2d9d-44f0-8531-d49b9f335a01\") " Oct 30 17:50:13.705060 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:50:13.705065 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e9cbcae-2d9d-44f0-8531-d49b9f335a01-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e9cbcae-2d9d-44f0-8531-d49b9f335a01-maas-api-xks-ca\") pod \"3e9cbcae-2d9d-44f0-8531-d49b9f335a01\" (UID: \"3e9cbcae-2d9d-44f0-8531-d49b9f335a01\") " Oct 30 17:50:13.705397 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:50:13.705133 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e9cbcae-2d9d-44f0-8531-d49b9f335a01-kube-api-access-d6tz9\" (UniqueName: \"kubernetes.io/projected/3e9cbcae-2d9d-44f0-8531-d49b9f335a01-kube-api-access-d6tz9\") pod \"3e9cbcae-2d9d-44f0-8531-d49b9f335a01\" (UID: \"3e9cbcae-2d9d-44f0-8531-d49b9f335a01\") " Oct 30 17:50:13.705503 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:50:13.705474 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e9cbcae-2d9d-44f0-8531-d49b9f335a01-maas-api-openshift-ca" pod "3e9cbcae-2d9d-44f0-8531-d49b9f335a01" (UID: "3e9cbcae-2d9d-44f0-8531-d49b9f335a01"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:50:13.705661 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:50:13.705606 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e9cbcae-2d9d-44f0-8531-d49b9f335a01-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:50:13.707372 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:50:13.707343 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e9cbcae-2d9d-44f0-8531-d49b9f335a01-kube-api-access-d6tz9" pod "3e9cbcae-2d9d-44f0-8531-d49b9f335a01" (UID: "3e9cbcae-2d9d-44f0-8531-d49b9f335a01"). InnerVolumeSpecName "kube-api-access-d6tz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:50:13.707372 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:50:13.707357 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e9cbcae-2d9d-44f0-8531-d49b9f335a01-maas-api-xks-ca" pod "3e9cbcae-2d9d-44f0-8531-d49b9f335a01" (UID: "3e9cbcae-2d9d-44f0-8531-d49b9f335a01"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:50:13.806397 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:50:13.806298 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e9cbcae-2d9d-44f0-8531-d49b9f335a01-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:50:13.806397 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:50:13.806345 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6tz9\" (UniqueName: \"kubernetes.io/projected/3e9cbcae-2d9d-44f0-8531-d49b9f335a01-kube-api-access-d6tz9\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:50:14.511020 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:50:14.510984 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lnn2v" event={"ID":"3e9cbcae-2d9d-44f0-8531-d49b9f335a01","Type":"ContainerDied","Data":"f5d50f884a669864d81147abebe9bc207b5ac1e71a5b2e2eb66cbcc2e6f63f08"} Oct 30 17:50:14.511504 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:50:14.511037 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5d50f884a669864d81147abebe9bc207b5ac1e71a5b2e2eb66cbcc2e6f63f08" Oct 30 17:50:14.511504 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:50:14.510992 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lnn2v" Oct 30 17:50:26.030836 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:50:26.030794 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ms2k9"] Oct 30 17:50:26.045204 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:50:26.045173 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ms2k9"] Oct 30 17:50:26.143539 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:50:26.143507 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="343b8093-c8a1-4dd2-b187-731295b0bc89" path="/var/lib/kubelet/pods/343b8093-c8a1-4dd2-b187-731295b0bc89/volumes" Oct 30 17:50:42.947276 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:50:42.947239 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:51:03.109102 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:03.109056 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-493ff32b-lndtb"] Oct 30 17:51:03.109677 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:03.109654 4522 cpu_manager.go:414] "containerMap: removing container" podUID="3e9cbcae-2d9d-44f0-8531-d49b9f335a01" containerName="revoke-keys" Oct 30 17:51:03.109761 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:03.109680 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e9cbcae-2d9d-44f0-8531-d49b9f335a01" containerName="revoke-keys" Oct 30 17:51:03.109819 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:03.109789 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e9cbcae-2d9d-44f0-8531-d49b9f335a01" containerName="revoke-keys" Oct 30 17:51:03.112946 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:03.112924 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-493ff32b-lndtb" Oct 30 17:51:03.145422 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:03.145386 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:51:03.145627 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:03.145422 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:51:03.145627 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:03.145430 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:51:03.145627 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:03.145587 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:51:03.147372 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:03.147353 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-493ff32b-lndtb"] Oct 30 17:51:03.279506 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:03.279470 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24f50285-f406-48d7-af07-33a1ebe5825a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-493ff32b-lndtb\" (UID: \"24f50285-f406-48d7-af07-33a1ebe5825a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-493ff32b-lndtb" Oct 30 17:51:03.279721 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:03.279522 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24f50285-f406-48d7-af07-33a1ebe5825a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-493ff32b-lndtb\" (UID: \"24f50285-f406-48d7-af07-33a1ebe5825a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-493ff32b-lndtb" Oct 30 17:51:03.279721 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:03.279604 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw6cg\" (UniqueName: \"kubernetes.io/projected/24f50285-f406-48d7-af07-33a1ebe5825a-kube-api-access-cw6cg\") pod \"maas-api-revoke-sub-493ff32b-lndtb\" (UID: \"24f50285-f406-48d7-af07-33a1ebe5825a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-493ff32b-lndtb" Oct 30 17:51:03.380349 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:03.380226 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24f50285-f406-48d7-af07-33a1ebe5825a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-493ff32b-lndtb\" (UID: \"24f50285-f406-48d7-af07-33a1ebe5825a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-493ff32b-lndtb" Oct 30 17:51:03.380349 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:03.380285 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24f50285-f406-48d7-af07-33a1ebe5825a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-493ff32b-lndtb\" (UID: \"24f50285-f406-48d7-af07-33a1ebe5825a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-493ff32b-lndtb" Oct 30 17:51:03.380349 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:03.380344 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw6cg\" (UniqueName: \"kubernetes.io/projected/24f50285-f406-48d7-af07-33a1ebe5825a-kube-api-access-cw6cg\") pod \"maas-api-revoke-sub-493ff32b-lndtb\" (UID: \"24f50285-f406-48d7-af07-33a1ebe5825a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-493ff32b-lndtb" Oct 30 17:51:03.380987 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:03.380960 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24f50285-f406-48d7-af07-33a1ebe5825a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-493ff32b-lndtb\" (UID: \"24f50285-f406-48d7-af07-33a1ebe5825a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-493ff32b-lndtb" Oct 30 17:51:03.382699 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:03.382679 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24f50285-f406-48d7-af07-33a1ebe5825a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-493ff32b-lndtb\" (UID: \"24f50285-f406-48d7-af07-33a1ebe5825a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-493ff32b-lndtb" Oct 30 17:51:03.409741 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:03.409715 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw6cg\" (UniqueName: \"kubernetes.io/projected/24f50285-f406-48d7-af07-33a1ebe5825a-kube-api-access-cw6cg\") pod \"maas-api-revoke-sub-493ff32b-lndtb\" (UID: \"24f50285-f406-48d7-af07-33a1ebe5825a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-493ff32b-lndtb" Oct 30 17:51:03.426230 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:03.426201 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-493ff32b-lndtb" Oct 30 17:51:03.727245 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:51:03.727213 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24f50285_f406_48d7_af07_33a1ebe5825a.slice/crio-17eb7d08a2b2a5261b907da9194fc4d70246eb662c91729686d820ac1127ed6b WatchSource:0}: Error finding container 17eb7d08a2b2a5261b907da9194fc4d70246eb662c91729686d820ac1127ed6b: Status 404 returned error can't find the container with id 17eb7d08a2b2a5261b907da9194fc4d70246eb662c91729686d820ac1127ed6b Oct 30 17:51:03.727397 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:03.727263 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-493ff32b-lndtb"] Oct 30 17:51:03.755363 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:03.755331 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-493ff32b-lndtb" event={"ID":"24f50285-f406-48d7-af07-33a1ebe5825a","Type":"ContainerStarted","Data":"17eb7d08a2b2a5261b907da9194fc4d70246eb662c91729686d820ac1127ed6b"} Oct 30 17:51:04.762627 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:04.762589 4522 generic.go:356] "Generic (PLEG): container finished" podID="24f50285-f406-48d7-af07-33a1ebe5825a" containerID="95f1457f8e3d983f237ea54392d8bb6b770b5ddbfdce03a5dd07b29acc448a89" exitCode=0 Oct 30 17:51:04.763035 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:04.762677 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-493ff32b-lndtb" event={"ID":"24f50285-f406-48d7-af07-33a1ebe5825a","Type":"ContainerDied","Data":"95f1457f8e3d983f237ea54392d8bb6b770b5ddbfdce03a5dd07b29acc448a89"} Oct 30 17:51:05.898968 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:05.898940 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-493ff32b-lndtb" Oct 30 17:51:06.004954 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:06.004919 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24f50285-f406-48d7-af07-33a1ebe5825a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24f50285-f406-48d7-af07-33a1ebe5825a-maas-api-xks-ca\") pod \"24f50285-f406-48d7-af07-33a1ebe5825a\" (UID: \"24f50285-f406-48d7-af07-33a1ebe5825a\") " Oct 30 17:51:06.005153 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:06.005043 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24f50285-f406-48d7-af07-33a1ebe5825a-kube-api-access-cw6cg\" (UniqueName: \"kubernetes.io/projected/24f50285-f406-48d7-af07-33a1ebe5825a-kube-api-access-cw6cg\") pod \"24f50285-f406-48d7-af07-33a1ebe5825a\" (UID: \"24f50285-f406-48d7-af07-33a1ebe5825a\") " Oct 30 17:51:06.005153 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:06.005082 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24f50285-f406-48d7-af07-33a1ebe5825a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24f50285-f406-48d7-af07-33a1ebe5825a-maas-api-openshift-ca\") pod \"24f50285-f406-48d7-af07-33a1ebe5825a\" (UID: \"24f50285-f406-48d7-af07-33a1ebe5825a\") " Oct 30 17:51:06.005511 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:06.005481 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24f50285-f406-48d7-af07-33a1ebe5825a-maas-api-openshift-ca" pod "24f50285-f406-48d7-af07-33a1ebe5825a" (UID: "24f50285-f406-48d7-af07-33a1ebe5825a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:51:06.007021 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:06.006999 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24f50285-f406-48d7-af07-33a1ebe5825a-maas-api-xks-ca" pod "24f50285-f406-48d7-af07-33a1ebe5825a" (UID: "24f50285-f406-48d7-af07-33a1ebe5825a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:51:06.007124 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:06.007104 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24f50285-f406-48d7-af07-33a1ebe5825a-kube-api-access-cw6cg" pod "24f50285-f406-48d7-af07-33a1ebe5825a" (UID: "24f50285-f406-48d7-af07-33a1ebe5825a"). InnerVolumeSpecName "kube-api-access-cw6cg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:51:06.106520 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:06.106431 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24f50285-f406-48d7-af07-33a1ebe5825a-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:51:06.106520 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:06.106466 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cw6cg\" (UniqueName: \"kubernetes.io/projected/24f50285-f406-48d7-af07-33a1ebe5825a-kube-api-access-cw6cg\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:51:06.106520 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:06.106479 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24f50285-f406-48d7-af07-33a1ebe5825a-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:51:06.774149 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:06.774112 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-493ff32b-lndtb" event={"ID":"24f50285-f406-48d7-af07-33a1ebe5825a","Type":"ContainerDied","Data":"17eb7d08a2b2a5261b907da9194fc4d70246eb662c91729686d820ac1127ed6b"} Oct 30 17:51:06.774360 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:06.774165 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17eb7d08a2b2a5261b907da9194fc4d70246eb662c91729686d820ac1127ed6b" Oct 30 17:51:06.774360 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:06.774123 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-493ff32b-lndtb" Oct 30 17:51:11.926396 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:11.926363 4522 scope.go:122] "RemoveContainer" containerID="a48e48c7ea1b212354a05a03ac6553a072ad27d03e5ae0859d7e7bf9ef20b700" Oct 30 17:51:11.935057 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:11.935036 4522 scope.go:122] "RemoveContainer" containerID="44fc1eb5d4f77810c87b45cb54b834630e2abe4ba6500a1ebef66efdd12302dd" Oct 30 17:51:12.948010 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:51:12.947977 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:51:14.183438 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:14.183400 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ce3c55-7kvh7"] Oct 30 17:51:14.183929 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:14.183811 4522 cpu_manager.go:414] "containerMap: removing container" podUID="24f50285-f406-48d7-af07-33a1ebe5825a" containerName="revoke-keys" Oct 30 17:51:14.183929 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:14.183824 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24f50285-f406-48d7-af07-33a1ebe5825a" containerName="revoke-keys" Oct 30 17:51:14.183929 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:14.183898 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="24f50285-f406-48d7-af07-33a1ebe5825a" containerName="revoke-keys" Oct 30 17:51:14.186778 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:14.186761 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ce3c55-7kvh7" Oct 30 17:51:14.208462 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:14.208433 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ce3c55-7kvh7"] Oct 30 17:51:14.220750 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:14.220717 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:51:14.220901 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:14.220764 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:51:14.220901 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:14.220785 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:51:14.220901 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:14.220880 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:51:14.284418 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:14.284386 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d061776f-e6c1-4e3e-806d-bcf3c538f130-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ce3c55-7kvh7\" (UID: \"d061776f-e6c1-4e3e-806d-bcf3c538f130\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ce3c55-7kvh7" Oct 30 17:51:14.284612 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:14.284426 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txt9v\" (UniqueName: \"kubernetes.io/projected/d061776f-e6c1-4e3e-806d-bcf3c538f130-kube-api-access-txt9v\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ce3c55-7kvh7\" (UID: \"d061776f-e6c1-4e3e-806d-bcf3c538f130\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ce3c55-7kvh7" Oct 30 17:51:14.284612 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:14.284570 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d061776f-e6c1-4e3e-806d-bcf3c538f130-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ce3c55-7kvh7\" (UID: \"d061776f-e6c1-4e3e-806d-bcf3c538f130\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ce3c55-7kvh7" Oct 30 17:51:14.386061 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:14.386022 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d061776f-e6c1-4e3e-806d-bcf3c538f130-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ce3c55-7kvh7\" (UID: \"d061776f-e6c1-4e3e-806d-bcf3c538f130\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ce3c55-7kvh7" Oct 30 17:51:14.386061 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:14.386061 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txt9v\" (UniqueName: \"kubernetes.io/projected/d061776f-e6c1-4e3e-806d-bcf3c538f130-kube-api-access-txt9v\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ce3c55-7kvh7\" (UID: \"d061776f-e6c1-4e3e-806d-bcf3c538f130\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ce3c55-7kvh7" Oct 30 17:51:14.386369 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:14.386106 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d061776f-e6c1-4e3e-806d-bcf3c538f130-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ce3c55-7kvh7\" (UID: \"d061776f-e6c1-4e3e-806d-bcf3c538f130\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ce3c55-7kvh7" Oct 30 17:51:14.386719 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:14.386696 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d061776f-e6c1-4e3e-806d-bcf3c538f130-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ce3c55-7kvh7\" (UID: \"d061776f-e6c1-4e3e-806d-bcf3c538f130\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ce3c55-7kvh7" Oct 30 17:51:14.388507 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:14.388488 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d061776f-e6c1-4e3e-806d-bcf3c538f130-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ce3c55-7kvh7\" (UID: \"d061776f-e6c1-4e3e-806d-bcf3c538f130\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ce3c55-7kvh7" Oct 30 17:51:14.412211 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:14.412171 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txt9v\" (UniqueName: \"kubernetes.io/projected/d061776f-e6c1-4e3e-806d-bcf3c538f130-kube-api-access-txt9v\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ce3c55-7kvh7\" (UID: \"d061776f-e6c1-4e3e-806d-bcf3c538f130\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ce3c55-7kvh7" Oct 30 17:51:14.499435 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:14.499401 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ce3c55-7kvh7" Oct 30 17:51:14.797901 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:51:14.797868 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd061776f_e6c1_4e3e_806d_bcf3c538f130.slice/crio-07bea4573d683a98bba9f002c405c0d9bb45d09e222cab09a2df0941f4e7cb3f WatchSource:0}: Error finding container 07bea4573d683a98bba9f002c405c0d9bb45d09e222cab09a2df0941f4e7cb3f: Status 404 returned error can't find the container with id 07bea4573d683a98bba9f002c405c0d9bb45d09e222cab09a2df0941f4e7cb3f Oct 30 17:51:14.798139 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:14.798119 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ce3c55-7kvh7"] Oct 30 17:51:14.814791 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:14.814761 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ce3c55-7kvh7" event={"ID":"d061776f-e6c1-4e3e-806d-bcf3c538f130","Type":"ContainerStarted","Data":"07bea4573d683a98bba9f002c405c0d9bb45d09e222cab09a2df0941f4e7cb3f"} Oct 30 17:51:15.822353 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:15.822288 4522 generic.go:356] "Generic (PLEG): container finished" podID="d061776f-e6c1-4e3e-806d-bcf3c538f130" containerID="15ddf78d1fb54d71b4117e9b321c910ba69371eb28f4cdb7b1222de2c23c1a50" exitCode=0 Oct 30 17:51:15.822811 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:15.822376 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ce3c55-7kvh7" event={"ID":"d061776f-e6c1-4e3e-806d-bcf3c538f130","Type":"ContainerDied","Data":"15ddf78d1fb54d71b4117e9b321c910ba69371eb28f4cdb7b1222de2c23c1a50"} Oct 30 17:51:16.963920 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:16.963895 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ce3c55-7kvh7" Oct 30 17:51:17.113890 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:17.113807 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d061776f-e6c1-4e3e-806d-bcf3c538f130-kube-api-access-txt9v\" (UniqueName: \"kubernetes.io/projected/d061776f-e6c1-4e3e-806d-bcf3c538f130-kube-api-access-txt9v\") pod \"d061776f-e6c1-4e3e-806d-bcf3c538f130\" (UID: \"d061776f-e6c1-4e3e-806d-bcf3c538f130\") " Oct 30 17:51:17.113890 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:17.113857 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d061776f-e6c1-4e3e-806d-bcf3c538f130-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d061776f-e6c1-4e3e-806d-bcf3c538f130-maas-api-openshift-ca\") pod \"d061776f-e6c1-4e3e-806d-bcf3c538f130\" (UID: \"d061776f-e6c1-4e3e-806d-bcf3c538f130\") " Oct 30 17:51:17.114109 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:17.113968 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d061776f-e6c1-4e3e-806d-bcf3c538f130-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d061776f-e6c1-4e3e-806d-bcf3c538f130-maas-api-xks-ca\") pod \"d061776f-e6c1-4e3e-806d-bcf3c538f130\" (UID: \"d061776f-e6c1-4e3e-806d-bcf3c538f130\") " Oct 30 17:51:17.114200 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:17.114175 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d061776f-e6c1-4e3e-806d-bcf3c538f130-maas-api-openshift-ca" pod "d061776f-e6c1-4e3e-806d-bcf3c538f130" (UID: "d061776f-e6c1-4e3e-806d-bcf3c538f130"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:51:17.115996 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:17.115971 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d061776f-e6c1-4e3e-806d-bcf3c538f130-maas-api-xks-ca" pod "d061776f-e6c1-4e3e-806d-bcf3c538f130" (UID: "d061776f-e6c1-4e3e-806d-bcf3c538f130"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:51:17.116132 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:17.116115 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d061776f-e6c1-4e3e-806d-bcf3c538f130-kube-api-access-txt9v" pod "d061776f-e6c1-4e3e-806d-bcf3c538f130" (UID: "d061776f-e6c1-4e3e-806d-bcf3c538f130"). InnerVolumeSpecName "kube-api-access-txt9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:51:17.214704 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:17.214667 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d061776f-e6c1-4e3e-806d-bcf3c538f130-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:51:17.214704 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:17.214697 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txt9v\" (UniqueName: \"kubernetes.io/projected/d061776f-e6c1-4e3e-806d-bcf3c538f130-kube-api-access-txt9v\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:51:17.214704 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:17.214707 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d061776f-e6c1-4e3e-806d-bcf3c538f130-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:51:17.840594 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:17.840560 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ce3c55-7kvh7" event={"ID":"d061776f-e6c1-4e3e-806d-bcf3c538f130","Type":"ContainerDied","Data":"07bea4573d683a98bba9f002c405c0d9bb45d09e222cab09a2df0941f4e7cb3f"} Oct 30 17:51:17.840594 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:17.840593 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ce3c55-7kvh7" Oct 30 17:51:17.840813 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:17.840611 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07bea4573d683a98bba9f002c405c0d9bb45d09e222cab09a2df0941f4e7cb3f" Oct 30 17:51:27.558217 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:27.558179 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ce3c55-7kvh7"] Oct 30 17:51:27.573265 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:27.573226 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ce3c55-7kvh7"] Oct 30 17:51:28.143168 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:51:28.143136 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d061776f-e6c1-4e3e-806d-bcf3c538f130" path="/var/lib/kubelet/pods/d061776f-e6c1-4e3e-806d-bcf3c538f130/volumes" Oct 30 17:51:42.986570 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:51:42.986523 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:52:02.996522 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:02.996453 4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-245xx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:52:02.997152 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:02.996562 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-245xx" podUID="fb8adf71-af34-4715-ada0-42494da2219a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:52:11.961829 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:11.961796 4522 scope.go:122] "RemoveContainer" containerID="deaad1d15800d0dd63f9a2a0d350a921c42e910a3780f779a5812c113984ef7d" Oct 30 17:52:11.970409 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:11.970387 4522 scope.go:122] "RemoveContainer" containerID="8e249bfd4d9f4bf7d0630effb1d9c579c8eb19a661d9f88d64a66463419d4612" Oct 30 17:52:11.978096 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:11.978076 4522 scope.go:122] "RemoveContainer" containerID="bac1cd701e4a86011a433d26dae4ed27a610c94a80526d7dc19ac15b7ac2e792" Oct 30 17:52:12.948880 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:52:12.948843 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:52:14.675061 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:14.675030 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-k844j"] Oct 30 17:52:14.675659 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:14.675445 4522 cpu_manager.go:414] "containerMap: removing container" podUID="d061776f-e6c1-4e3e-806d-bcf3c538f130" containerName="revoke-keys" Oct 30 17:52:14.675659 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:14.675457 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d061776f-e6c1-4e3e-806d-bcf3c538f130" containerName="revoke-keys" Oct 30 17:52:14.675659 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:14.675545 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="d061776f-e6c1-4e3e-806d-bcf3c538f130" containerName="revoke-keys" Oct 30 17:52:14.678627 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:14.678606 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-k844j" Oct 30 17:52:14.703876 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:14.703839 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-k844j"] Oct 30 17:52:14.708439 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:14.708394 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k484g\" (UniqueName: \"kubernetes.io/projected/ccd80853-0b0f-4875-ace4-b9d2cebbfc11-kube-api-access-k484g\") pod \"maas-api-revoke-sub-452b6d7b-k844j\" (UID: \"ccd80853-0b0f-4875-ace4-b9d2cebbfc11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-k844j" Oct 30 17:52:14.708439 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:14.708440 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccd80853-0b0f-4875-ace4-b9d2cebbfc11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-k844j\" (UID: \"ccd80853-0b0f-4875-ace4-b9d2cebbfc11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-k844j" Oct 30 17:52:14.708730 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:14.708605 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccd80853-0b0f-4875-ace4-b9d2cebbfc11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-k844j\" (UID: \"ccd80853-0b0f-4875-ace4-b9d2cebbfc11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-k844j" Oct 30 17:52:14.715116 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:14.715089 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:52:14.715405 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:14.715386 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:52:14.715591 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:14.715575 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:52:14.715780 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:14.715765 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:52:14.809860 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:14.809827 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccd80853-0b0f-4875-ace4-b9d2cebbfc11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-k844j\" (UID: \"ccd80853-0b0f-4875-ace4-b9d2cebbfc11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-k844j" Oct 30 17:52:14.810098 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:14.809909 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k484g\" (UniqueName: \"kubernetes.io/projected/ccd80853-0b0f-4875-ace4-b9d2cebbfc11-kube-api-access-k484g\") pod \"maas-api-revoke-sub-452b6d7b-k844j\" (UID: \"ccd80853-0b0f-4875-ace4-b9d2cebbfc11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-k844j" Oct 30 17:52:14.810098 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:14.809939 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccd80853-0b0f-4875-ace4-b9d2cebbfc11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-k844j\" (UID: \"ccd80853-0b0f-4875-ace4-b9d2cebbfc11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-k844j" Oct 30 17:52:14.810739 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:14.810709 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccd80853-0b0f-4875-ace4-b9d2cebbfc11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-k844j\" (UID: \"ccd80853-0b0f-4875-ace4-b9d2cebbfc11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-k844j" Oct 30 17:52:14.813055 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:14.813029 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccd80853-0b0f-4875-ace4-b9d2cebbfc11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-k844j\" (UID: \"ccd80853-0b0f-4875-ace4-b9d2cebbfc11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-k844j" Oct 30 17:52:14.844840 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:14.844795 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k484g\" (UniqueName: \"kubernetes.io/projected/ccd80853-0b0f-4875-ace4-b9d2cebbfc11-kube-api-access-k484g\") pod \"maas-api-revoke-sub-452b6d7b-k844j\" (UID: \"ccd80853-0b0f-4875-ace4-b9d2cebbfc11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-k844j" Oct 30 17:52:14.995463 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:14.995424 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-k844j" Oct 30 17:52:15.364676 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:15.364573 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-k844j"] Oct 30 17:52:15.385592 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:52:15.385543 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccd80853_0b0f_4875_ace4_b9d2cebbfc11.slice/crio-4451aeb65ebf1cb831d148e572323abe855f8d1b0c5d8cc1349acd4c9af0ac11 WatchSource:0}: Error finding container 4451aeb65ebf1cb831d148e572323abe855f8d1b0c5d8cc1349acd4c9af0ac11: Status 404 returned error can't find the container with id 4451aeb65ebf1cb831d148e572323abe855f8d1b0c5d8cc1349acd4c9af0ac11 Oct 30 17:52:16.167750 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:16.167712 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-k844j" event={"ID":"ccd80853-0b0f-4875-ace4-b9d2cebbfc11","Type":"ContainerStarted","Data":"7cfd818afd6316a95215d1809cfb2df6954f4d25aecc880e5498229e8df03794"} Oct 30 17:52:16.168377 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:16.168350 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-k844j" event={"ID":"ccd80853-0b0f-4875-ace4-b9d2cebbfc11","Type":"ContainerStarted","Data":"4451aeb65ebf1cb831d148e572323abe855f8d1b0c5d8cc1349acd4c9af0ac11"} Oct 30 17:52:16.243360 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:16.243265 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-k844j" podStartSLOduration=2.243245335 podStartE2EDuration="2.243245335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:52:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:52:16.229813202 +0000 UTC m=+3910.230568103" watchObservedRunningTime="2026-10-30 17:52:16.243245335 +0000 UTC m=+3910.244000237" Oct 30 17:52:17.175166 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:17.175134 4522 generic.go:356] "Generic (PLEG): container finished" podID="ccd80853-0b0f-4875-ace4-b9d2cebbfc11" containerID="7cfd818afd6316a95215d1809cfb2df6954f4d25aecc880e5498229e8df03794" exitCode=0 Oct 30 17:52:17.175610 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:17.175221 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-k844j" event={"ID":"ccd80853-0b0f-4875-ace4-b9d2cebbfc11","Type":"ContainerDied","Data":"7cfd818afd6316a95215d1809cfb2df6954f4d25aecc880e5498229e8df03794"} Oct 30 17:52:18.312629 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:18.312603 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-k844j" Oct 30 17:52:18.391060 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:18.391032 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccd80853-0b0f-4875-ace4-b9d2cebbfc11-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccd80853-0b0f-4875-ace4-b9d2cebbfc11-maas-api-xks-ca\") pod \"ccd80853-0b0f-4875-ace4-b9d2cebbfc11\" (UID: \"ccd80853-0b0f-4875-ace4-b9d2cebbfc11\") " Oct 30 17:52:18.391244 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:18.391145 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccd80853-0b0f-4875-ace4-b9d2cebbfc11-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccd80853-0b0f-4875-ace4-b9d2cebbfc11-maas-api-openshift-ca\") pod \"ccd80853-0b0f-4875-ace4-b9d2cebbfc11\" (UID: \"ccd80853-0b0f-4875-ace4-b9d2cebbfc11\") " Oct 30 17:52:18.391244 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:18.391220 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccd80853-0b0f-4875-ace4-b9d2cebbfc11-kube-api-access-k484g\" (UniqueName: \"kubernetes.io/projected/ccd80853-0b0f-4875-ace4-b9d2cebbfc11-kube-api-access-k484g\") pod \"ccd80853-0b0f-4875-ace4-b9d2cebbfc11\" (UID: \"ccd80853-0b0f-4875-ace4-b9d2cebbfc11\") " Oct 30 17:52:18.391570 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:18.391545 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccd80853-0b0f-4875-ace4-b9d2cebbfc11-maas-api-openshift-ca" pod "ccd80853-0b0f-4875-ace4-b9d2cebbfc11" (UID: "ccd80853-0b0f-4875-ace4-b9d2cebbfc11"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:52:18.393397 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:18.393366 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccd80853-0b0f-4875-ace4-b9d2cebbfc11-maas-api-xks-ca" pod "ccd80853-0b0f-4875-ace4-b9d2cebbfc11" (UID: "ccd80853-0b0f-4875-ace4-b9d2cebbfc11"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:52:18.393493 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:18.393368 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccd80853-0b0f-4875-ace4-b9d2cebbfc11-kube-api-access-k484g" pod "ccd80853-0b0f-4875-ace4-b9d2cebbfc11" (UID: "ccd80853-0b0f-4875-ace4-b9d2cebbfc11"). InnerVolumeSpecName "kube-api-access-k484g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:52:18.492159 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:18.492119 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k484g\" (UniqueName: \"kubernetes.io/projected/ccd80853-0b0f-4875-ace4-b9d2cebbfc11-kube-api-access-k484g\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:52:18.492159 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:18.492157 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccd80853-0b0f-4875-ace4-b9d2cebbfc11-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:52:18.492159 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:18.492166 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccd80853-0b0f-4875-ace4-b9d2cebbfc11-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:52:19.186375 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:19.186337 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-k844j" event={"ID":"ccd80853-0b0f-4875-ace4-b9d2cebbfc11","Type":"ContainerDied","Data":"4451aeb65ebf1cb831d148e572323abe855f8d1b0c5d8cc1349acd4c9af0ac11"} Oct 30 17:52:19.186579 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:19.186401 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4451aeb65ebf1cb831d148e572323abe855f8d1b0c5d8cc1349acd4c9af0ac11" Oct 30 17:52:19.186579 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:19.186347 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-k844j" Oct 30 17:52:29.030871 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:29.030831 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff8a412d-nd6g9"] Oct 30 17:52:29.046641 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:29.046601 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff8a412d-nd6g9"] Oct 30 17:52:30.151330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:30.150376 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="202abe8e-fbc7-4d3d-994e-e5fa7aea6657" path="/var/lib/kubelet/pods/202abe8e-fbc7-4d3d-994e-e5fa7aea6657/volumes" Oct 30 17:52:34.764330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:34.763555 4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-hgtsq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:52:34.764330 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:34.763617 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hgtsq" podUID="9d445c57-2efa-4ae2-8a16-9e4dcdac8544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:52:35.594051 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:35.594015 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9xhhn"] Oct 30 17:52:35.594439 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:35.594425 4522 cpu_manager.go:414] "containerMap: removing container" podUID="ccd80853-0b0f-4875-ace4-b9d2cebbfc11" containerName="revoke-keys" Oct 30 17:52:35.594493 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:35.594440 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccd80853-0b0f-4875-ace4-b9d2cebbfc11" containerName="revoke-keys" Oct 30 17:52:35.594537 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:35.594504 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccd80853-0b0f-4875-ace4-b9d2cebbfc11" containerName="revoke-keys" Oct 30 17:52:35.597967 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:35.597949 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9xhhn" Oct 30 17:52:35.635973 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:35.635941 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:52:35.635973 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:35.635956 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:52:35.638008 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:35.637986 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9xhhn"] Oct 30 17:52:35.708631 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:35.708598 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:52:35.708811 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:35.708652 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:52:35.723512 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:35.723473 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7rbp\" (UniqueName: \"kubernetes.io/projected/5e2a09d4-7a61-481b-9fac-047f911b7eaf-kube-api-access-k7rbp\") pod \"maas-api-revoke-sub-3e31a506-9xhhn\" (UID: \"5e2a09d4-7a61-481b-9fac-047f911b7eaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9xhhn" Oct 30 17:52:35.723690 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:35.723550 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e2a09d4-7a61-481b-9fac-047f911b7eaf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-9xhhn\" (UID: \"5e2a09d4-7a61-481b-9fac-047f911b7eaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9xhhn" Oct 30 17:52:35.723690 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:35.723637 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e2a09d4-7a61-481b-9fac-047f911b7eaf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-9xhhn\" (UID: \"5e2a09d4-7a61-481b-9fac-047f911b7eaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9xhhn" Oct 30 17:52:35.824937 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:35.824898 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7rbp\" (UniqueName: \"kubernetes.io/projected/5e2a09d4-7a61-481b-9fac-047f911b7eaf-kube-api-access-k7rbp\") pod \"maas-api-revoke-sub-3e31a506-9xhhn\" (UID: \"5e2a09d4-7a61-481b-9fac-047f911b7eaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9xhhn" Oct 30 17:52:35.824937 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:35.824941 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e2a09d4-7a61-481b-9fac-047f911b7eaf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-9xhhn\" (UID: \"5e2a09d4-7a61-481b-9fac-047f911b7eaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9xhhn" Oct 30 17:52:35.825536 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:35.824983 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e2a09d4-7a61-481b-9fac-047f911b7eaf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-9xhhn\" (UID: \"5e2a09d4-7a61-481b-9fac-047f911b7eaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9xhhn" Oct 30 17:52:35.825848 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:35.825820 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e2a09d4-7a61-481b-9fac-047f911b7eaf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-9xhhn\" (UID: \"5e2a09d4-7a61-481b-9fac-047f911b7eaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9xhhn" Oct 30 17:52:35.827352 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:35.827332 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e2a09d4-7a61-481b-9fac-047f911b7eaf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-9xhhn\" (UID: \"5e2a09d4-7a61-481b-9fac-047f911b7eaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9xhhn" Oct 30 17:52:35.850680 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:35.850620 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7rbp\" (UniqueName: \"kubernetes.io/projected/5e2a09d4-7a61-481b-9fac-047f911b7eaf-kube-api-access-k7rbp\") pod \"maas-api-revoke-sub-3e31a506-9xhhn\" (UID: \"5e2a09d4-7a61-481b-9fac-047f911b7eaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9xhhn" Oct 30 17:52:35.910151 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:35.910115 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9xhhn" Oct 30 17:52:36.207225 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:36.207193 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9xhhn"] Oct 30 17:52:36.207481 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:52:36.207460 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e2a09d4_7a61_481b_9fac_047f911b7eaf.slice/crio-03853b52fe3d926eb585a527b96234c3b3042ec8b8a1b1c9d1a8701b9af6aa42 WatchSource:0}: Error finding container 03853b52fe3d926eb585a527b96234c3b3042ec8b8a1b1c9d1a8701b9af6aa42: Status 404 returned error can't find the container with id 03853b52fe3d926eb585a527b96234c3b3042ec8b8a1b1c9d1a8701b9af6aa42 Oct 30 17:52:36.291643 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:36.291612 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9xhhn" event={"ID":"5e2a09d4-7a61-481b-9fac-047f911b7eaf","Type":"ContainerStarted","Data":"03853b52fe3d926eb585a527b96234c3b3042ec8b8a1b1c9d1a8701b9af6aa42"} Oct 30 17:52:37.299840 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:37.299791 4522 generic.go:356] "Generic (PLEG): container finished" podID="5e2a09d4-7a61-481b-9fac-047f911b7eaf" containerID="9c7b93c2d4cf888240988a941401b5a346b10eb0648a2350ddadd62ebeee88ba" exitCode=0 Oct 30 17:52:37.300407 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:37.299936 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9xhhn" event={"ID":"5e2a09d4-7a61-481b-9fac-047f911b7eaf","Type":"ContainerDied","Data":"9c7b93c2d4cf888240988a941401b5a346b10eb0648a2350ddadd62ebeee88ba"} Oct 30 17:52:38.655999 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:38.655966 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9xhhn" Oct 30 17:52:38.674804 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:38.674752 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e2a09d4-7a61-481b-9fac-047f911b7eaf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e2a09d4-7a61-481b-9fac-047f911b7eaf-maas-api-xks-ca\") pod \"5e2a09d4-7a61-481b-9fac-047f911b7eaf\" (UID: \"5e2a09d4-7a61-481b-9fac-047f911b7eaf\") " Oct 30 17:52:38.674804 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:38.674806 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e2a09d4-7a61-481b-9fac-047f911b7eaf-kube-api-access-k7rbp\" (UniqueName: \"kubernetes.io/projected/5e2a09d4-7a61-481b-9fac-047f911b7eaf-kube-api-access-k7rbp\") pod \"5e2a09d4-7a61-481b-9fac-047f911b7eaf\" (UID: \"5e2a09d4-7a61-481b-9fac-047f911b7eaf\") " Oct 30 17:52:38.675618 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:38.675587 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e2a09d4-7a61-481b-9fac-047f911b7eaf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e2a09d4-7a61-481b-9fac-047f911b7eaf-maas-api-openshift-ca\") pod \"5e2a09d4-7a61-481b-9fac-047f911b7eaf\" (UID: \"5e2a09d4-7a61-481b-9fac-047f911b7eaf\") " Oct 30 17:52:38.675986 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:38.675958 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e2a09d4-7a61-481b-9fac-047f911b7eaf-maas-api-openshift-ca" pod "5e2a09d4-7a61-481b-9fac-047f911b7eaf" (UID: "5e2a09d4-7a61-481b-9fac-047f911b7eaf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:52:38.677680 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:38.677653 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e2a09d4-7a61-481b-9fac-047f911b7eaf-maas-api-xks-ca" pod "5e2a09d4-7a61-481b-9fac-047f911b7eaf" (UID: "5e2a09d4-7a61-481b-9fac-047f911b7eaf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:52:38.677680 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:38.677674 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e2a09d4-7a61-481b-9fac-047f911b7eaf-kube-api-access-k7rbp" pod "5e2a09d4-7a61-481b-9fac-047f911b7eaf" (UID: "5e2a09d4-7a61-481b-9fac-047f911b7eaf"). InnerVolumeSpecName "kube-api-access-k7rbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:52:38.777405 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:38.777359 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e2a09d4-7a61-481b-9fac-047f911b7eaf-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:52:38.777690 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:38.777667 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7rbp\" (UniqueName: \"kubernetes.io/projected/5e2a09d4-7a61-481b-9fac-047f911b7eaf-kube-api-access-k7rbp\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:52:38.777855 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:38.777839 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e2a09d4-7a61-481b-9fac-047f911b7eaf-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:52:39.325132 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:39.325093 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9xhhn" event={"ID":"5e2a09d4-7a61-481b-9fac-047f911b7eaf","Type":"ContainerDied","Data":"03853b52fe3d926eb585a527b96234c3b3042ec8b8a1b1c9d1a8701b9af6aa42"} Oct 30 17:52:39.325384 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:39.325145 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03853b52fe3d926eb585a527b96234c3b3042ec8b8a1b1c9d1a8701b9af6aa42" Oct 30 17:52:39.325384 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:39.325102 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9xhhn" Oct 30 17:52:42.948534 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:52:42.948498 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:52:58.121975 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:58.121936 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98d632f6-wc7lx"] Oct 30 17:52:58.122583 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:58.122562 4522 cpu_manager.go:414] "containerMap: removing container" podUID="5e2a09d4-7a61-481b-9fac-047f911b7eaf" containerName="revoke-keys" Oct 30 17:52:58.122684 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:58.122586 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e2a09d4-7a61-481b-9fac-047f911b7eaf" containerName="revoke-keys" Oct 30 17:52:58.122733 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:58.122726 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e2a09d4-7a61-481b-9fac-047f911b7eaf" containerName="revoke-keys" Oct 30 17:52:58.126287 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:58.126272 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d632f6-wc7lx" Oct 30 17:52:58.145627 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:58.145589 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98d632f6-wc7lx"] Oct 30 17:52:58.156842 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:58.156812 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:52:58.156842 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:58.156812 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:52:58.157091 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:58.156852 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:52:58.158471 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:58.158449 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:52:58.291460 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:58.291426 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfgtj\" (UniqueName: \"kubernetes.io/projected/b7077219-7481-4bfe-9201-c429ac69d9ce-kube-api-access-wfgtj\") pod \"maas-api-revoke-sub-98d632f6-wc7lx\" (UID: \"b7077219-7481-4bfe-9201-c429ac69d9ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d632f6-wc7lx" Oct 30 17:52:58.291656 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:58.291475 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7077219-7481-4bfe-9201-c429ac69d9ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98d632f6-wc7lx\" (UID: \"b7077219-7481-4bfe-9201-c429ac69d9ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d632f6-wc7lx" Oct 30 17:52:58.291707 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:58.291647 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7077219-7481-4bfe-9201-c429ac69d9ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98d632f6-wc7lx\" (UID: \"b7077219-7481-4bfe-9201-c429ac69d9ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d632f6-wc7lx" Oct 30 17:52:58.392964 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:58.392878 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7077219-7481-4bfe-9201-c429ac69d9ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98d632f6-wc7lx\" (UID: \"b7077219-7481-4bfe-9201-c429ac69d9ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d632f6-wc7lx" Oct 30 17:52:58.392964 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:58.392920 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfgtj\" (UniqueName: \"kubernetes.io/projected/b7077219-7481-4bfe-9201-c429ac69d9ce-kube-api-access-wfgtj\") pod \"maas-api-revoke-sub-98d632f6-wc7lx\" (UID: \"b7077219-7481-4bfe-9201-c429ac69d9ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d632f6-wc7lx" Oct 30 17:52:58.392964 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:58.392951 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7077219-7481-4bfe-9201-c429ac69d9ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98d632f6-wc7lx\" (UID: \"b7077219-7481-4bfe-9201-c429ac69d9ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d632f6-wc7lx" Oct 30 17:52:58.393628 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:58.393596 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7077219-7481-4bfe-9201-c429ac69d9ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98d632f6-wc7lx\" (UID: \"b7077219-7481-4bfe-9201-c429ac69d9ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d632f6-wc7lx" Oct 30 17:52:58.395282 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:58.395260 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7077219-7481-4bfe-9201-c429ac69d9ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98d632f6-wc7lx\" (UID: \"b7077219-7481-4bfe-9201-c429ac69d9ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d632f6-wc7lx" Oct 30 17:52:58.419959 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:58.419929 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfgtj\" (UniqueName: \"kubernetes.io/projected/b7077219-7481-4bfe-9201-c429ac69d9ce-kube-api-access-wfgtj\") pod \"maas-api-revoke-sub-98d632f6-wc7lx\" (UID: \"b7077219-7481-4bfe-9201-c429ac69d9ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d632f6-wc7lx" Oct 30 17:52:58.438960 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:58.438923 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d632f6-wc7lx" Oct 30 17:52:58.734655 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:52:58.734615 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7077219_7481_4bfe_9201_c429ac69d9ce.slice/crio-4f2ea7ad2085bb499e51676bb182024ca7eb872107a4b47425f1f97cc780968c WatchSource:0}: Error finding container 4f2ea7ad2085bb499e51676bb182024ca7eb872107a4b47425f1f97cc780968c: Status 404 returned error can't find the container with id 4f2ea7ad2085bb499e51676bb182024ca7eb872107a4b47425f1f97cc780968c Oct 30 17:52:58.736231 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:58.736107 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98d632f6-wc7lx"] Oct 30 17:52:59.443417 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:59.443336 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d632f6-wc7lx" event={"ID":"b7077219-7481-4bfe-9201-c429ac69d9ce","Type":"ContainerStarted","Data":"1860c1504d0bbd902939088c7e87dab4ecaa5f51696b8cdc0c453ab6c06639c1"} Oct 30 17:52:59.443417 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:59.443392 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d632f6-wc7lx" event={"ID":"b7077219-7481-4bfe-9201-c429ac69d9ce","Type":"ContainerStarted","Data":"4f2ea7ad2085bb499e51676bb182024ca7eb872107a4b47425f1f97cc780968c"} Oct 30 17:52:59.491754 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:52:59.491701 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d632f6-wc7lx" podStartSLOduration=1.491684009 podStartE2EDuration="1.491684009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:52:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:52:59.489948433 +0000 UTC m=+3953.490703350" watchObservedRunningTime="2026-10-30 17:52:59.491684009 +0000 UTC m=+3953.492438910" Oct 30 17:53:00.449910 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:00.449872 4522 generic.go:356] "Generic (PLEG): container finished" podID="b7077219-7481-4bfe-9201-c429ac69d9ce" containerID="1860c1504d0bbd902939088c7e87dab4ecaa5f51696b8cdc0c453ab6c06639c1" exitCode=0 Oct 30 17:53:00.450338 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:00.449923 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d632f6-wc7lx" event={"ID":"b7077219-7481-4bfe-9201-c429ac69d9ce","Type":"ContainerDied","Data":"1860c1504d0bbd902939088c7e87dab4ecaa5f51696b8cdc0c453ab6c06639c1"} Oct 30 17:53:01.588680 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:01.588651 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d632f6-wc7lx" Oct 30 17:53:01.726189 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:01.726106 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7077219-7481-4bfe-9201-c429ac69d9ce-kube-api-access-wfgtj\" (UniqueName: \"kubernetes.io/projected/b7077219-7481-4bfe-9201-c429ac69d9ce-kube-api-access-wfgtj\") pod \"b7077219-7481-4bfe-9201-c429ac69d9ce\" (UID: \"b7077219-7481-4bfe-9201-c429ac69d9ce\") " Oct 30 17:53:01.726189 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:01.726156 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7077219-7481-4bfe-9201-c429ac69d9ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7077219-7481-4bfe-9201-c429ac69d9ce-maas-api-xks-ca\") pod \"b7077219-7481-4bfe-9201-c429ac69d9ce\" (UID: \"b7077219-7481-4bfe-9201-c429ac69d9ce\") " Oct 30 17:53:01.726407 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:01.726230 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7077219-7481-4bfe-9201-c429ac69d9ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7077219-7481-4bfe-9201-c429ac69d9ce-maas-api-openshift-ca\") pod \"b7077219-7481-4bfe-9201-c429ac69d9ce\" (UID: \"b7077219-7481-4bfe-9201-c429ac69d9ce\") " Oct 30 17:53:01.726635 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:01.726609 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7077219-7481-4bfe-9201-c429ac69d9ce-maas-api-openshift-ca" pod "b7077219-7481-4bfe-9201-c429ac69d9ce" (UID: "b7077219-7481-4bfe-9201-c429ac69d9ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:53:01.728415 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:01.728389 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7077219-7481-4bfe-9201-c429ac69d9ce-maas-api-xks-ca" pod "b7077219-7481-4bfe-9201-c429ac69d9ce" (UID: "b7077219-7481-4bfe-9201-c429ac69d9ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:53:01.728415 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:01.728401 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7077219-7481-4bfe-9201-c429ac69d9ce-kube-api-access-wfgtj" pod "b7077219-7481-4bfe-9201-c429ac69d9ce" (UID: "b7077219-7481-4bfe-9201-c429ac69d9ce"). InnerVolumeSpecName "kube-api-access-wfgtj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:53:01.827374 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:01.827333 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7077219-7481-4bfe-9201-c429ac69d9ce-maas-api-openshift-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:53:01.827374 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:01.827369 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfgtj\" (UniqueName: \"kubernetes.io/projected/b7077219-7481-4bfe-9201-c429ac69d9ce-kube-api-access-wfgtj\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:53:01.827374 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:01.827380 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7077219-7481-4bfe-9201-c429ac69d9ce-maas-api-xks-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:53:02.032010 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:02.031896 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gg6r8"] Oct 30 17:53:02.035593 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:02.035567 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gg6r8"] Oct 30 17:53:02.143483 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:02.143451 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54fb12af-bc22-4ed2-a3a0-d5930744641f" path="/var/lib/kubelet/pods/54fb12af-bc22-4ed2-a3a0-d5930744641f/volumes" Oct 30 17:53:02.460769 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:02.460737 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d632f6-wc7lx" Oct 30 17:53:02.460951 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:02.460763 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d632f6-wc7lx" event={"ID":"b7077219-7481-4bfe-9201-c429ac69d9ce","Type":"ContainerDied","Data":"4f2ea7ad2085bb499e51676bb182024ca7eb872107a4b47425f1f97cc780968c"} Oct 30 17:53:02.460951 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:02.460817 4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f2ea7ad2085bb499e51676bb182024ca7eb872107a4b47425f1f97cc780968c" Oct 30 17:53:10.557298 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:10.557260 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-xmplt"] Oct 30 17:53:10.557922 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:10.557674 4522 cpu_manager.go:414] "containerMap: removing container" podUID="b7077219-7481-4bfe-9201-c429ac69d9ce" containerName="revoke-keys" Oct 30 17:53:10.557922 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:10.557686 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7077219-7481-4bfe-9201-c429ac69d9ce" containerName="revoke-keys" Oct 30 17:53:10.557922 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:10.557786 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7077219-7481-4bfe-9201-c429ac69d9ce" containerName="revoke-keys" Oct 30 17:53:10.562233 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:10.562213 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-xmplt" Oct 30 17:53:10.597738 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:10.597709 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 17:53:10.597888 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:10.597804 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 17:53:10.597888 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:10.597827 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cdj9d\"" Oct 30 17:53:10.600050 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:10.600024 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-xmplt"] Oct 30 17:53:10.611776 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:10.611746 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879-openshift-service-ca\") pod \"maas-api-5c48877c75-xmplt\" (UID: \"ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-xmplt" Oct 30 17:53:10.611915 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:10.611809 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879-metrics-certs\") pod \"maas-api-5c48877c75-xmplt\" (UID: \"ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-xmplt" Oct 30 17:53:10.611977 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:10.611926 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879-maas-api-tls\") pod \"maas-api-5c48877c75-xmplt\" (UID: \"ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-xmplt" Oct 30 17:53:10.612065 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:10.612046 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x74kd\" (UniqueName: \"kubernetes.io/projected/ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879-kube-api-access-x74kd\") pod \"maas-api-5c48877c75-xmplt\" (UID: \"ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-xmplt" Oct 30 17:53:10.625812 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:10.625780 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:53:10.625936 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:10.625830 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:53:10.713109 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:10.713081 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879-openshift-service-ca\") pod \"maas-api-5c48877c75-xmplt\" (UID: \"ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-xmplt" Oct 30 17:53:10.713277 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:10.713132 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879-metrics-certs\") pod \"maas-api-5c48877c75-xmplt\" (UID: \"ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-xmplt" Oct 30 17:53:10.713277 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:10.713160 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879-maas-api-tls\") pod \"maas-api-5c48877c75-xmplt\" (UID: \"ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-xmplt" Oct 30 17:53:10.713277 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:10.713196 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x74kd\" (UniqueName: \"kubernetes.io/projected/ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879-kube-api-access-x74kd\") pod \"maas-api-5c48877c75-xmplt\" (UID: \"ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-xmplt" Oct 30 17:53:10.713785 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:10.713760 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879-openshift-service-ca\") pod \"maas-api-5c48877c75-xmplt\" (UID: \"ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-xmplt" Oct 30 17:53:10.715637 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:10.715618 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879-metrics-certs\") pod \"maas-api-5c48877c75-xmplt\" (UID: \"ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-xmplt" Oct 30 17:53:10.715703 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:10.715671 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879-maas-api-tls\") pod \"maas-api-5c48877c75-xmplt\" (UID: \"ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-xmplt" Oct 30 17:53:10.739164 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:10.739137 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x74kd\" (UniqueName: \"kubernetes.io/projected/ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879-kube-api-access-x74kd\") pod \"maas-api-5c48877c75-xmplt\" (UID: \"ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-xmplt" Oct 30 17:53:10.875056 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:10.874965 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-xmplt" Oct 30 17:53:11.177203 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:53:11.177171 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec02dc7c_8dd3_4ff2_baa6_11e1c43c5879.slice/crio-4c1b0389510633d916ea41677ba9f18fc65614a4a9760921eb538bd3be353934 WatchSource:0}: Error finding container 4c1b0389510633d916ea41677ba9f18fc65614a4a9760921eb538bd3be353934: Status 404 returned error can't find the container with id 4c1b0389510633d916ea41677ba9f18fc65614a4a9760921eb538bd3be353934 Oct 30 17:53:11.177435 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:11.177410 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-xmplt"] Oct 30 17:53:11.179154 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:11.179132 4522 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:53:11.508813 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:11.508780 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-xmplt" event={"ID":"ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879","Type":"ContainerStarted","Data":"4c1b0389510633d916ea41677ba9f18fc65614a4a9760921eb538bd3be353934"} Oct 30 17:53:12.004289 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:12.004253 4522 scope.go:122] "RemoveContainer" containerID="7f6939961bc601126f9159f7598534a30ed5ffbceabca55abbe768b3e473cecd" Oct 30 17:53:12.515278 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:12.515248 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-xmplt" event={"ID":"ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879","Type":"ContainerStarted","Data":"4056ff63393f25b4a8e01f546d1501b1b662afadabeaee036227b3f08546a8c9"} Oct 30 17:53:12.515474 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:12.515386 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-xmplt" Oct 30 17:53:12.563762 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:12.563707 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c48877c75-xmplt" podStartSLOduration=2.168445047 podStartE2EDuration="2.563693452s" totalImagesPullingTime="395.248405ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:53:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:53:12.563181461 +0000 UTC m=+3966.563936382" watchObservedRunningTime="2026-10-30 17:53:12.563693452 +0000 UTC m=+3966.564448385" Oct 30 17:53:12.947694 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:53:12.947614 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:53:18.324040 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:18.324003 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-xmplt"] Oct 30 17:53:18.324492 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:18.324191 4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5c48877c75-xmplt" podUID="ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879" containerName="maas-api" containerID="cri-o://4056ff63393f25b4a8e01f546d1501b1b662afadabeaee036227b3f08546a8c9" gracePeriod=30 Oct 30 17:53:18.329609 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:18.329579 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-xmplt" Oct 30 17:53:18.371321 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:18.371283 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-zmm6m"] Oct 30 17:53:18.374755 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:18.374739 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-zmm6m" Oct 30 17:53:18.388366 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:18.388337 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-zmm6m"] Oct 30 17:53:18.483706 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:18.483670 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba36565a-c5bd-4b39-b343-0dae046a76d6-maas-api-tls\") pod \"maas-api-595cc648c-zmm6m\" (UID: \"ba36565a-c5bd-4b39-b343-0dae046a76d6\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-zmm6m" Oct 30 17:53:18.483895 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:18.483814 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba36565a-c5bd-4b39-b343-0dae046a76d6-metrics-certs\") pod \"maas-api-595cc648c-zmm6m\" (UID: \"ba36565a-c5bd-4b39-b343-0dae046a76d6\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-zmm6m" Oct 30 17:53:18.483895 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:18.483859 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba36565a-c5bd-4b39-b343-0dae046a76d6-openshift-service-ca\") pod \"maas-api-595cc648c-zmm6m\" (UID: \"ba36565a-c5bd-4b39-b343-0dae046a76d6\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-zmm6m" Oct 30 17:53:18.483895 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:18.483887 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx2sb\" (UniqueName: \"kubernetes.io/projected/ba36565a-c5bd-4b39-b343-0dae046a76d6-kube-api-access-xx2sb\") pod \"maas-api-595cc648c-zmm6m\" (UID: \"ba36565a-c5bd-4b39-b343-0dae046a76d6\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-zmm6m" Oct 30 17:53:18.547158 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:18.547130 4522 generic.go:356] "Generic (PLEG): container finished" podID="ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879" containerID="4056ff63393f25b4a8e01f546d1501b1b662afadabeaee036227b3f08546a8c9" exitCode=0 Oct 30 17:53:18.547352 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:18.547206 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-xmplt" event={"ID":"ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879","Type":"ContainerDied","Data":"4056ff63393f25b4a8e01f546d1501b1b662afadabeaee036227b3f08546a8c9"} Oct 30 17:53:18.581031 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:18.580978 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-xmplt" Oct 30 17:53:18.585107 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:18.585087 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba36565a-c5bd-4b39-b343-0dae046a76d6-metrics-certs\") pod \"maas-api-595cc648c-zmm6m\" (UID: \"ba36565a-c5bd-4b39-b343-0dae046a76d6\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-zmm6m" Oct 30 17:53:18.585181 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:18.585137 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba36565a-c5bd-4b39-b343-0dae046a76d6-openshift-service-ca\") pod \"maas-api-595cc648c-zmm6m\" (UID: \"ba36565a-c5bd-4b39-b343-0dae046a76d6\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-zmm6m" Oct 30 17:53:18.585352 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:18.585329 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx2sb\" (UniqueName: \"kubernetes.io/projected/ba36565a-c5bd-4b39-b343-0dae046a76d6-kube-api-access-xx2sb\") pod \"maas-api-595cc648c-zmm6m\" (UID: \"ba36565a-c5bd-4b39-b343-0dae046a76d6\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-zmm6m" Oct 30 17:53:18.585434 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:18.585418 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba36565a-c5bd-4b39-b343-0dae046a76d6-maas-api-tls\") pod \"maas-api-595cc648c-zmm6m\" (UID: \"ba36565a-c5bd-4b39-b343-0dae046a76d6\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-zmm6m" Oct 30 17:53:18.585793 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:18.585759 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba36565a-c5bd-4b39-b343-0dae046a76d6-openshift-service-ca\") pod \"maas-api-595cc648c-zmm6m\" (UID: \"ba36565a-c5bd-4b39-b343-0dae046a76d6\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-zmm6m" Oct 30 17:53:18.587588 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:18.587565 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba36565a-c5bd-4b39-b343-0dae046a76d6-metrics-certs\") pod \"maas-api-595cc648c-zmm6m\" (UID: \"ba36565a-c5bd-4b39-b343-0dae046a76d6\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-zmm6m" Oct 30 17:53:18.587688 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:18.587656 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba36565a-c5bd-4b39-b343-0dae046a76d6-maas-api-tls\") pod \"maas-api-595cc648c-zmm6m\" (UID: \"ba36565a-c5bd-4b39-b343-0dae046a76d6\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-zmm6m" Oct 30 17:53:18.611483 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:18.611461 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx2sb\" (UniqueName: \"kubernetes.io/projected/ba36565a-c5bd-4b39-b343-0dae046a76d6-kube-api-access-xx2sb\") pod \"maas-api-595cc648c-zmm6m\" (UID: \"ba36565a-c5bd-4b39-b343-0dae046a76d6\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-zmm6m" Oct 30 17:53:18.686615 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:18.686583 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879-kube-api-access-x74kd\" (UniqueName: \"kubernetes.io/projected/ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879-kube-api-access-x74kd\") pod \"ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879\" (UID: \"ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879\") " Oct 30 17:53:18.686792 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:18.686624 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879-openshift-service-ca\") pod \"ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879\" (UID: \"ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879\") " Oct 30 17:53:18.686792 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:18.686722 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879-maas-api-tls\") pod \"ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879\" (UID: \"ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879\") " Oct 30 17:53:18.686792 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:18.686775 4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879-metrics-certs\") pod \"ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879\" (UID: \"ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879\") " Oct 30 17:53:18.687078 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:18.687049 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879-openshift-service-ca" pod "ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879" (UID: "ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:53:18.688844 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:18.688821 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879-maas-api-tls" pod "ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879" (UID: "ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:53:18.688957 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:18.688884 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879-metrics-certs" pod "ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879" (UID: "ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:53:18.688957 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:18.688911 4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879-kube-api-access-x74kd" pod "ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879" (UID: "ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879"). InnerVolumeSpecName "kube-api-access-x74kd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:53:18.720874 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:18.720851 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-zmm6m" Oct 30 17:53:18.788123 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:18.788090 4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879-maas-api-tls\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:53:18.788252 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:18.788126 4522 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879-metrics-certs\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:53:18.788252 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:18.788142 4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x74kd\" (UniqueName: \"kubernetes.io/projected/ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879-kube-api-access-x74kd\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:53:18.788252 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:18.788158 4522 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879-openshift-service-ca\") on node \"ip-10-0-128-218.ec2.internal\" DevicePath \"\"" Oct 30 17:53:19.017259 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:53:19.017226 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba36565a_c5bd_4b39_b343_0dae046a76d6.slice/crio-e86758a06f51436d279d97ea73e4257cc7c043198a6a7d22b1e190ea74bfa529 WatchSource:0}: Error finding container e86758a06f51436d279d97ea73e4257cc7c043198a6a7d22b1e190ea74bfa529: Status 404 returned error can't find the container with id e86758a06f51436d279d97ea73e4257cc7c043198a6a7d22b1e190ea74bfa529 Oct 30 17:53:19.018029 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:19.018000 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-zmm6m"] Oct 30 17:53:19.554803 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:19.554760 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-zmm6m" event={"ID":"ba36565a-c5bd-4b39-b343-0dae046a76d6","Type":"ContainerStarted","Data":"e822e071e7cb80254d63b241ded0e8f3a29eefa1256267f3d356d5b3415a31f0"} Oct 30 17:53:19.555296 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:19.554827 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-zmm6m" event={"ID":"ba36565a-c5bd-4b39-b343-0dae046a76d6","Type":"ContainerStarted","Data":"e86758a06f51436d279d97ea73e4257cc7c043198a6a7d22b1e190ea74bfa529"} Oct 30 17:53:19.555296 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:19.554856 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-zmm6m" Oct 30 17:53:19.556240 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:19.556211 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-xmplt" event={"ID":"ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879","Type":"ContainerDied","Data":"4c1b0389510633d916ea41677ba9f18fc65614a4a9760921eb538bd3be353934"} Oct 30 17:53:19.556395 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:19.556247 4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-xmplt" Oct 30 17:53:19.556395 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:19.556268 4522 scope.go:122] "RemoveContainer" containerID="4056ff63393f25b4a8e01f546d1501b1b662afadabeaee036227b3f08546a8c9" Oct 30 17:53:19.604238 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:19.604149 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-595cc648c-zmm6m" podStartSLOduration=1.298886741 podStartE2EDuration="1.604135685s" totalImagesPullingTime="305.248944ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:53:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:53:19.602913992 +0000 UTC m=+3973.603668893" watchObservedRunningTime="2026-10-30 17:53:19.604135685 +0000 UTC m=+3973.604890585" Oct 30 17:53:19.646437 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:19.646408 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-xmplt"] Oct 30 17:53:19.648187 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:19.648162 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-xmplt"] Oct 30 17:53:20.145261 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:20.145221 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879" path="/var/lib/kubelet/pods/ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879/volumes" Oct 30 17:53:21.677510 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:21.677464 4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-ljkmh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:53:21.677923 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:21.677544 4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ljkmh" podUID="41ac1e04-01e0-4a54-940a-5b2eb66f1e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:53:25.574160 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:53:25.574129 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-zmm6m" Oct 30 17:53:42.945800 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:53:42.945764 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:54:09.156956 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:54:09.156914 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-648b5d97c7-78fx4_8f03ee2a-146e-4c54-b83a-35637ceda4bb/authorino/0.log" Oct 30 17:54:12.025055 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:54:12.025023 4522 scope.go:122] "RemoveContainer" containerID="2aae67debaf80a9f23bd63dee9c59a2af10884faccf8457cc9311e8942b3f5ec" Oct 30 17:54:12.033513 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:54:12.033491 4522 scope.go:122] "RemoveContainer" containerID="555c19ed0cd13a835cb1709ca5a0f9f4fb21dcf46e999229fdc8da0c782a4d81" Oct 30 17:54:12.041445 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:54:12.041422 4522 scope.go:122] "RemoveContainer" containerID="5804ec0c3d4a1c3180797846f5e9a0e1ea16373e4b3e0f82289c941ad841b600" Oct 30 17:54:12.947625 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:54:12.947592 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:54:22.188809 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:54:22.188778 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68c46f97f5-bzljn_d8c68be0-0463-4240-ac31-bb221e85ba64/copy-manifests/0.log" Oct 30 17:54:22.218997 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:54:22.218965 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68c46f97f5-bzljn_d8c68be0-0463-4240-ac31-bb221e85ba64/manager/0.log" Oct 30 17:54:24.046527 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:54:24.046488 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-zmm6m_ba36565a-c5bd-4b39-b343-0dae046a76d6/maas-api/0.log" Oct 30 17:54:24.448776 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:54:24.448692 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-9xhhn_5e2a09d4-7a61-481b-9fac-047f911b7eaf/revoke-keys/0.log" Oct 30 17:54:24.636002 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:54:24.635968 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-k844j_ccd80853-0b0f-4875-ace4-b9d2cebbfc11/revoke-keys/0.log" Oct 30 17:54:24.818780 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:54:24.818751 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-493ff32b-lndtb_24f50285-f406-48d7-af07-33a1ebe5825a/revoke-keys/0.log" Oct 30 17:54:25.200796 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:54:25.200704 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-98d632f6-wc7lx_b7077219-7481-4bfe-9201-c429ac69d9ce/revoke-keys/0.log" Oct 30 17:54:25.389388 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:54:25.389359 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d2bc93bc-lnn2v_3e9cbcae-2d9d-44f0-8531-d49b9f335a01/revoke-keys/0.log" Oct 30 17:54:25.587052 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:54:25.587012 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d576e68b-x7zf9_111f192f-5ccc-454a-90de-d94b2f283bb0/revoke-keys/0.log" Oct 30 17:54:27.529885 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:54:27.529854 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-648b5d97c7-78fx4_8f03ee2a-146e-4c54-b83a-35637ceda4bb/authorino/0.log" Oct 30 17:54:28.187663 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:54:28.187630 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-klsm7_d03e75f7-ffc9-4069-963a-f6676c7e8922/kuadrant-console-plugin/0.log" Oct 30 17:54:28.921938 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:54:28.921906 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-tt5f8_3244034b-650b-4f3f-a651-0bf35bd9c8f1/manager/0.log" Oct 30 17:54:30.330126 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:54:30.330087 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-768ffddf88-qflf4_af45ce16-d0ac-4fdf-8623-a23d3e8240f4/kube-auth-proxy/0.log" Oct 30 17:54:30.739775 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:54:30.739740 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-245xx_fb8adf71-af34-4715-ada0-42494da2219a/istio-proxy/0.log" Oct 30 17:54:31.130260 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:54:31.130169 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-hgtsq_9d445c57-2efa-4ae2-8a16-9e4dcdac8544/istio-proxy/0.log" Oct 30 17:54:31.532571 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:54:31.532540 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-ljkmh_41ac1e04-01e0-4a54-940a-5b2eb66f1e2c/istio-proxy/0.log" Oct 30 17:54:34.256646 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:54:34.256622 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-9452k_99c2ec18-1d5d-4bb1-99ac-f16a0f753a2e/main/0.log" Oct 30 17:54:34.685545 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:54:34.685463 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-2xdvj_ce97dea1-392d-437d-bcb8-b1d73b3a7528/main/0.log" Oct 30 17:54:35.125710 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:54:35.125674 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-zcm49_e9116a9b-6aaf-4f77-8d89-8766d89b6048/main/0.log" Oct 30 17:54:35.331572 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:54:35.331542 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5_4360e9dc-c001-4102-8af6-b55b692fcf2c/main/0.log" Oct 30 17:54:36.293342 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:54:36.293294 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-648b5d97c7-78fx4_8f03ee2a-146e-4c54-b83a-35637ceda4bb/authorino/0.log" Oct 30 17:54:42.947461 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:54:42.947425 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:54:47.426183 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:54:47.426151 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68c46f97f5-bzljn_d8c68be0-0463-4240-ac31-bb221e85ba64/copy-manifests/0.log" Oct 30 17:54:47.457875 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:54:47.457842 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68c46f97f5-bzljn_d8c68be0-0463-4240-ac31-bb221e85ba64/manager/0.log" Oct 30 17:54:48.938390 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:54:48.938352 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-zmm6m_ba36565a-c5bd-4b39-b343-0dae046a76d6/maas-api/0.log" Oct 30 17:54:49.333392 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:54:49.333363 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-9xhhn_5e2a09d4-7a61-481b-9fac-047f911b7eaf/revoke-keys/0.log" Oct 30 17:54:49.522533 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:54:49.522502 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-k844j_ccd80853-0b0f-4875-ace4-b9d2cebbfc11/revoke-keys/0.log" Oct 30 17:54:49.702804 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:54:49.702780 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-493ff32b-lndtb_24f50285-f406-48d7-af07-33a1ebe5825a/revoke-keys/0.log" Oct 30 17:54:50.071796 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:54:50.071709 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-98d632f6-wc7lx_b7077219-7481-4bfe-9201-c429ac69d9ce/revoke-keys/0.log" Oct 30 17:54:50.253175 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:54:50.253144 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d2bc93bc-lnn2v_3e9cbcae-2d9d-44f0-8531-d49b9f335a01/revoke-keys/0.log" Oct 30 17:54:50.435619 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:54:50.435543 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d576e68b-x7zf9_111f192f-5ccc-454a-90de-d94b2f283bb0/revoke-keys/0.log" Oct 30 17:54:52.030711 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:54:52.030681 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-648b5d97c7-78fx4_8f03ee2a-146e-4c54-b83a-35637ceda4bb/authorino/0.log" Oct 30 17:54:52.693848 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:54:52.693821 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-klsm7_d03e75f7-ffc9-4069-963a-f6676c7e8922/kuadrant-console-plugin/0.log" Oct 30 17:54:53.391537 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:54:53.391498 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-tt5f8_3244034b-650b-4f3f-a651-0bf35bd9c8f1/manager/0.log" Oct 30 17:54:54.583835 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:54:54.583802 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-768ffddf88-qflf4_af45ce16-d0ac-4fdf-8623-a23d3e8240f4/kube-auth-proxy/0.log" Oct 30 17:54:54.992291 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:54:54.992260 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-245xx_fb8adf71-af34-4715-ada0-42494da2219a/istio-proxy/0.log" Oct 30 17:54:55.393952 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:54:55.393859 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-hgtsq_9d445c57-2efa-4ae2-8a16-9e4dcdac8544/istio-proxy/0.log" Oct 30 17:54:55.771972 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:54:55.771939 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-ljkmh_41ac1e04-01e0-4a54-940a-5b2eb66f1e2c/istio-proxy/0.log" Oct 30 17:54:57.044234 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:54:57.044200 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-x7zf9"] Oct 30 17:54:57.058682 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:54:57.058654 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-x7zf9"] Oct 30 17:54:57.938704 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:54:57.938674 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-9452k_99c2ec18-1d5d-4bb1-99ac-f16a0f753a2e/main/0.log" Oct 30 17:54:58.143658 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:54:58.143625 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="111f192f-5ccc-454a-90de-d94b2f283bb0" path="/var/lib/kubelet/pods/111f192f-5ccc-454a-90de-d94b2f283bb0/volumes" Oct 30 17:54:58.349209 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:54:58.349181 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-2xdvj_ce97dea1-392d-437d-bcb8-b1d73b3a7528/main/0.log" Oct 30 17:54:58.790430 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:54:58.790399 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-zcm49_e9116a9b-6aaf-4f77-8d89-8766d89b6048/main/0.log" Oct 30 17:54:59.014659 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:54:59.014626 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-jdvh5_4360e9dc-c001-4102-8af6-b55b692fcf2c/main/0.log" Oct 30 17:55:12.947010 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:55:12.946968 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:55:14.031747 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:55:14.031710 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lnn2v"] Oct 30 17:55:14.046226 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:55:14.046185 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lnn2v"] Oct 30 17:55:14.143181 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:55:14.143138 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e9cbcae-2d9d-44f0-8531-d49b9f335a01" path="/var/lib/kubelet/pods/3e9cbcae-2d9d-44f0-8531-d49b9f335a01/volumes" Oct 30 17:55:42.946608 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:55:42.946563 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:56:06.053032 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:06.052990 4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-493ff32b-lndtb"] Oct 30 17:56:06.067266 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:06.067199 4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-493ff32b-lndtb"] Oct 30 17:56:06.151364 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:06.151330 4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24f50285-f406-48d7-af07-33a1ebe5825a" path="/var/lib/kubelet/pods/24f50285-f406-48d7-af07-33a1ebe5825a/volumes" Oct 30 17:56:12.077663 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:12.077631 4522 scope.go:122] "RemoveContainer" containerID="8a8d588a4f5dc022324776359975de229ed31251a9695d95fd595e068ea909e8" Oct 30 17:56:12.086327 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:12.086290 4522 scope.go:122] "RemoveContainer" containerID="236c933883a37d7a1ebc2fa701ebf280491b78e97c43cb2889ebb787db06b4cf" Oct 30 17:56:12.947327 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:56:12.947273 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:56:17.260556 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:17.260524 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2kbm7_d52d0ddc-6bd1-4dcc-9d56-84213c328055/global-pull-secret-syncer/0.log" Oct 30 17:56:18.489364 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:18.489174 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mqhg8_d24bfb4d-652d-4e7d-bc67-43441164f652/konnectivity-agent/0.log" Oct 30 17:56:18.724398 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:18.724355 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-218.ec2.internal_bc58f75ef247c6cbf4faad53957c9f43/haproxy/0.log" Oct 30 17:56:25.112462 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:25.112427 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-648b5d97c7-78fx4_8f03ee2a-146e-4c54-b83a-35637ceda4bb/authorino/0.log" Oct 30 17:56:25.449072 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:25.448992 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-klsm7_d03e75f7-ffc9-4069-963a-f6676c7e8922/kuadrant-console-plugin/0.log" Oct 30 17:56:25.947545 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:25.947507 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-tt5f8_3244034b-650b-4f3f-a651-0bf35bd9c8f1/manager/0.log" Oct 30 17:56:27.260101 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:27.260070 4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2tldp/perf-node-gather-daemonset-v2gjc"] Oct 30 17:56:27.260632 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:27.260472 4522 cpu_manager.go:414] "containerMap: removing container" podUID="ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879" containerName="maas-api" Oct 30 17:56:27.260632 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:27.260493 4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879" containerName="maas-api" Oct 30 17:56:27.260632 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:27.260558 4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec02dc7c-8dd3-4ff2-baa6-11e1c43c5879" containerName="maas-api" Oct 30 17:56:27.263694 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:27.263675 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-v2gjc" Oct 30 17:56:27.282041 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:27.282018 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5ed4044f-063d-4be8-a4fe-9410ac57c1ac/alertmanager/0.log" Oct 30 17:56:27.284328 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:27.284284 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2tldp/perf-node-gather-daemonset-v2gjc"] Oct 30 17:56:27.296239 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:27.296217 4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2tldp\"/\"default-dockercfg-t845d\"" Oct 30 17:56:27.317536 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:27.317508 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2tldp\"/\"openshift-service-ca.crt\"" Oct 30 17:56:27.317536 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:27.317508 4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2tldp\"/\"kube-root-ca.crt\"" Oct 30 17:56:27.398261 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:27.398230 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5ed4044f-063d-4be8-a4fe-9410ac57c1ac/config-reloader/0.log" Oct 30 17:56:27.404530 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:27.404502 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e6c7fc70-20fd-4122-9d01-af7aa718b72f-lib-modules\") pod \"perf-node-gather-daemonset-v2gjc\" (UID: \"e6c7fc70-20fd-4122-9d01-af7aa718b72f\") " pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-v2gjc" Oct 30 17:56:27.404663 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:27.404569 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e6c7fc70-20fd-4122-9d01-af7aa718b72f-podres\") pod \"perf-node-gather-daemonset-v2gjc\" (UID: \"e6c7fc70-20fd-4122-9d01-af7aa718b72f\") " pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-v2gjc" Oct 30 17:56:27.404710 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:27.404664 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e6c7fc70-20fd-4122-9d01-af7aa718b72f-sys\") pod \"perf-node-gather-daemonset-v2gjc\" (UID: \"e6c7fc70-20fd-4122-9d01-af7aa718b72f\") " pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-v2gjc" Oct 30 17:56:27.404766 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:27.404751 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76wft\" (UniqueName: \"kubernetes.io/projected/e6c7fc70-20fd-4122-9d01-af7aa718b72f-kube-api-access-76wft\") pod \"perf-node-gather-daemonset-v2gjc\" (UID: \"e6c7fc70-20fd-4122-9d01-af7aa718b72f\") " pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-v2gjc" Oct 30 17:56:27.404821 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:27.404775 4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e6c7fc70-20fd-4122-9d01-af7aa718b72f-proc\") pod \"perf-node-gather-daemonset-v2gjc\" (UID: \"e6c7fc70-20fd-4122-9d01-af7aa718b72f\") " pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-v2gjc" Oct 30 17:56:27.506177 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:27.506138 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e6c7fc70-20fd-4122-9d01-af7aa718b72f-lib-modules\") pod \"perf-node-gather-daemonset-v2gjc\" (UID: \"e6c7fc70-20fd-4122-9d01-af7aa718b72f\") " pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-v2gjc" Oct 30 17:56:27.506394 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:27.506204 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e6c7fc70-20fd-4122-9d01-af7aa718b72f-podres\") pod \"perf-node-gather-daemonset-v2gjc\" (UID: \"e6c7fc70-20fd-4122-9d01-af7aa718b72f\") " pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-v2gjc" Oct 30 17:56:27.506394 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:27.506268 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e6c7fc70-20fd-4122-9d01-af7aa718b72f-sys\") pod \"perf-node-gather-daemonset-v2gjc\" (UID: \"e6c7fc70-20fd-4122-9d01-af7aa718b72f\") " pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-v2gjc" Oct 30 17:56:27.506394 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:27.506291 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e6c7fc70-20fd-4122-9d01-af7aa718b72f-lib-modules\") pod \"perf-node-gather-daemonset-v2gjc\" (UID: \"e6c7fc70-20fd-4122-9d01-af7aa718b72f\") " pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-v2gjc" Oct 30 17:56:27.506394 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:27.506376 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e6c7fc70-20fd-4122-9d01-af7aa718b72f-sys\") pod \"perf-node-gather-daemonset-v2gjc\" (UID: \"e6c7fc70-20fd-4122-9d01-af7aa718b72f\") " pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-v2gjc" Oct 30 17:56:27.506567 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:27.506292 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e6c7fc70-20fd-4122-9d01-af7aa718b72f-podres\") pod \"perf-node-gather-daemonset-v2gjc\" (UID: \"e6c7fc70-20fd-4122-9d01-af7aa718b72f\") " pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-v2gjc" Oct 30 17:56:27.506567 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:27.506383 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76wft\" (UniqueName: \"kubernetes.io/projected/e6c7fc70-20fd-4122-9d01-af7aa718b72f-kube-api-access-76wft\") pod \"perf-node-gather-daemonset-v2gjc\" (UID: \"e6c7fc70-20fd-4122-9d01-af7aa718b72f\") " pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-v2gjc" Oct 30 17:56:27.506567 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:27.506438 4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e6c7fc70-20fd-4122-9d01-af7aa718b72f-proc\") pod \"perf-node-gather-daemonset-v2gjc\" (UID: \"e6c7fc70-20fd-4122-9d01-af7aa718b72f\") " pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-v2gjc" Oct 30 17:56:27.506567 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:27.506523 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e6c7fc70-20fd-4122-9d01-af7aa718b72f-proc\") pod \"perf-node-gather-daemonset-v2gjc\" (UID: \"e6c7fc70-20fd-4122-9d01-af7aa718b72f\") " pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-v2gjc" Oct 30 17:56:27.514407 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:27.514346 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5ed4044f-063d-4be8-a4fe-9410ac57c1ac/kube-rbac-proxy-web/0.log" Oct 30 17:56:27.532268 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:27.532246 4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76wft\" (UniqueName: \"kubernetes.io/projected/e6c7fc70-20fd-4122-9d01-af7aa718b72f-kube-api-access-76wft\") pod \"perf-node-gather-daemonset-v2gjc\" (UID: \"e6c7fc70-20fd-4122-9d01-af7aa718b72f\") " pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-v2gjc" Oct 30 17:56:27.575976 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:27.575946 4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-v2gjc" Oct 30 17:56:27.630478 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:27.630461 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5ed4044f-063d-4be8-a4fe-9410ac57c1ac/kube-rbac-proxy/0.log" Oct 30 17:56:27.745242 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:27.745210 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5ed4044f-063d-4be8-a4fe-9410ac57c1ac/kube-rbac-proxy-metric/0.log" Oct 30 17:56:27.861070 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:27.860983 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5ed4044f-063d-4be8-a4fe-9410ac57c1ac/prom-label-proxy/0.log" Oct 30 17:56:27.871138 ip-10-0-128-218 kubenswrapper[4522]: W1030 17:56:27.871104 4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode6c7fc70_20fd_4122_9d01_af7aa718b72f.slice/crio-2a4b41bc4124d8ea90275b1184e13e592d8cb1b7419e436cda06c13787141142 WatchSource:0}: Error finding container 2a4b41bc4124d8ea90275b1184e13e592d8cb1b7419e436cda06c13787141142: Status 404 returned error can't find the container with id 2a4b41bc4124d8ea90275b1184e13e592d8cb1b7419e436cda06c13787141142 Oct 30 17:56:27.872064 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:27.872036 4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2tldp/perf-node-gather-daemonset-v2gjc"] Oct 30 17:56:27.978386 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:27.978357 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5ed4044f-063d-4be8-a4fe-9410ac57c1ac/init-config-reloader/0.log" Oct 30 17:56:28.245535 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:28.245502 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-c46xd_01e543d8-5483-4cc6-8af5-aaf0fc93b069/kube-state-metrics/0.log" Oct 30 17:56:28.350538 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:28.350494 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-c46xd_01e543d8-5483-4cc6-8af5-aaf0fc93b069/kube-rbac-proxy-main/0.log" Oct 30 17:56:28.466269 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:28.466237 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-c46xd_01e543d8-5483-4cc6-8af5-aaf0fc93b069/kube-rbac-proxy-self/0.log" Oct 30 17:56:28.543607 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:28.543510 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-v2gjc" event={"ID":"e6c7fc70-20fd-4122-9d01-af7aa718b72f","Type":"ContainerStarted","Data":"4b0a125643dfe69227df88f192604d3658ad1761a69550bea15fd2013b2882a4"} Oct 30 17:56:28.543607 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:28.543572 4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-v2gjc" event={"ID":"e6c7fc70-20fd-4122-9d01-af7aa718b72f","Type":"ContainerStarted","Data":"2a4b41bc4124d8ea90275b1184e13e592d8cb1b7419e436cda06c13787141142"} Oct 30 17:56:28.543849 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:28.543696 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-v2gjc" Oct 30 17:56:28.589953 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:28.589903 4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-v2gjc" podStartSLOduration=1.58989095 podStartE2EDuration="1.58989095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:56:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:28.589396056 +0000 UTC m=+4162.590150957" watchObservedRunningTime="2026-10-30 17:56:28.58989095 +0000 UTC m=+4162.590645851" Oct 30 17:56:29.179097 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:29.179063 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cvp4x_b0a9853e-3d0e-4385-954c-bf4e19c996b8/node-exporter/0.log" Oct 30 17:56:29.299813 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:29.299790 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cvp4x_b0a9853e-3d0e-4385-954c-bf4e19c996b8/kube-rbac-proxy/0.log" Oct 30 17:56:29.417069 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:29.417038 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cvp4x_b0a9853e-3d0e-4385-954c-bf4e19c996b8/init-textfile/0.log" Oct 30 17:56:29.915717 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:29.915684 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-6ljvl_48ac41e9-e6af-4f7e-925a-c871db2be357/kube-rbac-proxy-main/0.log" Oct 30 17:56:30.032103 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:30.032073 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-6ljvl_48ac41e9-e6af-4f7e-925a-c871db2be357/kube-rbac-proxy-self/0.log" Oct 30 17:56:30.150968 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:30.150938 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-6ljvl_48ac41e9-e6af-4f7e-925a-c871db2be357/openshift-state-metrics/0.log" Oct 30 17:56:31.443756 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:31.443724 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-79b7d6d8f-d5kcb_89202ceb-ee62-4a1a-8756-54a870c7b7ab/telemeter-client/0.log" Oct 30 17:56:31.560562 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:31.560535 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-79b7d6d8f-d5kcb_89202ceb-ee62-4a1a-8756-54a870c7b7ab/reload/0.log" Oct 30 17:56:31.675467 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:31.675437 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-79b7d6d8f-d5kcb_89202ceb-ee62-4a1a-8756-54a870c7b7ab/kube-rbac-proxy/0.log" Oct 30 17:56:34.560416 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:34.560392 4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-v2gjc" Oct 30 17:56:35.918012 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:35.917978 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7d89df5bcc-d5wph_b18a897c-62ee-482f-8a3a-306c95b41bce/console/0.log" Oct 30 17:56:40.011426 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:40.011397 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wc9zw_bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3/dns/0.log" Oct 30 17:56:40.126553 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:40.126527 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wc9zw_bcdeb755-d77b-4d2c-95e0-aa2ddf7426d3/kube-rbac-proxy/0.log" Oct 30 17:56:40.253766 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:40.253743 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mbwdh_c702aab4-18a3-4b3a-a629-7d87f15747e7/dns-node-resolver/0.log" Oct 30 17:56:41.571292 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:41.571255 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-59dd87c5f7-xhqgq_12b90e9c-b1ad-4b61-8372-8a3b5fb9e317/registry/0.log" Oct 30 17:56:41.667652 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:41.667620 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7dcvf_bbdceb2b-73c7-4501-994c-4e3533fa47d0/node-ca/0.log" Oct 30 17:56:42.630948 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:42.630918 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-768ffddf88-qflf4_af45ce16-d0ac-4fdf-8623-a23d3e8240f4/kube-auth-proxy/0.log" Oct 30 17:56:42.776589 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:42.776561 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-245xx_fb8adf71-af34-4715-ada0-42494da2219a/istio-proxy/0.log" Oct 30 17:56:42.910553 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:42.910481 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-hgtsq_9d445c57-2efa-4ae2-8a16-9e4dcdac8544/istio-proxy/0.log" Oct 30 17:56:42.947111 ip-10-0-128-218 kubenswrapper[4522]: E1030 17:56:42.947078 4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4wjcq" containerName="extractor" Oct 30 17:56:43.043769 ip-10-0-128-218 kubenswrapper[4522]: I1030 17:56:43.043720 4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-ljkmh_41ac1e04-01e0-4a54-940a-5b2eb66f1e2c/istio-proxy/0.log"